Deploy and manage functions worker

Pulsar functions-worker is a logic component to run Pulsar Functions in cluster mode. Two options are available, and you can select either of the two options based on your requirements.

• run with brokers
• run it separately in a different broker

note

The --- Service Urls--- lines in the following diagrams represent Pulsar service URLs that Pulsar client and admin use to connect to a Pulsar cluster.

Run Functions-worker with brokers

The following diagram illustrates the deployment of functions-workers running along with brokers.

To enable functions-worker running as part of a broker, you need to set functionsWorkerEnabled to true in the broker.conf file.

functionsWorkerEnabled=true

When you set functionsWorkerEnabled to true, it means that you start functions-worker as part of a broker. You need to configure the conf/functions_worker.yml file to customize your functions_worker.

Before you run Functions-work with broker, you have to configure Functions-worker, and then start it with brokers.

Configure Functions-Worker to run with brokers

In this mode, since functions-worker is running as part of broker, most of the settings already inherit from your broker configuration (for example, configurationStore settings, authentication settings, and so on).

Pay attention to the following required settings when configuring functions-worker in this mode.

• numFunctionPackageReplicas: The number of replicas to store function packages. The default value is 1, which is good for standalone deployment. For production deployment, to ensure high availability, set it to be more than 2 .
• pulsarFunctionsCluster: Set the value to your Pulsar cluster name (same as the clusterName setting in the broker configuration).

If authentication is enabled on the BookKeeper cluster, configure the following BookKeeper authentication settings.

• bookkeeperClientAuthenticationPlugin: the BookKeeper client authentication plugin name.
• bookkeeperClientAuthenticationParametersName: the BookKeeper client authentication plugin parameters name.
• bookkeeperClientAuthenticationParameters: the BookKeeper client authentication plugin parameters.

Start Functions-worker with broker

Once you have configured the functions_worker.yml file, you can start or restart your broker.

And then you can use the following command to verify if functions-worker is running well.

curl <broker-ip>:8080/admin/v2/worker/cluster

After entering the command above, a list of active function workers in the cluster is returned. The output is something similar as follows.

[{"workerId":"<worker-id>","workerHostname":"<worker-hostname>","port":8080}]

Run Functions-worker separately

This section illustrates how to run functions-worker as a separate process in separate machines.

Note In this mode, make sure functionsWorkerEnabled is set to false, so you won't start functions-worker with brokers by mistake.

Configure Functions-worker to run separately

To run function-worker separately, you have to configure the following parameters.

Worker parameters

• workerId: The type is string. It is unique across clusters, used to identify a worker machine.
• workerHostname: The hostname of the worker machine.
• workerPort: The port that the worker server listens on. Keep it as default if you don't customize it.
• workerPortTls: The TLS port that the worker server listens on. Keep it as default if you don't customize it.

Function package parameter

• numFunctionPackageReplicas: The number of replicas to store function packages. The default value is 1.

Function metadata parameter

• pulsarServiceUrl: The Pulsar service URL for your broker cluster.
• pulsarWebServiceUrl: The Pulsar web service URL for your broker cluster.
• pulsarFunctionsCluster: Set the value to your Pulsar cluster name (same as the clusterName setting in the broker configuration).

If authentication is enabled for your broker cluster, you should configure the authentication plugin and parameters for the functions worker to communicate with the brokers.

• clientAuthenticationPlugin
• clientAuthenticationParameters

Security settings

If you want to enable security on functions workers, you should:

• Enable TLS transport encryption
• Enable Authentication Provider
• Enable Authorization Provider

Enable TLS transport encryption

To enable TLS transport encryption, configure the following settings.

tlsEnabled: true
tlsCertificateFilePath: /path/to/functions-worker.cert.pem
tlsKeyFilePath:         /path/to/functions-worker.key-pk8.pem
tlsTrustCertsFilePath:  /path/to/ca.cert.pem

For details on TLS encryption, refer to Transport Encryption using TLS.

Enable Authentication Provider

To enable authentication on Functions Worker, configure the following settings.

Note Substitute the providers list with the providers you want to enable.

authenticationEnabled: true
authenticationProviders: [ provider1, provider2 ]

For SASL Authentication provider, add saslJaasClientAllowedIds and saslJaasBrokerSectionName under properties if needed.

properties:
  saslJaasClientAllowedIds: .*pulsar.*
  saslJaasBrokerSectionName: Broker

For Token Authentication prodivder, add necessary settings under properties if needed. See Token Authentication for more details.

properties:
  tokenSecretKey:       file://my/secret.key
  # If using public/private
  # tokenPublicKey:     file:///path/to/public.key

Enable Authorization Provider

To enable authorization on Functions Worker, you need to configure authorizationEnabled and configurationStoreServers. The authentication provider connects to configurationStoreServers to receive namespace policies.

authorizationEnabled: true
configurationStoreServers: <configuration-store-servers>

You should also configure a list of superuser roles. The superuser roles are able to access any admin API. The following is a configuration example.

superUserRoles:
  - role1
  - role2
  - role3

BookKeeper Authentication

If authentication is enabled on the BookKeeper cluster, you should configure the BookKeeper authentication settings as follows:

• bookkeeperClientAuthenticationPlugin: the plugin name of BookKeeper client authentication.
• bookkeeperClientAuthenticationParametersName: the plugin parameters name of BookKeeper client authentication.
• bookkeeperClientAuthenticationParameters: the plugin parameters of BookKeeper client authentication.

Start Functions-worker

Once you have finished configuring the functions_worker.yml configuration file, you can use the following command to start a functions-worker:

bin/pulsar functions-worker

Configure Proxies for Functions-workers

When you are running functions-worker in a separate cluster, the admin rest endpoints are split into two clusters. functions, function-worker, source and sink endpoints are now served by the functions-worker cluster, while all the other remaining endpoints are served by the broker cluster. Hence you need to configure your pulsar-admin to use the right service URL accordingly.

In order to address this inconvenience, you can start a proxy cluster for routing the admin rest requests accordingly. Hence you will have one central entry point for your admin service.

If you already have a proxy cluster, continue reading. If you haven't setup a proxy cluster before, you can follow the instructions to start proxies.

To enable routing functions related admin requests to functions-worker in a proxy, you can edit the proxy.conf file to modify the following settings:

functionWorkerWebServiceURL=<pulsar-functions-worker-web-service-url>
functionWorkerWebServiceURLTLS=<pulsar-functions-worker-web-service-url>

Compare the Run-with-Broker and Run-separately modes

As described above, you can run Function-worker with brokers, or run it separately. And it is more convenient to run functions-workers along with brokers. However, running functions-workers in a separate cluster provides better resource isolation for running functions in Process or Thread mode.

Use which mode for your cases, refer to the following guidelines to determine.

Use the Run-with-Broker mode in the following cases:

• a) if resource isolation is not required when running functions in Process or Thread mode;
• b) if you configure the functions-worker to run functions on Kubernetes (where the resource isolation problem is addressed by Kubernetes).

Use the Run-separately mode in the following cases:

• a) you don't have a Kubernetes cluster;
• b) if you want to run functions and brokers separately.

Troubleshooting

Error message: Namespace missing local cluster name in clusters list

Failed to get partitioned topic metadata: org.apache.pulsar.client.api.PulsarClientException$BrokerMetadataException: Namespace missing local cluster name in clusters list: local_cluster=xyz ns=public/functions clusters=[standalone]

The error message prompts when either of the cases occurs:

• a) a broker is started with functionsWorkerEnabled=true, but the pulsarFunctionsCluster is not set to the correct cluster in the conf/functions_worker.yaml file;
• b) setting up a geo-replicated Pulsar cluster with functionsWorkerEnabled=true, while brokers in one cluster run well, brokers in the other cluster do not work well.

Workaround

If any of these cases happens, follow the instructions below to fix the problem:

1. Get the current clusters list of public/functions namespace.

bin/pulsar-admin namespaces get-clusters public/functions

2. Check if the cluster is in the clusters list. If the cluster is not in the list, add it to the list and update the clusters list.

bin/pulsar-admin namespaces set-clusters --cluster=<existing-clusters>,<new-cluster> public/functions

3. Set the correct cluster name in pulsarFunctionsCluster in the conf/functions_worker.yml file.