Run function workers separately
The following diagram illustrates how function workers run as a separate process in separate machines.
Service URLs in the illustration represent Pulsar service URLs that Pulsar client and Pulsar admin use to connect to a Pulsar cluster.
To set up function workers that run separately, complete the following steps:
Configure function workers to run separately
To run function workers separately, you need to keep
functionsWorkerEnabled as its default value (
false) in the
Configure worker parameters
Configure the required parameters for workers in the
workerId: The identity of a worker node, which is unique across clusters. The type is string.
workerHostname: The hostname of the worker node.
workerPort: The port that the worker server listens on. Keep it as default if you don't customize it. Set it to
nullto disable the plaintext port.
workerPortTls: The TLS port that the worker server listens on. Keep it as default if you don't customize it. For more information about TLS encryption settings, refer to settings.
When accessing function workers to manage functions, the
pulsar-admin CLI or any of the clients should use the configured
workerPort to generate an
Configure function package parameters
numFunctionPackageReplicas parameter in the
conf/functions_worker.yml file. It indicates the number of replicas to store function packages.
To ensure high availability in a production deployment, set
numFunctionPackageReplicas to equal the number of bookies. The default value
1 is only for one-node cluster deployment.
Configure function metadata parameters
Configure the required parameter for function metadata in the
pulsarServiceUrl: The Pulsar service URL for your broker cluster.
pulsarWebServiceUrl: The Pulsar web service URL for your broker cluster.
pulsarFunctionsCluster: Set the value to your Pulsar cluster name (same as the
clusterNamesetting in the
If authentication is enabled on your broker cluster, you must configure the following authentication settings for the function workers to communicate with the brokers.
brokerClientAuthenticationEnabled: Whether to enable the broker client authentication used by function workers to talk to brokers.
clientAuthenticationPlugin: The authentication plugin to be used by the Pulsar client used in worker service.
clientAuthenticationParameters: The authentication parameter to be used by the Pulsar client used in worker service.
Enable security settings
When you run a function worker separately in a cluster configured with authentication, your function worker needs to communicate with the broker and authenticate incoming requests. Thus you need to configure the properties that the broker requires for authentication and authorization.
You must configure both the function worker authentication and authorization for the server to authenticate incoming requests and configure the client to be authenticated to communicate with the broker.
For example, if you use token authentication, you need to configure the following properties in the
configurationMetadataStoreUrl: zk:zookeeper-cluster:2181 # auth requires a connection to zookeeper
tokenSecretKey: file:///etc/pulsar/jwt/secret # if using a secret token, key file must be DER-encoded
tokenPublicKey: file:///etc/pulsar/jwt/public.key # if using public/private key tokens, key file must be DER-encoded
You can enable the following security settings on function workers.
- Enable TLS encryption
- Enable authentication providers
- Enable authorization providers
- Enable end-to-end encryption
Enable TLS encryption
To enable TLS encryption, configure the following settings.
// The path to trusted certificates used by the Pulsar client to authenticate with Pulsar brokers
For more details on TLS encryption, refer to Transport Encryption using TLS.
Enable authentication providers
To enable authentication providers on function workers, substitute the
authenticationProviders parameter with the providers you want to enable.
authenticationProviders: [provider1, provider2]
For mTLS authentication provider, follow the example below to add the required settings.
For SASL authentication provider, add
For token authentication provider, add the required settings under
# If using public/private
# tokenPublicKey: file://path/to/public.key
Key files must be DER (Distinguished Encoding Rules)-encoded.
Enable authorization providers
To enable authorization on function workers, complete the following steps.
functions_worker.ymlfile. The authentication provider connects to
configurationMetadataStoreUrlto receive namespace policies.
Configure a list of superuser roles. The superuser roles can access any admin API. The following configuration is an example.
Configure BookKeeper authentication
If authentication is enabled on the BookKeeper cluster, you need to configure the following BookKeeper authentication settings for your function workers.
bookkeeperClientAuthenticationPlugin: the authentication plugin name of BookKeeper client.
bookkeeperClientAuthenticationParametersName: the authentication plugin parameters of BookKeeper client, including names and values.
bookkeeperClientAuthenticationParameters: the authentication plugin parameters of BookKeeper client.
Start function workers
Before starting function workers, make sure function runtime is configured.
You can start a function worker in the background by using the
bin/pulsar-daemon start functions-worker
To start a function worker in the foreground, you can use the
pulsar-adminCLI as follows.
Configure proxies for standalone function workers
When you are running function workers in a separate cluster, the admin rest endpoints are split into two clusters as shown in the following figure. The
sink endpoints are now served by the worker cluster, while all the other remaining endpoints are served by the broker cluster. This requires you to use the right service URL accordingly in the
pulsar-admin CLI. To address this inconvenience, you can start a proxy cluster that serves as the central entry point of the admin service for routing admin rest requests.
If you haven't set up a proxy cluster yet, follow the instructions to deploy one.
To enable a proxy for routing function-related admin requests to function workers, you can edit the
conf/proxy.conf file to modify the following settings: