Run a standalone Pulsar cluster in Kubernetes
This section guides you through every step of installing and running Apache Pulsar with Helm on Kubernetes quickly.
For deploying a Pulsar cluster for production usage, read the documentation on how to configure and install a Pulsar Helm chart.
Prerequisite
- Kubernetes server 1.21.0+
- kubectl 1.21.0+
- Helm 3.10+
For the following steps, step 2 and step 3 are for developers and step 4 and step 5 are for administrators.
To run Pulsar with Helm on Kubernetes, follow the steps below.
Step 0: Prepare a Kubernetes cluster
Before installing a Pulsar Helm chart, you have to create a Kubernetes cluster. You can follow the instructions to prepare a Kubernetes cluster.
We use Minikube in this quick start guide. To prepare a Kubernetes cluster, follow these steps:
Create a Kubernetes cluster on Minikube.
minikube start --memory=8192 --cpus=4 --kubernetes-version=<k8s-version>
The
<k8s-version>
can be any Kubernetes version supported by your Minikube installation, such asv1.16.1
.Set
kubectl
to use Minikube.kubectl config use-context minikube
To use the Kubernetes Dashboard with the local Kubernetes cluster on Minikube, enter the command below:
minikube dashboard
The command automatically triggers opening a webpage in your browser.
Step 1: Install Pulsar Helm chart
Add Pulsar charts repo.
helm repo add apache https://pulsar.apache.org/charts
helm repo update
Clone the Pulsar Helm chart repository.
git clone https://github.com/apache/pulsar-helm-chart
cd pulsar-helm-chartRun the script
prepare_helm_release.sh
to create the secrets required for installing the Apache Pulsar Helm chart. The usernamepulsar
and passwordpulsar
are used for logging into the Grafana dashboard and Pulsar Manager. noteWhen running the script, you can use
-n
to specify the Kubernetes namespace where the Pulsar Helm chart is installed,-k
to define the Pulsar Helm release name, and-c
to create the Kubernetes namespace. For more information about the script, run./scripts/pulsar/prepare_helm_release.sh --help
../scripts/pulsar/prepare_helm_release.sh \
-n pulsar \
-k pulsar-mini \
-cUse the Pulsar Helm chart to install a Pulsar cluster to Kubernetes.
helm install \
--values examples/values-minikube.yaml \
--namespace pulsar \
pulsar-mini apache/pulsar
Make sure the values-minikube.yaml
file contains the following lines:
pulsar_manager:
configData:
ENV_SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
SPRING_CONFIGURATION_FILE: "/pulsar-manager/pulsar-manager/application.properties"
PULSAR_MANAGER_OPTS: " -Dlog4j2.formatMsgNoLookups=true"
Check the status of all pods.
kubectl get pods -n pulsar
If all pods start up successfully, you can see that the
STATUS
is changed toRunning
orCompleted
.Output
NAME READY STATUS RESTARTS AGE
pulsar-mini-bookie-0 1/1 Running 0 9m27s
pulsar-mini-bookie-init-5gphs 0/1 Completed 0 9m27s
pulsar-mini-broker-0 1/1 Running 0 9m27s
pulsar-mini-grafana-6b7bcc64c7-4tkxd 1/1 Running 0 9m27s
pulsar-mini-prometheus-5fcf5dd84c-w8mgz 1/1 Running 0 9m27s
pulsar-mini-proxy-0 1/1 Running 0 9m27s
pulsar-mini-pulsar-init-t7cqt 0/1 Completed 0 9m27s
pulsar-mini-pulsar-manager-9bcbb4d9f-htpcs 1/1 Running 0 9m27s
pulsar-mini-toolset-0 1/1 Running 0 9m27s
pulsar-mini-zookeeper-0 1/1 Running 0 9m27sCheck the status of all services in the namespace
pulsar
.kubectl get services -n pulsar
Output
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
pulsar-mini-bookie ClusterIP None <none> 3181/TCP,8000/TCP 11m
pulsar-mini-broker ClusterIP None <none> 8080/TCP,6650/TCP 11m
pulsar-mini-grafana LoadBalancer 10.106.141.246 <pending> 3000:31905/TCP 11m
pulsar-mini-prometheus ClusterIP None <none> 9090/TCP 11m
pulsar-mini-proxy LoadBalancer 10.97.240.109 <pending> 80:32305/TCP,6650:31816/TCP 11m
pulsar-mini-pulsar-manager LoadBalancer 10.103.192.175 <pending> 9527:30190/TCP 11m
pulsar-mini-toolset ClusterIP None <none> <none> 11m
pulsar-mini-zookeeper ClusterIP None <none> 2888/TCP,3888/TCP,2181/TCP 11m
Step 2: Use pulsar-admin to create Pulsar tenants/namespaces/topics
pulsar-admin
is the CLI (Command-Line Interface) tool for Pulsar. In this step, you can use pulsar-admin
to create resources, including tenants, namespaces, and topics.
Enter the
toolset
container.kubectl exec -it -n pulsar pulsar-mini-toolset-0 -- /bin/bash
tipTo perform a health check, you can use the
bin/pulsar-admin brokers healthcheck
command. For more information, see Pulsar admin docs.In the
toolset
container, create a tenant namedapache
.bin/pulsar-admin tenants create apache
Then you can list the tenants to see if the tenant is created successfully.
bin/pulsar-admin tenants list
You should see a similar output as below. The tenant
apache
has been successfully created."apache"
"public"
"pulsar"In the
toolset
container, create a namespace namedpulsar
in the tenantapache
.bin/pulsar-admin namespaces create apache/pulsar
Then you can list the namespaces of tenant
apache
to see if the namespace is created successfully.bin/pulsar-admin namespaces list apache
You should see a similar output as below. The namespace
apache/pulsar
has been successfully created."apache/pulsar"
In the
toolset
container, create a topictest-topic
with4
partitions in the namespaceapache/pulsar
.bin/pulsar-admin topics create-partitioned-topic apache/pulsar/test-topic -p 4
In the
toolset
container, list all the partitioned topics in the namespaceapache/pulsar
.bin/pulsar-admin topics list-partitioned-topics apache/pulsar
Then you can see all the partitioned topics in the namespace
apache/pulsar
."persistent://apache/pulsar/test-topic"
Step 3: Use Pulsar client to produce and consume messages
You can use the Pulsar client to create producers and consumers to produce and consume messages.
By default, the Pulsar Helm chart exposes the Pulsar cluster through a Kubernetes LoadBalancer
. In Minikube, you can use the following command to check the proxy service.
kubectl get services -n pulsar | grep pulsar-mini-proxy
You will see a similar output as below.
pulsar-mini-proxy LoadBalancer 10.97.240.109 <pending> 80:32305/TCP,6650:31816/TCP 28m
This output tells what are the node ports that Pulsar cluster's binary port and HTTP port are mapped to. The port after 80:
is the HTTP port while the port after 6650:
is the binary port.
Then you can find the IP address and exposed ports of your Minikube server by running the following command.
minikube service pulsar-mini-proxy -n pulsar
Output
|-----------|-------------------|-------------|-------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-------------------|-------------|-------------------------|
| pulsar | pulsar-mini-proxy | http/80 | http://172.17.0.4:32305 |
| | | pulsar/6650 | http://172.17.0.4:31816 |
|-----------|-------------------|-------------|-------------------------|
🏃 Starting tunnel for service pulsar-mini-proxy.
|-----------|-------------------|-------------|------------------------|
| NAMESPACE | NAME | TARGET PORT | URL |
|-----------|-------------------|-------------|------------------------|
| pulsar | pulsar-mini-proxy | | http://127.0.0.1:61853 |
| | | | http://127.0.0.1:61854 |
|-----------|-------------------|-------------|------------------------|
At this point, you can get the service URLs to connect to your Pulsar client. Here are URL examples:
webServiceUrl=http://127.0.0.1:61853/
brokerServiceUrl=pulsar://127.0.0.1:61854/
Then you can proceed with the following steps:
Download the Apache Pulsar tarball from the downloads page.
Decompress the tarball based on your download file.
tar -xf <file-name>.tar.gz
Expose
PULSAR_HOME
.(1) Enter the directory of the decompressed download file.
(2) Expose
PULSAR_HOME
as the environment variable.export PULSAR_HOME=$(pwd)
Configure the Pulsar client.
In the
${PULSAR_HOME}/conf/client.conf
file, replacewebServiceUrl
andbrokerServiceUrl
with the service URLs you get from the above steps.Create a subscription to consume messages from
apache/pulsar/test-topic
.bin/pulsar-client consume -s sub apache/pulsar/test-topic -n 0
Open a new terminal. In the new terminal, create a producer and send 10 messages to the
test-topic
topic.bin/pulsar-client produce apache/pulsar/test-topic -m "---------hello apache pulsar-------" -n 10
Verify the results.
From the producer side
Output
The messages have been produced successfully.
18:15:15.489 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 10 messages successfully produced
From the consumer side
Output
At the same time, you can receive the messages as below.
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
----- got message -----
---------hello apache pulsar-------
Step 4: Use Pulsar Manager to manage the cluster
Pulsar Manager is a web-based GUI management tool for managing and monitoring Pulsar.
- To create a superuser account, connect to the pulsar-manager pod and create the account:
kubectl exec -it YOUR_PULSAR_MANAGER_POD_NAME -n pulsar -- /bin/bash
CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)
curl \
-H "X-XSRF-TOKEN: $CSRF_TOKEN" \
-H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
-H 'Content-Type: application/json' \
-X PUT http://localhost:7750/pulsar-manager/users/superuser \
-d '{"name": "pulsar", "password": "pulsar", "description": "test", "email": "username@test.org"}'
By default, the
Pulsar Manager
is exposed as a separateLoadBalancer
. You can open the Pulsar Manager UI using the following command:minikube service -n pulsar pulsar-mini-pulsar-manager
The Pulsar Manager UI will be open in your browser. You can use the username
pulsar
and passwordpulsar
to log into Pulsar Manager.In Pulsar Manager UI, you can create an environment.
- Click New Environment in the upper-left corner.
- Type
pulsar-mini
for the fieldEnvironment Name
in the pop-up window. - Type
http://pulsar-mini-broker:8080
for the fieldService URL
in the pop-up window. - Type
http://pulsar-mini-bookie:8080
for the fieldBookie URL
in the pop-up window. - Click Confirm in the pop-up window.
After successfully creating an environment, you can create
tenants
,namespaces
, andtopics
using the Pulsar Manager.
Step 5: Use Prometheus and Grafana to monitor cluster
Grafana is an open-source visualization tool, which can be used for visualizing time series data into dashboards.
By default, the Grafana is exposed as a separate
LoadBalancer
. You can open the Grafana UI using the following command:minikube service pulsar-mini-grafana -n pulsar
The Grafana UI is open in your browser. You can use the username
pulsar
and passwordpulsar
to log into the Grafana Dashboard.You can view dashboards for different components of a Pulsar cluster.