Pulsar on DC/OS
DC/OS (the DataCenter Operating System) is a distributed operating system used for deploying and managing applications and systems on Apache Mesos. DC/OS is an open-source tool created and maintained by Mesosphere.
Apache Pulsar is available as a Marathon Application Groups, which runs multiple applications as manageable sets.
In order to run Pulsar on DC/OS, we will need:
- DC/OS version 1.9 or higher
- A DC/OS cluster with at least three agent nodes
- The DC/OS CLI tool installed
- Download and save
PulsarGroups.jsonfrom Pulsar github repo.
Each node in the DC/OS-managed Mesos cluster must have at least:
- 4 CPU
- 4 GB of memory
- 60 GB of total persistent disk
Or, we could change the configuration in
PulsarGroups.json according to the current cluster resources.
Deploy Pulsar use dcos command interface
$ dcos marathon group add PulsarGroups.json
This command will deploy some docker container instances in 3 groups, which consist a Pulsar cluster:
- deploy 3 bookies, 1
bookieon each agent node; and 1
- deploy 3 Pulsar brokers, 1
brokeron each node; and 1
- deploy 1
prometheusinstance and 1
Since DC/OS is already deployed a zookeeper cluster serving at
master.mesos:2181, we leverage it, and no need to deploy it ourselves.
After execute the command above, click on the Services tab in the DC/OS GUI interface, whose link is
http://m1.dcos in this example, and we should see applications deploying.
To watch the status of
BookKeeper Cluster deployment, we could click into the ‘bookkeeper’ group in parent pulsar group picture.
Now 3 bookies is shown as green colour, which means they are already deployed successfully and running.
We could also click into each Instance of bookie, to get more detailed info, such as the bookie running log.
And we could also get the BookKeeper cluster information in Zookeeper, whose link is
http://m1.dcos/exhibitor, and could see 3 bookies under
Pulsar Broker Group
Similar to BookKeeper Group above, click into
brokers group, and we will see brokers status.
We could also click into each Instance of broker, to get more detailed info, such as the broker running log.
And also get Broker cluster information in Zookeeper, which created directory
Monitor Group consist of Prometheus and Grafana.
We could click into the instance of
prom to get the endpoint of Prometheus, which is
192.168.65.121:9090 in this example.
While click this endpoint above, and we could get into Prometheus dashboard, and the targets
http://192.168.65.121:9090/targets tab, should contains all bookies and brokers.
We could click into the instance of
grafana to get the endpoint of Grafana, which is
192.168.65.121:3000 in this example.
While click this endpoint above, and we could get into Grafana dashboard.
Run simple consumer and producer on DC/OS deployment.
At present we could get a deployed Pulsar cluster. and we can run simple consumer and producer on it now.
Download and prepare pulsar java tutorial
This code contains a simple Pulsar Consumer and Producer, and the
README.md contains more information about it.
$ git clone https://github.com/streamlio/pulsar-java-tutorial.git
SERVICE_URL from “pulsar://localhost:6650” to “pulsar://a1.dcos:6650” in both ConsumerTutorial.java and ProducerTutorial.java.
“pulsar://a1.dcos:6650” is the endpoint of broker service, we could get the endpoint details from DC/OS GUI of each broker instance.
a1.dcos is one of DC/OS client agent, which runs a broker, it can also be replaced by the client agent IP address.
Change the message number from 10 to 10000000 in main method of ProducerTutorial.java, so it will produce more message.
Now compile the project code using command:
$ mvn clean package
Run consumer and producer
Execute command to run consumer:
$ mvn exec:java -Dexec.mainClass="tutorial.ConsumerTutorial"
Execute command to run producer:
$ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial"
And we could see the producer producing messages and consumer consuming messages.
View Grafana metric output
While producer and consumer is running, we could get some running metrics information from Grafana
We can shut down and uninstall the
pulsar from DC/OS at any time by 2 ways.
Using DC/OS GUI, we could choose
Delete at the right end of Pulsar group.
Or we could use command to do the uninstall:
$ dcos marathon group remove /pulsar