Pulsar on DC/OS

DC/OS (the DataCenter Operating System) is a distributed operating system used for deploying and managing applications and systems on Apache Mesos. DC/OS is an open-source tool created and maintained by Mesosphere.

Apache Pulsar is available as a Marathon Application Groups, which runs multiple applications as manageable sets.


In order to run Pulsar on DC/OS, we will need:

  • DC/OS version 1.9 or higher
  • A DC/OS cluster with at least three agent nodes
  • The DC/OS CLI tool installed
  • Download and save PulsarGroups.json from Pulsar github repo.

Each node in the DC/OS-managed Mesos cluster must have at least:

  • 4 CPU
  • 4 GB of memory
  • 60 GB of total persistent disk

Or, we could change the configuration in PulsarGroups.json according to the current cluster resources.

Deploy Pulsar use dcos command interface

$ dcos marathon group add PulsarGroups.json

This command will deploy some docker container instances in 3 groups, which consist a Pulsar cluster:

  • deploy 3 bookies, 1 bookie on each agent node; and 1 bookie_recovery instance.
  • deploy 3 Pulsar brokers, 1 broker on each node; and 1 broker_admin instance.
  • deploy 1 prometheus instance and 1 grafana instance.

Since DC/OS is already deployed a zookeeper cluster serving at master.mesos:2181, we leverage it, and no need to deploy it ourselves. After execute the command above, click on the Services tab in the DC/OS GUI interface, whose link is http://m1.dcos in this example, and we should see applications deploying.

DC/OS command executed

DC/OS command executed2

BookKeeper Group

To watch the status of BookKeeper Cluster deployment, we could click into the ‘bookkeeper’ group in parent pulsar group picture.

DC/OS bookkeeper status

Now 3 bookies is shown as green colour, which means they are already deployed successfully and running.

DC/OS bookkeeper running

We could also click into each Instance of bookie, to get more detailed info, such as the bookie running log.

DC/OS bookie log

And we could also get the BookKeeper cluster information in Zookeeper, whose link is http://m1.dcos/exhibitor, and could see 3 bookies under available directory.

DC/OS bookkeeper in zk

Pulsar Broker Group

Similar to BookKeeper Group above, click into brokers group, and we will see brokers status.

DC/OS broker status

DC/OS broker running

We could also click into each Instance of broker, to get more detailed info, such as the broker running log.

DC/OS broker log

And also get Broker cluster information in Zookeeper, which created directory loadbalance and managed-ledgers.

DC/OS broker in zk

Monitor Group

Monitor Group consist of Prometheus and Grafana.

DC/OS monitor status


We could click into the instance of prom to get the endpoint of Prometheus, which is in this example.

DC/OS prom endpoint

While click this endpoint above, and we could get into Prometheus dashboard, and the targets tab, should contains all bookies and brokers.

DC/OS prom targets


We could click into the instance of grafana to get the endpoint of Grafana, which is in this example.

DC/OS grafana endpoint

While click this endpoint above, and we could get into Grafana dashboard.

DC/OS grafana targets

Run simple consumer and producer on DC/OS deployment.

At present we could get a deployed Pulsar cluster. and we can run simple consumer and producer on it now.

Download and prepare pulsar java tutorial

This code contains a simple Pulsar Consumer and Producer, and the README.md contains more information about it.

$ git clone https://github.com/streamlio/pulsar-java-tutorial.git

Change SERVICE_URL from “pulsar://localhost:6650” to “pulsar://a1.dcos:6650” in both ConsumerTutorial.java and ProducerTutorial.java. “pulsar://a1.dcos:6650” is the endpoint of broker service, we could get the endpoint details from DC/OS GUI of each broker instance. a1.dcos is one of DC/OS client agent, which runs a broker, it can also be replaced by the client agent IP address.

Change the message number from 10 to 10000000 in main method of ProducerTutorial.java, so it will produce more message.

Now compile the project code using command:

$ mvn clean package

Run consumer and producer

Execute command to run consumer:

$ mvn exec:java -Dexec.mainClass="tutorial.ConsumerTutorial"

Execute command to run producer:

$ mvn exec:java -Dexec.mainClass="tutorial.ProducerTutorial"

And we could see the producer producing messages and consumer consuming messages.

DC/OS pulsar producer

DC/OS pulsar consumer

View Grafana metric output

While producer and consumer is running, we could get some running metrics information from Grafana

DC/OS pulsar dashboard

Uninstall Pulsar

We can shut down and uninstall the pulsar from DC/OS at any time by 2 ways.

Using DC/OS GUI, we could choose Delete at the right end of Pulsar group.

DC/OS pulsar uninstall

Or we could use command to do the uninstall:

$ dcos marathon group remove /pulsar