Skip to main content

Pulsar command-line tools

Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more.

All Pulsar command-line tools can be run from the bin directory of your installed Pulsar package. The following tools are currently documented:

Getting help

You can get help for any CLI tool, command, or subcommand using the --help flag, or -h for short. Here's an example:


$ bin/pulsar broker --help


pulsar

The pulsar tool is used to start Pulsar components, such as bookies and ZooKeeper, in the foreground.

These processes can also be started in the background, using nohup, using the pulsar-daemon tool, which has the same command interface as pulsar.

Usage:


$ pulsar command

Commands:

  • bookie
  • broker
  • compact-topic
  • discovery
  • configuration-store
  • initialize-cluster-metadata
  • proxy
  • standalone
  • websocket
  • zookeeper
  • zookeeper-shell

Example:


$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker

The table below lists the environment variables that you can use to configure the pulsar tool.

VariableDescriptionDefault
PULSAR_LOG_CONFLog4j configuration fileconf/log4j2.yaml
PULSAR_BROKER_CONFConfiguration file for brokerconf/broker.conf
PULSAR_BOOKKEEPER_CONFdescription: Configuration file for bookieconf/bookkeeper.conf
PULSAR_ZK_CONFConfiguration file for zookeeperconf/zookeeper.conf
PULSAR_CONFIGURATION_STORE_CONFConfiguration file for the configuration storeconf/global_zookeeper.conf
PULSAR_DISCOVERY_CONFConfiguration file for discovery serviceconf/discovery.conf
PULSAR_WEBSOCKET_CONFConfiguration file for websocket proxyconf/websocket.conf
PULSAR_STANDALONE_CONFConfiguration file for standaloneconf/standalone.conf
PULSAR_EXTRA_OPTSExtra options to be passed to the jvm
PULSAR_EXTRA_CLASSPATHExtra paths for Pulsar's classpath
PULSAR_PID_DIRFolder where the pulsar server PID file should be stored
PULSAR_STOP_TIMEOUTWait time before forcefully killing the Bookie server instance if attempts to stop it are not successful

bookie

Starts up a bookie server

Usage:


$ pulsar bookie options

Options

OptionDescriptionDefault
-readOnlyForce start a read-only bookie serverfalse
-withAutoRecoveryStart auto-recover service bookie serverfalse

Example


$ PULSAR_BOOKKEEPER_CONF=/path/to/bookkeeper.conf pulsar bookie \
-readOnly \
-withAutoRecovery

broker

Starts up a Pulsar broker

Usage


$ pulsar broker options

Options

OptionDescriptionDefault
-bc , --bookie-confConfiguration file for BookKeeper
-rb , --run-bookieRun a BookKeeper bookie on the same host as the Pulsar brokerfalse
-ra , --run-bookie-autorecoveryRun a BookKeeper autorecovery daemon on the same host as the Pulsar brokerfalse

Example


$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker

compact-topic

Run compaction against a Pulsar topic (in a new process)

Usage


$ pulsar compact-topic options

Options

FlagDescriptionDefault
-t , --topicThe Pulsar topic that you would like to compact

Example


$ pulsar compact-topic --topic topic-to-compact

discovery

Run a discovery server

Usage


$ pulsar discovery

Example


$ PULSAR_DISCOVERY_CONF=/path/to/discovery.conf pulsar discovery

configuration-store

Starts up the Pulsar configuration store

Usage


$ pulsar configuration-store

Example


$ PULSAR_CONFIGURATION_STORE_CONF=/path/to/configuration_store.conf pulsar configuration-store

initialize-cluster-metadata

One-time cluster metadata initialization

Usage


$ pulsar initialize-cluster-metadata options

Options

FlagDescriptionDefault
-ub , --broker-service-urlThe broker service URL for the new cluster
-tb , --broker-service-url-tlsThe broker service URL for the new cluster with TLS encryption
-c , --clusterCluster name
-cs , --configuration-storeThe configuration store quorum connection string
--existing-bk-metadata-service-uriThe metadata service URI of the existing BookKeeper cluster that you want to use
-h , --helpCluster namefalse
--initial-num-stream-storage-containersThe number of storage containers of BookKeeper stream storage16
--initial-num-transaction-coordinatorsThe number of transaction coordinators assigned in a cluster16
-uw , --web-service-urlThe web service URL for the new cluster
-tw , --web-service-url-tlsThe web service URL for the new cluster with TLS encryption
-zk , --zookeeperThe local ZooKeeper quorum connection string
--zookeeper-session-timeout-msThe local ZooKeeper session timeout. The time unit is in millisecond(ms)30000

proxy

Manages the Pulsar proxy

Usage


$ pulsar proxy options

Options

FlagDescriptionDefault
--configuration-storeConfiguration store connection string
-zk , --zookeeper-serversLocal ZooKeeper connection string

Example


$ PULSAR_PROXY_CONF=/path/to/proxy.conf pulsar proxy \
--zookeeper-servers zk-0,zk-1,zk2 \
--configuration-store zk-0,zk-1,zk-2

standalone

Run a broker service with local bookies and local ZooKeeper

Usage


$ pulsar standalone options

Options

FlagDescriptionDefault
-a , --advertised-addressThe standalone broker advertised address
--bookkeeper-dirLocal bookies’ base data directorydata/standalone/bookkeeper
--bookkeeper-portLocal bookies’ base port3181
--no-brokerOnly start ZooKeeper and BookKeeper services, not the brokerfalse
--num-bookiesThe number of local bookies1
--only-brokerOnly start the Pulsar broker service (not ZooKeeper or BookKeeper)
--wipe-dataClean up previous ZooKeeper/BookKeeper data
--zookeeper-dirLocal ZooKeeper’s data directorydata/standalone/zookeeper
--zookeeper-portLocal ZooKeeper’s port2181

Example


$ PULSAR_STANDALONE_CONF=/path/to/standalone.conf pulsar standalone

websocket

Usage


$ pulsar websocket

Example


$ PULSAR_WEBSOCKET_CONF=/path/to/websocket.conf pulsar websocket

zookeeper

Starts up a ZooKeeper cluster

Usage


$ pulsar zookeeper

Example


$ PULSAR_ZK_CONF=/path/to/zookeeper.conf pulsar zookeeper

zookeeper-shell

Connects to a running ZooKeeper cluster using the ZooKeeper shell

Usage


$ pulsar zookeeper-shell options

Options

FlagDescriptionDefault
-c, --confConfiguration file for ZooKeeper
-serverConfiguration zk address, eg: 127.0.0.1:2181

pulsar-client

The pulsar-client tool

Usage


$ pulsar-client command

Commands

  • produce
  • consume

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{\"key1\":\"val1\",\"key2\":\"val2\"}"{"saslJaasClientSectionName":"PulsarClient", "serverType":"broker"}
--auth-pluginAuthentication plugin class nameorg.apache.pulsar.client.impl.auth.AuthenticationSasl
--listener-nameListener name for the broker
--proxy-protocolProxy protocol to select type of routing at proxy
--proxy-urlProxy-server URL to which to connect
--urlBroker URL to which to connectpulsar://localhost:6650/
ws://localhost:8080
-v, --versionGet the version of the Pulsar client
-h, --helpShow this help

produce

Send a message or messages to a specific broker and topic

Usage


$ pulsar-client produce topic options

Options

FlagDescriptionDefault
-f, --filesComma-separated file paths to send; either -m or -f must be specified[]
-m, --messagesComma-separated string of messages to send; either -m or -f must be specified[]
-n, --num-produceThe number of times to send the message(s); the count of messages/files * num-produce should be below 10001
-r, --rateRate (in messages per second) at which to produce; a value 0 means to produce messages as fast as possible0.0
-c, --chunkingSplit the message and publish in chunks if the message size is larger than the allowed max sizefalse
-s, --separatorCharacter to split messages string with.","
-k, --keyMessage key to addkey=value string, like k1=v1,k2=v2.
-p, --propertiesProperties to add. If you want to add multiple properties, use the comma as the separator, e.g. k1=v1,k2=v2.
-ekn, --encryption-key-nameThe public key name to encrypt payload.
-ekv, --encryption-key-valueThe URI of public key to encrypt payload. For example, file:///path/to/public.key or data:application/x-pem-file;base64,*****.

consume

Consume messages from a specific broker and topic

Usage


$ pulsar-client consume topic options

Options

FlagDescriptionDefault
--hexDisplay binary messages in hexadecimal format.false
-n, --num-messagesNumber of messages to consume, 0 means to consume forever.1
-r, --rateRate (in messages per second) at which to consume; a value 0 means to consume messages as fast as possible0.0
--regexIndicate the topic name is a regex patternfalse
-s, --subscription-nameSubscription name
-t, --subscription-typeThe type of the subscription. Possible values: Exclusive, Shared, Failover, Key_Shared.Exclusive
-p, --subscription-positionThe position of the subscription. Possible values: Latest, Earliest.Latest
-m, --subscription-modeSubscription mode. Possible values: Durable, NonDurable.Durable
-q, --queue-sizeThe size of consumer's receiver queue.0
-mc, --max_chunked_msgMax pending chunk messages.0
-ac, --auto_ack_chunk_q_fullAuto ack for the oldest message in consumer's receiver queue if the queue full.false
--hide-contentDo not print the message to the console.false
-st, --schema-typeSet the schema type. Use auto_consume to dump AVRO and other structured data types. Possible values: bytes, auto_consume.bytes
-ekv, --encryption-key-valueThe URI of public key to encrypt payload. For example, file:///path/to/public.key or data:application/x-pem-file;base64,*****.
-pm, --pool-messagesUse the pooled message.true

pulsar-daemon

A wrapper around the pulsar tool that’s used to start and stop processes, such as ZooKeeper, bookies, and Pulsar brokers, in the background using nohup.

pulsar-daemon has a similar interface to the pulsar command but adds start and stop commands for various services. For a listing of those services, run pulsar-daemon to see the help output or see the documentation for the pulsar command.

Usage


$ pulsar-daemon command

Commands

  • start
  • stop

start

Start a service in the background using nohup.

Usage


$ pulsar-daemon start service

stop

Stop a service that’s already been started using start.

Usage


$ pulsar-daemon stop service options

Options

FlagDescriptionDefault
-forceStop the service forcefully if not stopped by normal shutdown.false

pulsar-perf

A tool for performance testing a Pulsar broker.

Usage


$ pulsar-perf command

Commands

  • consume
  • produce
  • read
  • websocket-producer
  • managed-ledger
  • monitor-brokers
  • simulation-client
  • simulation-controller
  • help

Environment variables

The table below lists the environment variables that you can use to configure the pulsar-perf tool.

VariableDescriptionDefault
PULSAR_LOG_CONFLog4j configuration fileconf/log4j2.yaml
PULSAR_CLIENT_CONFConfiguration file for the clientconf/client.conf
PULSAR_EXTRA_OPTSExtra options to be passed to the JVM
PULSAR_EXTRA_CLASSPATHExtra paths for Pulsar's classpath

consume

Run a consumer

Usage


$ pulsar-perf consume options

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth_pluginAuthentication plugin class name
-ac, --auto_ack_chunk_q_fullAuto ack for the oldest message in consumer's receiver queue if the queue fullfalse
--listener-nameListener name for the broker
--acks-delay-millisAcknowledgements grouping delay in millis100
--batch-index-ackEnable or disable the batch index acknowledgmentfalse
-bw, --busy-waitEnable or disable Busy-Wait on the Pulsar clientfalse
-v, --encryption-key-value-fileThe file which contains the private key to decrypt payload
-h, --helpHelp messagefalse
--conf-fileConfiguration file
-e, --expire_time_incomplete_chunked_messagesThe expiration time for incomplete chunk messages (in milliseconds)0
-c, --max-connectionsMax number of TCP connections to a single broker100
-mc, --max_chunked_msgMax pending chunk messages0
-n, --num-consumersNumber of consumers (per topic)1
-ioThreads, --num-io-threadsSet the number of threads to be used for handling connections to brokers1
-ns, --num-subscriptionsNumber of subscriptions (per topic)1
-t, --num-topicsThe number of topics1
-pm, --pool-messagesUse the pooled messagetrue
-r, --rateSimulate a slow message consumer (rate in msg/s)0
-q, --receiver-queue-sizeSize of the receiver queue1000
-p, --receiver-queue-size-across-partitionsMax total size of the receiver queue across partitions50000
--replicatedWhether the subscription status should be replicatedfalse
-u, --service-urlPulsar service URL
-i, --stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled0
-s, --subscriber-nameSubscriber name prefixsub
-ss, --subscriptionsA list of subscriptions to consume on (e.g. sub1,sub2)sub
-st, --subscription-typeSubscriber type. Possible values are Exclusive, Shared, Failover, Key_Shared.Exclusive
-sp, --subscription-positionSubscriber position. Possible values are Latest, Earliest.Latest
-time, --test-durationTest duration (in seconds). If the value is 0 or smaller than 0, it keeps consuming messages0
--trust-cert-filePath for the trusted TLS certificate file
--tls-allow-insecureAllow insecure TLS connection

produce

Run a producer

Usage


$ pulsar-perf produce options

Options

FlagDescriptionDefault
-am, --access-modeProducer access mode. Valid values are Shared, Exclusive and WaitForExclusiveShared
-au, --admin-urlPulsar admin URL
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth_pluginAuthentication plugin class name
--listener-nameListener name for the broker
-b, --batch-time-windowBatch messages in a window of the specified number of milliseconds1
-bb, --batch-max-bytesMaximum number of bytes per batch4194304
-bm, --batch-max-messagesMaximum number of messages per batch1000
-bw, --busy-waitEnable or disable Busy-Wait on the Pulsar clientfalse
-ch, --chunkingSplit the message and publish in chunks if the message size is larger than allowed max sizefalse
-d, --delayMark messages with a given delay in seconds0s
-z, --compressionCompress messages’ payload. Possible values are NONE, LZ4, ZLIB, ZSTD or SNAPPY.
--conf-fileConfiguration file
-k, --encryption-key-nameThe public key name to encrypt payload
-v, --encryption-key-value-fileThe file which contains the public key to encrypt payload
-ef, --exit-on-failureExit from the process on publish failurefalse
-fc, --format-classCustom Formatter class nameorg.apache.pulsar.testclient.DefaultMessageFormatter
-fp, --format-payloadFormat %i as a message index in the stream from producer and/or %t as the timestamp nanosecondsfalse
-h, --helpHelp messagefalse
-c, --max-connectionsMax number of TCP connections to a single broker100
-o, --max-outstandingMax number of outstanding messages1000
-p, --max-outstanding-across-partitionsMax number of outstanding messages across partitions50000
-mk, --message-key-generation-modeThe generation mode of message key. Valid options are autoIncrement, random
-ioThreads, --num-io-threadsSet the number of threads to be used for handling connections to brokers1
-m, --num-messagesNumber of messages to publish in total. If the value is 0 or smaller than 0, it keeps publishing messages.0
-n, --num-producersThe number of producers (per topic)1
-threads, --num-test-threadsNumber of test threads1
-t, --num-topicThe number of topics1
-np, --partitionsCreate partitioned topics with the given number of partitions. Setting this value to 0 means not trying to create a topic
-f, --payload-fileUse payload from an UTF-8 encoded text file and a payload will be randomly selected when publishing messages
-e, --payload-delimiterThe delimiter used to split lines when using payload from a file\n
-pn, --producer-nameProducer Name
-r, --ratePublish rate msg/s across topics100
--send-timeoutSet the sendTimeout0
--separatorSeparator between the topic and topic number-
-u, --service-urlPulsar service URL
-s, --sizeMessage size (in bytes)1024
-i, --stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled.0
-time, --test-durationTest duration (in seconds). If the value is 0 or smaller than 0, it keeps publishing messages.0
--trust-cert-filePath for the trusted TLS certificate file
--warmup-timeWarm-up time in seconds1
--tls-allow-insecureAllow insecure TLS connection

read

Run a topic reader

Usage


$ pulsar-perf read options

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth_pluginAuthentication plugin class name
--listener-nameListener name for the broker
--conf-fileConfiguration file
-h, --helpHelp messagefalse
-c, --max-connectionsMax number of TCP connections to a single broker100
-ioThreads, --num-io-threadsSet the number of threads to be used for handling connections to brokers1
-t, --num-topicsThe number of topics1
-r, --rateSimulate a slow message reader (rate in msg/s)0
-q, --receiver-queue-sizeSize of the receiver queue1000
-u, --service-urlPulsar service URL
-m, --start-message-idStart message id. This can be either 'earliest', 'latest' or a specific message id by using 'lid:eid'earliest
-i, --stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled.0
-time, --test-durationTest duration (in seconds). If the value is 0 or smaller than 0, it keeps consuming messages.0
--trust-cert-filePath for the trusted TLS certificate file
--use-tlsUse TLS encryption on the connectionfalse
--tls-allow-insecureAllow insecure TLS connection

websocket-producer

Run a websocket producer

Usage


$ pulsar-perf websocket-producer options

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth_pluginAuthentication plugin class name
--conf-fileConfiguration file
-h, --helpHelp messagefalse
-m, --num-messagesNumber of messages to publish in total. If the value is 0 or smaller than 0, it keeps publishing messages0
-t, --num-topicThe number of topics1
-f, --payload-fileUse payload from a file instead of empty buffer
-u, --proxy-urlPulsar Proxy URL, e.g., "ws://localhost:8080/"
-r, --ratePublish rate msg/s across topics100
-s, --sizeMessage size in byte1024
-time, --test-durationTest duration (in seconds). If the value is 0 or smaller than 0, it keeps publishing messages0

managed-ledger

Write directly on managed-ledgers

Usage


$ pulsar-perf managed-ledger options

Options

FlagDescriptionDefault
-a, --ack-quorumLedger ack quorum1
-dt, --digest-typeBookKeeper digest type. Possible Values: [CRC32, MAC, CRC32C, DUMMY]CRC32C
-e, --ensemble-sizeLedger ensemble size1
-h, --helpHelp messagefalse
-c, --max-connectionsMax number of TCP connections to a single bookie1
-o, --max-outstandingMax number of outstanding requests1000
-m, --num-messagesNumber of messages to publish in total. If the value is 0 or smaller than 0, it keeps publishing messages0
-t, --num-topicNumber of managed ledgers1
-r, --rateWrite rate msg/s across managed ledgers100
-s, --sizeMessage size in byte1024
-time, --test-durationTest duration (in seconds). If the value is 0 or smaller than 0, it keeps publishing messages0
--threadsNumber of threads writing1
-w, --write-quorumLedger write quorum1
-zk, --zookeeperServersZooKeeper connection string

monitor-brokers

Continuously receive broker data and/or load reports

Usage


$ pulsar-perf monitor-brokers options

Options

FlagDescriptionDefault
--connect-stringA connection string for one or more ZooKeeper servers
-h, --helpHelp messagefalse

simulation-client

Run a simulation server acting as a Pulsar client. Uses the client configuration specified in conf/client.conf.

Usage


$ pulsar-perf simulation-client options

Options

FlagDescriptionDefault
--portPort to listen on for controller0
--service-urlPulsar Service URL
-h, --helpHelp messagefalse

simulation-controller

Run a simulation controller to give commands to servers

Usage


$ pulsar-perf simulation-controller options

Options

FlagDescriptionDefault
--client-portThe port that the clients are listening on0
--clientsComma-separated list of client hostnames
--clusterThe cluster to test on
-h, --helpHelp messagefalse

help

This help message

Usage


$ pulsar-perf help

bookkeeper

A tool for managing BookKeeper.

Usage


$ bookkeeper command

Commands

  • autorecovery
  • bookie
  • localbookie
  • upgrade
  • shell

Environment variables

The table below lists the environment variables that you can use to configure the bookkeeper tool.

VariableDescriptionDefault
BOOKIE_LOG_CONFLog4j configuration fileconf/log4j2.yaml
BOOKIE_CONFBookKeeper configuration fileconf/bk_server.conf
BOOKIE_EXTRA_OPTSExtra options to be passed to the JVM
BOOKIE_EXTRA_CLASSPATHExtra paths for BookKeeper's classpath
ENTRY_FORMATTER_CLASSThe Java class used to format entries
BOOKIE_PID_DIRFolder where the BookKeeper server PID file should be stored
BOOKIE_STOP_TIMEOUTWait time before forcefully killing the Bookie server instance if attempts to stop it are not successful

auto-recovery

Runs an auto-recovery service

Usage


$ bookkeeper autorecovery options

Options

FlagDescriptionDefault
-c, --confConfiguration for the auto-recovery

bookie

Starts up a BookKeeper server (aka bookie)

Usage


$ bookkeeper bookie options

Options

FlagDescriptionDefault
-c, --confConfiguration for the auto-recovery
-readOnlyForce start a read-only bookie serverfalse
-withAutoRecoveryStart auto-recovery service bookie serverfalse

localbookie

Runs a test ensemble of N bookies locally

Usage


$ bookkeeper localbookie N

upgrade

Upgrade the bookie’s filesystem

Usage


$ bookkeeper upgrade options

Options

FlagDescriptionDefault
-c, --confConfiguration for the auto-recovery
-u, --upgradeUpgrade the bookie’s directories

shell

Run shell for admin commands. To see a full listing of those commands, run bookkeeper shell without an argument.

Usage


$ bookkeeper shell

Example


$ bookkeeper shell bookiesanity

broker-tool

The broker- tool is used for operations on a specific broker.

Usage


$ broker-tool command

Commands

  • load-report
  • help

Example Two ways to get more information about a command as below:


$ broker-tool help command
$ broker-tool command --help

load-report

Collect the load report of a specific broker. The command is run on a broker, and used for troubleshooting why broker can’t collect right load report.

Options

FlagDescriptionDefault
-i, --intervalInterval to collect load report, in milliseconds
-h, --helpDisplay help information