Skip to main content

Pulsar command-line tools

Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more.

All Pulsar command-line tools can be run from the bin directory of your installed Pulsar package. The following tools are currently documented:

You can get help for any CLI tool, command, or subcommand using the --help flag, or -h for short. Here's an example:

$ bin/pulsar broker --help

pulsar​

The pulsar tool is used to start Pulsar components, such as bookies and ZooKeeper, in the foreground.

These processes can also be started in the background, using nohup, using the pulsar-daemon tool, which has the same command interface as pulsar.

Usage:


$ pulsar command

Commands:

  • bookie
  • broker
  • compact-topic
  • configuration-store
  • initialize-cluster-metadata
  • proxy
  • standalone
  • websocket
  • zookeeper
  • zookeeper-shell
  • autorecovery

Example:


$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker

The table below lists the environment variables that you can use to configure the pulsar tool.

VariableDescriptionDefault
PULSAR_LOG_CONFLog4j configuration fileconf/log4j2.yaml
PULSAR_BROKER_CONFConfiguration file for brokerconf/broker.conf
PULSAR_BOOKKEEPER_CONFdescription: Configuration file for bookieconf/bookkeeper.conf
PULSAR_ZK_CONFConfiguration file for zookeeperconf/zookeeper.conf
PULSAR_CONFIGURATION_STORE_CONFConfiguration file for the configuration storeconf/global_zookeeper.conf
PULSAR_WEBSOCKET_CONFConfiguration file for websocket proxyconf/websocket.conf
PULSAR_STANDALONE_CONFConfiguration file for standaloneconf/standalone.conf
PULSAR_EXTRA_OPTSExtra options to be passed to the jvm
PULSAR_EXTRA_CLASSPATHExtra paths for Pulsar's classpath
PULSAR_PID_DIRFolder where the pulsar server PID file should be stored
PULSAR_STOP_TIMEOUTWait time before forcefully killing the Bookie server instance if attempts to stop it are not successful
PULSAR_GC_LOGGc options to be passed to the jvm

bookie​

Starts up a bookie server

Usage:


$ pulsar bookie options

Options

OptionDescriptionDefault
-readOnlyForce start a read-only bookie serverfalse
-withAutoRecoveryStart auto-recover service bookie serverfalse

Example


$ PULSAR_BOOKKEEPER_CONF=/path/to/bookkeeper.conf pulsar bookie \
-readOnly \
-withAutoRecovery

broker​

Starts up a Pulsar broker

Usage


$ pulsar broker options

Options

OptionDescriptionDefault
-bc , --bookie-confConfiguration file for BookKeeper
-rb , --run-bookieRun a BookKeeper bookie on the same host as the Pulsar brokerfalse
-ra , --run-bookie-autorecoveryRun a BookKeeper autorecovery daemon on the same host as the Pulsar brokerfalse

Example


$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker

compact-topic​

Run compaction against a Pulsar topic (in a new process)

Usage


$ pulsar compact-topic options

Options

FlagDescriptionDefault
-t , --topicThe Pulsar topic that you would like to compact

Example


$ pulsar compact-topic --topic topic-to-compact

configuration-store​

Starts up the Pulsar configuration store

Usage


$ pulsar configuration-store

Example


$ PULSAR_CONFIGURATION_STORE_CONF=/path/to/configuration_store.conf pulsar configuration-store

initialize-cluster-metadata​

One-time cluster metadata initialization

Usage


$ pulsar initialize-cluster-metadata options

Options

FlagDescriptionDefault
-ub , --broker-service-urlThe broker service URL for the new cluster
-tb , --broker-service-url-tlsThe broker service URL for the new cluster with TLS encryption
-c , --clusterCluster name
-cms , --configuration-metadata-storeThe configuration metadata store quorum connection string
--existing-bk-metadata-service-uriThe metadata service URI of the existing BookKeeper cluster that you want to use
-h , --helpHelp messagefalse
--initial-num-stream-storage-containersThe number of storage containers of BookKeeper stream storage16
--initial-num-transaction-coordinatorsThe number of transaction coordinators assigned in a cluster16
-uw , --web-service-urlThe web service URL for the new cluster
-tw , --web-service-url-tlsThe web service URL for the new cluster with TLS encryption
-md , --metadata-storeThe metadata store service url
--zookeeper-session-timeout-msThe local ZooKeeper session timeout. The time unit is in millisecond(ms)30000

proxy​

Manages the Pulsar proxy

Usage


$ pulsar proxy options

Options

FlagDescriptionDefault
-cms, --configuration-metadata-storeConfiguration metadata store connection string
-md , --metadata-storeMetadata Store service url

Example


$ PULSAR_PROXY_CONF=/path/to/proxy.conf pulsar proxy \
--metadata-store zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 \
--configuration-metadata-store zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181

standalone​

Run a broker service with local bookies and local ZooKeeper

Usage


$ pulsar standalone options

Options

FlagDescriptionDefault
-a , --advertised-addressThe standalone broker advertised address
--bookkeeper-dirLocal bookies’ base data directorydata/standalone/bookkeeper
--bookkeeper-portLocal bookies’ base port3181
--no-brokerOnly start ZooKeeper and BookKeeper services, not the brokerfalse
--num-bookiesThe number of local bookies1
--only-brokerOnly start the Pulsar broker service (not ZooKeeper or BookKeeper)
--wipe-dataClean up previous ZooKeeper/BookKeeper data
--zookeeper-dirLocal ZooKeeper’s data directorydata/standalone/zookeeper
--zookeeper-portLocal ZooKeeper’s port2181

Example


$ PULSAR_STANDALONE_CONF=/path/to/standalone.conf pulsar standalone

websocket​

Usage


$ pulsar websocket

Example


$ PULSAR_WEBSOCKET_CONF=/path/to/websocket.conf pulsar websocket

zookeeper​

Starts up a ZooKeeper cluster

Usage


$ pulsar zookeeper

Example


$ PULSAR_ZK_CONF=/path/to/zookeeper.conf pulsar zookeeper

zookeeper-shell​

Connects to a running ZooKeeper cluster using the ZooKeeper shell

Usage


$ pulsar zookeeper-shell options

Options

FlagDescriptionDefault
-c, --confConfiguration file for ZooKeeper
-serverConfiguration zk address, eg: 127.0.0.1:2181

autorecovery​

Runs an auto-recovery service.

Usage


$ pulsar autorecovery options

Options

FlagDescriptionDefault
-c, --confConfiguration for the autorecoveryN/A

pulsar-client​

The pulsar-client tool

Usage


$ pulsar-client command

Commands

  • produce
  • consume

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}"{"saslJaasClientSectionName":"PulsarClient", "serverType":"broker"}
--auth-pluginAuthentication plugin class nameorg.apache.pulsar.client.impl.auth.AuthenticationSasl
--listener-nameListener name for the broker
--proxy-protocolProxy protocol to select type of routing at proxy
--proxy-urlProxy-server URL to which to connect
--urlBroker URL to which to connectpulsar://localhost:6650/
ws://localhost:8080
-v, --versionGet the version of the Pulsar client
-h, --helpShow this help

produce​

Send a message or messages to a specific broker and topic

Usage


$ pulsar-client produce topic options

Options

FlagDescriptionDefault
-f, --filesComma-separated file paths to send; either -m or -f must be specified[]
-m, --messagesComma-separated string of messages to send; either -m or -f must be specified[]
-n, --num-produceThe number of times to send the message(s); the count of messages/files * num-produce should be below 10001
-r, --rateRate (in messages per second) at which to produce; a value 0 means to produce messages as fast as possible0.0
-db, --disable-batchingDisable batch sending of messagesfalse
-c, --chunkingSplit the message and publish in chunks if the message size is larger than the allowed max sizefalse
-s, --separatorCharacter to split messages string with.","
-k, --keyMessage key to addkey=value string, like k1=v1,k2=v2.
-p, --propertiesProperties to add. If you want to add multiple properties, use the comma as the separator, e.g. k1=v1,k2=v2.
-ekn, --encryption-key-nameThe public key name to encrypt payload.
-ekv, --encryption-key-valueThe URI of public key to encrypt payload. For example, file:///path/to/public.key or data:application/x-pem-file;base64,*****.

consume​

Consume messages from a specific broker and topic

Usage


$ pulsar-client consume topic options

Options

FlagDescriptionDefault
--hexDisplay binary messages in hexadecimal format.false
-n, --num-messagesNumber of messages to consume, 0 means to consume forever.1
-r, --rateRate (in messages per second) at which to consume; a value 0 means to consume messages as fast as possible0.0
--regexIndicate the topic name is a regex patternfalse
-s, --subscription-nameSubscription name
-t, --subscription-typeThe type of the subscription. Possible values: Exclusive, Shared, Failover, Key_Shared.Exclusive
-p, --subscription-positionThe position of the subscription. Possible values: Latest, Earliest.Latest
-m, --subscription-modeSubscription mode. Possible values: Durable, NonDurable.Durable
-q, --queue-sizeThe size of consumer's receiver queue.0
-mc, --max_chunked_msgMax pending chunk messages.0
-ac, --auto_ack_chunk_q_fullAuto ack for the oldest message in consumer's receiver queue if the queue full.false
--hide-contentDo not print the message to the console.false
-st, --schema-typeSet the schema type. Use auto_consume to dump AVRO and other structured data types. Possible values: bytes, auto_consume.bytes
-ekv, --encryption-key-valueThe URI of public key to encrypt payload. For example, file:///path/to/public.key or data:application/x-pem-file;base64,*****.
-pm, --pool-messagesUse the pooled message.true

pulsar-daemon​

A wrapper around the pulsar tool that’s used to start and stop processes, such as ZooKeeper, bookies, and Pulsar brokers, in the background using nohup.

pulsar-daemon has a similar interface to the pulsar command but adds start and stop commands for various services. For a listing of those services, run pulsar-daemon to see the help output or see the documentation for the pulsar command.

Usage


$ pulsar-daemon command

Commands

  • start
  • stop
  • restart

start​

Start a service in the background using nohup.

Usage


$ pulsar-daemon start service

stop​

Stop a service that’s already been started using start.

Usage


$ pulsar-daemon stop service options

Options

FlagDescriptionDefault
-forceStop the service forcefully if not stopped by normal shutdown.false

restart​

Restart a service that has already been started.


$ pulsar-daemon restart service

pulsar-perf​

A tool for performance testing a Pulsar broker.

Usage


$ pulsar-perf command

Commands

  • consume
  • produce
  • read
  • websocket-producer
  • managed-ledger
  • monitor-brokers
  • simulation-client
  • simulation-controller
  • transaction
  • help

Environment variables

The table below lists the environment variables that you can use to configure the pulsar-perf tool.

VariableDescriptionDefault
PULSAR_LOG_CONFLog4j configuration fileconf/log4j2.yaml
PULSAR_CLIENT_CONFConfiguration file for the clientconf/client.conf
PULSAR_EXTRA_OPTSExtra options to be passed to the JVM
PULSAR_EXTRA_CLASSPATHExtra paths for Pulsar's classpath
PULSAR_GC_LOGGc options to be passed to the jvm

consume​

Run a consumer

Usage


$ pulsar-perf consume options

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth-pluginAuthentication plugin class name
-ac, --auto_ack_chunk_q_fullAuto ack for the oldest message in consumer's receiver queue if the queue fullfalse
--listener-nameListener name for the broker
--acks-delay-millisAcknowledgements grouping delay in millis100
--batch-index-ackEnable or disable the batch index acknowledgmentfalse
-bw, --busy-waitEnable or disable Busy-Wait on the Pulsar clientfalse
-v, --encryption-key-value-fileThe file which contains the private key to decrypt payload
-h, --helpHelp messagefalse
-cf, --conf-fileConfiguration file
-m, --num-messagesNumber of messages to consume in total. If the value is equal to or smaller than 0, it keeps consuming messages.0
-e, --expire_time_incomplete_chunked_messagesThe expiration time for incomplete chunk messages (in milliseconds)0
-c, --max-connectionsMax number of TCP connections to a single broker100
-mc, --max_chunked_msgMax pending chunk messages0
-n, --num-consumersNumber of consumers (per topic)1
-ioThreads, --num-io-threadsSet the number of threads to be used for handling connections to brokers1
-lt, --num-listener-threadsSet the number of threads to be used for message listeners1
-ns, --num-subscriptionsNumber of subscriptions (per topic)1
-t, --num-topicsThe number of topics1
-pm, --pool-messagesUse the pooled messagetrue
-r, --rateSimulate a slow message consumer (rate in msg/s)0
-q, --receiver-queue-sizeSize of the receiver queue1000
-p, --receiver-queue-size-across-partitionsMax total size of the receiver queue across partitions50000
--replicatedWhether the subscription status should be replicatedfalse
-u, --service-urlPulsar service URL
-i, --stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled0
-s, --subscriber-nameSubscriber name prefix
-ss, --subscriptionsA list of subscriptions to consume on (e.g. sub1,sub2)sub
-st, --subscription-typeSubscriber type. Possible values are Exclusive, Shared, Failover, Key_Shared.Exclusive
-sp, --subscription-positionSubscriber position. Possible values are Latest, Earliest.Latest
-time, --test-durationTest duration (in seconds). If this value is less than or equal to 0, it keeps consuming messages.0
--trust-cert-filePath for the trusted TLS certificate file
--tls-allow-insecureAllow insecure TLS connection

produce​

Run a producer

Usage


$ pulsar-perf produce options

Options

FlagDescriptionDefault
-am, --access-modeProducer access mode. Valid values are Shared, Exclusive and WaitForExclusiveShared
-au, --admin-urlPulsar admin URL
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth-pluginAuthentication plugin class name
--listener-nameListener name for the broker
-b, --batch-time-windowBatch messages in a window of the specified number of milliseconds1
-bb, --batch-max-bytesMaximum number of bytes per batch4194304
-bm, --batch-max-messagesMaximum number of messages per batch1000
-bw, --busy-waitEnable or disable Busy-Wait on the Pulsar clientfalse
-ch, --chunkingSplit the message and publish in chunks if the message size is larger than allowed max sizefalse
-d, --delayMark messages with a given delay in seconds0s
-z, --compressionCompress messages’ payload. Possible values are NONE, LZ4, ZLIB, ZSTD or SNAPPY.
-cf, --conf-fileConfiguration file
-k, --encryption-key-nameThe public key name to encrypt payload
-v, --encryption-key-value-fileThe file which contains the public key to encrypt payload
-ef, --exit-on-failureExit from the process on publish failurefalse
-fc, --format-classCustom Formatter class nameorg.apache.pulsar.testclient.DefaultMessageFormatter
-fp, --format-payloadFormat %i as a message index in the stream from producer and/or %t as the timestamp nanosecondsfalse
-h, --helpHelp messagefalse
-c, --max-connectionsMax number of TCP connections to a single broker100
-o, --max-outstandingMax number of outstanding messages1000
-p, --max-outstanding-across-partitionsMax number of outstanding messages across partitions50000
-m, --num-messagesNumber of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages.0
-mk, --message-key-generation-modeThe generation mode of message key. Valid options are autoIncrement, random
-ioThreads, --num-io-threadsSet the number of threads to be used for handling connections to brokers1
-n, --num-producersThe number of producers (per topic)1
-threads, --num-test-threadsNumber of test threads1
-t, --num-topicThe number of topics1
-np, --partitionsCreate partitioned topics with the given number of partitions. Setting this value to 0 means not trying to create a topic
-f, --payload-fileUse payload from an UTF-8 encoded text file and a payload will be randomly selected when publishing messages
-e, --payload-delimiterThe delimiter used to split lines when using payload from a file\n
-pn, --producer-nameProducer Name
-r, --ratePublish rate msg/s across topics100
--send-timeoutSet the sendTimeout0
--separatorSeparator between the topic and topic number-
-u, --service-urlPulsar service URL
-s, --sizeMessage size (in bytes)1024
-i, --stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled.0
-time, --test-durationTest duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages.0
--trust-cert-filePath for the trusted TLS certificate file
--warmup-timeWarm-up time in seconds1
--tls-allow-insecureAllow insecure TLS connection

read​

Run a topic reader

Usage


$ pulsar-perf read options

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth-pluginAuthentication plugin class name
--listener-nameListener name for the broker
-cf, --conf-fileConfiguration file
-h, --helpHelp messagefalse
-n, --num-messagesNumber of messages to consume in total. If the value is equal to or smaller than 0, it keeps consuming messages.0
-c, --max-connectionsMax number of TCP connections to a single broker100
-ioThreads, --num-io-threadsSet the number of threads to be used for handling connections to brokers1
-lt, --num-listener-threadsSet the number of threads to be used for message listeners1
-t, --num-topicsThe number of topics1
-r, --rateSimulate a slow message reader (rate in msg/s)0
-q, --receiver-queue-sizeSize of the receiver queue1000
-u, --service-urlPulsar service URL
-m, --start-message-idStart message id. This can be either 'earliest', 'latest' or a specific message id by using 'lid:eid'earliest
-i, --stats-interval-secondsStatistics interval seconds. If 0, statistics will be disabled.0
-time, --test-durationTest duration (in seconds). If this value is less than or equal to 0, it keeps consuming messages.0
--trust-cert-filePath for the trusted TLS certificate file
--use-tlsUse TLS encryption on the connectionfalse
--tls-allow-insecureAllow insecure TLS connection

websocket-producer​

Run a websocket producer

Usage


$ pulsar-perf websocket-producer options

Options

FlagDescriptionDefault
--auth-paramsAuthentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"}.
--auth-pluginAuthentication plugin class name
-cf, --conf-fileConfiguration file
-h, --helpHelp messagefalse
-m, --num-messagesNumber of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages.0
-t, --num-topicThe number of topics1
-f, --payload-fileUse payload from a file instead of empty buffer
-e, --payload-delimiterThe delimiter used to split lines when using payload from a file\n
-fp, --format-payloadFormat %i as a message index in the stream from producer and/or %t as the timestamp nanosecondsfalse
-fc, --format-classCustom formatter class nameorg.apache.pulsar.testclient.DefaultMessageFormatter
-u, --proxy-urlPulsar Proxy URL, e.g., "ws://localhost:8080/"
-r, --ratePublish rate msg/s across topics100
-s, --sizeMessage size in byte1024
-time, --test-durationTest duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages.0

managed-ledger​

Write directly on managed-ledgers

Usage


$ pulsar-perf managed-ledger options

Options

FlagDescriptionDefault
-a, --ack-quorumLedger ack quorum1
-dt, --digest-typeBookKeeper digest type. Possible Values: [CRC32, MAC, CRC32C, DUMMY]CRC32C
-e, --ensemble-sizeLedger ensemble size1
-h, --helpHelp messagefalse
-c, --max-connectionsMax number of TCP connections to a single bookie1
-o, --max-outstandingMax number of outstanding requests1000
-m, --num-messagesNumber of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages.0
-t, --num-topicNumber of managed ledgers1
-r, --rateWrite rate msg/s across managed ledgers100
-s, --sizeMessage size in byte1024
-time, --test-durationTest duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages.0
--threadsNumber of threads writing1
-w, --write-quorumLedger write quorum1
-md, --metadata-storeMetadata store service URL. For example: zk:my-zk:2181

monitor-brokers​

Continuously receive broker data and/or load reports

Usage


$ pulsar-perf monitor-brokers options

Options

FlagDescriptionDefault
--connect-stringA connection string for one or more ZooKeeper servers
-h, --helpHelp messagefalse

simulation-client​

Run a simulation server acting as a Pulsar client. Uses the client configuration specified in conf/client.conf.

Usage


$ pulsar-perf simulation-client options

Options

FlagDescriptionDefault
--portPort to listen on for controller0
--service-urlPulsar Service URL
-h, --helpHelp messagefalse

simulation-controller​

Run a simulation controller to give commands to servers

Usage


$ pulsar-perf simulation-controller options

Options

FlagDescriptionDefault
--client-portThe port that the clients are listening on0
--clientsComma-separated list of client hostnames
--clusterThe cluster to test on
-h, --helpHelp messagefalse

help​

This help message

Usage


$ pulsar-perf help

bookkeeper​

A tool for managing BookKeeper.

Usage


$ bookkeeper command

Commands

  • autorecovery
  • bookie
  • localbookie
  • upgrade
  • shell

Environment variables

The table below lists the environment variables that you can use to configure the bookkeeper tool.

VariableDescriptionDefault
BOOKIE_LOG_CONFLog4j configuration fileconf/log4j2.yaml
BOOKIE_CONFBookKeeper configuration fileconf/bk_server.conf
BOOKIE_EXTRA_OPTSExtra options to be passed to the JVM
BOOKIE_EXTRA_CLASSPATHExtra paths for BookKeeper's classpath
ENTRY_FORMATTER_CLASSThe Java class used to format entries
BOOKIE_PID_DIRFolder where the BookKeeper server PID file should be stored
BOOKIE_STOP_TIMEOUTWait time before forcefully killing the Bookie server instance if attempts to stop it are not successful
BOOKIE_GC_LOGGc options to be passed to the jvm

autorecovery​

Runs an auto-recovery service

Usage


$ bookkeeper autorecovery options

Options

FlagDescriptionDefault
-c, --confConfiguration for the auto-recovery

bookie​

Starts up a BookKeeper server (aka bookie)

Usage


$ bookkeeper bookie options

Options

FlagDescriptionDefault
-c, --confConfiguration for the auto-recovery
-readOnlyForce start a read-only bookie serverfalse
-withAutoRecoveryStart auto-recovery service bookie serverfalse

localbookie​

Runs a test ensemble of N bookies locally

Usage


$ bookkeeper localbookie N

upgrade​

Upgrade the bookie’s filesystem

Usage


$ bookkeeper upgrade options

Options

FlagDescriptionDefault
-c, --confConfiguration for the auto-recovery
-u, --upgradeUpgrade the bookie’s directories

shell​

Run shell for admin commands. To see a full listing of those commands, run bookkeeper shell without an argument.

Usage


$ bookkeeper shell

Example


$ bookkeeper shell bookiesanity

broker-tool​

The broker- tool is used for operations on a specific broker.

Usage


$ broker-tool command

Commands

  • load-report
  • help

Example Two ways to get more information about a command as below:


$ broker-tool help command
$ broker-tool command --help

load-report​

Collect the load report of a specific broker. The command is run on a broker, and used for troubleshooting why broker can’t collect right load report.

Options

FlagDescriptionDefault
-i, --intervalInterval to collect load report, in milliseconds
-h, --helpDisplay help information