Pulsar command-line tools
Pulsar offers several command-line tools that you can use for managing Pulsar installations, performance testing, using command-line producers and consumers, and more.
All Pulsar command-line tools can be run from the bin
directory of your installed Pulsar package. The following tools are currently documented:
Important
This page only shows some frequently used commands. For the latest information about
pulsar
,pulsar-client
, andpulsar-perf
, including commands, flags, descriptions, and more information, see the reference site.You can get help for any CLI tool, command, or subcommand using the
--help
flag, or-h
for short. Here's an example:
$ bin/pulsar broker --help
pulsar
The pulsar tool is used to start Pulsar components, such as bookies and ZooKeeper, in the foreground.
These processes can also be started in the background, using nohup, using the pulsar-daemon tool, which has the same command interface as pulsar.
Usage:
$ pulsar command
Commands:
bookie
broker
compact-topic
configuration-store
initialize-cluster-metadata
proxy
standalone
websocket
zookeeper
zookeeper-shell
autorecovery
Example:
$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker
The table below lists the environment variables that you can use to configure the pulsar
tool.
Variable | Description | Default |
---|---|---|
PULSAR_LOG_CONF | Log4j configuration file | conf/log4j2.yaml |
PULSAR_BROKER_CONF | Configuration file for broker | conf/broker.conf |
PULSAR_BOOKKEEPER_CONF | description: Configuration file for bookie | conf/bookkeeper.conf |
PULSAR_ZK_CONF | Configuration file for zookeeper | conf/zookeeper.conf |
PULSAR_CONFIGURATION_STORE_CONF | Configuration file for the configuration store | conf/global_zookeeper.conf |
PULSAR_WEBSOCKET_CONF | Configuration file for websocket proxy | conf/websocket.conf |
PULSAR_STANDALONE_CONF | Configuration file for standalone | conf/standalone.conf |
PULSAR_EXTRA_OPTS | Extra options to be passed to the jvm | |
PULSAR_EXTRA_CLASSPATH | Extra paths for Pulsar's classpath | |
PULSAR_PID_DIR | Folder where the pulsar server PID file should be stored | |
PULSAR_STOP_TIMEOUT | Wait time before forcefully killing the Bookie server instance if attempts to stop it are not successful | |
PULSAR_GC_LOG | Gc options to be passed to the jvm |
bookie
Starts up a bookie server
Usage:
$ pulsar bookie options
Options
Option | Description | Default |
---|---|---|
-readOnly | Force start a read-only bookie server | false |
-withAutoRecovery | Start auto-recover service bookie server | false |
Example
$ PULSAR_BOOKKEEPER_CONF=/path/to/bookkeeper.conf pulsar bookie \
-readOnly \
-withAutoRecovery
broker
Starts up a Pulsar broker
Usage
$ pulsar broker options
Options
Option | Description | Default |
---|---|---|
-bc , --bookie-conf | Configuration file for BookKeeper | |
-rb , --run-bookie | Run a BookKeeper bookie on the same host as the Pulsar broker | false |
-ra , --run-bookie-autorecovery | Run a BookKeeper autorecovery daemon on the same host as the Pulsar broker | false |
Example
$ PULSAR_BROKER_CONF=/path/to/broker.conf pulsar broker
compact-topic
Run compaction against a Pulsar topic (in a new process)
Usage
$ pulsar compact-topic options
Options
Flag | Description | Default |
---|---|---|
-t , --topic | The Pulsar topic that you would like to compact |
Example
$ pulsar compact-topic --topic topic-to-compact
configuration-store
Starts up the Pulsar configuration store
Usage
$ pulsar configuration-store
Example
$ PULSAR_CONFIGURATION_STORE_CONF=/path/to/configuration_store.conf pulsar configuration-store
initialize-cluster-metadata
One-time cluster metadata initialization
Usage
$ pulsar initialize-cluster-metadata options
Options
Flag | Description | Default |
---|---|---|
-ub , --broker-service-url | The broker service URL for the new cluster | |
-tb , --broker-service-url-tls | The broker service URL for the new cluster with TLS encryption | |
-c , --cluster | Cluster name | |
-cms , --configuration-metadata-store | The configuration metadata store quorum connection string | |
--existing-bk-metadata-service-uri | The metadata service URI of the existing BookKeeper cluster that you want to use | |
-h , --help | Help message | false |
--initial-num-stream-storage-containers | The number of storage containers of BookKeeper stream storage | 16 |
--initial-num-transaction-coordinators | The number of transaction coordinators assigned in a cluster | 16 |
-uw , --web-service-url | The web service URL for the new cluster | |
-tw , --web-service-url-tls | The web service URL for the new cluster with TLS encryption | |
-md , --metadata-store | The metadata store service url | |
--zookeeper-session-timeout-ms | The local ZooKeeper session timeout. The time unit is in millisecond(ms) | 30000 |
proxy
Manages the Pulsar proxy
Usage
$ pulsar proxy options
Options
Flag | Description | Default |
---|---|---|
-cms , --configuration-metadata-store | Configuration metadata store connection string | |
-md , --metadata-store | Metadata Store service url |
Example
$ PULSAR_PROXY_CONF=/path/to/proxy.conf pulsar proxy \
--metadata-store zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181 \
--configuration-metadata-store zk:my-zk-1:2181,my-zk-2:2181,my-zk-3:2181
standalone
Run a broker service with local bookies and local ZooKeeper
Usage
$ pulsar standalone options
Options
Flag | Description | Default |
---|---|---|
-a , --advertised-address | The standalone broker advertised address | |
--bookkeeper-dir | Local bookies’ base data directory | data/standalone/bookkeeper |
--bookkeeper-port | Local bookies’ base port | 3181 |
--no-broker | Only start ZooKeeper and BookKeeper services, not the broker | false |
--num-bookies | The number of local bookies | 1 |
--only-broker | Only start the Pulsar broker service (not ZooKeeper or BookKeeper) | |
--wipe-data | Clean up previous ZooKeeper/BookKeeper data | |
--zookeeper-dir | Local ZooKeeper’s data directory | data/standalone/zookeeper |
--zookeeper-port | Local ZooKeeper’s port | 2181 |
Example
$ PULSAR_STANDALONE_CONF=/path/to/standalone.conf pulsar standalone
websocket
Usage
$ pulsar websocket
Example
$ PULSAR_WEBSOCKET_CONF=/path/to/websocket.conf pulsar websocket
zookeeper
Starts up a ZooKeeper cluster
Usage
$ pulsar zookeeper
Example
$ PULSAR_ZK_CONF=/path/to/zookeeper.conf pulsar zookeeper
zookeeper-shell
Connects to a running ZooKeeper cluster using the ZooKeeper shell
Usage
$ pulsar zookeeper-shell options
Options
Flag | Description | Default |
---|---|---|
-c , --conf | Configuration file for ZooKeeper | |
-server | Configuration zk address, eg: 127.0.0.1:2181 |
autorecovery
Runs an auto-recovery service.
Usage
$ pulsar autorecovery options
Options
Flag | Description | Default |
---|---|---|
-c , --conf | Configuration for the autorecovery | N/A |
pulsar-client
The pulsar-client tool
Usage
$ pulsar-client command
Commands
produce
consume
Options
Flag | Description | Default |
---|---|---|
--auth-params | Authentication parameters, whose format is determined by the implementation of method configure in authentication plugin class, for example "key1:val1,key2:val2" or "{"key1":"val1","key2":"val2"}" | {"saslJaasClientSectionName":"PulsarClient", "serverType":"broker"} |
--auth-plugin | Authentication plugin class name | org.apache.pulsar.client.impl.auth.AuthenticationSasl |
--listener-name | Listener name for the broker | |
--proxy-protocol | Proxy protocol to select type of routing at proxy | |
--proxy-url | Proxy-server URL to which to connect | |
--url | Broker URL to which to connect | pulsar://localhost:6650/ ws://localhost:8080 |
-v , --version | Get the version of the Pulsar client | |
-h , --help | Show this help |
produce
Send a message or messages to a specific broker and topic
Usage
$ pulsar-client produce topic options
Options
Flag | Description | Default |
---|---|---|
-f , --files | Comma-separated file paths to send; either -m or -f must be specified | [] |
-m , --messages | Comma-separated string of messages to send; either -m or -f must be specified | [] |
-n , --num-produce | The number of times to send the message(s); the count of messages/files * num-produce should be below 1000 | 1 |
-r , --rate | Rate (in messages per second) at which to produce; a value 0 means to produce messages as fast as possible | 0.0 |
-db , --disable-batching | Disable batch sending of messages | false |
-c , --chunking | Split the message and publish in chunks if the message size is larger than the allowed max size | false |
-s , --separator | Character to split messages string with. | "," |
-k , --key | Message key to add | key=value string, like k1=v1,k2=v2. |
-p , --properties | Properties to add. If you want to add multiple properties, use the comma as the separator, e.g. k1=v1,k2=v2 . | |
-ekn , --encryption-key-name | The public key name to encrypt payload. | |
-ekv , --encryption-key-value | The URI of public key to encrypt payload. For example, file:///path/to/public.key or data:application/x-pem-file;base64,***** . |
consume
Consume messages from a specific broker and topic
Usage
$ pulsar-client consume topic options
Options
Flag | Description | Default |
---|---|---|
--hex | Display binary messages in hexadecimal format. | false |
-n , --num-messages | Number of messages to consume, 0 means to consume forever. | 1 |
-r , --rate | Rate (in messages per second) at which to consume; a value 0 means to consume messages as fast as possible | 0.0 |
--regex | Indicate the topic name is a regex pattern | false |
-s , --subscription-name | Subscription name | |
-t , --subscription-type | The type of the subscription. Possible values: Exclusive, Shared, Failover, Key_Shared. | Exclusive |
-p , --subscription-position | The position of the subscription. Possible values: Latest, Earliest. | Latest |
-m , --subscription-mode | Subscription mode. Possible values: Durable, NonDurable. | Durable |
-q , --queue-size | The size of consumer's receiver queue. | 0 |
-mc , --max_chunked_msg | Max pending chunk messages. | 0 |
-ac , --auto_ack_chunk_q_full | Auto ack for the oldest message in consumer's receiver queue if the queue full. | false |
--hide-content | Do not print the message to the console. | false |
-st , --schema-type | Set the schema type. Use auto_consume to dump AVRO and other structured data types. Possible values: bytes, auto_consume. | bytes |
-ekv , --encryption-key-value | The URI of public key to encrypt payload. For example, file:///path/to/public.key or data:application/x-pem-file;base64,***** . | |
-pm , --pool-messages | Use the pooled message. | true |
pulsar-daemon
A wrapper around the pulsar tool that’s used to start and stop processes, such as ZooKeeper, bookies, and Pulsar brokers, in the background using nohup.
pulsar-daemon has a similar interface to the pulsar command but adds start and stop commands for various services. For a listing of those services, run pulsar-daemon to see the help output or see the documentation for the pulsar command.
Usage
$ pulsar-daemon command
Commands
start
stop
restart
start
Start a service in the background using nohup.
Usage
$ pulsar-daemon start service
stop
Stop a service that’s already been started using start.
Usage
$ pulsar-daemon stop service options
Options
Flag | Description | Default |
---|---|---|
-force | Stop the service forcefully if not stopped by normal shutdown. | false |
restart
Restart a service that has already been started.
$ pulsar-daemon restart service
pulsar-perf
A tool for performance testing a Pulsar broker.
Usage
$ pulsar-perf command
Commands
consume
produce
read
websocket-producer
managed-ledger
monitor-brokers
simulation-client
simulation-controller
transaction
help
Environment variables
The table below lists the environment variables that you can use to configure the pulsar-perf tool.
Variable | Description | Default |
---|---|---|
PULSAR_LOG_CONF | Log4j configuration file | conf/log4j2.yaml |
PULSAR_CLIENT_CONF | Configuration file for the client | conf/client.conf |
PULSAR_EXTRA_OPTS | Extra options to be passed to the JVM | |
PULSAR_EXTRA_CLASSPATH | Extra paths for Pulsar's classpath | |
PULSAR_GC_LOG | Gc options to be passed to the jvm |
consume
Run a consumer
Usage
$ pulsar-perf consume options
Options
Flag | Description | Default |
---|---|---|
--auth-params | Authentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"} . | |
--auth-plugin | Authentication plugin class name | |
-ac , --auto_ack_chunk_q_full | Auto ack for the oldest message in consumer's receiver queue if the queue full | false |
--listener-name | Listener name for the broker | |
--acks-delay-millis | Acknowledgements grouping delay in millis | 100 |
--batch-index-ack | Enable or disable the batch index acknowledgment | false |
-bw , --busy-wait | Enable or disable Busy-Wait on the Pulsar client | false |
-v , --encryption-key-value-file | The file which contains the private key to decrypt payload | |
-h , --help | Help message | false |
-cf , --conf-file | Configuration file | |
-m , --num-messages | Number of messages to consume in total. If the value is equal to or smaller than 0, it keeps consuming messages. | 0 |
-e , --expire_time_incomplete_chunked_messages | The expiration time for incomplete chunk messages (in milliseconds) | 0 |
-c , --max-connections | Max number of TCP connections to a single broker | 100 |
-mc , --max_chunked_msg | Max pending chunk messages | 0 |
-n , --num-consumers | Number of consumers (per topic) | 1 |
-ioThreads , --num-io-threads | Set the number of threads to be used for handling connections to brokers | 1 |
-lt , --num-listener-threads | Set the number of threads to be used for message listeners | 1 |
-ns , --num-subscriptions | Number of subscriptions (per topic) | 1 |
-t , --num-topics | The number of topics | 1 |
-pm , --pool-messages | Use the pooled message | true |
-r , --rate | Simulate a slow message consumer (rate in msg/s) | 0 |
-q , --receiver-queue-size | Size of the receiver queue | 1000 |
-p , --receiver-queue-size-across-partitions | Max total size of the receiver queue across partitions | 50000 |
--replicated | Whether the subscription status should be replicated | false |
-u , --service-url | Pulsar service URL | |
-i , --stats-interval-seconds | Statistics interval seconds. If 0, statistics will be disabled | 0 |
-s , --subscriber-name | Subscriber name prefix | |
-ss , --subscriptions | A list of subscriptions to consume on (e.g. sub1,sub2) | sub |
-st , --subscription-type | Subscriber type. Possible values are Exclusive, Shared, Failover, Key_Shared. | Exclusive |
-sp , --subscription-position | Subscriber position. Possible values are Latest, Earliest. | Latest |
-time , --test-duration | Test duration (in seconds). If this value is less than or equal to 0, it keeps consuming messages. | 0 |
--trust-cert-file | Path for the trusted TLS certificate file | |
--tls-allow-insecure | Allow insecure TLS connection |
Below are transaction related options.
If you want --txn-timeout
, --numMessage-perTransaction
, -nmt
, -ntxn
, or -abort
take effect, set --txn-enable
to true.
Flag | Description | Default |
---|---|---|
-tto , --txn-timeout | Set the time of transaction timeout (in second). | 10 |
-nmt , --numMessage-perTransaction | The number of messages acknowledged by a transaction. | 50 |
-txn , --txn-enable | Enable or disable a transaction. | false |
-ntxn | The number of opened transactions. 0 means the number of transactions is unlimited. | 0 |
-abort | Abort a transaction. | true |
produce
Run a producer
Usage
$ pulsar-perf produce options
Options
Flag | Description | Default |
---|---|---|
-am , --access-mode | Producer access mode. Valid values are Shared , Exclusive and WaitForExclusive | Shared |
-au , --admin-url | Pulsar admin URL | |
--auth-params | Authentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"} . | |
--auth-plugin | Authentication plugin class name | |
--listener-name | Listener name for the broker | |
-b , --batch-time-window | Batch messages in a window of the specified number of milliseconds | 1 |
-bb , --batch-max-bytes | Maximum number of bytes per batch | 4194304 |
-bm , --batch-max-messages | Maximum number of messages per batch | 1000 |
-bw , --busy-wait | Enable or disable Busy-Wait on the Pulsar client | false |
-ch , --chunking | Split the message and publish in chunks if the message size is larger than allowed max size | false |
-d , --delay | Mark messages with a given delay in seconds | 0s |
-z , --compression | Compress messages’ payload. Possible values are NONE, LZ4, ZLIB, ZSTD or SNAPPY. | |
-cf , --conf-file | Configuration file | |
-k , --encryption-key-name | The public key name to encrypt payload | |
-v , --encryption-key-value-file | The file which contains the public key to encrypt payload | |
-ef , --exit-on-failure | Exit from the process on publish failure | false |
-fc , --format-class | Custom Formatter class name | org.apache.pulsar.testclient.DefaultMessageFormatter |
-fp , --format-payload | Format %i as a message index in the stream from producer and/or %t as the timestamp nanoseconds | false |
-h , --help | Help message | false |
-c , --max-connections | Max number of TCP connections to a single broker | 100 |
-o , --max-outstanding | Max number of outstanding messages | 1000 |
-p , --max-outstanding-across-partitions | Max number of outstanding messages across partitions | 50000 |
-m , --num-messages | Number of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages. | 0 |
-mk , --message-key-generation-mode | The generation mode of message key. Valid options are autoIncrement , random | |
-ioThreads , --num-io-threads | Set the number of threads to be used for handling connections to brokers | 1 |
-n , --num-producers | The number of producers (per topic) | 1 |
-threads , --num-test-threads | Number of test threads | 1 |
-t , --num-topic | The number of topics | 1 |
-np , --partitions | Create partitioned topics with the given number of partitions. Setting this value to 0 means not trying to create a topic | |
-f , --payload-file | Use payload from an UTF-8 encoded text file and a payload will be randomly selected when publishing messages | |
-e , --payload-delimiter | The delimiter used to split lines when using payload from a file | \n |
-pn , --producer-name | Producer Name | |
-r , --rate | Publish rate msg/s across topics | 100 |
--send-timeout | Set the sendTimeout | 0 |
--separator | Separator between the topic and topic number | - |
-u , --service-url | Pulsar service URL | |
-s , --size | Message size (in bytes) | 1024 |
-i , --stats-interval-seconds | Statistics interval seconds. If 0, statistics will be disabled. | 0 |
-time , --test-duration | Test duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages. | 0 |
--trust-cert-file | Path for the trusted TLS certificate file | |
--warmup-time | Warm-up time in seconds | 1 |
--tls-allow-insecure | Allow insecure TLS connection |
Below are transaction related options.
If you want --txn-timeout
, --numMessage-perTransaction
, or -abort
take effect, set --txn-enable
to true.
Flag | Description | Default |
---|---|---|
-tto , --txn-timeout | Set the time of transaction timeout (in second). | 5 |
-nmt , --numMessage-perTransaction | The number of messages acknowledged by a transaction. | 50 |
-txn , --txn-enable | Enable or disable a transaction. | true |
-abort | Abort a transaction. | true |
read
Run a topic reader
Usage
$ pulsar-perf read options
Options
Flag | Description | Default |
---|---|---|
--auth-params | Authentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"} . | |
--auth-plugin | Authentication plugin class name | |
--listener-name | Listener name for the broker | |
-cf , --conf-file | Configuration file | |
-h , --help | Help message | false |
-n , --num-messages | Number of messages to consume in total. If the value is equal to or smaller than 0, it keeps consuming messages. | 0 |
-c , --max-connections | Max number of TCP connections to a single broker | 100 |
-ioThreads , --num-io-threads | Set the number of threads to be used for handling connections to brokers | 1 |
-lt , --num-listener-threads | Set the number of threads to be used for message listeners | 1 |
-t , --num-topics | The number of topics | 1 |
-r , --rate | Simulate a slow message reader (rate in msg/s) | 0 |
-q , --receiver-queue-size | Size of the receiver queue | 1000 |
-u , --service-url | Pulsar service URL | |
-m , --start-message-id | Start message id. This can be either 'earliest', 'latest' or a specific message id by using 'lid:eid' | earliest |
-i , --stats-interval-seconds | Statistics interval seconds. If 0, statistics will be disabled. | 0 |
-time , --test-duration | Test duration (in seconds). If this value is less than or equal to 0, it keeps consuming messages. | 0 |
--trust-cert-file | Path for the trusted TLS certificate file | |
--use-tls | Use TLS encryption on the connection | false |
--tls-allow-insecure | Allow insecure TLS connection |
websocket-producer
Run a websocket producer
Usage
$ pulsar-perf websocket-producer options
Options
Flag | Description | Default |
---|---|---|
--auth-params | Authentication parameters, whose format is determined by the implementation of method configure in authentication plugin class. For example, key1:val1,key2:val2 or {"key1":"val1","key2":"val2"} . | |
--auth-plugin | Authentication plugin class name | |
-cf , --conf-file | Configuration file | |
-h , --help | Help message | false |
-m , --num-messages | Number of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages. | 0 |
-t , --num-topic | The number of topics | 1 |
-f , --payload-file | Use payload from a file instead of empty buffer | |
-e , --payload-delimiter | The delimiter used to split lines when using payload from a file | \n |
-fp , --format-payload | Format %i as a message index in the stream from producer and/or %t as the timestamp nanoseconds | false |
-fc , --format-class | Custom formatter class name | org.apache.pulsar.testclient.DefaultMessageFormatter |
-u , --proxy-url | Pulsar Proxy URL, e.g., "ws://localhost:8080/" | |
-r , --rate | Publish rate msg/s across topics | 100 |
-s , --size | Message size in byte | 1024 |
-time , --test-duration | Test duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages. | 0 |
managed-ledger
Write directly on managed-ledgers
Usage
$ pulsar-perf managed-ledger options
Options
Flag | Description | Default |
---|---|---|
-a , --ack-quorum | Ledger ack quorum | 1 |
-dt , --digest-type | BookKeeper digest type. Possible Values: [CRC32, MAC, CRC32C, DUMMY] | CRC32C |
-e , --ensemble-size | Ledger ensemble size | 1 |
-h , --help | Help message | false |
-c , --max-connections | Max number of TCP connections to a single bookie | 1 |
-o , --max-outstanding | Max number of outstanding requests | 1000 |
-m , --num-messages | Number of messages to publish in total. If this value is less than or equal to 0, it keeps publishing messages. | 0 |
-t , --num-topic | Number of managed ledgers | 1 |
-r , --rate | Write rate msg/s across managed ledgers | 100 |
-s , --size | Message size in byte | 1024 |
-time , --test-duration | Test duration (in seconds). If this value is less than or equal to 0, it keeps publishing messages. | 0 |
--threads | Number of threads writing | 1 |
-w , --write-quorum | Ledger write quorum | 1 |
-md , --metadata-store | Metadata store service URL. For example: zk:my-zk:2181 |
monitor-brokers
Continuously receive broker data and/or load reports
Usage
$ pulsar-perf monitor-brokers options
Options
Flag | Description | Default |
---|---|---|
--connect-string | A connection string for one or more ZooKeeper servers | |
-h , --help | Help message | false |
simulation-client
Run a simulation server acting as a Pulsar client. Uses the client configuration specified in conf/client.conf
.
Usage
$ pulsar-perf simulation-client options
Options
Flag | Description | Default |
---|---|---|
--port | Port to listen on for controller | 0 |
--service-url | Pulsar Service URL | |
-h , --help | Help message | false |