Skip to main content

Types

Pulsar provides three types of broker load balancers:

The modular and extensible broker load balancer implements similar load balance functionalities with different system designs. For example, they both employ a similar approach to distributing data loads among brokers, including:

However, for bundle ownership and load data stores, the modular load balancer uses ZooKeeper, whereas the extensible load balancer uses System topics and Table views.

Side-by-side comparisons

DimensionModular broker load balancerExtensible broker load balancer
Available versionPulsar 1.7.0

2017
Pulsar 3.0.0

2023
Load Balance Metadata Store and ReplicationDependent on ZooKeeper for load balance metadata store and replication.

- All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers.

- All bundle ownerships are stored in ZooKeeper as ephemeral locks.
Dependent on system topics and table views for load balance metadata store and replication.

- All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views.

- Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view.

- Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views.

Note: The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics.
Load Balance Decision
(Load Balance leader dependency)
Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading.

Note: The leader broker requires global load information
Each broker decides and runs bundle-broker assignment (lookup) and bundle splitting based on the local (replicated) information.

The leader broker still decides on bundle unloading globally.
Pub/sub reconnection upon bundle unloadingWhen bundles are unloaded, the topics in the bundles are temporarily closed, and producers and consumers connect to new owner brokers.

- Connecting to the new owner brokers involves redirecting lookup requests via the leader broker.
Any broker can respond to the lookup requests to locate the new owner brokers without asking the leader broker. The reconnection does not go through the leader broker.
ObservabilityYou can monitor and analyze load balance with various metricsIt adds additional metrics and debug mode. For example:

- Check the status of load balance decision breakdown and failure.

- Use the dynamic debug mode configuration to dynamically turn on and off more load balance decision logs.