Pulsar provides three types of broker load balancers:
Simple broker load balancer: it existed since Pulsar's inception, but now is deprecated.
The modular and extensible broker load balancer implements similar load balance functionalities with different system designs. For example, they both employ a similar approach to distributing data loads among brokers, including:
|Modular broker load balancer
|Extensible broker load balancer
|Load Balance Metadata Store and Replication
|Dependent on ZooKeeper for load balance metadata store and replication.
- All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers.
- All bundle ownerships are stored in ZooKeeper as ephemeral locks.
|Dependent on system topics and table views for load balance metadata store and replication.
- All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views.
- Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view.
- Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views.
Note: The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics.
|Load Balance Decision
(Load Balance leader dependency)
|Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading.
Note: The leader broker requires global load information
|Each broker decides and runs bundle-broker assignment (lookup) and bundle splitting based on the local (replicated) information.
The leader broker still decides on bundle unloading globally.
|Pub/sub reconnection upon bundle unloading
|When bundles are unloaded, the topics in the bundles are temporarily closed, and producers and consumers connect to new owner brokers.
- Connecting to the new owner brokers involves redirecting lookup requests via the leader broker.
|Any broker can respond to the lookup requests to locate the new owner brokers without asking the leader broker. The reconnection does not go through the leader broker.
|You can monitor and analyze load balance with various metrics
|It adds additional metrics and debug mode. For example:
- Check the status of load balance decision breakdown and failure.
- Use the dynamic debug mode configuration to dynamically turn on and off more load balance decision logs.
To get a comprehensive understanding and discover the key insights, see Broker load balancing | Overview.
To discover different usage scenarios, see Broker load balancing | Use cases.
To explore functionalities, see Broker load balancing | Features.
To understand advantages, see Broker load balancing | Benefits.
To learn essential fundamentals, see Broker load balancing | Concepts.
To get up quickly, see Broker load balancing | Quick start.
To migrate one broker load balancer type to another, see Broker load balancing | Migration.