Types
Pulsar provides three types of broker load balancers:
-
Simple broker load balancer: it existed since Pulsar's inception, but now is deprecated.
The modular and extensible broker load balancer implements similar load balance functionalities with different system designs. For example, they both employ a similar approach to distributing data loads among brokers, including:
-
Dynamic bundle-broker assignment
-
Dynamic bundle splitting
-
Dynamic bundle unloading (shedding)
However, for bundle ownership and load data stores, the modular load balancer uses ZooKeeper, whereas the extensible load balancer uses System topics and Table views.
Side-by-side comparisons
Dimension | Modular broker load balancer | Extensible broker load balancer |
---|---|---|
Available version | Pulsar 1.7.0 2017 | Pulsar 3.0.0 2023 |
Load Balance Metadata Store and Replication | Dependent on ZooKeeper for load balance metadata store and replication. - All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. - All bundle ownerships are stored in ZooKeeper as ephemeral locks. | Dependent on system topics and table views for load balance metadata store and replication. - All broker load data are stored in a non-persistent system topic and replicated to all brokers via table views. - Each broker publishes only top k bundles' load data in a non-persistent system topic, and only the leader broker consumes it via a table view. - Bundle ownerships are stored in a persistent system topic and replicated to all brokers via table views. Note: The absolute size of the replicated load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. |
Load Balance Decision (Load Balance leader dependency) | Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading. Note: The leader broker requires global load information | Each broker decides and runs bundle-broker assignment (lookup) and bundle splitting based on the local (replicated) information. The leader broker still decides on bundle unloading globally. |
Pub/sub reconnection upon bundle unloading | When bundles are unloaded, the topics in the bundles are temporarily closed, and producers and consumers connect to new owner brokers. - Connecting to the new owner brokers involves redirecting lookup requests via the leader broker. | Any broker can respond to the lookup requests to locate the new owner brokers without asking the leader broker. The reconnection does not go through the leader broker. |
Observability | You can monitor and analyze load balance with various metrics | It adds additional metrics and debug mode. For example: - Check the status of load balance decision breakdown and failure. - Use the dynamic debug mode configuration to dynamically turn on and off more load balance decision logs. |
Related topics
-
To get a comprehensive understanding and discover the key insights, see Broker load balancing | Overview.
-
To discover different usage scenarios, see Broker load balancing | Use cases.
-
To explore functionalities, see Broker load balancing | Features.
-
To understand advantages, see Broker load balancing | Benefits.
-
To learn essential fundamentals, see Broker load balancing | Concepts.
-
To get up quickly, see Broker load balancing | Quick start.
-
To migrate one broker load balancer type to another, see Broker load balancing | Migration.