Apache Pulsar 2.5.0
2019-12-06
PIPs
- [PIP-41] Introduce Protocol Handler API #5130 #5743
- [PIP-45] Pluggable metadata interface #5330
- [PIP-44] Separate schema compatibility checker for producer and consumer #5227
- [PIP-43] Producer send messages with different schema #5141 #5517
- [PIP-51] Introduce sticky consumer #5388
- [PIP-38] Support batch receive in java client. #4621
- [PIP-52] PIP-52: [pulsar-sever] Add support of dispatch throttling relative to publish-rate #5797
Fixes
- [Broker] Avoid retrying deleting namespace when topic is already deleted/fenced #4665
- [Broker] Fix expiry monitor to continue on non-recoverable error #4818
- [Broker] fix ns-isolation api to fetch policy for specific broker #5314
- [Broker] external protocols not set to local broker data #5749
- [Broker] Add handle exception KeeperException.BadVersionException #5563
- [Broker] Fix message deduplicate issue while using external sequence id with batch produce #5491
- [Broker] Remove cursor while remove non-durable subscription #5719
- [Broker] Fix potential read 0 entries cause dispatcher stop dispatch #5894
- [Proxy] Proxy doesn't use the right ca certicate to connect to brokers #5971
- [Client] Add SentConnectFrame state check when running
handleError
#5913
Enhancements
- [Zookeeper] Bump zookeeper to version 3.5.6 #5043
- [BookKeeper] Upgrade bk version to 4.10.0 #5607
- [Broker] Process requests asynchronously on some REST APIs 4765 4778 4795
- [Broker] Fixes not owned bundles got selected when broker overloading #5002
- [Broker] Support update partition for global topic #5306
- [Broker] Ensure the handling of PartitionMetadataRequest is async end-to-end #5307
- [Broker] Allow to automatically assign TCP ports when starting a broker #3555
- [Broker] Introduce publish rate-limiting on topic #3986
- [Broker] Add publish rate limit for each broker to avoid OOM #5710
- [Broker] Allow for namespace default of offload threshold #5872
- [Broker] Avoid unsafe split when validate hostname which might be ipv6 address #5713
- [Broker] Support batch authorization of partitioned topic #5767
- [Client][Java] Introduce
batchingMaxBytes
setting in pulsar producer #5045 - [Client][Java] Add epoch for connection handler to handle create producer timeout #5571
- [Performance] Reduce char[] creation on jvm heap #5055
- [CLI] Add a broker tool for operations of a specific broker #5768
- [CLI] Validate topic name before creating partition/non partition topic via admin cli #5148
- [CLI] Make PulsarClusterMetadataSetup idempotent #5879
- [CLI] Allow for topic deletions with regex consumers #5230
Stats & Monitoring
- [Broker] Added delayed messages in Prometheus when using namespace-level metrics aggregation #4691
- [Dashboard] Increasing Dashboard consumerName field to 256 varchar 4716
- [Dashboard] integrate peek into messages page #4966
- [Dashboard] Support parse batch entry #4992
Security
- [Broker] Add broker-bookie mTLS support #5042
Tiered Storage
Pulsar Schema
- [Broker] Pulsar schema api should respect to namespace level compatibility #4821
- [Client][Java] NPE is thrown when a consumer consumes a partitioned topic with struct schema #4960
- [Broker] Add compatibility check for primitive schema types #5051
- [Broker] Support uploading key/value schema using Pulsar admin #5000
- [Client][Java] Schema support encoding & encoding ByteBuf #5123
Pulsar IO
- [Broker] Support reload Source and Sink for Pulsar IO 5008
- [Connector] Added Kinesis Source Connector #3784
- [Connector] Add a source connector for MongoDB #5316
- [Connector] Support CDC Connector for MongoDB #5590
- [Connector] Improve hbase sink performance #5705
Pulsar Functions
- [Function Worker] Allow resource overcommitting when running functions in Kubernetes 4829
- [Function Worker] Make Function Authentication Provider pluggable #5404
- [Function Worker] Added deletion of state for Functions #5469
- [Function Worker] Distribute the CA for KubernetesSecretsTokenAuthProvider #5398
- [Function Runtime] Function runtime pluggable #5463
- [Function Runtime] Allow functions to pass runtime specific options #5400
Pulsar SQL
- Support for other schema name separators in pulsar SQL 4732
- Reuse ManagedLedgerFactory instances across SQL queries 4813
- Using pulsar SQL query messages will appear
NoSuchLedger
when zk root directory changed #5001
Adaptors
- Add support of pulsar-kafka-adapter for kafka-0.9 API #4886
- Add support of pulsar-kafka-adapter for kafka-0.8 API #4797
- Make client keepalive interval configurable on pulsar-client-kafka #5131
Transaction
The development of Pulsar Transaction is still ongoing
- [Buffer] Add new marker to show which message belongs to transaction #4776
- [Buffer] Add data ledger position in txn commit marker #4826
- [Buffer] Add basic operation of transaction #4738
- [Buffer] Add new commands for the transaction #4866
- [Protocol] Add default handler to handle transaction related commands #4891
- [Client] Introduce Transaction Client API #4952
- [Client] Add transaction coordinator client #4953
- [Broker] Ownership change listeners #5457
- [Coordinator] Bootstrap pulsar system namespace and create TC assign topic #5515
- [Coordinator] Add transaction metadata store service #5504
For a complete list of issues fixed, see