The Apache Pulsar community releases version 2.11! 61 contributors provided feature enhancements and fixes that delivered 1617 commits. Thanks for all your contributions.
Highlights of the 2.11 release
- Upgrade Pulsar server Java build version to 17 (PIP-156)
- Support chunking with shared subscriptions (PR-16202)
- Support pluggable topic factory (PR-12235)
- Support configurable compression type for
ManagedCursorInfoto balance storage resource usage (PR-14542)
This blog documents the most noteworthy changes in this release. For the complete list, including all feature enhancements and bug fixes, check out the Pulsar 2.11 Release Notes.
Notable feature enhancements and fixes
Store subscription properties within metadata service (PR-15757)
Restarting the broker or unloading topics may lead to the loss of subscription properties since they are not stored in the metadata service.
Store the subscription properties within the metadata service (
Support configurable compression type for
When the cursor data expands, the data size increases and it takes a lot of time to pull the data.
Introduce various compression algorithms to support cursor compression.
Reference documentation: managedCursorInfoCompressionType
Sync Pulsar metadata across multiple clouds (PR-16425)
Synchronizing configuration metadata (policies) is a critical path to sharing tenant/namespace/topic policies among clusters and administering policies uniformly across all clusters. However, syncing the metadata store (global zookeeper) among Pulsar clusters deployed on separate cloud platforms is not supported in earlier versions.
Use two system topics
configurationmetadataSyncEventTopic along with a metadata synchronizer to sync metadata among clusters deployed on different cloud platforms and persist local topic policies with it.
Fix client memory limit
currentUsage leak in
The client memory limit
currentUsage leaks in
ProducerImpl and causes the producer’s send rate to be slow.
Release the memory when there are failed pending batch messages.
Release semaphore before discarding messages in
There is a race condition in
batchMessageContainer when discarding the messages and releasing the semaphore, which causes the semaphore to be held.
Release the semaphore before discarding the messages in
Support consumer client memory limit (PR-15216)
It takes time for an application with a large number of producers/consumers to select an appropriate value for its queue size. The same applies to topics with many partitions.
- Block the expansion of the consumer receiver’s queue size if the memory usage exceeds 75%.
- Trigger the shrinking of the consumer receiver’s queue size if the memory usage exceeds 95%.
Support adding interceptors for readers (PR-14729)
Pulsar supports adding interceptors for producers and consumers to implement message tracing, but there is no way to add interceptors for readers.
- Add a new interface
ReaderInterceptorto customize the reader interceptor.
- Support setting reader interceptor and auto-update partition configurations by using
User documentation: Create a reader with an interceptor
Record as a new output type of Functions (PR-16041)
In earlier versions, if you want to dynamically set an output topic, message properties, or change the output schema in a Pulsar function, the only way is to create a function that returns
Void. It would be more intuitive to return a structure like
Record that carries this information.
Add a utility method
Context that returns a
FunctionRecord builder initialized with the information from the source record. The builder methods can be used to override these values as needed.
User documentation: Use
Record as function output
Add a universal S3 provider for the offloader (PR-15710)
Pulsar supports cloud storage compatible with S3 APIs, such as AWS and Aliyun, through existing offloaders, but the forced registration of specific metadata limits their use.
Provide a more general offloader
S3 to serve more S3-compatible storage, which uses pure JClouds S3 metadata and allows overriding the default JClouds properties through system properties.
User documentation: Use S3 offloader with Pulsar
PrometheusRawMetricsProvider for Pulsar proxies (PR-14681)
The metrics of plugins could not be exposed to Prometheus for monitoring.
PrometheusRawMetricsProvider in Pulsar proxies so that plugins can add their metrics to Prometheus.
If you are interested in learning more about Pulsar 2.11, you can download and try it out now!
For more information about the Apache Pulsar project and current progress, visit the Pulsar website, follow the project on Twitter @apache_pulsar, and join Pulsar Slack!