Apache Pulsar 3.2.0
2024-02-05β
PIPsβ
- PIP-323: Complete Backlog Quota Telemetry #21709
- PIP-322: Pulsar Rate Limiting Refactoring #21680
- PIP-320: OpenTelemetry Scaffolding #21635
- PIP-318: Don't retain null-key messages during topic compaction #21541
- PIP-315: Configurable max delay limit for delayed delivery #21490
- PIP-313: Support force unsubscribe using consumer api #21452
- PIP-312:
- PIP-307:
- PIP-305: Customize DNS servers to use for Pulsar Client #21352
- PIP-303: Add optional parameters for getPartitionedStats #21228
- PIP-302: Introduce refreshAsync API for TableView #21271
- PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD #21129
- PIP-300: Add custom dynamic configuration for plugins #21127
- PIP-299: Stop dispatch messages if the individual acks will be lost in the persistent storage #21118
- PIP-297: Support terminating Function & Connector with the fatal exception #21079
- PIP-293: Delete config
disableBrokerInterceptors
#20993 - PIP-290: Provide a way to implement WSS E2E encryption and not need to expose the private key to the WebSocket Proxy #20923
- PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index #20867
- PIP-277: Add
current
option in the Clusters list cmd #20614 - PIP-264: Enhanced OTel-based metric system #21080
Brokerβ
- Replication stopped due to unload topic failed #21947
- Fix getMessageById throws 500 #21919
- Fix write all compacted out entry into compacted topic #21917
- Fix deadlock while skip non-recoverable ledgers #21915
- Fix leader broker cannot be determined when the advertised address and advertised listeners are configured #21894
- Fix DLQ producer name conflicts when multiples consumers send messages to DLQ #21890
- Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 #21885
- Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException #21856
- Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21842
- Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher #21765
- Fix the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764
- Fix closing of AbstractDispatcherSingleActiveConsumer and reduce flakiness of test #21736
- Skip topic auto-creation for ExtensibleLoadManager internal topics #21729
- Fix ServiceUnitStateChannel monitor to tombstone only inactive bundle states #21721
- Avoid compaction task stuck when the last message to compact is a marker #21718
- Fix the issue of topics possibly being deleted #21704
- CheckTopicExists supports checking partitioned topic without index #21701
- Remove internalGetListAsync and validatePartitionTopicUpdateAsync #21699
- Cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission #21696
- Fix typo in the config key #21690
- network package lost if enable haProxyProtocolEnabled #21684
- Pulsar Rate Limiting Refactoring changes (PIP-322) #21681
- Record GeoPersistentReplicator.msgOut before producersendAsync #21673
- Print recoverBucketSnapshot log if cursorProperties are empty #21651
- Fix memory leak during topic compaction #21647
- Fix lookupRequestSemaphore leak when topic not found #21646
- Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21633
- Fix getting incorrect KeyValue schema version #21632
- Add optional parameters for getPartitionedStats #21611
- PIP-301 Part-3: Add QuotaResources #21596
- Delete topic timeout due to NPE #21595
- Fix incorrect unack count when using shared subscription on non-persistent topic #21592
- Duplicate LedgerOffloader creation when namespace/topic⦠#21591
- Support not retaining null-key message during topic compaction #21578
- Correct schema deletion for partitioned topic #21574
- Fix create topic with different auto creation strategies causes race condition #21545
- Fix thousands orphan PersistentTopic caused OOM #21540
- Fix setReplicatedSubscriptionStatus incorrect behavior #21510
- Fix failure while creating non-durable cursor with inactive managed-ledger #21508
- Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 #21496
- Do not write replicated snapshot marker when the topic which is not enable replication #21495
- Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
- Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
- Fix issue with consumer read uncommitted messages from compacted topic #21465
- Delete methods marked Lombok @Getter #21462
- Fix resource_quota_zpath #21461
- Fix namespace bundle stuck in unloading status #21445
- Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered #21434
- Fix MultiRoles token provider NPE when using anonymous clients #21429
- Allow broker deployment in heterogeneous hw config cluster without restricting nic speed detection #21409
- Fix avoid creating new topic after migration is started #21368
- Fix race condition of replication cluster connection during migration topic #21364
- Optimize and clean up aggregation of topic stats #21361
- Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
- PIP-188 Fix cluster migration state store into local metadatastore #21359
- PIP-188 Add support to auto create topic resources into green cluster before migration [part-3][#21354](https://github.com/apache/pulsar/pull/21354)
- Fix heartbeat namespace create transaction internal topic #21348
- Fix unload operation stuck when use ExtensibleLoadManager #21332
- Fix typos in Subscription interface #21321
- Use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error #21282
- Cleanup deprecated method of metadata store etcd #21253
- Miss headersAndPayload and messageIdData in MessagePublishContext #21245
- Improve error messages when updating partition count #21243
- Fixed produce and consume when anonymousUserRole enabled #21237
- Fix inconsistent topic policy #21231
- Fixed reset for AggregatedNamespaceStats #21225
- fix flaky test PatternTopicsConsumerImplTest #21222
- Let the producer request success at the first time if the previous one is inactive #21220
- Fix lookup heartbeat and sla namespace bundle when using extensible load manager #21213
- Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211
- Make the TopicCompactionService to support find entry based on publishTime or index #21208
- Replicator leak when removeReplicator in NonPersistentTopic #21205
- Print warn log if ssl handshake error & print ledger id when switch ledger #21201
- Avoid record inactiveproducers when deduplication is disable. #21193
- Make the new exclusive consumer instead the inactive one faster #21183
- improve read entry error log for troubleshooting #21169
- Disable balancing based on DirectMemory. #21168
- Make specified producer could override the previous one #21155
- Improve logs for troubleshooting #21141
- Fix web tls url null cause NPE #21137
- Fix unack count when mixing non batch index and batch index acks #21126
- Fix unsubscribe non-durable subscription error #21099
- Fix isolated group not work problem. #21096
- Fix deleting topic not delete the related topic policy and schema. #21093
- revert remove duplicate topics name when deleteNamespace #21087
- Cleanup correctly heartbeat bundle ownership when handling broker deletion event #21083
- Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
- Remove bundle-data in local metadata store. #21078
- Fix typos: lister -> listener #21068
- Make sure all inflight writes have finished before completion of compaction #21067
- Make read compacted entries support maxReadSizeBytes limitation #21065
- Improve cache handling for partitioned topic metadata when doing lookup #21063
- Use MessageDigest.isEqual when comparing digests #21061
- Removing webURL check null #21043
- Fix potential case cause retention policy not working on topic level #21041
- Fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21025
- Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled #21015
- PIP-293: Delete config disableBrokerInterceptors #20995
- Fix incorrect unack msk count when dup ack a message #20990
- Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
- Fix get outdated compactedTopicContext after compactionHorizon has been updated #20984
- Fix compaction subscription delete by inactive subscription check. #20983
- Fix message loss during topic compaction #20980
- Fix incorrect number of read compacted entries #20978
- Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there #20951
- Fix http error message of Bookie rack placement config API when bookie-rack is not configured. #20949
- Fix chunked messages will be filtered by duplicating #20948
- Rackaware policy is ineffective when delete zk rack info after bkclient initialize #20944
- Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin #20939
- Fix consumers are not redirected to migrated cluster #20928
- PIP-300: Add custom dynamic configuration for plugins #20884
- Delete unused param #20866
- Improve performance of
StatsOutputStream
by usingArrayDeque
instead ofStack
#20808 - Fix missing generate some metrics in BrokerOperabilityMetrics #20720
- Fix ModularLoadManagerImpl always delete active bundle-data. sec ver. #20620
- Consider iowait as idle. #19110
- Fix bug caused by optimistic locking #18390
Pulsar IO and Pulsar Functionsβ
- [IO] Make connectors load sensitive fields from secrets #21675
- [IO] PIP-297: Support terminating Function & Connector with the fatal exception #21143
- [IO] Fix --retain[-key]-ordering not working error for sink #21060
- [IO] Improve kinesis connector config. #21004
- [IO] Update test certs for Elasticsearch #21001
- [IO] Support configuration secret interpolation #20901
- [FN] Add missing version field back to querystate API #21966
- [FN] Throw 404 RestException when state key not found #21921
- [FN] Fix Deadlock in Functions Worker LeaderService #21711
- [FN] Fix functions_log4j2.xml delete strategy config #21215
- [FN] Fix ProducerConfig cannot update error #21037
- [FN] Fix the --batch-builder not working error for functions #21023
- [FN] Make python install dependencies from requirements.txt #20174g function removal to avoid label errors #19584
CLIβ
- [Admin] Set ns level backlog quota does not take effect if retention exists #20690
- [Admin] Add clusters check when set replication clusters #21650
- [Admin] Fix KeyValue schema compatibility check caused OOM #21645
- [Admin] Do not print full stacktrace when get partitioned metadata not found #20979
- [Admin] Fix get topic stats fail if a subscription catch up concurrently #20971
- [CLI] Fix the variable producerName of BatchMsgContainer is null #20819
- [CLI] Introduce Regex and File Input Parameters for Enhanced Topic Deletion Command #21664
- [CLI] Add
get-cluster-migration
cmd #21473 - [CLI] Add current option in the Clusters list cmd #21139
- [CLI] Add some checks for topic-level
setOffloadPolicies
#20943 - [CLI] Create new
pulsar-cli-utils
module #20782 - [CLI] Refactor
pulsar-client-tools
module #20764 - [CLI] Add command line option for configuring the memory limit #20663
Othersβ
- [TXN] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly #21505
- [TXN] Fix getting last message ID when there are ongoing transactions #21466
- [TXN] Fix the consumer stuck due to deduplicated messages in pending ack state #21177
- [ML] Fix orphan scheduled task for ledger create timeout check #21542
- [ML] Fix unfinished callback when deleting managed ledger #21530
- [ML] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads #21333
- [ML] Fix thread safe issue with RangeCache.put and RangeCache.clear #21302
- [ML] Fix wrong msg backlog of non-durable cursor after trim ledgers #21250
- [ML] Reader can set read-pos to a deleted ledger #21248
- [ML] Persist mark deleted ops to ZK if create cursor ledger was failed #20935
- [ML] Fix discontinuous ledger deletion #20898
- [META] Improve fault tolerance of blocking calls by supporting timeout #21028
- [META] Fix deadlock in AutoRecovery. #21010
- [WS] Support subscribing multi/pattern topic for Websocket #21379
- [WS] PIP-290 Make WSS support E2E encryption #20958
- [MISC] Add Pulsar BOM (Bill of Materials) #21871
- [MISC] Pin Netty version in pulsar-io/alluxio #21728
- [MISC] Disable rebase and merge to prevent unsquashed PRs #21517
- [Build] Increase apt's initial timeout from 250ms to 2000ms and increase retries #21727
- [Build] Configure https timeout for apt, copy config to java-test-image #21724
- [Build] Add a default username in the image #21695
- [Build] Fix Stage Docker images fail on M1 Mac #21659
- [Build] Change UBUNTU_MIRROR default value #21520
- [Build] Unified the version of the library org.checkerframework #21503
- [Build] Fix apt download issue in building the docker image #21489
- [Build] Rename schema_example.conf to schema_example.json #21447
- [Build] Use git-archive to generate source release #20981
Library updatesβ
- Upgrade spotbugs maven plugin version for Java 21 compatibility #21389
- Upgrade bookkeeper to 4.16.3 #21146
- Upgrade Netty to 4.1.104 and io_uring to 0.0.24 #21723
- Upgrade Jacoco version to 0.8.11 to support Java 21 #21388
- Bump broker okio version to 3.4.0 #21064
- Bump GRPC version to 1.55.3 to fix CVE #21057
- Upgrade alluxio version to 2.9.3 to fix CVE-2023-38889 #21715
- Bumped version to 3.2.0-SNAPSHOT #21147
- Upgrade Lombok to 1.18.30 to support compiling with JDK21 #21278
- Upgrade Guava to 32.1.2-jre #21090
- Upgrade Apache ZooKeeper to 3.9.1 #20933
A special thanks to the following contributors who contributed to Pulsar 3.2.0: lhotari, jiangpengcheng, merlimat, poorbarcode, nodece, thetumbled, hrzzzz, BewareMyPower, RobertIndie, onobc, Technoboy-, wenbingshen, aloyszhang, heesung-sn, massakam, liangyepianzhou, coderzc, asafm, crossoverJie, TakaHiR07, zymap, yaalsn, rdhabalia, dragosvictor, pandalee99, AnonHxy, hangc0276, pengxiangrui127, mattisonchao, Denovo1998, 1Jack2, shibd, horizonzy, liudezhi2098, KevinLiLu, hanmz, erobot, Demogorgon314, tisonkun, diegosalvi, liangyuanpeng, kecona, alpreu, tuteng, frankjkelly, lifepuzzlefun, Shawyeok, Apurva007, Crispy-fried-chicken, michaeljmarshall, chenhongSZ, eolivelli, vineeth1995, codelipenghui, zhanghaou, JooHyukKim, klevy-toast.