Skip to main content

Apache Pulsar 3.2.0

2024-02-05​

PIPs​

  • PIP-323: Complete Backlog Quota Telemetry #21709
  • PIP-322: Pulsar Rate Limiting Refactoring #21680
  • PIP-320: OpenTelemetry Scaffolding #21635
  • PIP-318: Don't retain null-key messages during topic compaction #21541
  • PIP-315: Configurable max delay limit for delayed delivery #21490
  • PIP-313: Support force unsubscribe using consumer api #21452
  • PIP-312:
    • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21597
    • Use StateStoreProvider to manage state in Pulsar Functions endpoints #21438
  • PIP-307:
    • Support subscribing multi topics for WebSocket #21390
    • Optimize Bundle Unload(Transfer) Protocol for ExtensibleLoadManager #20748
  • PIP-305: Customize DNS servers to use for Pulsar Client #21352
  • PIP-303: Add optional parameters for getPartitionedStats #21228
  • PIP-302: Introduce refreshAsync API for TableView #21271
  • PIP-301: Introduce LoadBalanceResources to unify the load-date CRUD #21129
  • PIP-300: Add custom dynamic configuration for plugins #21127
  • PIP-299: Stop dispatch messages if the individual acks will be lost in the persistent storage #21118
  • PIP-297: Support terminating Function & Connector with the fatal exception #21079
  • PIP-293: Delete config disableBrokerInterceptors #20993
  • PIP-290: Provide a way to implement WSS E2E encryption and not need to expose the private key to the WebSocket Proxy #20923
  • PIP-286: Make the TopicCompactionService to support find entry based on publishTime or index #20867
  • PIP-277: Add current option in the Clusters list cmd #20614
  • PIP-264: Enhanced OTel-based metric system #21080

Broker​

  • Replication stopped due to unload topic failed #21947
  • Fix getMessageById throws 500 #21919
  • Fix write all compacted out entry into compacted topic #21917
  • Fix deadlock while skip non-recoverable ledgers #21915
  • Fix leader broker cannot be determined when the advertised address and advertised listeners are configured #21894
  • Fix DLQ producer name conflicts when multiples consumers send messages to DLQ #21890
  • Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 #21885
  • Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException #21856
  • Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21842
  • Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher #21765
  • Fix the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes #21764
  • Fix closing of AbstractDispatcherSingleActiveConsumer and reduce flakiness of test #21736
  • Skip topic auto-creation for ExtensibleLoadManager internal topics #21729
  • Fix ServiceUnitStateChannel monitor to tombstone only inactive bundle states #21721
  • Avoid compaction task stuck when the last message to compact is a marker #21718
  • Fix the issue of topics possibly being deleted #21704
  • CheckTopicExists supports checking partitioned topic without index #21701
  • Remove internalGetListAsync and validatePartitionTopicUpdateAsync #21699
  • Cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission #21696
  • Fix typo in the config key #21690
  • network package lost if enable haProxyProtocolEnabled #21684
  • Pulsar Rate Limiting Refactoring changes (PIP-322) #21681
  • Record GeoPersistentReplicator.msgOut before producersendAsync #21673
  • Print recoverBucketSnapshot log if cursorProperties are empty #21651
  • Fix memory leak during topic compaction #21647
  • Fix lookupRequestSemaphore leak when topic not found #21646
  • Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set #21633
  • Fix getting incorrect KeyValue schema version #21632
  • Add optional parameters for getPartitionedStats #21611
  • PIP-301 Part-3: Add QuotaResources #21596
  • Delete topic timeout due to NPE #21595
  • Fix incorrect unack count when using shared subscription on non-persistent topic #21592
  • Duplicate LedgerOffloader creation when namespace/topic… #21591
  • Support not retaining null-key message during topic compaction #21578
  • Correct schema deletion for partitioned topic #21574
  • Fix create topic with different auto creation strategies causes race condition #21545
  • Fix thousands orphan PersistentTopic caused OOM #21540
  • Fix setReplicatedSubscriptionStatus incorrect behavior #21510
  • Fix failure while creating non-durable cursor with inactive managed-ledger #21508
  • Fix PulsarService/BrokerService shutdown when brokerShutdownTimeoutMs=0 #21496
  • Do not write replicated snapshot marker when the topic which is not enable replication #21495
  • Avoid pass null role in MultiRolesTokenAuthorizationProvider #21486
  • Fix the deadlock when using BookieRackAffinityMapping with rackaware policy #21481
  • Fix issue with consumer read uncommitted messages from compacted topic #21465
  • Delete methods marked Lombok @Getter #21462
  • Fix resource_quota_zpath #21461
  • Fix namespace bundle stuck in unloading status #21445
  • Ignore individual acknowledgment for CompactorSubscription when an entry has been filtered #21434
  • Fix MultiRoles token provider NPE when using anonymous clients #21429
  • Allow broker deployment in heterogeneous hw config cluster without restricting nic speed detection #21409
  • Fix avoid creating new topic after migration is started #21368
  • Fix race condition of replication cluster connection during migration topic #21364
  • Optimize and clean up aggregation of topic stats #21361
  • Fix heartbeat namespace create event topic and cannot delete heartbeat topic #21360
  • PIP-188 Fix cluster migration state store into local metadatastore #21359
  • PIP-188 Add support to auto create topic resources into green cluster before migration [part-3][#21354](https://github.com/apache/pulsar/pull/21354)
  • Fix heartbeat namespace create transaction internal topic #21348
  • Fix unload operation stuck when use ExtensibleLoadManager #21332
  • Fix typos in Subscription interface #21321
  • Use ConcurrentHashMap in ServiceUnitStateChannel and avoid recursive update error #21282
  • Cleanup deprecated method of metadata store etcd #21253
  • Miss headersAndPayload and messageIdData in MessagePublishContext #21245
  • Improve error messages when updating partition count #21243
  • Fixed produce and consume when anonymousUserRole enabled #21237
  • Fix inconsistent topic policy #21231
  • Fixed reset for AggregatedNamespaceStats #21225
  • fix flaky test PatternTopicsConsumerImplTest #21222
  • Let the producer request success at the first time if the previous one is inactive #21220
  • Fix lookup heartbeat and sla namespace bundle when using extensible load manager #21213
  • Not close the socket if lookup failed caused by bundle unloading or metadata ex #21211
  • Make the TopicCompactionService to support find entry based on publishTime or index #21208
  • Replicator leak when removeReplicator in NonPersistentTopic #21205
  • Print warn log if ssl handshake error & print ledger id when switch ledger #21201
  • Avoid record inactiveproducers when deduplication is disable. #21193
  • Make the new exclusive consumer instead the inactive one faster #21183
  • improve read entry error log for troubleshooting #21169
  • Disable balancing based on DirectMemory. #21168
  • Make specified producer could override the previous one #21155
  • Improve logs for troubleshooting #21141
  • Fix web tls url null cause NPE #21137
  • Fix unack count when mixing non batch index and batch index acks #21126
  • Fix unsubscribe non-durable subscription error #21099
  • Fix isolated group not work problem. #21096
  • Fix deleting topic not delete the related topic policy and schema. #21093
  • revert remove duplicate topics name when deleteNamespace #21087
  • Cleanup correctly heartbeat bundle ownership when handling broker deletion event #21083
  • Fix write duplicate entries into the compacted ledger after RawReader reconnects #21081
  • Remove bundle-data in local metadata store. #21078
  • Fix typos: lister -> listener #21068
  • Make sure all inflight writes have finished before completion of compaction #21067
  • Make read compacted entries support maxReadSizeBytes limitation #21065
  • Improve cache handling for partitioned topic metadata when doing lookup #21063
  • Use MessageDigest.isEqual when comparing digests #21061
  • Removing webURL check null #21043
  • Fix potential case cause retention policy not working on topic level #21041
  • Fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #21025
  • Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled #21015
  • PIP-293: Delete config disableBrokerInterceptors #20995
  • Fix incorrect unack msk count when dup ack a message #20990
  • Fix can't stop phase-two of compaction even though messageId read reaches lastReadId #20988
  • Fix get outdated compactedTopicContext after compactionHorizon has been updated #20984
  • Fix compaction subscription delete by inactive subscription check. #20983
  • Fix message loss during topic compaction #20980
  • Fix incorrect number of read compacted entries #20978
  • Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there #20951
  • Fix http error message of Bookie rack placement config API when bookie-rack is not configured. #20949
  • Fix chunked messages will be filtered by duplicating #20948
  • Rackaware policy is ineffective when delete zk rack info after bkclient initialize #20944
  • Check that the super user role is in the MultiRolesTokenAuthorizationProvider plugin #20939
  • Fix consumers are not redirected to migrated cluster #20928
  • PIP-300: Add custom dynamic configuration for plugins #20884
  • Delete unused param #20866
  • Improve performance of StatsOutputStream by using ArrayDeque instead of Stack #20808
  • Fix missing generate some metrics in BrokerOperabilityMetrics #20720
  • Fix ModularLoadManagerImpl always delete active bundle-data. sec ver. #20620
  • Consider iowait as idle. #19110
  • Fix bug caused by optimistic locking #18390

Pulsar IO and Pulsar Functions​

  • [IO] Make connectors load sensitive fields from secrets #21675
  • [IO] PIP-297: Support terminating Function & Connector with the fatal exception #21143
  • [IO] Fix --retain[-key]-ordering not working error for sink #21060
  • [IO] Improve kinesis connector config. #21004
  • [IO] Update test certs for Elasticsearch #21001
  • [IO] Support configuration secret interpolation #20901
  • [FN] Add missing version field back to querystate API #21966
  • [FN] Throw 404 RestException when state key not found #21921
  • [FN] Fix Deadlock in Functions Worker LeaderService #21711
  • [FN] Fix functions_log4j2.xml delete strategy config #21215
  • [FN] Fix ProducerConfig cannot update error #21037
  • [FN] Fix the --batch-builder not working error for functions #21023
  • [FN] Make python install dependencies from requirements.txt #20174g function removal to avoid label errors #19584

CLI​

  • [Admin] Set ns level backlog quota does not take effect if retention exists #20690
  • [Admin] Add clusters check when set replication clusters #21650
  • [Admin] Fix KeyValue schema compatibility check caused OOM #21645
  • [Admin] Do not print full stacktrace when get partitioned metadata not found #20979
  • [Admin] Fix get topic stats fail if a subscription catch up concurrently #20971
  • [CLI] Fix the variable producerName of BatchMsgContainer is null #20819
  • [CLI] Introduce Regex and File Input Parameters for Enhanced Topic Deletion Command #21664
  • [CLI] Add get-cluster-migration cmd #21473
  • [CLI] Add current option in the Clusters list cmd #21139
  • [CLI] Add some checks for topic-level setOffloadPolicies #20943
  • [CLI] Create new pulsar-cli-utils module #20782
  • [CLI] Refactor pulsar-client-tools module #20764
  • [CLI] Add command line option for configuring the memory limit #20663

Others​

  • [TXN] OpRequestSend reuse problem cause tbClient commitTxnOnTopic timeout unexpectedly #21505
  • [TXN] Fix getting last message ID when there are ongoing transactions #21466
  • [TXN] Fix the consumer stuck due to deduplicated messages in pending ack state #21177
  • [ML] Fix orphan scheduled task for ledger create timeout check #21542
  • [ML] Fix unfinished callback when deleting managed ledger #21530
  • [ML] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads #21333
  • [ML] Fix thread safe issue with RangeCache.put and RangeCache.clear #21302
  • [ML] Fix wrong msg backlog of non-durable cursor after trim ledgers #21250
  • [ML] Reader can set read-pos to a deleted ledger #21248
  • [ML] Persist mark deleted ops to ZK if create cursor ledger was failed #20935
  • [ML] Fix discontinuous ledger deletion #20898
  • [META] Improve fault tolerance of blocking calls by supporting timeout #21028
  • [META] Fix deadlock in AutoRecovery. #21010
  • [WS] Support subscribing multi/pattern topic for Websocket #21379
  • [WS] PIP-290 Make WSS support E2E encryption #20958
  • [MISC] Add Pulsar BOM (Bill of Materials) #21871
  • [MISC] Pin Netty version in pulsar-io/alluxio #21728
  • [MISC] Disable rebase and merge to prevent unsquashed PRs #21517
  • [Build] Increase apt's initial timeout from 250ms to 2000ms and increase retries #21727
  • [Build] Configure https timeout for apt, copy config to java-test-image #21724
  • [Build] Add a default username in the image #21695
  • [Build] Fix Stage Docker images fail on M1 Mac #21659
  • [Build] Change UBUNTU_MIRROR default value #21520
  • [Build] Unified the version of the library org.checkerframework #21503
  • [Build] Fix apt download issue in building the docker image #21489
  • [Build] Rename schema_example.conf to schema_example.json #21447
  • [Build] Use git-archive to generate source release #20981

Library updates​

  • Upgrade spotbugs maven plugin version for Java 21 compatibility #21389
  • Upgrade bookkeeper to 4.16.3 #21146
  • Upgrade Netty to 4.1.104 and io_uring to 0.0.24 #21723
  • Upgrade Jacoco version to 0.8.11 to support Java 21 #21388
  • Bump broker okio version to 3.4.0 #21064
  • Bump GRPC version to 1.55.3 to fix CVE #21057
  • Upgrade alluxio version to 2.9.3 to fix CVE-2023-38889 #21715
  • Bumped version to 3.2.0-SNAPSHOT #21147
  • Upgrade Lombok to 1.18.30 to support compiling with JDK21 #21278
  • Upgrade Guava to 32.1.2-jre #21090
  • Upgrade Apache ZooKeeper to 3.9.1 #20933

A special thanks to the following contributors who contributed to Pulsar 3.2.0: lhotari, jiangpengcheng, merlimat, poorbarcode, nodece, thetumbled, hrzzzz, BewareMyPower, RobertIndie, onobc, Technoboy-, wenbingshen, aloyszhang, heesung-sn, massakam, liangyepianzhou, coderzc, asafm, crossoverJie, TakaHiR07, zymap, yaalsn, rdhabalia, dragosvictor, pandalee99, AnonHxy, hangc0276, pengxiangrui127, mattisonchao, Denovo1998, 1Jack2, shibd, horizonzy, liudezhi2098, KevinLiLu, hanmz, erobot, Demogorgon314, tisonkun, diegosalvi, liangyuanpeng, kecona, alpreu, tuteng, frankjkelly, lifepuzzlefun, Shawyeok, Apurva007, Crispy-fried-chicken, michaeljmarshall, chenhongSZ, eolivelli, vineeth1995, codelipenghui, zhanghaou, JooHyukKim, klevy-toast.

Was this helpful?
💡 Suggest changes🛟 Get support