Skip to main content

Apache Pulsar 4.1.0

2025-09-08​

Approved PIPs​

  • [improve][pip] PIP-292: Enforce token expiration time in the Websockets plugin (#20953)
  • [improve][pip] PIP-391: Enable batch index ACK by default (#23567)
  • [improve][pip] PIP-409: support producer configuration for retry/dead letter topic producer (#24022)
  • [improve][pip] PIP-416: Add a new topic method to implement trigger offload by size threshold (#24276)
  • [improve][pip] PIP-425: Support connecting with next available endpoint for multi-endpoint serviceUrls (#24394)
  • [improve][pip] PIP-427: Align pulsar-admin Default for Mark-Delete Rate with Broker Configuration (#24425)
  • [improve][pip] PIP-429: Optimize Handling of Compacted Last Entry by Skipping Payload Buffer Parsing (#24439)
  • [pip] PIP-430: Pulsar Broker cache improvements: refactoring eviction and adding a new cache strategy based on expected read count (#24444)
  • [improve][pip] PIP-431: Add Creation and Last Publish Timestamps to Topic Stats (#24469)
  • [improve][pip] PIP-432: Add isEncrypted field to EncryptionContext (#24481)
  • [improve][pip] PIP-433: Optimize the conflicts of the replication and automatic creation mechanisms, including the automatic creation of topics and schemas (#24485)
  • [improve][pip] PIP-435: Add startTimestamp and endTimestamp for consuming messages in client cli (#24524)
  • [improve][pip]PIP-422 Support global topic-level policy: replicated clusters and new API to delete topic-level policies (#24368)
  • [improve][pip]PIP-436: Add decryptFailListener to Consumer (#24572)
  • [feat][pip] PIP-420: Provide ability for Pulsar clients to integrate with third-party schema registry service (#24328)
  • [improve] [pip] PIP-373: Add a topic's system prop that indicates whether users have published TXN messages in before. (#23210)
  • [improve] [pip] PIP-375 Expose the Admin client configs: readTimeout, requestTimeout, and connectionTimeout (#23222)
  • [improve] [pip] PIP-382: Add a label named reason for topic_load_failed_total (#23351)
  • [pip] PIP-428: Change TopicPoliciesService interface to fix consistency issues (#24428)

Library updates​

  • [improve][broker] Upgrade avro version to 1.12.0 (#24617)
  • [improve][broker] Upgrade bookkeeper to 4.17.2/commons-configuration to 2.x/grpc to 1.72.0 and enable ZooKeeper client to establish connection in read-only mode (#24468)
  • [fix][sec] Bump commons-io version to 2.18.0 (#23684)
  • [fix][sec] Mitigate CVE-2024-53990 by disabling AsyncHttpClient CookieStore (#23725)
  • [fix][sec] Remove dependency on out-dated commons-configuration 1.x (#24562)
  • [fix][sec] Replace bcprov-jdk15on dependency with bcprov-jdk18-on (#23532)
  • [fix][sec] Upgrade async-http-client to 2.12.4 to address CVE-2024-53990 (#23732)
  • [fix][sec] Upgrade bouncycastle bcpkix-fips version to 1.79 to address CVE-2025-8916 (#24650)
  • [fix][sec] Upgrade golang.org/x/crypto from 0.21.0 to 0.31.0 in pulsar-function-go (#23743)
  • [fix][sec] Upgrade Jetty to 9.4.57.v20241219 to mitigate CVE-2024-6763 (#24232)
  • [fix][sec] Upgrade jwt/v5 to 5.2.2 to address CVE-2025-30204 (#24140)
  • [fix][sec] Upgrade Kafka connector and clients version to 3.9.1 to address CVE-2025-27818 (#24564)
  • [fix][sec] Upgrade pulsar-function-go dependencies to address CVE-2025-22868 (#24547)
  • [fix][sec] Upgrade to Netty 4.1.115.Final to address CVE-2024-47535 (#23596)
  • [fix][sec] Upgrade to Netty 4.1.118 (#23965)
  • [fix][sec] Upgrade to Netty 4.1.124.Final to address CVE-2025-55163 (#24637)
  • [fix][sec] Upgrade Zookeeper to 3.9.3 to address CVE-2024-51504 (#23581)
  • [fix][build] Upgrade json-smart to 2.5.2 (#23966)
  • [improve][io] Upgrade AWS SDK v1 & v2, Kinesis KPL and KPC versions (#24661)
  • [improve][io] Upgrade Kafka client and compatible Confluent platform version (#24201)
  • [improve][io] Upgrade Spring version to 6.1.13 in IO Connectors (#23459)
  • [improve][io] Upgrade Spring version to 6.1.14 in IO Connectors (#23481)
  • [improve][monitor] Upgrade OTel to 1.41.0 (#23484)
  • [improve][monitor] Upgrade OTel to 1.45.0 (#23756)
  • [fix][misc] Upgrade dependencies to fix critical security vulnerabilities (#24532)
  • [fix][misc] Upgrade fastutil to 8.5.16 (#24659)
  • [improve][build] Upgrade Caffeine from 2.9.1 to 3.2.1 (#24527)
  • [improve][build] Upgrade commons-compress version from 1.27.0 to 1.27.1 (#24270)
  • [improve][build] Upgrade errorprone to 2.38.0 (#24242)
  • [improve][build] Upgrade Gradle Develocity Maven Extension dependencies (#24260)
  • [improve][build] Upgrade Lombok to 1.18.38 to support JDK 24 (#24237)
  • [improve][build] Upgrade Mockito to 5.17.0 and byte-buddy to 1.15.11 (#24241)
  • [improve][build] Upgrade SpotBugs to 4.9.x (#24243)
  • [improve][build] Upgrade to jacoco 0.8.13 (#24240)
  • [improve][build] Upgrade zstd version from 1.5.2-3 to 1.5.7-3 (#24263)
  • [improve][ci] Upgrade Gradle Develocity Maven Extension to 1.23.1 (#24004)
  • [improve][test] Upgrade Testcontainers to 1.20.4 and docker-java to 3.4.0 (#24003)
  • [improve][misc] Upgrade Guava to 33.4.8 that uses JSpecify annotations (#24267)
  • [improve][misc] Upgrade Netty to 4.1.122.Final and tcnative to 2.0.72.Final (#24397)
  • [improve][misc] Upgrade RE2/J to 1.8 (#24530)
  • [feat][misc] Upgrade oxia version to 0.4.9 (#23607)
  • [improve] Upgrade Apache Commons library versions to compatible versions (#24205)
  • [improve] Upgrade lombok to 1.18.36 (#23752)
  • [improve] Upgrade Netty to 4.1.119.Final (#24049)
  • [improve] Upgrade Netty to 4.1.121.Final (#24214)
  • [improve] Upgrade OpenTelemetry library to 1.44.1 version (#23656)
  • [improve] Upgrade oxia-java to 0.4.10 and fix closing of OxiaMetadataStore (#23653)
  • [improve] Upgrade pulsar-client-python to 3.7.0 in Docker image (#24302)
  • [improve] Upgrade to Netty 4.1.116.Final and io_uring to 0.0.26.Final (#23813)
  • [improve] Upgrade to Netty 4.1.117.Final (#23863)

Broker​

  • [fix][broker] Add double-check for non-durable cursor creation (#24643)
  • [fix][broker] Add expire check for replicator (#23975)
  • [fix][broker] Add topic consistency check (#24118)
  • [fix][broker] Allow recreation of partitioned topic after metadata loss (#24225)
  • [fix][broker] Apply dispatcherMaxReadSizeBytes also for replay reads for Shared and Key_Shared subscriptions (#23894)
  • [fix][broker] Avoid block markDeletePosition forward when skip lost entries (#21210)
  • [fix][broker] Avoid IllegalStateException when marker_type field is not set in publishing (#24087)
  • [fix][broker] Broker is failing to create non-durable sub if topic is fenced (#23579)
  • [fix][broker] Catch exception for entry payload interceptor processor (#23683)
  • [fix][broker] Consumer stuck when delete subscription __compaction failed (#23980)
  • [fix][broker] Continue using the next provider for authentication if one fails (#23797)
  • [fix][broker] Continue using the next provider for http authentication if one fails (#23842)
  • [fix][broker] Directly query single topic existence when the topic is partitioned (#24154)
  • [fix][broker] expose consumer name for partitioned topic stats (#24360)
  • [fix][broker] fix broker may lost rack information (#23331)
  • [fix][broker] Fix Broker migration NPE while broker tls url not configured (#23534)
  • [fix][broker] Fix Broker OOM due to too many waiting cursors and reuse a recycled OpReadEntry incorrectly (#24551)
  • [fix][broker] Fix broker shutdown delay by resolving hanging health checks (#24210)
  • [fix][broker] Fix bug causing loss of migrated information when setting other localPolicies in namespace (#23764)
  • [fix][broker] Fix ByteBuf memory leak in REST API for publishing messages (#24228)
  • [fix][broker] Fix compaction service log's wrong condition (#24207)
  • [fix][broker] Fix deadlock in Key_Shared PIP-379 implementation (#23854)
  • [fix][broker] fix delay queue sequence issue. (#24035)
  • [fix][broker] Fix duplicate increment of ADD_OP_COUNT_UPDATER in OpAddEntry (#24506)
  • [fix][broker] Fix duplicate watcher registration after SessionReestablished (#24621)
  • [fix][broker] Fix enableReplicatedSubscriptions (#23781)
  • [fix][broker] Fix exclusive producer creation when last shared producer closes (#24516)
  • [fix][broker] fix ExtensibleLoadManager to override the ownerships concurrently without blocking load manager thread (#24156)
  • [fix][broker] Fix failed TokenAuthenticatedProducerConsumerTest (#23602)
  • [fix][broker] Fix flaky testReplicatorsInflightTaskListIsEmptyAfterReplicationFinished (#24590)
  • [fix][broker] Fix HashedWheelTimer leak in PulsarService by stopping it in shutdown (#24275)
  • [fix][broker] Fix HealthChecker deadlock in shutdown (#24216)
  • [fix][broker] Fix incorrect API documentation for non-persistent topic (#24413)
  • [fix][broker] Fix incorrect AuthData passed to AuthorizationService in proxy scenarios (#24593)
  • [fix][broker] Fix incorrect blockedConsumerOnUnackedMsgs value when maxUnackedMessagesPerConsumer is 1 (#23796)
  • [fix][broker] Fix issue that topic policies was deleted after a sub topic deleted, even if the partitioned topic still exists (#24350)
  • [fix][broker] fix logging with correct error message while loading the topic (#23544)
  • [fix][broker] Fix ManagedCursor state management race conditions and lifecycle issues (#24569)
  • [fix][broker] Fix matching of topicsPattern for topic names which contain non-ascii characters (#24543)
  • [fix][broker] Fix Metadata Event Synchronizer producer creation retry so that the producer gets created eventually (#24081)
  • [fix][broker] Fix Metadata event synchronizer should not fail with bad version (#24080)
  • [fix][broker] Fix missing validation when setting retention policy on topic level (#24032)
  • [fix][broker] fix namespace deletion TLS URL selection for geo-replication (#24591)
  • [fix][broker] Fix NPE being logged if load manager class name is blank (#24570)
  • [fix][broker] Fix NPE when getting delayed delivery policy (#24512)
  • [fix][broker] Fix NPE while publishing Metadata-Event with not init producer (#24079)
  • [fix][broker] fix null lookup result when brokers are starting (#23642)
  • [fix][broker] Fix ownership loss (#23515)
  • [fix][broker] Fix possible mark delete NPE when batch index ack is enabled (#23833)
  • [fix][broker] Fix potential deadlock when creating partitioned topic (#24313)
  • [fix][broker] Fix print cluster migration state response (#23535)
  • [fix][broker] Fix rate limiter token bucket and clock consistency issues causing excessive throttling and connection timeouts (#23930)
  • [fix][broker] Fix repeatedly acquired pending reads quota (#23869)
  • [fix][broker] Fix REST API to produce messages to single-partitioned topics (#24450)
  • [fix][broker] Fix seeking by timestamp can be reset the cursor position to earliest (#23919)
  • [fix][broker] Fix some problems in calculate totalAvailableBookies in method getExcludedBookiesWithIsolationGroups when some bookies belongs to multiple isolation groups. (#24091)
  • [fix][broker] Fix the non-persistenttopic's replicator always get error "Producer send queue is full" if set a small value of the config replicationProducerQueueSize (#24424)
  • [fix][broker] Fix the retry mechanism in MetadataCache#readModifyUpdateOrCreate (#23686)
  • [fix][broker] Fix the wrong cache name (#24407)
  • [fix][broker] Fix UnsupportedOperationException while setting subscription level dispatch rate policy (#24048)
  • [fix][broker] Fix wrong backlog age metrics when the mark delete position point to a deleted ledger (#24518)
  • [fix][broker] Fixes Inconsistent ServiceUnitStateData View (ExtensibleLoadManagerImpl only) (#24186)
  • [fix][broker] Geo Replication lost messages or frequently fails due to Deduplication is not appropriate for Geo-Replication (#23697)
  • [fix][broker] Ignore metadata changes when broker is not in the Started state (#24352)
  • [fix][broker] Increase readBuffer size for bookkeeper.DLOutputStream (#23548)
  • [fix][broker] Invalid regex in PulsarLedgerManager causes zk data notification to be ignored (#23977)
  • [fix][broker] Key_Shared subscription: Reject consumers with incompatible policy (#23449)
  • [fix][broker] Make InflightReadsLimiter asynchronous and apply it for replay queue reads (#23901)
  • [fix][broker] Once the cluster is configured incorrectly, the broker maintains the incorrect cluster configuration even if you removed it (#24419)
  • [fix][broker] Orphan schema after disabled a cluster for a namespace (#24223)
  • [fix][broker] PIP-322 Fix issue with rate limiters where rates can exceed limits initially and consumption pauses until token balance is positive (#24012)
  • [fix][broker] PIP-379 Key_Shared implementation race condition causing out-of-order message delivery (#23874)
  • [fix][broker] PIP-399: Fix Metric Name for Delayed Queue (#23712)
  • [fix][broker] PIP-428: Fix corrupted topic policies issues with sequential topic policy updates (#24427)
  • [fix][broker] Remove blocking calls from internalGetPartitionedStats (#23832)
  • [fix][broker] Remove failed OpAddEntry from pendingAddEntries (#23817)
  • [fix][broker] replication does not work due to the mixed and repetitive sending of user messages and replication markers (#24453)
  • [fix][broker] Restore the behavior to dispatch batch messages according to consumer permits (#24092)
  • [fix][broker] Revert "[fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)" (#23855)
  • [fix][broker] Revert "[improve][client] Add log when can't add message to the container (#23657)
  • [fix][broker] Skip to persist cursor info if it failed by cursor closed (#23615)
  • [fix][broker] support missing cluster level fine-granted permissions (#23675)
  • [fix][broker] support missing tenant level fine-granted permissions (#23660)
  • [fix][broker] System topic should not be migrated during blue-green cluster migration (#23767)
  • [fix][broker] The feature brokerDeleteInactivePartitionedTopicMetadataEnabled leaves orphan topic policies and topic schemas (#24150)
  • [fix][broker] Unregister non-static metrics collectors registered in Prometheus default registry (#24257)
  • [fix][broker]Avoid read a entry that entry id is -1 when calling getLastMessagePublishTime (#24579)
  • [fix][broker]Data lost due to conflict loaded up a topic for two brokers, when enabled ServiceUnitStateMetadataStoreTableViewImpl (#24478)
  • [fix][broker]excessive replication speed leads to error: Producer send queue is full (#24189)
  • [fix][broker]Failed to create partitions after the partitions were deleted because topic GC (#24651)
  • [fix][broker]Fix deadlock when compaction and topic deletion execute concurrently (#24366)
  • [fix][broker]Fix failed consumption after loaded up a terminated topic (#24063)
  • [fix][broker]Fix flaky test PartitionCreationTest.testCreateMissedPartitions (#24679)
  • [fix][broker]Fix incorrect priority between topic policies and global topic policies (#24254)
  • [fix][broker]fix memory leak, messages lost, incorrect replication state if using multiple schema versions(auto_produce) (#24178)
  • [fix][broker]Fix thread safety issues in BucketDelayedDeliveryTracker with StampedLock optimistic reads (#24542)
  • [fix][broker]Global topic policies do not affect after unloading topic and persistence global topic policies never affect (#24279)
  • [fix][broker]Non-global topic policies and global topic policies overwrite each other (#24286)
  • [fix][broker]User topic failed to delete after removed cluster because of failed delete data from transaction buffer topic (#24648)
  • [fix][admin] Backlog quota's policy is null which causes a NPE (#24192)
  • [fix][admin] Fix exception loss in getMessageId method (#23766)
  • [fix][admin] Fix exception thrown in getMessageId method (#23784)
  • [fix][admin] Listen partitioned topic creation event (#23680)
  • [fix][admin] Verify is policies read only before revoke permissions on topic (#23730)
  • [fix][ml] Corrected pulsar_storage_size metric to not multiply offloaded storage by the write quorum (#24054)
  • [fix][ml] Cursor ignores the position that has an empty ack-set if disabled deletionAtBatchIndexLevelEnabled (#24406)
  • [fix][ml] Don't estimate number of entries when ledgers are empty, return 1 instead (#24125)
  • [fix][ml] Fix deadlock in PendingReadsManager (#23958)
  • [fix][ml] Fix issues in estimateEntryCountBySize (#24089)
  • [fix][ml] Fix ManagedCursorImpl.individualDeletedMessages concurrent issue (#24338)
  • [fix][ml] Fix memory leaks in ManagedCursorInfo and ManagedLedgerInfo decompression and compression (#23960)
  • [fix][ml] Return 1 when bytes size is 0 or negative for entry count estimation (#24131)
  • [fix][ml] Skip deleting cursor if it was already deleted before calling unsubscribe (#24098)
  • [fix][ml] Topic load timeout due to ml data ledger future never finishes (#23772)
  • [fix][ml]Received more than once callback when calling cursor.delete (#24405)
  • [fix][ml]Revert a behavior change of releasing idle offloaded ledger handle: only release idle BlobStoreBackedReadHandle (#24384)
  • [fix][ml]Still got BK ledger, even though it has been deleted after offloaded (#24432)
  • [fix][offload] Complete the future outside of the reading loop in BlobStoreBackedReadHandleImplV2.readAsync (#24331)
  • [fix][offload] Exclude unnecessary dependencies from tiered storage provider / offloader nar files (#24649)
  • [fix][meta] Fix ephemeral handling of ZK nodes and fix MockZooKeeper ephemeral and ZK stat handling (#23988)
  • [fix][meta] Fix ephemeral Zookeeper put which creates a persistent znode (#23984)
  • [fix][meta] Use getChildrenFromStore to read children data to avoid lost data (#24665)
  • [improve][broker] Add managedCursor/LedgerInfoCompressionType settings to broker.conf (#24391)
  • [improve][broker] add warn log when the delayedMessagesCount is incorrect. (#24460)
  • [improve][broker] Added synchronized for sendMessages in Non-Persistent message dispatchers (#24386)
  • [improve][broker] Avoid logging errors when there is a connection issue during subscription. (#23939)
  • [improve][broker] Avoid PersistentReplicator.expireMessages logic compute backlog twice (#23957)
  • [improve][broker] Avoid printing log for IncompatibleSchemaException in ServerCnx (#23938)
  • [improve][broker] Deny removing local cluster from topic level replicated cluster policy (#24351)
  • [improve][broker] Do not print error logs for NotFound or Conflict errors when using the Admin API (#23928)
  • [improve][broker] Don't print error logs for ProducerBusyException (#23929)
  • [improve][broker] Enable concurrent processing of pending read Entries to avoid duplicate Reads (#24346)
  • [improve][broker] Enhance compaction triggering with message check to avoid unnecessary operations (#24449)
  • [improve][broker] Exclude system topics from namespace level publish and dispatch rate limiting (#23589)
  • [improve][broker] Extract duplication in AbstractTopic#incrementTopicEpochIfNeeded (#24520)
  • [improve][broker] extract getMaxEntriesInThisBatch into a method and add unit test for it (#24117)
  • [improve][broker] Implement PIP-430 Pulsar Broker cache improvements (#24623)
  • [improve][broker] Improve Consumer.equals performance (#23864)
  • [improve][broker] Improve SystemTopicBasedTopicPoliciesService reader to reduce GC pressure (#23780)
  • [improve][broker] Make maxBatchDeletedIndexToPersist configurable and document other related configs (#24392)
  • [improve][broker] Optimize message deduplication by removing redundant null checks and variable declarations (#24467)
  • [improve][broker] Optimize message expiration rate repeated update issues (#24073)
  • [improve][broker] Optimize message TTL check (#24271)
  • [improve][broker] Optimize subscription seek (cursor reset) by timestamp (#22792)
  • [improve][broker] Optimize ThresholdShedder with improved boundary checks and parameter reuse (#24064)
  • [improve][broker] PIP-406: Introduce metrics related to dispatch throttled events (#23946)
  • [improve][broker] PIP-429: Optimize Handling of Compacted Last Entry by Skipping Payload Buffer Parsing (#24523)
  • [improve][broker] PIP427:Align pulsar-admin Default for Mark-Delete Rate with Broker Configuration (#24470)
  • [improve][broker] re-elect the channel owner if no channel owner is found (#23516)
  • [improve][broker] Reduce cpu usage of InMemoryDelayedDeliveryTracker. (#24430)
  • [improve][broker] Reduce memory occupation of InMemoryRedeliveryTracker. (#23640)
  • [improve][broker] Reduce memory occupation of the delayed message queue (#23611)
  • [improve][broker] Refactor a private method to eliminate an unnecessary parameter (#23915)
  • [improve][broker] Skip unloading when bundle throughput is zero (ExtensibleLoadManagerImpl only) (#23626)
  • [improve][broker] Support showing client ip address in client stats while using reverse proxy (#23974)
  • [improve][broker] Support values up to 2^32 in ConcurrentBitmapSortedLongPairSet (#23878)
  • [improve][broker][pip-431] PIP-431: Add Creation and Last Publish Timestamps to Topic Stats (#24471)
  • [improve][broker]clean up duplicate code and fix typo (#24069)
  • [improve][broker]Find the target position at most once, during expiring messages for a topic, even though there are many subscriptions (#24622)
  • [improve][broker]Improve the anti-concurrency mechanism expirationCheckInProgress (#24607)
  • [improve][broker]Improve the feature "Optimize subscription seek (cursor reset) by timestamp": search less entries (#24219)
  • [improve][broker]Improve the log when encountered in-flight read limitation (#24359)
  • [improve][broker]Part-2 Add Admin API to delete topic policies (#24602)
  • [improve][broker]Remove block calling that named cursor.asyncGetNth when expiring messages (#24606)
  • [improve][admin] Opt-out of topic-existence check (#23709)
  • [improve][admin] PIP-422 part 1: Support global topic-level replicated clusters policy (#24390)
  • [improve][admin] Print error log if handle http response fails (#23563)
  • [fix][txn] Fix negative unacknowledged messages in transactions by ensuring that the batch size is added into CommandAck (#24443)
  • [improve][ml] Avoid repetitive nested lock for isMessageDeleted in ManagedCursorImpl (#23609)
  • [improve][ml] Offload ledgers without check ledger length (#24344)
  • [improve][ml] Optimize BlobStoreManagedLedgerOffloader.getOffloadPolicies (#23776)
  • [improve][ml] Optimize ledger opening by skipping fully acknowledged ledgers (#24655)
  • [improve][ml] Use lock-free queue in InflightReadsLimiter since there's no concurrent access (#23962)
  • [improve][ml]Release idle offloaded read handle only the ref count is 0 (#24381)
  • [improve][ml]Set default value of managedLedgerPersistIndividualAckAsLongArray to true (#23888)
  • [fix][schema] Reject unsupported Avro schema types during schema registration (#24103)
  • [improve][offload] Skip tiered-storage deployment (#23933)
  • [improve][offload] Use filesystemURI as the storage path (#23591)
  • [improve][meta] Simplify getting parent path in ZKMetadataStore without using java.io.File (#23996)
  • [improve][monitor] Add version=0.0.4 to /metrics content type for Prometheus 3.x compatibility (#24060)
  • [improve][txn] Improve Reader in TransactionBuffer to reduce GC pressure (#23779)
  • [refactor][broker] Expose the managedLedger field for the sub class (#24448)
  • [refactor][broker] fix wrong method name checkTopicExists. (#24293)
  • [refactor][ml] Replace cache eviction algorithm with centralized removal queue and job (#24363)
  • [revert][broker] Revert #24200: Cleanup OneWayReplicatorUsingGlobalPartitionedTest and OneWayReplicatorUsingGlobalZKTest (#24211)
  • [feat][broker] Implement allowBrokerOperationAsync in PulsarAuthorizationProvider to avoid exception thrown (#23663)
  • [feat][broker] Prevent auto-creation of topics using legacy cluster-based naming scheme (#23620)
  • [feat][monitor] Add offloader stats grafana dashboard (#23479)

Client​

  • [fix][client] Cannot access message data inside ProducerInterceptor#onSendAcknowledgement (#23791)
  • [fix][client] Close orphan producer or consumer when the creation is interrupted (#24539)
  • [fix][client] Copy eventTime to retry letter topic and DLQ messages (#24059)
  • [fix][client] fix ArrayIndexOutOfBoundsException in SameAuthParamsLookupAutoClusterFailover (#24662)
  • [fix][client] Fix ArrayIndexOutOfBoundsException when using SameAuthParamsLookupAutoClusterFailover (#23336)
  • [fix][client] Fix building broken batched message when publishing (#24061)
  • [fix][client] Fix consumer leak when thread is interrupted before subscribe completes (#24100)
  • [fix][client] Fix consumer not returning encrypted messages on decryption failure with compression enabled (#24356)
  • [fix][client] Fix ConsumerStats.getRateMsgsReceived javadoc (#21618)
  • [fix][client] Fix deadlock of NegativeAcksTracker (#23651)
  • [fix][client] Fix DLQ producer name conflicts when there are same name consumers (#23577)
  • [fix][client] Fix enableRetry for consumers using legacy topic naming where cluster name is included (#23753)
  • [fix][client] fix incomingMessageSize and client memory usage is negative (#23624)
  • [fix][client] Fix inconsistent compression threshold behavior across batching modes (#24102)
  • [fix][client] Fix incorrect producer.getPendingQueueSize due to incomplete queue implementation (#24184)
  • [fix][client] Fix issue in auto releasing of idle connection with topics pattern consumer (#24528)
  • [fix][client] Fix LoadManagerReport not found (#23886)
  • [fix][client] Fix memory leak in ClientCnx.newLookup when there's TooManyRequestsException (#23971)
  • [fix][client] Fix memory leak when message size exceeds max message size and batching is enabled (#23967)
  • [fix][client] Fix producer publishing getting stuck after message with incompatible schema is discarded (#24282)
  • [fix][client] Fix race-condition causing doReconsumeLater to hang when creating retryLetterProducer has failed (#23560)
  • [fix][client] Fix reader message filtering issue during blue-green cluster switch (#23693)
  • [fix][client] Fix Reader.hasMessageAvailable return wrong value after seeking by timestamp with startMessageIdInclusive (#23502)
  • [fix][client] fix the beforeConsume() method earlier hit with message listener (#23578)
  • [fix][client] Fix the javadoc for startMessageIdInclusive (#23508)
  • [fix][client] Fix wrong start message id when it's a chunked message id (#23713)
  • [fix][client] Fixed an issue where a cert chain could not be used in TLS authentication (#23644)
  • [fix][client] Initializing client-authentication using configured auth params (#23610)
  • [fix][client] Make DeadLetterPolicy & KeySharedPolicy serializable (#23718)
  • [fix][client] Make protobuf-java dependency optional in java client libraries (#23632)
  • [fix][client] NPE in MultiTopicsConsumerImpl.negativeAcknowledge (#24476)
  • [fix][client] Orphan producer when concurrently calling producer closing and reconnection (#23853)
  • [fix][client] PIP-425: fix updateServiceUrl and fix flaky ServiceUrlQuarantineTest (#24574)
  • [fix][client] Prevent embedding protobuf-java class files in pulsar-client-admin and pulsar-client-all (#23468)
  • [fix][client] Prevent NPE when seeking with null topic in TopicMessageId (#24404)
  • [fix][client] Prevent retry topic and dead letter topic producer leaks when sending of message fails (#23824)
  • [fix][client] Skip schema validation when sending messages to DLQ to avoid infinite loop when schema validation fails on an incoming message (#24663)
  • [fix][client] The partitionedProducer maxPendingMessages always is 0 (#23593)
  • [fix][client] Use dedicated executor for requests in BinaryProtoLookupService (#23378)
  • [fix][client]Add setCompressMinMsgBodySize method to ProducerBuilder for compression configuration flexibility (#24164)
  • [fix][client]Prevent ZeroQueueConsumer from receiving batch messages when using MessagePayloadProcessor (#24610)
  • [improve][client] Add startTimestamp and endTimestamp for consuming message in client cli (#24521)
  • [improve][client] Add OpenTelemetry metrics for client memory buffer usage (#24647)
  • [improve][client] Enhance error handling for non-exist subscription in consumer creation (#23254)
  • [improve][client] Make replicateSubscriptionState nullable (#23757)
  • [improve][client] PIP-391: Enable batch index ACK by default (#24203)
  • [improve][client] PIP-393: Improve performance of Negative Acknowledgement (#23600)
  • [improve][client] PIP-420: Supports users implement external schemas (#24488)
  • [improve][client] PIP-421: Require Java 17 as the minimum for Pulsar Java client SDK (#24475)
  • [improve][client] Print consumer stats log if prefetched messages are not zero (#23698)
  • [improve][client] Reduce unshaded dependencies and shading warnings in shaded Java client modules (#23647)
  • [improve][client] Replace NameUtil#generateRandomName with RandomStringUtils#randomAlphanumeric (#23645)
  • [improve][client] Support load RSA PKCS#8 private key (#24582)
  • [improve][client] Terminate consumer.receive() when consumer is closed (#24550)
  • [improve][client] Update TypedMessageBuilder deliverAfter and deliverAt api comment (#23969)
  • [improve][client] validate ClientConfigurationData earlier to avoid resource leaks (#24187)
  • [improve][client][PIP-389] Add a producer config to improve compression performance (#23525)
  • [improve][client][pip] Require Java 17 as the minimum for Pulsar Java client SDK (#24364)
  • [improve][client]Improve transaction log when a TXN command timeout (#24230)
  • [improve][client]PIP-425:Support connecting with next available endpoint for multi-endpoint serviceUrls (#24387)
  • [clean][client] Clean code for the construction of retry/dead letter topic name (#24082)
  • [feat][client] Support forward proxy for the ZTS server in pulsar-client-auth-athenz (#23947)

Pulsar IO and Pulsar Functions​

  • [fix][fn] Align WindowContext with BaseContext (#23628)
  • [fix][fn] Record Pulsar Function processing time properly for asynchronous functions (#23811)
  • [fix][io] Acknowledge RabbitMQ message after processing the message successfully (#24354)
  • [fix][io] Fix kinesis avro bytes handling (#24316)
  • [fix][io] Fix KinesisSink json flattening for AVRO's SchemaType.BYTES (#24132)
  • [fix][io] Fix pulsar-io:pom not found (#23979)
  • [fix][io] Fix SyntaxWarning in Pulsar Python functions (#24297)
  • [improve][fn] Implement PIP-412: Support setting messagePayloadProcessor in Pulsar Functions and Sinks (#24163)
  • [improve][fn] Improve closing of producers in Pulsar Functions ProducerCache invalidation (#23734)
  • [improve][fn] Improve implementation for maxPendingAsyncRequests async concurrency limit when return type is CompletableFuture<Void> (#23708)
  • [improve][fn] Introduce NewOutputMessageWithError to enable error handling (#24122)
  • [improve][fn] Set default tenant and namespace for ListFunctions cmd (#23881)
  • [improve][io] Add configuration parameter for disabling aggregation for Kinesis Producers (#24289)
  • [improve][io] Add dependency file name information to error message when .nar file validation fails with ZipException (#24604)
  • [improve][io] Add support for the complete KinesisProducerConfiguration in KinesisSinkConfig (#24489)
  • [improve][io] Allow skipping connector deployment (#23932)
  • [improve][io] Bump io.lettuce:lettuce-core from 5.0.2.RELEASE to 6.5.1.RELEASE in /pulsar-io/redis (#23685)
  • [improve][io] Enhance Kafka connector logging with focused bootstrap server information (#24128)
  • [improve][io] support kafka connect transforms and predicates (#24221)
  • [improve][io] Support specifying Kinesis KPL native binary path with 1.0 version specific path (#24669)
  • [improve][io][kca] support fully-qualified topic names in source records (#24248)

Others​

  • [fix][proxy] Fix default value of connectionMaxIdleSeconds in Pulsar Proxy (#24529)
  • [fix][proxy] Fix incorrect client error when calling get topic metadata (#24181)
  • [fix][proxy] Fix proxy OOM by replacing TopicName with a simple conversion method (#24465)
  • [fix][proxy] Fix TooLongFrameException with Pulsar Proxy (#24626)
  • [fix][proxy] Propagate client connection feature flags through Pulsar Proxy to Broker (#24158)
  • [fix][doc] Refine ClientBuilder#memoryLimit and ConsumerBuilder#autoScaledReceiverQueueSizeEnabled javadoc (#23687)
  • [fix][doc] Workaround Go Yaml issue go-yaml/yaml#789 in docker-compose example (#24040)
  • [fix][doc]fix typo in Message and RawMessage (#24416)
  • [improve][proxy] Make keep-alive interval configurable in Pulsar Proxy (#23981)
  • [fix][ws] Allow websocket principals to specify originalPrincipal without proxy role (#24642)
  • [fix][ws] Fix WebSocket authentication with authenticateOriginalAuthData enabled (#24615)
  • [fix][ws] Fix WebSocket authorization issue due to originalPrincipal must be provided (#24533)
  • [fix][ws] Fix WebSocket proxy originalPrincipal for HTTP admin API calls (#24613)
  • [fix][ws] Implement missing http header data functions in AuthenticationDataSubscription (#23638)
  • [fix][cli] Fix pulsar-shell cannot produce message with quotes and space (#24320)
  • [fix][cli] Fix set topic retention policy failed (#23688)
  • [fix][cli] Fix set-retention with >2GB size value for topic policy (#23689)
  • [fix][cli] Print result of GetMessageIdByIndex command (#24446)
  • [fix][misc] Exclude commons-configuration2 and commons-beanutils in pulsar-common (#24612)
  • [fix][misc] Fix ByteBuf leak in SchemaUtils (#24274)
  • [fix][misc] Fix ByteBuf leaks in tests by making ByteBufPair.coalesce release the input ByteBufPair (#24273)
  • [fix][misc] Fix topics pattern consumer backwards compatibility (#24537)
  • [fix][misc] Honor dynamic log levels in log4j2.yaml (#23847)
  • [fix][misc] Unable to connect an etcd metastore with recent releases due to jetc-core sharding problem (#23604)
  • [fix][misc]: ignore deleted ledger when tear down cluster (#23831)
  • [improve][doc] Add all legacy PIPs from Pulsar wiki and GitHub issues as files to pip directory (#24660)
  • [improve][doc] Cleanup some legacy PIP documents and improve PIP listing (#24684)
  • [improve][doc] Improve the JavaDocs of sendAsync to avoid improper use (#24601)
  • [improve][doc] Update PIP links in PIP documents converted from the wiki and remove trailing whitespace (#24678)
  • [improve][cli] Make pulsar-perf termination more responsive by using Thread interrupt status (#24309)
  • [improve][misc] Disable OTel by default when running the pulsar-perf tool (#23585)
  • [improve][misc] Ingore claude code generated file (#24474)
  • [improve][misc] Migrate from multiple nullness annotation libraries to JSpecify annotations (#24239)
  • [improve][misc] Optimize topic list hashing so that potentially large String allocation is avoided (#24525)
  • [cleanup][misc] Add override annotation (#24033)
  • [fix][common] TopicName: Throw IllegalArgumentException if localName is whitespace only (#23691)
  • [improve][log] Print ZK path if write to ZK fails due to data being too large to persist (#23652)
  • [improve][offloaders] Automatically evict Offloaded Ledgers from memory (#19783)
  • [cleanup] Remove unused config autoShrinkForConsumerPendingAcksMap (#24315)
  • [cleanup] remove unused config messagePublishBufferCheckIntervalInMillis (#24252)
  • [cleanup] Remove unused static fields in BrokerService (#24251)
  • [fix] [admin] Fix lookup get a null result if uses proxy (#23556)
  • [fix] [broker] Add consumer name for subscription stats (#23671)
  • [fix] [broker] Fix acknowledgeCumulativeAsync block when ackReceipt is enabled (#23841)
  • [fix] [broker] Fix config replicationStartAt does not work when set it to earliest (#23719)
  • [fix] [broker] Fix items in dispatcher.recentlyJoinedConsumers are out-of-order, which may cause a delivery stuck (#23802)
  • [fix] [broker] fix NPE when calculating a topic's backlogQuota (#23720)
  • [fix] [broker] Fix race-condition causing repeated delete topic (#23522)
  • [fix] [broker] No longer allow creating subscription that contains slash (#23594)
  • [fix] [broker] topics infinitely failed to delete after remove cluster from replicated clusters modifying when using partitioned system topic (#24097)
  • [fix] [client] call redeliver 1 msg but did 2 msgs (#23943)
  • [fix] [client] Fix memory leak when publishing encountered a corner case error (#23738)
  • [fix] [common] Fix RawMessageImpl.getProperties() failed when the message metadata contains the same key but with different values (#23927)
  • [fix] [ml] Fix cursor metadata compatability issue when switching the config unackedRangesOpenCacheSetEnabled (#23759)
  • [fix] [ml] incorrect non-durable cursor's backlog due to concurrently trimming ledger and non-durable cursor creation (#23951)
  • [fix] [proxy] Fix pattern consumer does not work when using Proxy (#23489)
  • [fix] Avoid negative estimated entry count (#24055)
  • [fix] Avoid NPE when closing an uninitialized SameAuthParamsLookupAutoClusterFailover (#23911)
  • [fix] Bring GitHub Discussions back by enabling discussions in .asf.yaml (#24077)
  • [fix] Bump org.apache.solr:solr-core from 8.11.3 to 9.8.0 in /pulsar-io/solr (#23899)
  • [fix] chore: remove unused preciseTopicPublishRateLimiterEnable (#24249)
  • [fix] fix for code scanning alert no. 48: Uncontrolled data used in path expression (#23985)
  • [fix] Fix issues with Pulsar Alpine docker image stability: remove glibc-compat (#23762)
  • [fix] Initialize UrlServiceProvider before trying to use transaction coordinator (#23914)
  • [fix] Prevent IllegalStateException: Field 'message' is not set (#24472)
  • [fix] Use Alpine 3.21 in base image (#23964)
  • [fix][broker Fix bug in RangeCache where different instance of the key wouldn't ever match (#23903)
  • [Fix][Client] Fix pending message not complete when closeAsync (#23761)
  • [improve] [broker] Make the estimated entry size more accurate (#23931)
  • [improve] [broker] Separate offload read and write thread pool (#24025)
  • [improve] [test] Add more test for the case that client receives a SendError, which relates to the PR #23038 (#23721)
  • [improve] Adapt startup scripts for Java 24 changes (#24236)
  • [improve] Change PersistentMessageFinder's fields modifier to reuse in plugins (#24308)
  • [improve] Enable metrics for all broker caches (#24365)
  • [improve] Improve logic for enabling Netty leak detection (#23613)
  • [improve] Improve the GitHub issue templates (#24264)
  • [improve] Install coreutils in docker image to improve compatibility (#23667)
  • [improve] Support overriding java.net.preferIPv4Stack with OPTS (#23846)
  • [improve] Update to Oxia 0.6.0 and use new group-id (#24438)
  • [improve] Use single buffer for metrics when noUnsafe use (#23612)
  • [revert] Revert "[improve][broker] Support showing client ip address in client stats while using reverse proxy (#23974)" (#24177)

Tests & CI​

  • [fix][build] Add develops for buildtools (#23992)
  • [fix][build] Add missing <name> to submodules (#24421)
  • [fix][build] Ensure that buildtools is Java 8 compatible and fix remaining compatibility issue (#24307)
  • [fix][build] Fix docker image building by replacing deprecated and removed compress argument (#24155)
  • [fix][build] Fix error "Element encoding is not allowed here" in pom.xml (#23655)
  • [fix][build] Fix errorprone maven profile configuration (#24246)
  • [fix][build] Fix skipTag and use explicit tag for image name (#24168)
  • [fix][build] propose wagon-ssh-external version (#24462)
  • [fix][build] Set project version to 4.1.0-SNAPSHOT (#23442)
  • [fix][build] Use amazoncorretto:21-alpine3.20 JDK build for Alpine 3.20 (#23898)
  • [fix][ci] Bump dependency-check to 12.1.0 to fix OWASP Dependency Check job (#24083)
  • [fix][ci] Fix code coverage metrics in Pulsar CI (#24595)
  • [fix][ci] Temporarily move OneWayReplicatorTestBase tests to flaky group to unblock CI (#23478)
  • [fix][test] Add build-helper-maven-plugin for jetcd-core-shaded (#24099)
  • [fix][test] Fix DeadLetterTopicTest.testDeadLetterTopicWithInitialSubscriptionAndMultiConsumers (#23552)
  • [fix][test] Fix flaky BrokerServiceChaosTest.testFetchPartitionedTopicMetadataWithCacheRefresh (#24161)
  • [fix][test] Fix flaky EntryCacheManagerTest.simple (#24609)
  • [fix][test] fix flaky GrowableArrayBlockingQueueTest.testPollBlockingThreadsTermination (#24576)
  • [fix][test] Fix flaky KeySharedSubscriptionTest.testNoKeySendAndReceiveWithHashRangeAutoSplitStickyKeyConsumerSelector (#23747)
  • [fix][test] Fix flaky ManagedCursorTest.testLastActiveAfterResetCursor and disable failing SchemaTest (#24261)
  • [fix][test] Fix flaky ManagedLedgerTest.testDoNotGetOffloadPoliciesMultipleTimesWhenTrimLedgers (#24330)
  • [fix][test] Fix flaky MultiTopicsReaderTest.testMultiNonPartitionedTopicWithRollbackDuration (#24318)
  • [fix][test] Fix flaky NamespacesTest.testNamespacesApiRedirects (#24194)
  • [fix][test] Fix flaky NonPersistentTopicTest.testMsgDropStat (#24134)
  • [fix][test] Fix flaky PrometheusMetricsTest.testBrokerMetrics (#24042)
  • [fix][test] Fix flaky PulsarBrokerStatsClientTest.testTopicInternalStats (#24341)
  • [fix][test] Fix flaky SimpleBrokerStartTest.testNoNICSpeed (#24415)
  • [fix][test] Fix flaky test MetadataStoreTest.emptyStoreTest (#23998)
  • [fix][test] Fix flaky test OneWayReplicatorUsingGlobalZKTest.testConfigReplicationStartAt (#24011)
  • [fix][test] fix flaky testNegativeAcksWithBackoff when batch enabled. (#23986)
  • [fix][test] Fix invalid test CompactionTest.testDeleteCompactedLedgerWithSlowAck (#24166)
  • [fix][test] Fix ManagedCursorTest.testForceCursorRecovery (#23518)
  • [fix][test] Fix memory leak via OTel shutdown hooks in tests (#23483)
  • [fix][test] Fix more ByteBuf resource leaks in tests, part 4 (#24332)
  • [fix][test] Fix more Netty ByteBuf leaks in tests (#24299)
  • [fix][test] Fix more resource leaks in tests (#24314)
  • [fix][test] Fix multiple ByteBuf leaks in tests (#24281)
  • [fix][test] Fix multiple resource leaks in tests (#24218)
  • [fix][test] Fix NonDurableSubscriptionTest.testInitReaderAtSpecifiedP… (#23293)
  • [fix][test] Fix quiet time implementation in BrokerTestUtil.receiveMessages (#23876)
  • [fix][test] Fix remaining UnfinishedStubbingException issue with AuthZTests (#24174)
  • [fix][test] Fix resource leaks in ProxyTest and fix invalid tests (#24204)
  • [fix][test] Fix resource leaks in PulsarBrokerStarterTest (#24235)
  • [fix][test] Fix running ClusterMetadataSetupTest in IDE (#23492)
  • [fix][test] Fix SimpleProducerConsumerTest.testMultiTopicsConsumerImplPauseForManualSubscription (#23546)
  • [fix][test] Fix TestNG BetweenTestClassesListenerAdapter listener (#24258)
  • [fix][test] Fix UnfinishedStubbing issue in AuthZTests (#24165)
  • [fix][test] Improve reliability of IncrementPartitionsTest (#24172)
  • [fix][test] Prevent OOM in test by not spying invocations in SimpleProducerConsumerTest (#23486)
  • [fix][test] Remove useless test code (#23823)
  • [fix][test] Simplify BetweenTestClassesListenerAdapter and fix issue with BeforeTest/AfterTest annotations (#24304)
  • [fix][test] Update partitioned topic subscription assertions in IncrementPartitionsTest (#24056)
  • [fix][test]: Flaky-test: GetPartitionMetadataMultiBrokerTest.testCompatibilityDifferentBrokersForNonPersistentTopic (#23666)
  • [fix][test]fix flaky test BrokerServiceAutoTopicCreationTest.testDynamicConfigurationTopicAutoCreationPartitioned (#24505)
  • [fix][test]Fix flaky test testTopicUnloadAfterSessionRebuild (#23852)
  • [fix][test]Fix flaky test V1_ProducerConsumerTest.testConcurrentConsumerReceiveWhileReconnect (#24019)
  • [fix][test]Flaky-test: SchemaServiceTest.testSchemaRegistryMetrics (#23566)
  • [fix][test]flaky-test:ManagedLedgerInterceptorImplTest.testManagedLedgerPayloadInputProcessorFailure (#24170)
  • [improve][build] Allow building and running tests on JDK 24 and upcoming JDK 25 LTS (#24268)
  • [improve][build] Build apachepulsar/pulsar-io-kinesis-sink-kinesis_producer with Alpine 3.21 (#24180)
  • [improve][build] Bump org.apache.commons:commons-lang3 from 3.17.0 to 3.18.0 (#24514)
  • [improve][build] Improve thread leak detector by ignoring "Attach Listener" thread (#24277)
  • [improve][build] Increase maven resolver's sync context timeout (#24666)
  • [improve][build] replace org.apache.commons.lang to org.apache.commons.lang3 (#24473)
  • [improve][build] Skip top level hidden directories from license check and add instructions about AI tool directories (#24614)
  • [improve][build] Suppress JVM class sharing warning when running tests (#24278)
  • [improve][build] Use org.apache.nifi:nifi-nar-maven-plugin:2.1.0 with skipDocGeneration=true (#24668)
  • [improve][ci] Add Netty leak detection reporting to Pulsar CI (#24272)
  • [improve][ci] Disable detailed console logging for integration tests in CI (#24266)
  • [improve][ci] Increase Maven max heap size to 2048M and tune GCLockerRetryAllocationCount (#23883)
  • [improve][ci] Move ZkSessionExpireTest to flaky group to unblock CI (#23810)
  • [improve][ci] Prevent git force push to more recent maintenance branches (#23464)
  • [improve][ci] Publish build scans to develocity.apache.org (#23851)
  • [improve][ci] Skip "OWASP dependency check" when data wasn't found in cache (#23970)
  • [improve][test] Add solution to PulsarMockBookKeeper for intercepting reads (#23875)
  • [improve][test] Add test for concurrent processing of pending read Entries (#24519)
  • [improve][test] Add test for dead letter topic with max unacked messages blocking (#24535)
  • [improve][test] Add test for retrieving bookie metrics via HTTP (#24408)
  • [improve][test] Added message properties tests for batch and non-batch messages (#23473)
  • [improve][test] Clarify method signatures in Bookkeeper mock client (#23598)
  • [improve][test] Disable OTel autoconfigured exporters in tests (#23540)
  • [improve][test] Reduce OneWayReplicatorUsingGlobalZKTest.testRemoveCluster execution time (#23633)
  • [improve][test] Refactor the way way pulsar-io-debezium-oracle nar file is patched when building the test image (#24586)
  • [improve][test] Remove EntryCacheCreator from ManagedLedgerFactoryImpl (#24552)
  • [improve][test] Support decorating topic, subscription, dispatcher, ManagedLedger and ManagedCursors instances in tests (#23892)
  • [improve][test] Use configured session timeout for MockZooKeeper and TestZKServer in PulsarTestContext (#24171)
  • [improve][test]Add new test PartitionCreationTest.testGetPoliciesIfPartitionsNotCreated (#24681)
  • [cleanup][build] skip generating pom.xml.versionsBackup (#23639)
  • [cleanup][test] Cleanup OneWayReplicatorUsingGlobalPartitionedTest and OneWayReplicatorUsingGlobalZKTest (#24200)
  • [cleanup][test] Remove unused parameter from deleteNamespaceWithRetry method in MockedPulsarServiceBaseTest (#24283)

For the complete list, check the full changelog.

Was this helpful?
💡 Suggest changes🛟 Get support