Skip to main content

Apache Pulsar 5.0.1

2026-10-26​

Preparing to upgrade to Pulsar 5.0?​

Approved PIPs​

  • [improve][pip] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf (#26695)

Library updates​

  • [improve][broker] Upgrade bookkeeper to 4.18.2 (#27009)
  • [fix][sec] Bump github.com/klauspost/compress from 1.18.0 to 1.18.7 in /pulsar-function-go (#27023)
  • [improve][build] Upgrade LightProto to 0.9.3 (#26893)
  • [improve][build] Upgrade Netty to 4.2.19.Final (#26831)
  • [improve][build] Upgrade slog to 0.10.1 (#26835)

Broker​

  • [fix][broker] Answer a scalable consumer's subscribe in order with its assignment updates (#26857)
  • [fix][broker] Apply dedup disable after recovery (#25982)
  • [fix][broker] Authorize cluster listing requests (#26267)
  • [fix][broker] Authorize scalable topic segments with the permissions of the parent topic (#26949)
  • [fix][broker] Avoid false WARN log when replicator is already terminated (#25259)
  • [fix][broker] Clean up orphan schema ledgers after write failures (#26902)
  • [fix][broker] Close Conscrypt SSL engines when Jetty closes a TLS connection (#26812)
  • [fix][broker] Delete scalable topics when deleting a namespace or a tenant (#26869)
  • [fix][broker] Delete the ack records of open transactions when deleting a segment or a subscription (#26887)
  • [fix][broker] Delete the transaction state of a segment when deleting it (#26882)
  • [fix][broker] Don't continue loading a topic after its load has timed out (#20540)
  • [fix][broker] Don't continue the REST topic listings that waited for a size estimate one after another on a metadata store thread (#27017)
  • [fix][broker] Don't dispatch a replay read that was pending when all consumers of a Shared subscription disconnected (#27007)
  • [fix][broker] Don't let an in-flight read dispatch a Key_Shared entry before an older one added to the replay queue (#26868)
  • [fix][broker] Fail compacted reads when the compacted ledger is missing (#26631)
  • [fix][broker] Fence non-persistent subscriptions before asynchronous … (#26991)
  • [fix][broker] Fix dispatcher permit accounting race with recycled batch index acks (#26809)
  • [fix][broker] Fix Exclusive/Failover dispatch rate limiting using counts from another dispatch (#26810)
  • [fix][broker] Fix inflated rate stats of newly created consumers (#26928)
  • [fix][broker] Fix managed-ledger scheduler thread stalls when estimating offline backlog (#26972)
  • [fix][broker] Fix non-persistent subscribe cleanup during migration (#27003)
  • [fix][broker] Fix partitioned topic auto deletion http request timeout (#24912)
  • [fix][broker] Fix readCompacted skipping tombstones within compaction horizon (#26190)
  • [fix][broker] Fix Shared and Key_Shared consumer stalls during FLOW and unblock races (#26797)
  • [fix][broker] Fix spurious placement failures when IsolatedBookieEnsemblePlacementPolicy counts unavailable secondary bookies (#26701)
  • [fix][broker] Fix the broker's WebSocket service ignoring configurationStoreConfigPath (#27015)
  • [fix][broker] Fix topic listing memory limit enforcement (#26945)
  • [fix][broker] Fix typo in broker.conf (#23295)
  • [fix][broker] Fix web service close hang when the Jetty server fails to start (#26895)
  • [fix][broker] Handle flow_or_qps bundle split requests in ELM (#25442)
  • [fix][broker] Handle synchronous schema lookup failures in replication (#26108)
  • [fix][broker] Hold a scalable-topic segment until every ancestor is drained (#26850)
  • [fix][broker] Honor dynamic decreases of dispatcherMaxReadBatchSize (#26908)
  • [fix][broker] Honor secondary tenant admin roles in HTTP admin requests (#26911)
  • [fix][broker] Ignore replicated subscription updates for the compaction subscription (#27032)
  • [fix][broker] Keep the scalable topic schema when deleting a regular topic with the same name (#26870)
  • [fix][broker] Keep the scalable topic schema when deleting a segment topic (#26852)
  • [fix][broker] Keep the scalable-topics watch's initial listing from undoing newer events (#26876)
  • [fix][broker] Keep the topic-level permissions of a topic migrated to a scalable topic (#26953)
  • [fix][broker] Log replicated subscription update ack failures (#26109)
  • [fix][broker] Make the scalable topic controller follow its leader election (#26878)
  • [fix][broker] Manage in-flight bucket work during clear (#26401)
  • [fix][broker] Match namespace isolation policy brokers given as host:port (#26829)
  • [fix][broker] Never share a segment between members of a checkpoint consumer group (#26836)
  • [fix][broker] PIP-442: Fix unbounded executor queue growth and single-thread bottleneck in the topic list memory limiter (#26929)
  • [fix][broker] Preserve non-durable cursor ownership during cleanup (#26925)
  • [fix][broker] Preserve unacked backpressure for stale redelivery requests (#26910)
  • [fix][broker] Prevent bundle split hangs on executor rejection (#26905)
  • [fix][broker] Prevent changing shadow sources through topic properties (#26900)
  • [fix][broker] Prevent duplicate messages when reconnect races with purge (#26907)
  • [fix][broker] Prevent stale consumer removal from removing replacements (#26652)
  • [fix][broker] Prevent stale messages from surviving compaction retries (#26664)
  • [fix][broker] Producer still publishes messages to Broker after Topic was unloaded (#24793)
  • [fix][broker] Prune sealed scalable-topic segments ancestors first (#26842)
  • [fix][broker] Read again when a Key_Shared read's only recipients have blocked writes (#26963)
  • [fix][broker] Recover bucket delayed delivery tracker asynchronously to avoid blocking dispatcher threads (#26105)
  • [fix][broker] Reject replicated status for the compaction subscription (#27037)
  • [fix][broker] Reject unsupported subscription-only backlog requests (#26906)
  • [fix][broker] Remove the sticky key hashes of a sparse replay prefix when the mark-delete position moves (#26976)
  • [fix][broker] Reset fixed delay detection when the in-memory delayed delivery tracker is cleared (#27021)
  • [fix][broker] Resume snapshots after re-enabling deduplication (#26996)
  • [fix][broker] Retry incompatible replication schemas without topic unload (#27026)
  • [fix][broker] Return retryable error for timed-out end transactions (#26917)
  • [fix][broker] Run the inactive topic GC when it is enabled by dynamic configuration (#26894)
  • [fix][broker] Shared subscription stalls forever on a chunked message (#26396)
  • [fix][broker] Stop checkpoint consumer groups from triggering rebucket rollovers (#26838)
  • [fix][broker] Stop scalable topic controllers from re-electing while the topic is deleted (#26875)
  • [fix][broker] Stop serving bundles that another broker took over while the metadata session is lost (#26866)
  • [fix][broker] Stop the scalable-topics namespace watch from re-adding a deleted topic (#26853)
  • [fix][broker] The configuration subscriptionExpirationTimeMinutes does not work if the topic was reloaded frequently (#25244)
  • [fix][broker] Use the persistent:// topic of a migrated scalable topic's legacy segments (#26871)
  • [fix][broker] Use the uncompressed payload when finding the last message position of a compacted batch (#26962)
  • [fix][broker] Incorrect source side topic permission deletion when disabling replication (#26479)
  • [fix][broker] The partitions of the topic was wrongly created even though this cluster was not allowed to access it (#25443)
  • [fix][admin] Encode a space in a path component as %20, not as '+' (#26957)
  • [fix][admin] Use ClientConfigurationData timeout params to set AsyncHttpConnector timeout params (#25040)
  • [fix][ml] Complete the original future after ledger deletion retries (#26742)
  • [fix][ml] Decode the shared cached message metadata before publishing it to other threads (#26824)
  • [fix][ml] Don't drop a durable cursor reset when an ack is queued behind it (#26305)
  • [fix][ml] Don't fail an add on a terminated managed ledger before the adds in flight ahead of it (#27018)
  • [fix][ml] Enforce the add-entry handover batch byte limit (#26901)
  • [fix][ml] Fix race in ManagedCursorImpl skip and getNthEntry that could skip unacknowledged entries (#20488)
  • [fix][ml] Fix races in managed ledger initialization timeout handling (#23470)
  • [fix][ml] Fix underReplicated always true in topic internal stats (#26834)
  • [fix][ml] Materialize compressed ManagedCursorInfo before releasing the decompressed buffer (#27050)
  • [fix][ml] Preserve individual batch ACKs when merging cumulative ACKs (#26998)
  • [fix][ml] Serialize a cursor's reused position info under its monitor (#26944)
  • [fix][ml] Wait for managed ledgers to close on factory shutdown, bounding each close (#26063)
  • [fix][offload] Fix final ledger not being offloaded for terminated topics (#25955)
  • [fix][offload] Fix NPE when reading from a closed offloaded read handle by keeping its ledger metadata (#26891)
  • [fix][meta] Avoid duplicate sync listener registration in OxiaMetadataStore (#26856)
  • [fix][meta] Don't leak closed leader elections and their metadata store listeners (#26879)
  • [fix][meta] Don't lose SessionReestablished when a listener invalidates the new session (#26861)
  • [fix][meta] Find the records written with a partition key in Oxia secondary-index scans (#26920)
  • [fix][meta] Fix leader election close deleting a leader node owned by another session (#26828)
  • [fix][meta] Fix lost ledger metadata updates during concurrent listener registration and removal (#26819)
  • [fix][meta] Fix repeated metadata recovery failures after a ZooKeeper session expires (#26845)
  • [fix][meta] Fix the upper bound of the Oxia secondary-index scans (#26890)
  • [fix][meta] Fix underreplication locks left behind when closing by handling each deletion independently (#26674)
  • [fix][meta] Preserve ownership during underreplication lock cleanup (#26794)
  • [fix][meta] Recover ambiguous ledger metadata creation using cToken (#26896)
  • [fix][meta] Return LedgerExistException for duplicate ledger creation (#26669)
  • [fix][meta] Unregister metadata store listeners when their owners are closed (#24256)
  • [improve][broker] Add a random value upto 5% to avoid rollover multiple cursor ledgers (#26542)
  • [improve][broker] Add scalable topic segment-owner metrics (#26981)
  • [improve][broker] Add scalable topic stats diagnostics (#26984)
  • [improve][broker] Add scalable-topic controller and layout metrics (#26983)
  • [improve][broker] Cache schema payloads to stop re-reading BookKeeper history on every explicit-version request (#26960)
  • [improve][broker] Coalesce small messages dispatched to a consumer into one buffer (#26926)
  • [improve][broker] Default numIOThreads to half the available processors, at least 8 (#26825)
  • [improve][broker] Don't compute Key_Shared consumers' key hash ranges for stats without the consumers (#26965)
  • [improve][broker] Enable Log4j2 async loggers again (#26979)
  • [improve][broker] Hash Key_Shared sticky keys without allocation (#26968)
  • [improve][broker] improve readability for the class TopicPolicyListenerWrapper (#26277)
  • [improve][broker] Keep each ledger's pending acks in sorted blocks instead of an AVL tree (#26955)
  • [improve][broker] Keep the Key_Shared consistent hashing ring in arrays (#26863)
  • [improve][broker] Let authorization providers restrict topic auto-creation (#17411)
  • [improve][broker] log incompatible schema failures at warn level, rather than logging at error level (#26804)
  • [improve][broker] PIP-246: Improved PROTOBUF_NATIVE schema compatibility checks without using avro-protobuf (#19566)
  • [improve][broker] Record the deduplication's persisted sequence IDs in place (#26822)
  • [improve][broker] Reduce allocations on dispatch and managed-ledger read hot paths (#26956)
  • [improve][broker] Reduce the lookups and allocations of an individual ack (#26948)
  • [improve][broker] Skip the size cap estimate when there are no messages to replay (#26954)
  • [improve][broker] Write async REST responses on the web service executor (#26946)
  • [improve][ml] Bound the requeues of size-based cache eviction to one cycle through the removal queue (#26933)
  • [improve][ml] Don't visit a ledger's other single-entry pending reads for a read of a single entry (#27005)
  • [improve][ml] Optimization method getNumberOfEntries (#23576)
  • [improve][ml] Read a cursor's close replay positions with one range read (#27006)
  • [improve][ml] Read ahead the next range of a cursor's read into the entry cache (#27039)
  • [improve][ml] Store the broker entry cache in pages of consecutive entry IDs (#26821)
  • [fix][schema] Fix BookKeeper read handle leaks on schema read failures (#26818)
  • [fix][schema] Fix Protobuf native descriptor deserialization (#26725)
  • [improve][bk] Lower the bookies' read-ahead after a read cache miss from 1,000 to 100 entries (#27004)
  • [improve][bk] Partition index and filter blocks in the entry location RocksDB config (#27008)
  • [fix][meta] Fix stats in PulsarZooKeeperClient multi. (#25766)
  • [improve][meta] Correct the MetadataStore.scanByIndex javadoc on the fallback filter and bounds (#26886)
  • [improve][meta] Log metadata store deletes at DEBUG (#26978)
  • [feat][broker] Add scalable topic consumer coordination metrics (#26982)
  • [feat][broker] Add scalable-topic session and request metrics (#26980)
  • [feat][monitor] PIP-320: Add OpenTelemetry metrics for the Function Worker (#26823)

Client​

  • [fix][client] Apply layout changes that arrive while a V5 producer or consumer is created (#26854)
  • [fix][client] Apply topic changes that arrive while a V5 namespace consumer is created (#26840)
  • [fix][client] Avoid duplicate reconcile retries and queue layout stalls in V5 consumers (#26874)
  • [fix][client] chunked message uuid-queue never removes entries on successful processing (#26658)
  • [fix][client] Complete a TableView refresh waiting for a message that could not be applied (#26986)
  • [fix][client] Don't apply an older segment assignment after a newer one in V5 consumer groups (#26858)
  • [fix][client] Don't retry a Pulsar Admin request after its request timeout has elapsed (#26881)
  • [fix][client] Don't skip the first messages of new segments in a V5 checkpoint consumer started at latest (#26844)
  • [fix][client] Fix message leaks and size and permit accounting races when clearing the consumer's incoming queue (#21767)
  • [fix][client] Fix retry logic for BinaryProtoLookupService.getTopicsUnderNamespace method (#24974)
  • [fix][client] Fix the embedded native image configuration and make its smoke test run (#26990)
  • [fix][client] Fix ZeroQueueConsumer stuck on pending receiveAsync after reconnection (#26814)
  • [fix][client] Include segments without received messages in V5 checkpoints (#26839)
  • [fix][client] Keep per-key order of V5 async sends across scalable topic layout changes (#26820)
  • [fix][client] Make chunking inert on non-persistent topics, including the message-size checks (#26550)
  • [fix][client] Preserve per-key order across splits and merges in the V5 checkpoint consumer (#26837)
  • [fix][client] Prevent chunk message ID leaks after seek (#26909)
  • [fix][client] Release connections on partition metadata rejection (#26912)
  • [fix][client] Release the output buffer when a compression codec fails to decode (#26889)
  • [fix][client] Retry failed initial topic attaches and drop topics removed while attaching in V5 namespace consumers (#26848)
  • [fix][client] Return flow-control permit when client seek by messageID (#26726)
  • [fix][client] Schedule TableView tail-read retries instead of sleeping on the callback thread (#26672)
  • [fix][client] Serialize chunked-message bookkeeping to fix use-after-free and count/queue drift (#26084)
  • [fix][client] Skip zero-queue permits for completed async receives (#27001)
  • [fix][client] Stop V5 namespace stream consumers from recreating deleted topics (#26851)
  • [fix][client] Surface V5 client failures as the typed exception of their error category (#26950)
  • [improve][client] Don't start a batch receive on an internal consumer without queued messages (#26966)
  • [improve][client] Flush consumers' grouped acks with one task per event loop, only for consumers with new acks (#26947)
  • [improve][client] Keep only its own topic in the configuration of a multi-topic consumer's internal consumer (#26923)
  • [improve][client] Make Consumer.resume() eagerly top up the broker's permit window (#26919)
  • [improve][doc] add missing .enableRetry(true) in DeadLetterPolicy examples (#26892)

Pulsar IO and Pulsar Functions​

  • [fix][fn] Fix Go function incorrectly using nanoseconds for timestamp (#25869)
  • [fix][fn] Honour consumerCryptoFailureAction in the Python function runtime (#26490)
  • [fix][fn] Only load crypto_key_reader for python function when cryptoKeyReaderConfig is provided (#24783)
  • [fix][fn] Support deadLetterTopic and maxMessageRetries for Python functions (#26400)
  • [improve][fn] Provide a way to set Pulsar client memory limit in Pulsar Function (#23959)
  • [feat][fn] Expose the Prometheus metrics registry to Go functions via FunctionContext (#26458)

Others​

  • [fix][proxy] Fix connection and topic stats that stop updating after the initial update (#26961)
  • [fix][proxy] Fix direct memory leaks from dropped reference-counted messages (#26811)
  • [fix][proxy] Fix Proxy Protocol forwarding for UNKNOWN and IPv6 (#26999)
  • [fix][misc] Fix Backoff mandatory-stop timer tracking (#25282)
  • [improve][doc] Clarify AI-assisted contribution and review expectations (#26964)
  • [improve][doc] Note in CODING.md that a delayedExecutor() task cannot be taken back (#26988)
  • [improve][doc] Update the PIP listing and fill in mailing list thread links (#26883)
  • [improve][doc] Warn about /tmp cleanup in narExtractionDirectory docs (#27011)
  • [fix][metadata] Invalidate ledger metadata cache on version conflicts (#26940)
  • [fix][managed-ledger] Honor batch-index cap in isCursorDataFullyPersistable (#26553)
  • [fix][managed-ledger] Persist partial batch ACKs on flush (#26765)

Tests & CI​

  • [fix][build] Cap test compilation release at the server Java baseline (#26994)
  • [fix][build] Include the java.compiler module in the Pulsar image JVM (#27035)
  • [fix][ci] Fix flaky native image build caused by a GraalVM class initializer simulation race (#26989)
  • [fix][test] Fix flaky NegativeAcksTest.testNegativeAcksWithBackoff (#24342)
  • [fix][test] Fix flaky OffloadPrefixTest.terminateTriggersAutomaticOffloadForLastLedger (#27040)
  • [fix][test] Fix flaky PulsarLedgerUnderreplicationManagerCloseTest.testCloseAggregatesFailuresAndRetriesOnlyUnreleasedLocks (#27013)
  • [fix][test] Fix flaky ScalableTopicControllerAutoScaleTest.testRebucketCooldownSurvivesLeaderFailover (#26806)
  • [fix][test] Fix flaky V5ProducerBackpressureTest.asyncSendsBlockTheCallerAtTheMemoryLimit (#26855)
  • [fix][test] Fix order-dependent AdminApiTest.testUpdateDynamicLoadBalancerSheddingIntervalMinutes (#26872)
  • [fix][test] Fix vacuous assertion in testInvalidBrokerLevelPolicyDisablesAutoScale (#26841)
  • [fix][test] Give each geo-replicated cluster's bookies and ZooKeeper a hostname of their own (#26873)
  • [fix][test] Make transaction buffer recovery placement deterministic (#26913)
  • [fix][test] Run the Oxia-backed tests against oxia/oxia:nightly-beta (#26880)
  • [fix][test] Split and fix flaky SASL authentication first-stage auth cache tests (#25949)
  • [fix][test] Stub the cancellable listener in PulsarLedgerCreateRecoveryTest (#27025)
  • [improve][ci] Publish Develocity Build Scans for the Docker images (#26832)
  • [improve][test] Add a catch-up scenario: consumers joining a live stream later (#26826)
  • [improve][test] Add a script that builds and installs hsdis for perfasm (#26995)
  • [improve][test] Add deduplication contract coverage across broker restart (#27041)
  • [improve][test] Add IoT scenarios for Failover, compaction, retries, delayed delivery, chunking, pattern subscriptions and TTL to the performance tests (#26993)
  • [improve][test] Add IoT telemetry scenarios with partitioned topics to the performance tests (#27028)
  • [improve][test] Add Key_Shared replay queue and stuck consumer scenarios to the performance tests (#26975)
  • [improve][test] Add network conditions, network faults and geo-replication to the performance tests (#26859)
  • [improve][test] Add offloading performance scenarios that read backlogs and seek by timestamp in tiered storage (#26884)
  • [improve][test] Add Shared subscriptions and alert fan-out scenarios to the performance tests (#26922)
  • [improve][test] Add TLS and JWT authentication to the performance testing framework (#27012)
  • [improve][test] Compare two Pulsar images, or an image and the checkout, in the performance tests (#26847)
  • [improve][test] Cover the cleanup of a scalable topic controller that fails to initialize (#26877)
  • [improve][test] Create the IoT gateways' producers concurrently in a random order (#26827)

For the complete list, check the full changelog.

Was this helpful?
✏️ Edit this page💡 Suggest changes🛟 Get support