Apache Pulsar 3.3.1

2024-08-01

Broker

[feat][broker] PIP-264: Add OpenTelemetry consumer metrics (#22693)
[feat][broker] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22961)
[fix] Make operations on individualDeletedMessages in lock scope (#22966)
[fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
[fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
[fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
[fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
[fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
[fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
[fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
[fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
[fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
[fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
[fix][broker] Fix broker OOM when upload a large package. (#22989)
[fix][broker] Fix cursor should use latest ledger config (#22644)
[fix][broker] Fix geo-replication admin client url (#22584)
[fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
[fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
[fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
[fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
[fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
[fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
[fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
[fix][broker] Remove blocking calls from Subscription.getStats (#23088)
[fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
[fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
[fix][broker] The topic might reference a closed ledger (#22860)
[fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
[fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
[fix][broker] disable loadBalancerDirectMemoryResourceWeight by default (#22821)
[fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
[fix][broker] fix replicated subscriptions for transactional messages (#22452)
[fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
[fix][broker] maintain last active info in memory only. (#22794)
[fix][broker] rename to changeMaxReadPositionCount (#22656)
[fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
[fix][broker] usedLocallySinceLastReport should always be reset (#22672)
[fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
[fix][broker][branch-3.3] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23085)
[fix][misc] Disable JFR based telemetry collection since it's not used (#22869)
[fix][misc] Topic name from persistence name should decode local name (#22879)
[fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
[fix][ml] Fix race conditions in RangeCache (#22789)
[improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
[improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
[fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
[improve] Refactored BK ClientFactory to return futures (#22853)
[improve] Validate range of argument before long -> int conversion (#22830)
[improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
[improve][broker] Add subscription prefix for internal reader (#23044)
[improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
[improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
[improve][broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)
[improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
[improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
[improve][broker] Improve exception for topic does not have schema to check (#22974)
[improve][broker] PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22779)
[improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
[improve][broker] Reduce number of OpenTelemetry consumer attributes (#22837)
[improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
[improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
[improve][broker] Reuse topic OpenTelemetry attributes (#22876)
[improve][broker] Trigger offload on topic load (#22652)
[improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
[improve][broker] avoid creating new objects when intercepting (#22790)
[improve][broker] high CPU usage caused by list topics under namespace (#23049)
[improve][broker] make system topic distribute evenly. (#22953)
[improve][broker]Ensure namespace deletion doesn't fail (#22627)
[improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

[fix][client] Fix negative acknowledgement by messageId (#23060)
[fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
[fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted (#22854)
[fix][client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed (#22858)
[fix][client] PIP-344 Do not create partitioned metadata when calling pulsarClient.getPartitionsForTopic(topicName) (#22206)
[improve][client] PIP-344 support feature flag supportsGetPartitionedMetadataWithoutAutoCreation (#22773)
[improve][client] improve the class GetTopicsResult (#22766)

Pulsar IO and Pulsar Functions

[fix][fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes (#22910)
[fix][fn] Support compression type and crypto config for all producers in Functions and Connectors (#22950)
[improve][fn] Make producer cache bounded and expiring in Functions/Connectors (#22945)
[improve][io] The JDBC connector supports JSON substructure schema (#23043)

Others

[fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
[fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
[fix][cli] Fix expiration of tokens created with "pulsar tokens create" (#22815)
[fix][cli] Fix healthcheck script pulsar-zookeeper-ruok.sh (#22873)
[fix][cli] Fix the pulsar-daemon parameter passthrough syntax (#22905)
[fix][conf] fix configuration name and typo. (#22822)
[fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)
[improve][offload] Allow to disable the managedLedgerOffloadDeletionLagInMillis (#22689)

Library updates

[fix][sec] Upgrade Bouncycastle libraries to address CVEs (#22826)
[fix][sec] Upgrade postgresql version to avoid CVE-2024-1597 (#22635)
[fix] Bump io.airlift:aircompressor from 0.20 to 0.27 (#22819)
[improve] Upgrade IPAddress to 5.5.0 (#22886)
[improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
[improve] Upgrade to Oxia client 0.3.0 (#22807)
[fix] Upgrade to Oxia 0.3.1 (#23048)
[cleanup][misc] Remove classifier from netty-transport-native-unix-common dependency (#22951)
[improve][misc] Set Alpine base image to 3.20 instead of 3.19.1 (#22941)
[improve][misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin (#22957)
[improve][misc] Upgrade Bookkeeper to 4.17.1 (#22962)
[improve][misc] Upgrade OTel library to 1.38.0 version (#22825)
[improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
[fix][misc] Add proper nslookup (included in bind-tools) to docker image (#22878)
[fix][misc] Remove RoaringBitmap dependency from pulsar-common (#23008)
[fix][misc] Rename netty native libraries in pulsar-client-admin-shaded (#22954)

Tests & CI

[fix][ci] Fix OWASP Dependency Check download by using NVD API key (#22999)
[fix][ci] Fix jacoco code coverage report aggregation (#22964)
[fix][ci] Replace removed macos-11 with macos-latest in GitHub Actions (#22965)
[fix][test] Fix TableViewBuilderImplTest NPE and infinite loop (#22924)
[fix][test] Update OpenTelemetry receiver endpoint in integration test (#22998)
[improve][build] Support git worktree working directory while building docker images (#22851)
[improve][build] Upgrade dependency-check-maven-plugin to 10.0.2 (#23012)
[improve][ci] Add arm64 image build (#22755)
[improve][ci] Migrate from Gradle Enterprise to Develocity (#22880)
[improve][ci] Switch to use DEVELOCITY_ACCESS_KEY from GRADLE_ENTERPRISE_ACCESS_KEY (#23090)
[improve][test] Add a test to guarantee the TNX topics will not be replicated (#22721)