Apache Pulsar 3.0.3

2024-03-09

Broker

[improve][broker] Consistently add fine-grain authorization to REST API 22202
[improve][broker] Add fine-grain authorization to retention admin API 22163
[improve][broker] Cache the internal writer when sent to system topic. 22099
[improve][broker] Add an error log to troubleshoot the failure of starting broker registry. 22065
[improve][broker] Do not retain the data in the system topic 22022
[improve] [broker] Do not try to open ML when the topic meta does not exist and do not expect to create a new one. #21995 22004
[improve] [broker] Do not print an Error log when responding to HTTP-404 when calling Admin API and the topic does not exist. 21995
[improve][broker] Do not close the socket if lookup failed due to LockBusyException 21993
[improve][broker] Don't rollover empty ledgers based on inactivity 21893
[improve][broker] Improve NamespaceUnloadStrategy error message 21880
[improve][broker] Skip loading the NAR packages if not configured 21867
[improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21857
[improve][broker] defer the ownership checks if the owner is inactive (ExtensibleLoadManager) 21811
[improve][broker] cleanup the empty subscriptionAuthenticationMap in zk when revoke subscription permission 21696
[improve] [broker] Let the producer request success at the first time if the previous one is inactive 21220
[improve][broker] Avoid record inactiveproducers when deduplication is disable. 21193
[improve][broker] Support not retaining null-key message during topic compaction (#21578) 21662
[improve][broker] Print recoverBucketSnapshot log if cursorProperties are empty 21651
[improve][broker][PIP-318] Support not retaining null-key message during topic compaction 21578
[fix] [broker] print non log when delete partitioned topic failed 22153
[fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled 22139
[fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init 22112
[fix] [broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency 22090
[fix][broker] Support running docker container with gid != 0 22081
[fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs 22064
[fix][broker] Fix hash collision when using a consumer name that ends with a number 22053
[fix][broker] Sanitize values before logging in apply-config-from-env.py script 22044
[fix] [broker] Fix can not subscribe partitioned topic with a suffix-matched regexp 22025
[fix] [broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog 22019
[fix][broker][branch-3.1] Avoid PublishRateLimiter use an already closed RateLimiter 22011
[fix] [broker] [branch-3.0] Fast fix infinite HTTP call createSubscriptions caused by wrong topicName 21997
[fix][broker]Support setting autoSkipNonRecoverableData dynamically in expiryMon… 21991
[fix] [broker] add timeout for health check read. 21990
[fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema 21977
[fix] [broker] Fix reader stuck when read from compacted topic with read compact mode disable 21969
[fix] [broker] Replication stopped due to unload topic failed 21947
[fix] [broker] Expire messages according to ledger close time to avoid client clock skew 21940
[fix][broker] Restore the broker id to match the format used in existing Pulsar releases 21937
[fix][broker] Fix getMessageById throws 500 21919
[fix] [broker] fix write all compacted out entry into compacted topic 21917
[fix][broker] Fix deadlock while skip non-recoverable ledgers. 21915
[fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured 21894
[fix] [broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 21885
[fix][broker] Fix compaction/replication data loss when expire messages 21865
[fix][broker]Fix NonPersistentDispatcherMultipleConsumers ArrayIndexOutOfBoundsException 21856
[fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21842
[fix][broker] Fix String wrong format 21829
[fix] [broker] Update topic policies as much as possible when some ex was thrown 21810
[fix][broker] Fix TableViewLoadDataStoreImpl NPE 21777
[fix][broker] Fix the wrong value of BrokerSrevice.maxUnackedMsgsPerDispatcher 21765
[fix][broker] Fixed the ExtensibleLoadManagerImpl internal system getTopic failure when the leadership changes 21764
[fix][broker] Delete compacted ledger when topic is deleted 21745
[fix][broker] Skip topic auto-creation for ExtensibleLoadManager internal topics 21729
[fix][broker] Fixed ServiceUnitStateChannel monitor to tombstone only inactive bundle states 21721
[fix][broker] Avoid compaction task stuck when the last message to compact is a marker 21718
[fix][broker] Fix the issue of topics possibly being deleted. 21704
[fix][broker] Fix typo in the config key 21690
[fix] [broker] network package lost if enable haProxyProtocolEnabled 21684
[fix][broker] Record GeoPersistentReplicator.msgOut before producer#sendAsync 21673
[fix][broker] Fix memory leak during topic compaction 21647
[fix][broker] Fix lookupRequestSemaphore leak when topic not found 21646
[fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set 21633
[fix][broker] Fixed getting incorrect KeyValue schema version 21632 (https://github.com/apache/pulsar/pull/21620)
[fix] [broker] Delete topic timeout due to NPE 21595
[fix][broker] Fix incorrect unack count when using shared subscription on non-persistent topic 21592
[fix][broker] Duplicate LedgerOffloader creation when namespace/topic… 21591
[fix][broker] Correct schema deletion for partitioned topic 21574
[fix][broker] Fix setReplicatedSubscriptionStatus incorrect behavior 21510
[fix][broker] Do not write replicated snapshot marker when the topic which is not enable replication 21495
[fix][broker] Fix issue with consumer read uncommitted messages from compacted topic 21465
[fix][broker] Fix resource_quota_zpath 21461
[fix][broker] Fix MultiRoles token provider NPE when using anonymous clients 21429
[fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection 21187
[fix] [broker] Make the new exclusive consumer instead the inactive one faster 21183
[fix][broker] Fix PulsarService.getLookupServiceAddress returns wrong port if TLS is enabled 21015
[fix][broker] fix Update contains no change error when use --update-auth-data flag to update function/sink/source 19450
[improve] [proxy] Add a check for brokerServiceURL that does not support multi uri yet 21972
[improve][proxy] Fix comment about enableProxyStatsEndpoints 21757
[improve][proxy] When adding new brokers resolve the DNS name more quickly 21207
[fix][txn]Fix TopicTransactionBuffer potential thread safety issue 22149
[fix] [txn] Get previous position by managed ledger. 22024
[fix][txn] Ack all message ids when ack chunk messages with transaction. 21268
[fix][txn] Fix getting last message ID when there are ongoing transactions 21466
[fix] [ml] Fix retry mechanism of deleting ledgers to invalidate 21869
[fix][ml] Make mlOwnershipChecker asynchronous so that it doesn't block/deadlock threads 21333
[improve][ml] Filter out deleted entries before read entries from ledger. 21739
[fix][offload] Fix Offload readHandle cannot close multi times. 22162
[fix][offload] Don't cleanup data when offload met MetaStore exception 21686
[fix] [ci] [branch-3.0] Fix the build issue from cherry-pick 22042
[fix][sec] Add a check for the input time value 22023
[fix] [log] fix the vague response if topic not found 20932

Client

[fix] [client] fix huge permits if acked a half batched message 22091
[fix] [client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation 22078
[fix][client] Fix ConsumerBuilderImpl#subscribe silent stuck when using pulsar-client:3.0.x with jackson-annotations prior to 2.12.0 21985
[fix][client] Fix multi-topics consumer could receive old messages after seek 21945
[cleanup][client] Fix inconsistent API annotations of getTopicName [21620]
[fix][client] fix negative message re-delivery twice issue 20750
[fix][client] Fix messages in the batch container timed out unexpectedly 21889
[improve] [client] Prevent reserve memory with a negative memory size to avoid send task stuck 21804
[fix][client] Fix producer thread block forever on memory limit controller 21790
[fix] [client] Messages lost due to TopicListWatcher reconnect 21853

CLI

[fix][admin] Fix KeyValue schema compatibility check caused OOM 21645
[improve][admin] Add clusters check when set replication clusters 21650-

Pulsar IO and Pulsar Functions

[improve][fn] Add missing "exception" argument to some log.error 22140
[improve][fn] Optimize Function Worker startup by lazy loading and direct zip/bytecode access 22122
[fix][fn] Fix Deadlock in Functions Worker LeaderService 21711
[fix][fn] Use unified PackageManagement service to download packages 21955
[improve][io] Make connectors load sensitive fields from secrets 21675

Others

[fix][test] Fix test testAsyncFunctionMaxPending 22121
[fix][test] fix test testSyncNormalPositionWhenTBRecover 22120
[fix][test] Fix test testTransactionBufferMetrics 22117
[fix][test] Fix PerformanceProducer send count error 21706
[fix][test] ProxyWithoutServiceDiscoveryTest should enable authz 20348
[cleanup] Consolidate certs used in tests 20336
[cleanup] Deduplicate test certificates to simplify management 20289
[cleanup] Consolidate certs in broker (and some proxy) tests 20353
[fix][test] Make base test class method protected so that it passes ReportUnannotatedMethods validation 21976
[improve][build] Add a default username in the image 21695
[fix][build] Fix Stage Docker images fail on M1 Mac 21659

Library updates

[fix][sec] Upgrade Jetty to 9.4.54.v20240208 to address CVE-2024-22201 22144
[fix][sec] Upgrade rabbitmq client to address CVE-2023-46120 21619
[fix][sec] Upgrade commons-compress to 1.26.0 22086
[fix][sec] Exclude avro from hadoop-client 21719
[improve][ci] Upgrade pulsar-client-python to 3.4.0 to avoid CVE-2023-1428 21899
[fix][sec] Upgrade org.bouncycastle:bc-fips to 1.0.2.4 21730
[improve][build] Upgrade Apache ZooKeeper to 3.9.1 20933
[fix][sec] Upgrade snappy-java to 1.1.10.5 21280
[improve] [bk] Upgrade BookKeeper dependency to 4.16.4 21983