Apache Pulsar 3.1.0
2023-08-10
Important notice
- Fix the reason label of authentication metrics #20030
PIPs
- PIP-275: Introduce topicOrderedExecutorThreadNum to deprecate numWorkerThreadsForNonPersistentTopic #20504
- PIP-255: Make the partition assignment strategy pluggable #20537
- PIP-278: Support pluggable topic compaction service #20624
Broker
- Fix MessageDeduplication throw NPE when enabling broker deduplication and disabling namespace deduplication #20905
- In replication scenarios, remote consumers could not be registered if there were no message sent #20888
- Fix IndexOutOfBoundsException in the CompactedTopicUtils #20887
- Add annotation for topic compaction strategy #20858
- Fix inconsensus namespace policies by
getPoliciesIfCached
#20855 - Pass
bytesToRead
when reading compacted entries #20850 - Fix the built-in admin failed to delete a topic with a custom authz provider #20848
- Avoid print redirect exception log when getting a list from bundle #20846
- Inconsistent behaviour for topic auto_creation #20843
- Follow up 4196 use
PulsarByteBufAllocator
handle OOM #20837 - Add broker filter sync method back to guarantee the API compatibility #20826
- Fix direct memory leak by delayed index OutOfDirectMemory #20823
- Avoid infinite bundle unloading #20822
- Add the MessageExpirer interface to make code clear #20800
- Broker failed to load v1 namespace resources cache #20783
- Cannot receive any messages after switching to a standby cluster #20767
- Fix get topic policies as null during clean cache #20763
- Fix typo in ServerCnx.java #20762
- Avoid throwing RestException in BrokerService #20761
- Call ServerCnxcloseProducer from correct thread #20747
- Ignore openIDTokenIssuerTrustCertsFilePath conf when blank #20745
- Support pluggable topic compaction service - part2 #20718
- Revert "Skip loading broker interceptor when disableBrokerInterceptors is true 20422" #20710
- Gracefully shutdown does not work with admin cli in standalone #20709
- Improve CompactedTopicImpl lock #20697
- Added the skipped message handler for ServiceUnitStateChannel #20677
- Make ExtensibleLoadManagerImpl's broker filter pure async #20666
- Add logs for topics or subscriptions when enabling rate limit #20660
- Make get list from bundle Admin API async #20652
- Fix the publish latency spike from the contention of MessageDeduplication #20647
- Support pluggable topic compaction service - part1 #20645
- Fix get owned service units NPE #20625
- Topic policy cannot work well if the replay policy message has any exceptions #20613
- Fix the publish latency spike issue with a large number of producers #20607
- AuthenticationFilter.doFilter #20598
- Fix NPE when resetting Replicator's cursor by position. #20597
- Add consumer-id into the log when doing subscribe #20568
- Release orphan replicator after topic closed #20567
- New load balancer system topic should not be auto-created now #20566
- Remove unused cache executor in
PulsarService
#20563 - Handle get owned namespaces admin API in ExtensibleLoadManager #20552
- Handle heartbeat namespace in ExtensibleLoadManager #20551
- Validate authz earlier in delete subscription logic #20549
- Should not throw NotFoundException when metadata already exists #20539
- REST Client Producer fails with TLS only #20535
- Fix redirect loop when using ExtensibleLoadManager and list in bundle admin API #20528
- Restore solution for certain topic unloading race conditions #20527
- Emit the namespace bundle listener event on extensible load manager #20525
- Choose random thread for consumerFlow in PersistentDispatcherSingleActiveConsumer #20522
- Disable EntryFilters for system topics #20514
- Support get/remove permissions for AuthorizationProvider #20496
- Improve the efficiency of checking message deletion #20490
- Reformat property in generateResponseWithEntry #20481
- Add metric prefix for
topic_load_times
#20472 - Fix return the earliest position when query position by timestamp. #20457
- Support revoking permission for AuthorizationProvider #20456
- Replace String.intern() with Guava Interner #20432
- Remove ineffective solution for reducing GC pressure #20428
- Skip loading broker interceptor when disableBrokerInterceptors is true #20422
- Avoid
PersistentSubscription.expireMessages
logic check backlog twice #20416 - Change limitStatsLogging config default value to true #20409
- PIP-255 Part-1: Add listener interface for namespace service #20406
- Use immutable base cursor properties #20400
- Fix partitioned __change_events topic is policy topic #20392
- Do not expose bucketDelayedIndexStats #20383
- Fix flaky test - testClusterMigrationWithReplica… #20379
- Change some static fields referencing mutable objects to ordinary instance fields in BrokerService #20372
- Fix pre-create non-partitioned system topics for load balance extension #20370
- Fix memory leak when rebatchMessage #20369
- Invalidate metadata children cache after key deleted #20363
- Fix managedLedger.getConfig().getProperties().putAll(properties) NPE #20361
- Update rest endpoint method names to avoid swagger conflict #20359
- Fix broker load manager class filter NPE #20350
- Copy BrokerEntryMetadata when rebatchMessage #20337
- In Key_Shared mode: remove unnecessary mechanisms of message skip to avoid unnecessary consumption stuck #20335
- When creating a namespace anti-affinity group with the default local policies, use the bundle size supplied by the user. #20327
- Fix skip message API when hole messages exist #20326
- Gracefully shut down load balancer extension #20315
- Fix class name typo
PrecisPublishLimiter
to "Precise" #20310 - Allow Access to System Topic Metadata for Reader Creation Post-Namespace Deletion #20304
- Fix NPE cause by topic publish rate limiter. #20302
- Remove Streaming Dispatcher Code #20279
- Fix
UnsupportedOperationException
when updating topic properties. #20261 - Fix ledger cachemiss size metric #20257
- Fix default bundle size used while setting bookie affinity #20250
- Fix the behavior of delayed message in Key_Shared mode #20233
- Fix infinite ack of Replicator after topic is closed #20232
- Producer created by replicator is not displayed in topic stats #20229
- Use bulk operation instead of iteration #20219
- Fix Return value of getPartitionedStats doesn't contain subscription type #20210
- Improve knownBrokers update in ModularLoadManagerImpl #20196
- Skip split bundle if only one broker #20190
- Fix deadlock of metadata store #20189
- Fix
RoaringBitmap.contains
can't check value 65535 #20176 - Release EntryBuffer after parsing proto object #20170
- Remove useless field in the DelayedMessageIndexBucketSegment.proto #20166
- Optimization protobuf code in the bucket delayed tracker #20158
- Move bitmap from lastMutableBucket to ImmutableBucket #20156
- Producer/Consumer should call allowTopicOperationAsync #20142
- Fix entry filter feature for the non-persistent topic #20141
- Optimize delayed metadata index bitmap #20136
- Fast fix infinite HTTP call getSubscriptions caused by wrong topicName #20131
- Make timer execute immediately after load index #20126
- Cache LedgerHandle in BookkeeperBucketSnapshotStorage #20117
- Pin AppendIndexMetadataInterceptor to field in
ManagedLedgerInterceptorImpl
#20112 - Close temporary open ledger in BookkeeperBucketSnapshotStorage #20111
- Fix use error TimeUnit to record publish latency #20074
- Fix the reason label of authentication metrics #20030
- Save createIfMissing in TopicLoadingContext #19993
- Log resource usage rate of brokers that need to be offloaded in ThresholdShedder #19983
- Fix getPartitionedStats miss subscription's messageAckRate #19870
- Fix wrong logic of method TopicName.getPartition(int index) #19841
- Fix
Update contains no change
error when using--update-auth-data
flag to update function/sink/source #19450 - Add
topic_load_failed
metric #19236 - Support getStats/update partitioned topic with
-partition-
#19235 - Do not filter system topics while shedding. #18949
- If ledger lost, cursor mark delete position can not forward #18620
- Do not grant permission for each partition to reduce unnecessary zk metadata #18222
- Support cgroup v2 by using
jdk.internal.platform.Metrics
in Pulsar Loadbalancer #16832
Pulsar IO and Pulsar Functions
- [IO] Not restart instance when kafka source poll exception. #20795
- [IO] Add notifyError method on PushSource. #20791
- [IO] elastic-search sink connector not support JSON.String schema #20741
- [IO] Close the Kafka source connector got stuck #20698
- [IO] Close the Kafka source connector if there is uncaught exception #20424
- [IO] add protobuf ByteString to pulsar-io jdbc core #20259
- [IO] KCA: handle kafka sources that use commitRecord #20121
- [IO] Fix the wrong format of the logs #20907
- [IO] Remove useless configuration for Pulsar SQL #20605
- [IO] Some optimized points in 9224 #19027
- [FN] Exit JVM when the main thread throws exceptions #20689
- [FN] TLS args admin download command use zero arity #20513
- [FN] Optimize string concatenation in user metrics gen #20499
- [FN] Support customizing TLS config for function download command #20482
- [FN] enable Go function token auth and TLS #20468
- [FN] Reset idle timer correctly #20450
- [FN] Add stateStorageURL and pulsarWebService URL to go InstanceConfig #20443
- [FN] Go functions must retrieve consumers by non-partitioned topic ID #20413
- [FN] Go functions need to use static grpcPort in k8s runtime #20404
- [FN] Correct TLS cert config translation from broker to fn worker #20297
- [FN] JavaInstanceStarter --tls_allow_insecure default to false #20267
- [FN] Allow unknown fields in connectors config #20116
- [FN] Use functions classloader in TopicSchema.newSchemaInstance() to fix ClassNotFoundException when using custom SerDe classes. (targeted for master) #20115
- [FN] Upgrade Kotlin version from 1.4.32 to 1.8.20. (targeted for master) #20114
- [FN] Check user metric len before iterating #20021
- [FN] Support multiple input topics for Go runtime #20000
- [FN] Make pulsar-admin support update py/go with package url #19897
- [FN] Fix JavaInstanceStarter inferring type class name error #19896
- [FN] Fix function update error #19895
- [FN] Make KubernetesRuntime translate characters in function tenant, namespace, and name during function removal to avoid label errors #19584
Observability
- [Metrics] Add JVM start time metric #20381
- [Metrics] Topic with double quote breaks the prometheus format #20230
CLI
- [Admin] Set ns level backlog quota does not take effect if retention exists #20690
- [Admin] Remove duplicate topics name when
deleteNamespace
#20683 - [Admin] Make admin.cli throw exception instead of silent catch #20530
- [Admin] Pretty print bookies racks-placement command output #20516
- [Admin] Make failed
bin/pulsar-admin source
command exit with code1 (failed)
instead of0 (success)
#20503 - [Admin] Using AuthorizationService to revoke the permissions #20478
- [Admin] Return BAD_REQUEST on cluster data is null for createCluster #20346
- [Admin] Enhancing Transaction Buffer Stats and Introducing TransactionBufferInternalStats API #20330
- [Admin] Fix examine messages if total message is zero #20152
- [Admin] Make subscriptions on NonPersistentTopic non-durable #19741
- [Admin] Report earliest msg in partitioned backlog #19465
- [CLI] Fix the variable producerName of BatchMsgContainer is null #20819
- [CLI] Fix error messages in
CmdNameSpaces
andCmdTopics
#20701 - [CLI] Dedicated docs tools #20700
- [CLI] pulsar-client produce: allow to produce AVRO message from strings #20654
- [CLI] Client: allow to specify KeyValue's schema key (Avro Key support) #20447
- [CLI] Fulfill add-opens to function-localrunner also #20417
- [CLI] Allow pulser-client consume create a replicated subscription #20316
- [CLI] Add
--cleanupSubscription
to pulsar-admin #20028 - [CLI] Fix logging noise while admin tool exit #19884
- [CLI] PerformanceProducer do not produce expected number of messages #19775
- [CLI] Pulsar-perf: refactor to reduce code duplication #19279
Others
- [TXN] Add log when can't send TcClientConnectRequest to broker #20880
- [TXN] Use PulsarResource check for topic existence instead of brokerservice.getTopic() #20569
- [TXN] Unwrap the completion exception. #20396
- [TXN] Implement compatibility for transaction buffer segmented snapshot feature upgrade #20235
- [TXN] Fix transaction is not aborted when send or ACK failed #20055
- [TXN] fix consumer can receive aborted txn message when readType is replay #19815
- [ML] cleanup numManagedLedgerWorkerThreads #20454
- [ML] Fix ledger left in OPEN state when enabling
inactiveLedgerRollOverTimeMs
#20276 - [ML] There are two same-named managed ledgers in the one broker #18688
- [META] Bookie Info lost by notification race condition. #20642
- [META] Adding the missed bookie id in the registration manager #20641
- [META] Upgrade to jetcd to 0.7.5 #20339
- [META] Switch to the metadata store thread after zk operation #20303
- [META] deadlock of zkSessionWatcher when zkConnection loss #20122
- [META] Support set metadata size threshold for compression #19561
- [WS] Add cryptoKeyReaderFactoryClassName into the file websocket.conf #20840
- [WS] Remove unnecessary ping/pong implementation #20733
- [WS] Use allowTopicOperationAsync for authz checks #20299
- [WS] Use async auth method to support OIDC #20238
- [MISC] Stop logging full stack trace when
SecurityUtility.loadConscryptProvider()
fails to find Conscrypt with an unknown error #20705 - [MISC] Fix typo in pip template #20556
- [MISC] Use ubuntu 22.04 for Pulsar images #20475
- [MISC] Upgrade Netty to 4.1.93.Final #20423
- [misc] Consolidate certs in broker (and some proxy) tests #20353
- [MISC] Catch TimeoutException when logging about time outs #20349
- [Schema] Only handle exceptions when there has #20730
- [Schema] Add admin cli for testCompatibility #19974
- [Build] Fix generating javadoc issue #20922
- [build] Do not bundle swagger dependencies #20687
- [Build] Build function-go in golang image #20665
- [Build] Configure git-commit-id-plugin to skip git describe #20550
- [Build] Fix the pulsar-all image may use the wrong upstream image #20435
- [Build] Fix publish image script #20305
- [Build] Capture local build scans on ge.apache.org to benefit from deep build insights #20187
Library updates
- Upgrade Guava to 32.1.1 to address CVE-2023-2976 #20699
- Upgrade snappy-java to address multiple CVEs #20604
- Upgrade Jetty to address CVEs #20461
- Suppress etcd CVE warnings #20342
- Upgrade PyYaml version to 6.0.1 #20831
- Upgrade jackson version to 2.15.0 for CVE-2022-1471 #20177
- Upgrade prometheus client_golang to v1.12.2 to fix CVE-2022-21698 #20579
- Upgrade Netty to 4.1.93.Final #20423
- Upgrade to jetcd to 0.7.5 #20339
- Upgrade Kotlin version from 1.4.32 to 1.8.20. (targeted for master) #20114
Documentation
- [feat][doc] add overview for broker load balancing #621
- [feat][doc] add concepts for broker load balancing #644
- [feat][doc] add use cases + features + benefits + types for broker load balancing #652
- [feat][doc] add Quick Start for broker load balancing #660
- [feat][doc] add migration tutorials for broker load balancing #657
- [feat][doc] add metrics for broker load balancing #646
- [feat][doc] add configurations for broker load balancing #661
- [feat][doc] add development docs for broker load balancing #662
- [improve][doc] clarify detailed behaviors for failover subscription type #633
- [improve][doc] add docs for supporting reading config options from files in Python runner #544
- [improve][doc] add solutions for debezium-source-mysql connection errors #649
- [feat][doc] add docs for OpenID Connect Auth Provider #555
- [feat][doc] add docs for openIDKeyIdCacheMissRefreshSeconds setting #580
- [feat][doc] add docs for KubernetesServiceAccountTokenAuthProvider #570
- [feat][doc] add client matrix page #513
- [feature][doc] add docs for client memory limits #571
- [feature][doc] add docs for CPP client table view #565
- [improve] [doc] add compatibility reminders for retry queues #651
- [feat][doc] add docs for JVM metrics in Pulsar broker #567
- [feat][doc] add docs for bucket delayed message tracker metrics #564
- [fix][doc] exclude outdated versions of API documentation from search engine results #623
- [feat][website] implement 'How does Pulsar work' homepage screen #614
For the complete list, including all features, enhancements, and bug fixes, check out the following resources.
A special thanks to the following contributors (alphabetic Github IDs) who contributed to Pulsar 3.1.0: Technoboy-, crossoverJie, lifepuzzlefun, poorbarcode, coderzc, liangyepianzhou, Demogorgon314, mattisonchao, BewareMyPower, yaalsn, zymap, lhotari, daziz, shibd, rdhabalia, eltociear, tisonkun, aloyszhang, michaeljmarshall, 315157973, JooHyukKim, zhanghaou, heesung-sn, codelipenghui, eolivelli, ethqunzhong, massakam, StevenLuMT, gaoran10, ericsyh, zuobiao-zhou, AnonHxy, xiaotongwang1, maanders-tibco, yuruguo, vineeth1995, flowchartsman, hanmz, nicoloboschi, Shawyeok, RobertIndie, nodece, TakaHiR07, hleecs, MMirelli, thetumbled, pgier, syk-coder, dragonls, bpereto, liqiangcc, clayburn, 1Jack2, asafm, tiny-rain, dlg99, gmiklos-ltg, cbornet, aymkhalil, jiangpengcheng, congbobo184, wangjialing218, dao-jun, csthomas1, teabot, labuladong, tjiuming, ziang123, shashrath.