Skip to main content

Apache Pulsar 2.4.0

Penghui Li
· 3 min read

We are glad to publish Apache Pulsar 2.4.0. This is the result of a huge effort from the community, with over 460 commits and a long list of new features, general improvements and bug fixes.

Check out the release notes for a detailed list of the changes, with links to the relevant pull requests, discussions and documentation.

Regarding new features introduced, I just want to highlight here a tiny subset of them:

Delayed message delivery​

It's now possible to send a delayed message by Pulsar producer, and a delayed message will be available after a delay time.

The Java code for a client using delayed messages delivery looks as follows:


producer.newMessage().value("delayed message").deliverAfter(10, TimeUnit.SECONDS).send()

note
  1. Messages are only delayed on shared subscriptions, other subscriptions will deliver immediately.
  2. Delayed messages are sent individually even if you enable message batching on producer.

Go Functions​

Before 2.4.0 release, Java and Python are supported to write Pulsar Functions. Now, you can use Go to write Pulsar Functions, the following is an example of a Pulsar Function written in Go.


import (
    "fmt"
    "context"

    "github.com/apache/pulsar/pulsar-function-go/pf"
)

func HandleRequest(ctx context.Context, in []byte) error {
    fmt.Println(string(in) + "!")
    return nil
}

func main() {
    pf.Start(HandleRequest)
}

Key_Shared subscription​

A new subscription mode Key_shared is introduced in 2.4.0. In Key_shared subscription mode, one partition could have several consumers to consume messages in parallelism and ensure messages with the same key are distributed to a consumer in order. Here is architecture for Key_Shared.

The following is an example to use Key_shared subscription:


client.newConsumer()
        .topic("topic")
        .subscriptionType(SubscriptionType.Key_Shared)
        .subscriptionName("sub-1")
        .subscribe();

Schema versioning​

Before 2.4.0 release, Avro schema used one schema for both writer schema and reader schema. Multiple schemas version is supported now.

With multiple schemas, a producer can send messages with different schema versions and a consumer can read messages with different schemas.

In 2.4.0 release, FORWARD_TRANSITIVE, BACKWARD_TRANSITIVE and FULL_TRANSITIVE compatibility strategies are added to check the compatibility with all existing schema version.

Replicated subscription​

In 2.4.0 release, a mechanism is added to keep subscription state in sync, within a sub-second timeframe, in the context of a topic that is being asynchronously replicated across multiple geographical regions. Here is architecture for replicated subscription.

The following is an example to use replicated subscription:


Consumer<String> consumer = client.newConsumer(Schema.STRING)
            .topic("my-topic")
            .subscriptionName("my-subscription")
            .replicateSubscriptionState(true)
            .subscribe();

New IO connectors​

A new batch of connectors is added, including Flume, Redis sink, Solr sink, RabbitMQ sink. The following lists builtin connectors that Pulsar supports.

Security​

In 2.4.0 release, Kerberos is supported in Apache Pulsar broker and client. To enable Kerberos authentication, refer to the document.

Also added role based Pulsar Function authentication and authorization.

Conclusion​

If you want to download Pulsar 2.4.0, click here. You can send any questions or suggestions to our mailing lists, contribute to Pulsar on GitHub or join the Apache Pulsar community on Slack.