Skip to main content
Version: Next

End-to-End Encryption

Applications can use Pulsar end-to-end encryption (E2EE) to encrypt messages on the producer side and decrypt messages on the consumer side. You can use the public and private key pair that the application configures to perform encryption and decryption. Only the consumers with a valid key can decrypt the encrypted messages.

How it works in Pulsar

Pulsar uses a dynamically generated symmetric AES key to encrypt messages (data). You can use the application-provided ECDSA (Elliptic Curve Digital Signature Algorithm) or RSA (Rivest–Shamir–Adleman) key pair to encrypt the AES key (data key), so you do not have to share the secret with everyone.

The application configures the producer with the public key for encryption. You can use this key to encrypt the AES data key. The encrypted data key is sent as part of the message header. Only entities with the private key (in this case the consumer) can decrypt the data key which is used to decrypt the message.

The following figure illustrates how Pulsar encrypts messages on the producer side and decrypts messages on the consumer side.

alt text

If produced messages are consumed across application boundaries, you need to ensure that consumers in other applications have access to one of the private keys that can decrypt the messages. You can do this in two ways:

  1. The consumer application provides you access to the public key, which you add to your producer keys.
  2. You grant access to one of the private keys from the pairs that the producer uses.
tip
  • Pulsar does not store the encryption key anywhere in the Pulsar service. If you lose or delete the private key, your message is irretrievably lost and unrecoverable.
  • Pulsar generates a new AES data key every 4 hours or after publishing a certain number of messages. Producers fetch the asymmetric public key every 4 hours by calling CryptoKeyReader.getPublicKey() to retrieve the latest version.

Get started

Prerequisites

  • Pulsar Java/Python/C++/Node.js client 2.7.1 or later versions.
  • Pulsar Go client 0.6.0 or later versions.

Configure end-to-end encryption

  1. Create both public and private key pairs.

    openssl ecparam -name secp521r1 -genkey -param_enc explicit -out test_ecdsa_privkey.pem
    openssl ec -in test_ecdsa_privkey.pem -pubout -outform pem -out test_ecdsa_pubkey.pem
  2. Configure a CryptoKeyReader on producers, consumers or readers.

    PulsarClient pulsarClient = PulsarClient.builder().serviceUrl("pulsar://localhost:6650").build();
    String topic = "persistent://my-tenant/my-ns/my-topic";
    // RawFileKeyReader is just an example implementation that's not provided by Pulsar
    CryptoKeyReader keyReader = new RawFileKeyReader("test_ecdsa_pubkey.pem", "test_ecdsa_privkey.pem");

    Producer<byte[]> producer = pulsarClient.newProducer()
    .topic(topic)
    .cryptoKeyReader(keyReader)
    .addEncryptionKey("myappkey")
    .create();

    Consumer<byte[]> consumer = pulsarClient.newConsumer()
    .topic(topic)
    .subscriptionName("my-subscriber-name")
    .cryptoKeyReader(keyReader)
    .subscribe();

    Reader<byte[]> reader = pulsarClient.newReader()
    .topic(topic)
    .startMessageId(MessageId.earliest)
    .cryptoKeyReader(keyReader)
    .create();
  3. Optional: customize the CryptoKeyReader implementation.

    class RawFileKeyReader implements CryptoKeyReader {

    String publicKeyFile = "";
    String privateKeyFile = "";

    RawFileKeyReader(String pubKeyFile, String privKeyFile) {
    publicKeyFile = pubKeyFile;
    privateKeyFile = privKeyFile;
    }

    @Override
    public EncryptionKeyInfo getPublicKey(String keyName, Map<String, String> keyMeta) {
    EncryptionKeyInfo keyInfo = new EncryptionKeyInfo();
    try {
    keyInfo.setKey(Files.readAllBytes(Paths.get(publicKeyFile)));
    } catch (IOException e) {
    System.out.println("ERROR: Failed to read public key from file " + publicKeyFile);
    e.printStackTrace();
    }
    return keyInfo;
    }

    @Override
    public EncryptionKeyInfo getPrivateKey(String keyName, Map<String, String> keyMeta) {
    EncryptionKeyInfo keyInfo = new EncryptionKeyInfo();
    try {
    keyInfo.setKey(Files.readAllBytes(Paths.get(privateKeyFile)));
    } catch (IOException e) {
    System.out.println("ERROR: Failed to read private key from file " + privateKeyFile);
    e.printStackTrace();
    }
    return keyInfo;
    }
    }

Encrypt a message with multiple keys

note

This is only available for Java clients.

You can encrypt a message with more than one key. Producers add all such keys to the config and consumers can decrypt the message as long as they have access to at least one of the keys. Any one of the keys used for encrypting the message is sufficient to decrypt the message.

For example, encrypt the messages using 2 keys (myapp.messagekey1 and myapp.messagekey2):

PulsarClient.newProducer().addEncryptionKey("myapp.messagekey1").addEncryptionKey("myapp.messagekey2");

Troubleshoot

  • Producer/Consumer loses access to the key
    • Producer action fails to indicate the cause of the failure. Application has the option to proceed with sending unencrypted messages in such cases. Call PulsarClient.newProducer().cryptoFailureAction(ProducerCryptoFailureAction) to control the producer behavior. The default behavior is to fail the request.
    • If consumption fails due to decryption failure or missing keys in the consumer, the application has the option to consume the encrypted message or discard it. Call PulsarClient.newConsumer().cryptoFailureAction(ConsumerCryptoFailureAction) to control the consumer behavior. The default behavior is to fail the request. Application is never able to decrypt the messages if the private key is permanently lost.
  • Batch messaging
    • If decryption fails and the message contains batch messages, client is not able to retrieve individual messages in the batch, hence message consumption fails even if cryptoFailureAction() is set to ConsumerCryptoFailureAction.CONSUME.
  • If decryption fails, the message consumption stops and the application notices backlog growth in addition to decryption failure messages in the client log. If the application does not have access to the private key to decrypt the message, the only option is to skip or discard backlogged messages.