Using TLS with KeyStore configure
Overviewβ
Apache Pulsar supports TLS encryption and TLS authentication between clients and Apache Pulsar service. By default it uses PEM format file configuration. This page tries to describe use KeyStore type configure for TLS.
TLS encryption with KeyStore configureβ
Generate TLS key and certificateβ
The first step of deploying TLS is to generate the key and the certificate for each machine in the cluster.
You can use Javaβs keytool
utility to accomplish this task. We will generate the key into a temporary keystore
initially for broker, so that we can export and sign it later with CA.
keytool -keystore broker.keystore.jks -alias localhost -validity {validity} -genkeypair -keyalg RSA
You need to specify two parameters in the above command:
keystore
: the keystore file that stores the certificate. The keystore file contains the private key of the certificate; hence, it needs to be kept safely.validity
: the valid time of the certificate in days.
Ensure that common name (CN) matches exactly with the fully qualified domain name (FQDN) of the server. The client compares the CN with the DNS domain name to ensure that it is indeed connecting to the desired server, not a malicious one.
Creating your own CAβ
After the first step, each broker in the cluster has a public-private key pair, and a certificate to identify the machine. The certificate, however, is unsigned, which means that an attacker can create such a certificate to pretend to be any machine.
Therefore, it is important to prevent forged certificates by signing them for each machine in the cluster.
A certificate authority (CA)
is responsible for signing certificates. CA works likes a government that issues passports β
the government stamps (signs) each passport so that the passport becomes difficult to forge. Other governments verify the stamps
to ensure the passport is authentic. Similarly, the CA signs the certificates, and the cryptography guarantees that a signed
certificate is computationally difficult to forge. Thus, as long as the CA is a genuine and trusted authority, the clients have
high assurance that they are connecting to the authentic machines.
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
The generated CA is simply a public-private key pair and certificate, and it is intended to sign other certificates.
The next step is to add the generated CA to the clients' truststore so that the clients can trust this CA:
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
NOTE: If you configure the brokers to require client authentication by setting tlsRequireTrustedClientCertOnConnect
to true
on the
broker configuration, then you must also provide a truststore for the brokers and it should have all the CA certificates that clients keys were signed by.
keytool -keystore broker.truststore.jks -alias CARoot -import -file ca-cert
In contrast to the keystore, which stores each machineβs own identity, the truststore of a client stores all the certificates that the client should trust. Importing a certificate into oneβs truststore also means trusting all certificates that are signed by that certificate. As the analogy above, trusting the government (CA) also means trusting all passports (certificates) that it has issued. This attribute is called the chain of trust, and it is particularly useful when deploying TLS on a large BookKeeper cluster. You can sign all certificates in the cluster with a single CA, and have all machines share the same truststore that trusts the CA. That way all machines can authenticate all other machines.
Signing the certificateβ
The next step is to sign all certificates in the keystore with the CA we generated. First, you need to export the certificate from the keystore:
keytool -keystore broker.keystore.jks -alias localhost -certreq -file cert-file
Then sign it with the CA:
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}
Finally, you need to import both the certificate of the CA and the signed certificate into the keystore:
keytool -keystore broker.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore broker.keystore.jks -alias localhost -import -file cert-signed
The definitions of the parameters are the following:
keystore
: the location of the keystoreca-cert
: the certificate of the CAca-key
: the private key of the CAca-password
: the passphrase of the CAcert-file
: the exported, unsigned certificate of the brokercert-signed
: the signed certificate of the broker
Configuring brokersβ
Brokers enable TLS by provide valid brokerServicePortTls
and webServicePortTls
, and also need set tlsEnabledWithKeyStore
to true
for using KeyStore type configuration.
Besides this, KeyStore path, KeyStore password, TrustStore path, and TrustStore password need to provided.
And since broker will create internal client/admin client to communicate with other brokers, user also need to provide config for them, this is similar to how user config the outside client/admin-client.
If tlsRequireTrustedClientCertOnConnect
is true
, broker will reject the Connection if the Client Certificate is not trusted.
The following TLS configs are needed on the broker side:
tlsEnabledWithKeyStore=true
# key store
tlsKeyStoreType=JKS
tlsKeyStore=/var/private/tls/broker.keystore.jks
tlsKeyStorePassword=brokerpw
# trust store
tlsTrustStoreType=JKS
tlsTrustStore=/var/private/tls/broker.truststore.jks
tlsTrustStorePassword=brokerpw
# internal client/admin-client config
brokerClientTlsEnabled=true
brokerClientTlsEnabledWithKeyStore=true
brokerClientTlsTrustStoreType=JKS
brokerClientTlsTrustStore=/var/private/tls/client.truststore.jks
brokerClientTlsTrustStorePassword=clientpw
NOTE: it is important to restrict access to the store files via filesystem permissions.
If you have configured TLS on the broker, to disable non-TLS ports, you can set the values of the following configurations to empty as below.
brokerServicePort=
webServicePort=
In this case, you need to set the following configurations.
brokerClientTlsEnabled=true // Set this to true
brokerClientTlsEnabledWithKeyStore=true // Set this to true
brokerClientTlsTrustStore= // Set this to your desired value
brokerClientTlsTrustStorePassword= // Set this to your desired value
Optional settings that may worth consider:
- tlsClientAuthentication=false: Enable/Disable using TLS for authentication. This config when enabled will authenticate the other end of the communication channel. It should be enabled on both brokers and clients for mutual TLS.
- tlsCiphers=[TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256], A cipher suite is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS network protocol. By default, it is null. OpenSSL CiphersJDK Ciphers
- tlsProtocols=[TLSv1.3,TLSv1.2] (list out the TLS protocols that you are going to accept from clients). By default, it is not set.
Configuring Clientsβ
This is similar to TLS encryption configuing for client with PEM type. For a minimal configuration, you need to provide the TrustStore information.
For example:
for Command-line tools like
pulsar-admin
,pulsar-perf
, andpulsar-client
use theconf/client.conf
config file in a Pulsar installation.
webServiceUrl=https://broker.example.com:8443/
brokerServiceUrl=pulsar+ssl://broker.example.com:6651/
useKeyStoreTls=true
tlsTrustStoreType=JKS
tlsTrustStorePath=/var/private/tls/client.truststore.jks
tlsTrustStorePassword=clientpwfor java client
import org.apache.pulsar.client.api.PulsarClient;
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://broker.example.com:6651/")
.enableTls(true)
.useKeyStoreTls(true)
.tlsTrustStorePath("/var/private/tls/client.truststore.jks")
.tlsTrustStorePassword("clientpw")
.allowTlsInsecureConnection(false)
.build();for java admin client
PulsarAdmin amdin = PulsarAdmin.builder().serviceHttpUrl("https://broker.example.com:8443")
.useKeyStoreTls(true)
.tlsTrustStorePath("/var/private/tls/client.truststore.jks")
.tlsTrustStorePassword("clientpw")
.allowTlsInsecureConnection(false)
.build();
Note: Please configure
tlsTrustStorePath
when you setuseKeyStoreTls
totrue
.
TLS authentication with KeyStore configureβ
This similar to TLS authentication with PEM type
broker authentication configβ
broker.conf
# Configuration to enable authentication
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderTls
# this should be the CN for one of client keystore.
superUserRoles=admin
# Enable KeyStore type
tlsEnabledWithKeyStore=true
requireTrustedClientCertOnConnect=true
# key store
tlsKeyStoreType=JKS
tlsKeyStore=/var/private/tls/broker.keystore.jks
tlsKeyStorePassword=brokerpw
# trust store
tlsTrustStoreType=JKS
tlsTrustStore=/var/private/tls/broker.truststore.jks
tlsTrustStorePassword=brokerpw
# internal client/admin-client config
brokerClientTlsEnabled=true
brokerClientTlsEnabledWithKeyStore=true
brokerClientTlsTrustStoreType=JKS
brokerClientTlsTrustStore=/var/private/tls/client.truststore.jks
brokerClientTlsTrustStorePassword=clientpw
# internal auth config
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls
brokerClientAuthenticationParameters={"keyStoreType":"JKS","keyStorePath":"/var/private/tls/client.keystore.jks","keyStorePassword":"clientpw"}
# currently websocket not support keystore type
webSocketServiceEnabled=false
client authentication configuringβ
Besides the TLS encryption configuring. The main work is configuring the KeyStore, which contains a valid CN as client role, for client.
For example:
for Command-line tools like
pulsar-admin
,pulsar-perf
, andpulsar-client
use theconf/client.conf
config file in a Pulsar installation.
webServiceUrl=https://broker.example.com:8443/
brokerServiceUrl=pulsar+ssl://broker.example.com:6651/
useKeyStoreTls=true
tlsTrustStoreType=JKS
tlsTrustStorePath=/var/private/tls/client.truststore.jks
tlsTrustStorePassword=clientpw
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls
authParams={"keyStoreType":"JKS","keyStorePath":"/path/to/keystorefile","keyStorePassword":"keystorepw"}for java client
import org.apache.pulsar.client.api.PulsarClient;
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://broker.example.com:6651/")
.enableTls(true)
.useKeyStoreTls(true)
.tlsTrustStorePath("/var/private/tls/client.truststore.jks")
.tlsTrustStorePassword("clientpw")
.allowTlsInsecureConnection(false)
.authentication(
"org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
"keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
.build();for java admin client
PulsarAdmin amdin = PulsarAdmin.builder().serviceHttpUrl("https://broker.example.com:8443")
.useKeyStoreTls(true)
.tlsTrustStorePath("/var/private/tls/client.truststore.jks")
.tlsTrustStorePassword("clientpw")
.allowTlsInsecureConnection(false)
.authentication(
"org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
"keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
.build();
Note: Please configure
tlsTrustStorePath
when you setuseKeyStoreTls
totrue
.
Enabling TLS Loggingβ
You can enable TLS debug logging at the JVM level by starting the brokers and/or clients with javax.net.debug
system property. For example:
-Djavax.net.debug=all
You can find more details on this in Oracle documentation on debugging SSL/TLS connections.