Skip to main content

AWS Kinesis Connector

Sink​

The Kinesis Sink connector is used to pull data from Pulsar topics and persist the data into AWS Kinesis.

Sink Configuration Options​

NameRequiredDefaultDescription
awsEndpointtruenullkinesis end-point url can be found at : https://docs.aws.amazon.com/general/latest/gr/rande.html
awsRegiontruenullappropriate aws region eg: us-west-1, us-west-2
awsKinesisStreamNametruenullkinesis stream name
awsCredentialPluginNamefalsenullFully-Qualified class name of implementation of AwsCredentialProviderPlugin. It is a factory class which creates an AWSCredentialsProvider that will be used by Kinesis Sink. If it is empty then KinesisSink will create a default AWSCredentialsProvider which accepts json-map of credentials in awsCredentialPluginParam
awsCredentialPluginParamfalsenulljson-parameters to initialize AwsCredentialsProviderPlugin
messageFormattrueONLY_RAW_PAYLOADMessage format in which kinesis sink converts pulsar messages and publishes to kinesis streams

Message Formats​

The available message formats are listed as below:

ONLY_RAW_PAYLOAD​

Kinesis sink directly publishes pulsar message payload as a message into the configured kinesis stream.

FULL_MESSAGE_IN_JSON​

Kinesis sink creates a json payload with pulsar message payload, properties and encryptionCtx, and publishes json payload into the configured kinesis stream.

FULL_MESSAGE_IN_FB​

Kinesis sink creates a flatbuffer serialized paylaod with pulsar message payload, properties and encryptionCtx, and publishes flatbuffer payload into the configured kinesis stream.