Kinesis source connector
The Kinesis source connector pulls data from Amazon Kinesis and persists data into Pulsar.
This connector uses the Kinesis Consumer Library (KCL) to do the actual consuming of messages. The KCL uses DynamoDB to track state for consumers.
Note: currently, the Kinesis source connector only supports raw messages. If you use KMS encrypted messages, the encrypted messages are sent to downstream. This connector will support decrypting messages in the future release.
The configuration of the Kinesis source connector has the following properties.
|InitialPositionInStream||false||LATEST||The position where the connector starts from.|
Below are the available options:
|Date||false||" " (empty string)||If set to |
|String||false||Pulsar IO connector||The name of the Amazon Kinesis application. |
By default, the application name is included in the user agent string used to make AWS requests. This can assist with troubleshooting, for example, distinguish requests made by separate connector instances.
|long||false||60000||The frequency of the Kinesis stream checkpoint in milliseconds.|
|long||false||3000||The amount of time to delay between requests when the connector encounters a throttling exception from AWS Kinesis in milliseconds.|
|int||false||3||The number of re-attempts when the connector encounters an exception while trying to set a checkpoint.|
|int||false||1000||The maximum number of AWS records that can be buffered inside the connector. |
|String||false||" " (empty string)||The Dynamo end-point URL, which can be found at here.|
|String||false||" " (empty string)||The Cloudwatch end-point URL, which can be found at here.|
|boolean||false||true||If set to true, it uses Kinesis enhanced fan-out.|
If set to false, it uses polling.
|String||false||" " (empty string)||The Kinesis end-point URL, which can be found at here.|
|String||false||" " (empty string)||The AWS region. |
|String||true||" " (empty string)||The Kinesis stream name.|
|String||false||" " (empty string)||The fully-qualified class name of implementation of AwsCredentialProviderPlugin.|
this plugin uses the default AWS provider chain.
For more information, see using the default credential provider chain.
this plugin takes a configuration via the
JSON configuration example
|String||false||" " (empty string)||The JSON parameter to initialize |
Before using the Kinesis source connector, you need to create a configuration file through one of the following methods.
"applicationName": "My test application",
applicationName: "My test application"