Hdfs Connector

Sink

The Hdfs Sink Connector is used to pull messages from Pulsar topics and persist the messages to an hdfs file.

Sink Configuration Options

Name	Default	Required	Description
hdfsConfigResources	null	true	A file or comma separated list of files which contains the Hadoop file system configuration, e.g. 'core-site.xml', 'hdfs-site.xml'.
directory	null	true	The HDFS directory from which files should be read from or written to.
encoding	null	false	The character encoding for the files, e.g. UTF-8, ASCII, etc.
compression	null	false	The compression codec used to compress/de-compress the files on HDFS.
kerberosUserPrincipal	null	false	The Kerberos user principal account to use for authentication.
keytab	null	false	The full pathname to the Kerberos keytab file to use for authentication.
filenamePrefix	null	false	The prefix of the files to create inside the HDFS directory, i.e. a value of "topicA" will result in files named topicA-, topicA-, etc being produced.
fileExtension	null	false	The extension to add to the files written to HDFS, e.g. '.txt', '.seq', etc.
separator	null	false	The character to use to separate records in a text file. If no value is provided then the content from all of the records will be concatenated together in one continuous byte array.
syncInterval	null	false	The interval (in milliseconds) between calls to flush data to HDFS disk.
maxPendingRecords	Integer.MAX_VALUE	false	The maximum number of records that we hold in memory before acking. Default is Integer.MAX_VALUE. Setting this value to one, results in every record being sent to disk before the record is acked, while setting it to a higher values allows us to buffer records before flushing them all to disk.