Netty source connector
The Netty source connector opens a port that accepts incoming data via the configured network protocol and publish it to user-defined Pulsar topics.
This connector can be used in a containerized (for example, k8s) deployment. Otherwise, if the connector is running in process or thread mode, the instance may be conflicting on listening to ports.
Configuration​
The configuration of the Netty source connector has the following properties.
Property​
Name | Type | Required | Default | Description |
---|---|---|---|---|
type | String | true | tcp | The network protocol over which data is transmitted to netty. Below are the available options: |
host | String | true | 127.0.0.1 | The host name or address on which the source instance listen. |
port | int | true | 10999 | The port on which the source instance listen. |
numberOfThreads | int | true | 1 | The number of threads of Netty TCP server to accept incoming connections and handle the traffic of accepted connections. |
Example​
Before using the Netty source connector, you need to create a configuration file through one of the following methods.
JSON
{
"configs": {
"type": "tcp",
"host": "127.0.0.1",
"port": "10911",
"numberOfThreads": "1"
}
}YAML
configs:
type: "tcp"
host: "127.0.0.1"
port: 10999
numberOfThreads: 1
Usage​
The following examples show how to use the Netty source connector with TCP and HTTP.
TCP​
Start Pulsar standalone.
$ docker pull apachepulsar/pulsar:{version}
$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar standaloneCreate a configuration file netty-source-config.yaml.
configs:
type: "tcp"
host: "127.0.0.1"
port: 10999
numberOfThreads: 1Copy the configuration file netty-source-config.yaml to Pulsar server.
$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/Download the Netty source connector.
$ docker exec -it pulsar-netty-standalone /bin/bash
curl -O http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.narStart the Netty source connector.
$ ./bin/pulsar-admin sources localrun \
--archive pulsar-io-2.10.6.nar \
--tenant public \
--namespace default \
--name netty \
--destination-topic-name netty-topic \
--source-config-file netty-source-config.yaml \
--parallelism 1Consume data.
$ docker exec -it pulsar-netty-standalone /bin/bash
$ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0Open another terminal window to send data to the Netty source.
$ docker exec -it pulsar-netty-standalone /bin/bash
$ apt-get update
$ apt-get -y install telnet
$ root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
hello
worldThe following information appears on the consumer terminal window.
----- got message -----
hello
----- got message -----
world
HTTP​
Start Pulsar standalone.
$ docker pull apachepulsar/pulsar:{version}
$ docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-netty-standalone apachepulsar/pulsar:{version} bin/pulsar standaloneCreate a configuration file netty-source-config.yaml.
configs:
type: "http"
host: "127.0.0.1"
port: 10999
numberOfThreads: 1Copy the configuration file netty-source-config.yaml to Pulsar server.
$ docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/Download the Netty source connector.
$ docker exec -it pulsar-netty-standalone /bin/bash
curl -O http://mirror-hk.koddos.net/apache/pulsar/pulsar-{version}/connectors/pulsar-io-netty-{version}.narStart the Netty source connector.
$ ./bin/pulsar-admin sources localrun \
--archive pulsar-io-2.10.6.nar \
--tenant public \
--namespace default \
--name netty \
--destination-topic-name netty-topic \
--source-config-file netty-source-config.yaml \
--parallelism 1Consume data.
$ docker exec -it pulsar-netty-standalone /bin/bash
$ ./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0Open another terminal window to send data to the Netty source.
$ docker exec -it pulsar-netty-standalone /bin/bash
$ curl -X POST --data 'hello, world!' http://127.0.0.1:10999/The following information appears on the consumer terminal window.
----- got message -----
hello, world!