Skip to main content

Netty Tcp or Udp Connector

Source​

The Netty Source connector opens a port that accept incoming data via the configured network protocol and publish it to a user-defined Pulsar topic. Also, this connector is suggested to be used in a containerized (e.g. k8s) deployment. Otherwise, if the connector is running in process or thread mode, the instances may be conflicting on listening to ports.

Source Configuration Options​

NameRequiredDefaultDescription
typefalsetcpThe network protocol over which data is trasmitted to netty. Valid values include HTTP, TCP, and UDP
hostfalse127.0.0.1The host name or address that the source instance to listen on.
portfalse10999The port that the source instance to listen on.
numberOfThreadsfalse1The number of threads of Netty Tcp Server to accept incoming connections and handle the traffic of the accepted connections.

Configuration Example​

Here is a configuration Json example:


{
"type": "tcp",
"host": "127.0.0.1",
"port": "10911",
"numberOfThreads": "5"
}

Here is a configuration Yaml example:


configs:
type: "tcp"
host: "127.0.0.1"
port: 10999
numberOfThreads: 1

Usage example​

  • Start pulsar standalone

docker pull apachepulsar/pulsar:2.4.0
docker run -d -it -p 6650:6650 -p 8080:8080 -v $PWD/data:/pulsar/data --name pulsar-netty-standalone apachepulsar/pulsar:2.4.0 bin/pulsar standalone

  • Start pulsar-io in standalone

Tcp example​

  • Config file netty-source-config.yaml

configs:
type: "tcp"
host: "127.0.0.1"
port: 10999
numberOfThreads: 1

  • Copy configuration file to pulsar server

docker cp netty-source-config.yaml pulsar-netty-standalone:/pulsar/conf/

  • Download netty connector and start netty connector

docker exec -it pulsar-netty-standalone /bin/bash
curl -O http://mirror-hk.koddos.net/apache/pulsar/pulsar-2.4.0/connectors/pulsar-io-netty-2.4.0.nar
./bin/pulsar-admin sources localrun --archive pulsar-io-2.4.2.nar --tenant public --namespace default --name netty --destination-topic-name netty-topic --source-config-file netty-source-config.yaml --parallelism 1

  • Consume data

docker exec -it pulsar-netty-standalone /bin/bash
./bin/pulsar-client consume -t Exclusive -s netty-sub netty-topic -n 0

  • Open another window for send data to netty source

docker exec -it pulsar-netty-standalone /bin/bash
apt-get update
apt-get -y install telnet
root@1d19327b2c67:/pulsar# telnet 127.0.0.1 10999
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
hello
world

  • Verification results

In the consumer window just opened, you can see the following data


----- got message -----
hello

----- got message -----
world

Http example​

  • Config file netty-source-config.yaml

configs:
type: "http"
host: "127.0.0.1"
port: 10999
numberOfThreads: 1

  • Start netty source

docker exec -it pulsar-netty-standalone /bin/bash
./bin/pulsar-admin sources localrun --archive pulsar-io-2.4.2.nar --tenant public --namespace default --name netty --destination-topic-name netty-topic --source-config-file netty-source-config.yaml --parallelism 1

  • Verification results

curl -X POST --data 'hello, world!' http://127.0.0.1:10999/

  • Verification results

In the consumer window just opened, you can see the following data


----- got message -----
hello, world!