Skip to main content

RabbitMQ source connector

note

You can download all the Pulsar connectors on download page.

The RabbitMQ source connector receives messages from RabbitMQ clusters and writes messages to Pulsar topics.

Configuration

The configuration of the RabbitMQ source connector has the following properties.

Property

NameTypeRequiredDefaultDescription
connectionNameStringtrue(empty string)The connection name.
hostStringtrue(empty string)The RabbitMQ host.
portinttrue5672The RabbitMQ port.
virtualHostStringtrue/The virtual host used to connect to RabbitMQ.
usernameStringfalseguestThe username used to authenticate to RabbitMQ.
passwordStringfalseguestThe password used to authenticate to RabbitMQ.
queueNameStringtrue(empty string)The RabbitMQ queue name that messages should be read from or written to.
requestedChannelMaxintfalse0The initially requested maximum channel number.
0 means unlimited.
requestedFrameMaxintfalse0The initially requested maximum frame size in octets.
0 means unlimited.
connectionTimeoutintfalse60000The timeout of TCP connection establishment in milliseconds.
0 means infinite.
handshakeTimeoutintfalse10000The timeout of AMQP0-9-1 protocol handshake in milliseconds.
requestedHeartbeatintfalse60The requested heartbeat timeout in seconds.
prefetchCountintfalse0The maximum number of messages that the server delivers.
0 means unlimited.
prefetchGlobalbooleanfalsefalseWhether the setting should be applied to the entire channel rather than each consumer.
passivebooleanfalsefalseWhether the rabbitmq consumer should create its own queue or bind to an existing one.

Example

Before using the RabbitMQ source connector, you need to create a configuration file through one of the following methods.

  • JSON

    {
    "configs": {
    "host": "localhost",
    "port": "5672",
    "virtualHost": "/",
    "username": "guest",
    "password": "guest",
    "queueName": "test-queue",
    "connectionName": "test-connection",
    "requestedChannelMax": "0",
    "requestedFrameMax": "0",
    "connectionTimeout": "60000",
    "handshakeTimeout": "10000",
    "requestedHeartbeat": "60",
    "prefetchCount": "0",
    "prefetchGlobal": "false",
    "passive": "false"
    }
    }
  • YAML

    configs:
    host: "localhost"
    port: 5672
    virtualHost: "/"
    username: "guest"
    password: "guest"
    queueName: "test-queue"
    connectionName: "test-connection"
    requestedChannelMax: 0
    requestedFrameMax: 0
    connectionTimeout: 60000
    handshakeTimeout: 10000
    requestedHeartbeat: 60
    prefetchCount: 0
    prefetchGlobal: "false"
    passive: "false"

Usage

Standalone mode

This example describes how to use the RabbitMQ source connector to feed data from RabbitMQ and write data to Pulsar topics in the standalone mode.

Prerequisites

  • There is a RabbitMQ server with some history messages in the queue.

Steps

  1. Get a Pulsar package and start Pulsar in standalone mode.

    wget https://archive.apache.org/dist/pulsar/pulsar-3.0.8/apache-pulsar-3.0.8-bin.tar.gz
    tar xvfz apache-pulsar-3.0.8-bin.tar.gz
    cd apache-pulsar-3.0.8
    bin/pulsar standalone
  2. Download the nar package corresponding to Pulsar's version and copy the following file to Pulsar's directory.

    wget https://archive.apache.org/dist/pulsar/pulsar-3.0.8/connectors/pulsar-io-rabbitmq-3.0.8.nar
    cp pulsar-io-rabbitmq-3.0.8.nar ./connectors
  3. Set the retention of the namespace, otherwise the messages into the Pulsar's topic which have not the subscription have been immediately deleted.

    ./bin/pulsar-admin namespaces set-retention -s 100M -t 3d public/default
  4. Prepare a configuration file with name is rabbitmq-source-queue-name.yaml.

     configs:
    host: "localhost"
    port: 5672
    virtualHost: "/"
    username: "guest"
    password: "guest"
    queueName: "test-queue"
    connectionName: "test-connection"
    requestedChannelMax: 0
    requestedFrameMax: 0
    connectionTimeout: 60000
    handshakeTimeout: 10000
    requestedHeartbeat: 60
    prefetchCount: 0
    prefetchGlobal: "false"
    passive: "false"

    Copy the configuration file to Pulsar‘s conf directory.

    cp rabbitmq-source-queue-name.yaml ./conf
  5. Open a new terminal window and start the connector in local run mode.

    ./bin/pulsar-admin source localrun \
    --source-config-file $PWD/conf/rabbitmq-source-queue-name.yaml \
    --archive $PWD/connectors/pulsar-io-rabbitmq-3.0.8.nar \
    --name rabbitmq-source \
    --destination-topic-name pulsar-rabbitmq-test-topic \
    --broker-service-url pulsar://{ip}:{port}
  6. Open a new terminal window and check the topic is created automatically.

    ./bin/pulsar-admin topics list public/default \

    This topic is created automatically as follows:

    persistent://public/default/pulsar-rabbitmq-test-topic-partition-0
  7. Consume this topic.

    ./bin/pulsar-client consume persistent://public/default/pulsar-rabbitmq-test-topic-partition-0 -s s1 -n 0 -p Earliest

    The following information appears on the consumer terminal window.

       ----- got message -----
    key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 0
    ----- got message -----
    key:[quick.orange.pulsar], properties:[], content:message-topic-O(range) 1

    ... ...