Pulsar SQl Deployment and Configuration

Below is a list configurations for the Presto Pulsar connector and instruction on how to deploy a cluster.

Presto Pulsar Connector Configurations

There are several configurations for the Presto Pulsar Connector. The properties file that contain these configurations can be found at ${project.root}/conf/presto/catalog/pulsar.properties. The configurations for the connector and its default values are described below.

# name of the connector to be displayed in the catalog
connector.name=pulsar

# the url of Pulsar broker service
pulsar.broker-service-url=http://localhost:8080

# URI of Zookeeper cluster
pulsar.zookeeper-uri=localhost:2181

# minimum number of entries to read at a single time
pulsar.entry-read-batch-size=100

# default number of splits to use per query
pulsar.target-num-splits=4

Query Pulsar from Existing Presto Cluster

If you already have an existing Presto cluster, you can copy Presto Pulsar connector plugin to your existing cluster. You can download the archived plugin package via:

$ wget https://archive.apache.org/dist/pulsar/pulsar-2.2.0/apache-pulsar-2.2.0-bin.tar.gz

Deploying a new cluster

Please note that the Getting Started guide shows you how to easily setup a standalone single node environment to experiment with.

Pulsar SQL is powered by Presto thus many of the configurations for deployment is the same for the Pulsar SQL worker.

You can use the same CLI args as the Presto launcher:

$ ./bin/pulsar sql-worker --help
Usage: launcher [options] command

Commands: run, start, stop, restart, kill, status

Options:
  -h, --help            show this help message and exit
  -v, --verbose         Run verbosely
  --etc-dir=DIR         Defaults to INSTALL_PATH/etc
  --launcher-config=FILE
                        Defaults to INSTALL_PATH/bin/launcher.properties
  --node-config=FILE    Defaults to ETC_DIR/node.properties
  --jvm-config=FILE     Defaults to ETC_DIR/jvm.config
  --config=FILE         Defaults to ETC_DIR/config.properties
  --log-levels-file=FILE
                        Defaults to ETC_DIR/log.properties
  --data-dir=DIR        Defaults to INSTALL_PATH
  --pid-file=FILE       Defaults to DATA_DIR/var/run/launcher.pid
  --launcher-log-file=FILE
                        Defaults to DATA_DIR/var/log/launcher.log (only in
                        daemon mode)
  --server-log-file=FILE
                        Defaults to DATA_DIR/var/log/server.log (only in
                        daemon mode)
  -D NAME=VALUE         Set a Java system property

There is a set of default configs for the cluster located in ${project.root}/conf/presto that will be used by default. You can change them to customize your deployment

You can also set the worker to read from a different configuration directory as well as set a different directory for writing its data:

$ ./bin/pulsar sql-worker run --etc-dir /tmp/incubator-pulsar/conf/presto --data-dir /tmp/presto-1

You can also start the worker as daemon process:

$ ./bin/pulsar sql-worker start

Deploying to a 3 node cluster

For example, if I wanted to deploy a Pulsar SQL/Presto cluster on 3 nodes, you can do the following:

First, copy the Pulsar binary distribution to all three nodes.

The first node, will run the Presto coordinator. The minimal configuration in ${project.root}/conf/presto/config.properties can be the following

coordinator=true
node-scheduler.include-coordinator=true
http-server.http.port=8080
query.max-memory=50GB
query.max-memory-per-node=1GB
discovery-server.enabled=true
discovery.uri=<coordinator-url>

Also, modify pulsar.broker-service-url and pulsar.zookeeper-uri configs in ${project.root}/conf/presto/catalog/pulsar.properties on those nodes accordingly

Afterwards, you can start the coordinator by just running

$ ./bin/pulsar sql-worker run

For the other two nodes that will only serve as worker nodes, the configurations can be the following:

properties
coordinator=false
http-server.http.port=8080
query.max-memory=50GB
query.max-memory-per-node=1GB
discovery.uri=<coordinator-url>

Also, modify

pulsar.broker-service-url``` and  ```pulsar.zookeeper-uri``` configs in ```${project.root}/conf/presto/catalog/pulsar.properties``` accordingly

You can also start the worker by just running: ```$ ./bin/pulsar sql-worker run```

You can check the status of your cluster from the SQL CLI.  To start the SQL CLI:

```bash

$ ./bin/pulsar sql --server <coordinate_url>

You can then run the following command to check the status of your nodes:

presto> SELECT * FROM system.runtime.nodes;
 node_id |        http_uri         | node_version | coordinator | state
---------+-------------------------+--------------+-------------+--------
 1       | http://192.168.2.1:8081 | testversion  | true        | active
 3       | http://192.168.2.2:8081 | testversion  | false       | active
 2       | http://192.168.2.3:8081 | testversion  | false       | active