Connect to RabbitMQ Data as an External Source in Dremio

The CData API Driver for JDBC implements JDBC Standards and allows various applications, including Dremio, to work with live RabbitMQ data. Dremio is a data lakehouse platform designed to empower self-service, interactive analytics on the data lake. With the CData JDBC driver, you can include live RabbitMQ data as a part of your enterprise data lake. This article describes how to connect to RabbitMQ data from Dremio as an External Source.

The CData JDBC Driver enables high-speed access to live RabbitMQ data in Dremio. Once you install the driver, authenticate with RabbitMQ and gain immediate access to RabbitMQ data within your data lake. By surfacing RabbitMQ data using native data types and handling complex filters, aggregations, & other operations automatically, the CData JDBC Driver grants seamless access to RabbitMQ data.

Prerequisites

This article assumes you are utilizing Docker to run Dremio. You can create a Docker container with the Dremio service using a command similar to the follow:

docker run -d --name dremio -p 9047:9047 -p 31010:31010 dremio/dremio-oss

Where dremio is the name of the container, 9047 is the container's port for the Dremio web interface and 31010 is the port that maps to the Dremio query service. dremio/dremio-oss specifies the image to use.

Build the ARP Connector

To use the CData JDBC Driver in Dremio, you need to build an Advanced Relation Pushdown (ARP) Connector. You can view the source code for the Connector on GitHub or download the ZIP file (GitHub.com) directly. Once you copy or extract the files, run the following command from the root directory of the connector (the directory containing the pom.xml file) to build the connector.

mvn clean install

NOTE: The CData ARP Connectors are build to be compiled with Java 11. Be sure to install Java 11 and use the correct version. You can update your Java version using a command similar to the following:

sudo update-alternatives --config java

Once the JAR file for the connector is built (in the target directory), you are ready to copy the ARP connector and JDBC Driver to your Dremio instance.

Installing the Connector and JDBC Driver

Install the ARP Connector to %DREMIO_HOME%/jars/ and the JDBC Driver for RabbitMQ to %DREMIO_HOME%/jars/3rdparty. You can use commands similar to the following:

ARP Connector

docker cp PATH\TO\dremio-api-plugin-{DREMIO_VERSION}.jar dremio_image_name:/opt/dremio/jars/

JDBC Driver for RabbitMQ

docker cp PATH\TO\cdata.jdbc.api.jar dremio_image_name:/opt/dremio/jars/3rdparty/

Connecting to RabbitMQ

RabbitMQ will now appear as an External Source option in Dremio. The ARP Connector built uses a JDBC URL to connect to RabbitMQ data. The JDBC Driver has a built-in connection string designer that you can use (see below).

Built-in Connection String Designer

For assistance in constructing the JDBC URL, use the connection string designer built into the RabbitMQ JDBC Driver. Double-click the JAR file or execute the jar file from the command line.

java -jar cdata.jdbc.api.jar

Fill in the connection properties and copy the connection string to the clipboard.

About RabbitMQ Management HTTP API

RabbitMQ is an open-source message broker that supports multiple messaging protocols. The RabbitMQ Management HTTP API provides HTTP-based access to management and monitoring data for a RabbitMQ server. The API exposes information about virtual hosts, exchanges, queues, bindings, connections, channels, consumers, users, permissions, policies, and cluster-wide statistics.

The Management plugin must be enabled on the RabbitMQ server for the HTTP API to be available. By default, the management interface listens on port 15672.

Using Basic Authentication

RabbitMQ Management HTTP API uses HTTP Basic authentication. You must supply the username and password of a RabbitMQ management user.

To enable access to the management API:

1. Ensure the RabbitMQ Management plugin is enabled on your server (rabbitmq-plugins enable rabbitmq_management).
2. Use an existing management user or create one with the appropriate management tag (management, policymaker, monitoring, or administrator).
3. Note the full base URL of your RabbitMQ Management HTTP API (e.g., http://localhost:15672).

After configuring your RabbitMQ server, set the following connection properties to connect:

• AuthScheme: Set this to Basic.
• URL: Set this to the base URL of your RabbitMQ Management HTTP API (e.g., http://localhost:15672).
• User: Set this to your RabbitMQ management username (e.g., guest).
• Password: Set this to your RabbitMQ management password.

Example connection string:

Profile=C:\profiles\RabbitMQ.apip;AuthScheme=Basic;URL=http://localhost:15672;User=guest;Password=guest;

Available Tables

The RabbitMQ profile provides access to the following tables:

• Overview - Cluster-wide statistics and information about the RabbitMQ node
• Nodes - Information about individual nodes in the RabbitMQ cluster
• NodeMemory - Detailed memory usage breakdown for a specific cluster node
• Connections - List of all open AMQP connections to the broker
• Channels - List of all open AMQP channels across all connections
• Consumers - List of all consumers registered across all queues
• Exchanges - List of exchanges declared across all virtual hosts
• Queues - List of queues declared across all virtual hosts
• Bindings - List of all bindings between exchanges and queues
• VirtualHosts - List of virtual hosts configured on the broker
• VhostPermissions - User permissions within a specific virtual host
• Users - List of all RabbitMQ users
• Permissions - Permission records for all users across all virtual hosts
• TopicPermissions - Topic-level permission records for all users
• Policies - List of policies applied to queues and exchanges in virtual hosts
• OperatorPolicies - List of operator policies applied to queues in virtual hosts
• Parameters - List of component parameters (e.g., federation, shovel) per virtual host
• GlobalParameters - List of global parameters that apply across all virtual hosts
• VhostLimits - Resource limits configured for specific virtual hosts
• UserLimits - Resource limits configured for specific users
• FeatureFlags - List of feature flags and their enabled/disabled state on the node
• DeprecatedFeatures - List of deprecated features and their usage state
• AuthAttempts - Authentication attempt statistics for the node
• ClusterName - The name of the RabbitMQ cluster
• WhoAmI - Information about the currently authenticated management user
• ExchangeBindingsSource - Bindings for which a specific exchange is the source
• ExchangeBindingsDestination - Bindings for which a specific exchange is the destination
• QueueBindings - Bindings for a specific queue within a virtual host

NOTE: To use the JDBC Driver in Dremio, you will need a license (full or trial) and a Runtime Key (RTK). For more information on obtaining this license (or a trial), contact our sales team.

Add the Runtime Key (RTK) to the JDBC URL. You will end up with a JDBC URL similar to the following:

jdbc:api:RTK=5246...;Profile=C:\profiles\\RabbitMQ.apip;AuthScheme=Basic;URL=http://localhost:15672;User=guest;Password=guest;

Access RabbitMQ as an External Source

To add RabbitMQ as an External Source, click to add a new source and select API. Copy the JDBC URL and paste it into the New API Source wizard.

Save the connection and you are ready to query live RabbitMQ data in Dremio, easily incorporating RabbitMQ data into your data lake.

More Information & Free Trial

Using the CData API Driver for JDBC in Dremio, you can incorporate live RabbitMQ data into your data lake. Check out our CData API Driver for JDBC page for more information about connecting to RabbitMQ. Download a free, 30 day trial of the CData API Driver for JDBC and get started today.