Build a PostgreSQL Interface for Apache Airflow Data using the CData JDBC Driver

There are a vast number of PostgreSQL clients available on the Internet. From standard Drivers to BI and Analytics tools, PostgreSQL is a popular interface for data access. Using our JDBC Drivers, you can now create PostgreSQL entry-points that you can connect to from any standard client.

To access Apache Airflow data as a PostgreSQL database, use the CData JDBC Driver for Apache Airflow and a JDBC foreign data wrapper (FDW). In this article, we compile the FDW, install it, and query Apache Airflow data from PostgreSQL Server.

Connect to Apache Airflow Data as a JDBC Data Source

To connect to Apache Airflow as a JDBC data source, you will need the following:

• Driver JAR path: The JAR is located in the lib subfolder of the installation directory.

• Driver class:

  
  cdata.jdbc.api.APIDriver
  

• JDBC URL: The URL must start with "jdbc:api:" and can include any of the connection properties in name-value pairs separated with semicolons.

  Start by setting the Profile connection property to the location of the ApacheAirflow Profile on disk (e.g. C:\profiles\ApacheAirflow.apip). Next, set the ProfileSettings connection property to the connection string for ApacheAirflow (see below).

  ApacheAirflow API Profile Settings

  Apache Airflow 3 uses JWT Bearer tokens for API authentication. You can generate a token from the Airflow web UI under Settings or via the Airflow CLI using airflow users create and the /api/v2/auth/token endpoint. Note that this profile targets Apache Airflow 3.x using the /api/v2 REST API. The legacy /api/v1 endpoint used by Airflow 2.x is not supported.

  After setting the following connection properties, you are ready to connect:

  • AuthScheme: Set this to APIKey.
  • APIKey: Set this to your Apache Airflow JWT Bearer token.
  • Server: Set this to the base URL of your Airflow instance (e.g. http://localhost:8080).

  Built-in Connection String Designer

  For assistance in constructing the JDBC URL, use the connection string designer built into the Apache Airflow JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

  
  java -jar cdata.jdbc.api.jar
  

  Fill in the connection properties and copy the connection string to the clipboard.

  

  A typical JDBC URL is below:

  
  jdbc:api:Profile=C:\profiles\ApacheAirflow.apip;AuthScheme=APIKey;ProfileSettings='APIKey=your_jwt_token;Server=http://localhost:8080';
  

Build the JDBC Foreign Data Wrapper

The Foreign Data Wrapper can be installed as an extension to PostgreSQL, without recompiling PostgreSQL. The jdbc2_fdw extension is used as an example (downloadable here).

1. Add a symlink from the shared object for your version of the JRE to /usr/lib/libjvm.so. For example:

   
   ln -s /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server/libjvm.so /usr/lib/libjvm.so
   

2. Start the build:

   
   make install USE_PGXS=1
   

Query Apache Airflow Data as a PostgreSQL Database

After you have installed the extension, follow the steps below to start executing queries to Apache Airflow data:

1. Log into your database.
2. Load the extension for the database:

   
   CREATE EXTENSION jdbc2_fdw;
   

3. Create a server object for Apache Airflow:

   
   CREATE SERVER API 
   FOREIGN DATA WRAPPER jdbc2_fdw OPTIONS (
   drivername 'cdata.jdbc.api.APIDriver',
   url 'jdbc:api:Profile=C:\profiles\ApacheAirflow.apip;AuthScheme=APIKey;ProfileSettings='APIKey=your_jwt_token;Server=http://localhost:8080';',
   querytimeout '15',
   jarfile '/home/MyUser/CData/CData\ JDBC\ Driver\ for\ Salesforce MyDriverEdition/lib/cdata.jdbc.api.jar'); 
   

4. Create a user mapping for the username and password of a user known to the MySQL daemon.

   
   CREATE USER MAPPING for postgres SERVER API OPTIONS (
   username 'admin', 
   password 'test');
   

5. Create a foreign table in your local database:

   
   postgres=# CREATE FOREIGN TABLE dagruns (
   dagruns_id text,
   dagruns_DagRunId text,
   dagruns_State numeric) 
   SERVER API OPTIONS (
   table_name 'dagruns');
   

postgres=# SELECT * FROM dagruns;