Connect to Apache Airflow Data in CloverDX (formerly CloverETL)

The CData JDBC Driver for Apache Airflow enables you to use the data transformation components in CloverDX (formerly CloverETL) to work with Apache Airflow as sources. In this article, you will use the JDBC Driver for Apache Airflow to set up a simple transfer into a flat file. The CData JDBC Driver for Apache Airflow enables you to use the data transformation components in CloverDX (formerly CloverETL) to work with Apache Airflow as sources. In this article, you will use the JDBC Driver for Apache Airflow to set up a simple transfer into a flat file.

Connect to Apache Airflow as a JDBC Data Source

1. Create the connection to Apache Airflow data. In a new CloverDX graph, right-click the Connections node in the Outline pane and click Connections -> Create Connection. The Database Connection wizard is displayed.
2. Click the plus icon to load a driver from a JAR. Browse to the lib subfolder of the installation directory and select the cdata.jdbc.api.jar file.
3. Enter the JDBC URL.

   Start by setting the Profile connection property to the location of the ApacheAirflow Profile on disk (e.g. C:\profiles\ApacheAirflow.apip). Next, set the ProfileSettings connection property to the connection string for ApacheAirflow (see below).

   ApacheAirflow API Profile Settings

   Apache Airflow 3 uses JWT Bearer tokens for API authentication. You can generate a token from the Airflow web UI under Settings or via the Airflow CLI using airflow users create and the /api/v2/auth/token endpoint. Note that this profile targets Apache Airflow 3.x using the /api/v2 REST API. The legacy /api/v1 endpoint used by Airflow 2.x is not supported.

   After setting the following connection properties, you are ready to connect:

   • AuthScheme: Set this to APIKey.
   • APIKey: Set this to your Apache Airflow JWT Bearer token.
   • Server: Set this to the base URL of your Airflow instance (e.g. http://localhost:8080).

   Built-in Connection String Designer

   For assistance in constructing the JDBC URL, use the connection string designer built into the Apache Airflow JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

   
   java -jar cdata.jdbc.api.jar
   

   Fill in the connection properties and copy the connection string to the clipboard.

   

   A typical JDBC URL is below:

   jdbc:api:Profile=C:\profiles\ApacheAirflow.apip;AuthScheme=APIKey;ProfileSettings='APIKey=your_jwt_token;Server=http://localhost:8080';

Query Apache Airflow Data with the DBInputTable Component

1. Drag a DBInputTable from the Readers selection of the Palette onto the job flow and double-click it to open the configuration editor.
2. In the DB connection property, select the Apache Airflow JDBC data source from the drop-down menu.
3. Enter the SQL query. For example:

   SELECT DagRunId, State FROM DagRuns WHERE DagId = 'example_dag'

Write the Output of the Query to a UniversalDataWriter

1. Drag a UniversalDataWriter from the Writers selection onto the jobflow.
2. Double-click the UniversalDataWriter to open the configuration editor and add a file URL.
3. Right-click the DBInputTable and then click Extract Metadata.
4. Connect the output port of the DBInputTable to the UniversalDataWriter.
5. In the resulting Select Metadata menu for the UniversalDataWriter, choose the DagRuns table. (You can also open this menu by right-clicking the input port for the UniversalDataWriter.)
6. Click Run to write to the file.