Automate Apache Airflow Integration Tasks from PowerShell

The CData API Driver for ADO.NET is a standard ADO.NET Provider that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time access to Apache Airflow.

ADO.NET Provider

The ADO.NET Provider provides a SQL interface for Apache Airflow; this tutorial shows how to use the Provider to retrieve Apache Airflow data.

Once you have acquired the necessary connection properties, accessing Apache Airflow data in PowerShell can be enabled in three steps.

Start by setting the Profile connection property to the location of the ApacheAirflow Profile on disk (e.g. C:\profiles\ApacheAirflow.apip). Next, set the ProfileSettings connection property to the connection string for ApacheAirflow (see below).

ApacheAirflow API Profile Settings

Apache Airflow 3 uses JWT Bearer tokens for API authentication. You can generate a token from the Airflow web UI under Settings or via the Airflow CLI using airflow users create and the /api/v2/auth/token endpoint. Note that this profile targets Apache Airflow 3.x using the /api/v2 REST API. The legacy /api/v1 endpoint used by Airflow 2.x is not supported.

After setting the following connection properties, you are ready to connect:

• AuthScheme: Set this to APIKey.
• APIKey: Set this to your Apache Airflow JWT Bearer token.
• Server: Set this to the base URL of your Airflow instance (e.g. http://localhost:8080).

1. Load the provider's assembly:

   
   [Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData API Driver for ADO.NET\lib\System.Data.CData.API.dll")
       

2. Connect to Apache Airflow:

    
   $conn= New-Object System.Data.CData.API.APIConnection("Profile=C:\profiles\ApacheAirflow.apip;AuthScheme=APIKey;ProfileSettings='APIKey=your_jwt_token;Server=http://localhost:8080';")
   $conn.Open()
   

3. Instantiate the APIDataAdapter, execute an SQL query, and output the results:

   
   $sql="SELECT DagRunId, State from DagRuns"
   
   $da= New-Object System.Data.CData.API.APIDataAdapter($sql, $conn)
   $dt= New-Object System.Data.DataTable
   $da.Fill($dt)
   
   $dt.Rows | foreach {
   	Write-Host $_.dagrunid $_.state
   }