LINQ to Apache Airflow Data

This article illustrates using LINQ to access tables within the Apache Airflow via the CData ADO.NET Data Provider for Apache Airflow. To achieve this, we will use LINQ to Entity Framework, which facilitates the generation of connections and can be seamlessly employed with any CData ADO.NET Data Providers to access data through LINQ.

See the help documentation for a guide to setting up an EF 6 project to use the provider.

1. In a new project in Visual Studio, right-click on the project and choose to add a new item. Add an ADO.NET Entity Data Model.
2. Choose EF Designer from Database and click Next.
3. Add a new Data Connection, and change your data source type to "CData Apache Airflow Data Source".

4. Enter your data source connection information.

   Start by setting the Profile connection property to the location of the ApacheAirflow Profile on disk (e.g. C:\profiles\ApacheAirflow.apip). Next, set the ProfileSettings connection property to the connection string for ApacheAirflow (see below).

   ApacheAirflow API Profile Settings

   Apache Airflow 3 uses JWT Bearer tokens for API authentication. You can generate a token from the Airflow web UI under Settings or via the Airflow CLI using airflow users create and the /api/v2/auth/token endpoint. Note that this profile targets Apache Airflow 3.x using the /api/v2 REST API. The legacy /api/v1 endpoint used by Airflow 2.x is not supported.

   After setting the following connection properties, you are ready to connect:

   • AuthScheme: Set this to APIKey.
   • APIKey: Set this to your Apache Airflow JWT Bearer token.
   • Server: Set this to the base URL of your Airflow instance (e.g. http://localhost:8080).

   Below is a typical connection string:

   Profile=C:\profiles\ApacheAirflow.apip;AuthScheme=APIKey;ProfileSettings='APIKey=your_jwt_token;Server=http://localhost:8080';


5. If saving your entity connection to App.Config, set an entity name. In this example we are setting APIEntities as our entity connection in App.Config.
6. Enter a model name and select any tables or views you would like to include in the model.

Using the entity you created, you can now perform select commands. For example:

APIEntities context = new APIEntities();

var dagrunsQuery = from dagruns in context.DagRuns
  select dagruns;

foreach (var result in dagrunsQuery) {
  Console.WriteLine("{0} {1} ", result.Id, result.DagRunId);
}

See "LINQ and Entity Framework" chapter in the help documentation for example queries of the supported LINQ.