How to Access Apache Airflow Data Using Entity Framework

Microsoft Entity Framework serves as an object-relational mapping framework for working with data represented as objects. Although Visual Studio offers the ADO.NET Entity Data Model wizard to automatically generate the Entity Model, this model-first approach may present challenges when your data source undergoes changes or when you require greater control over entity operations. In this article, we will delve into the code-first approach for accessing Apache Airflow data through the CData ADO.NET Provider, providing you with more flexibility and control.

1. Open Visual Studio and create a new Windows Form Application. This article uses a C# project with .NET 4.5.
2. Run the command 'Install-Package EntityFramework' in the Package Manger Console in Visual Studio to install the latest release of Entity Framework.

3. Modify the App.config file in the project to add a reference to the Apache Airflow Entity Framework 6 assembly and the connection string.

   Start by setting the Profile connection property to the location of the ApacheAirflow Profile on disk (e.g. C:\profiles\ApacheAirflow.apip). Next, set the ProfileSettings connection property to the connection string for ApacheAirflow (see below).

   ApacheAirflow API Profile Settings

   Apache Airflow 3 uses JWT Bearer tokens for API authentication. You can generate a token from the Airflow web UI under Settings or via the Airflow CLI using airflow users create and the /api/v2/auth/token endpoint. Note that this profile targets Apache Airflow 3.x using the /api/v2 REST API. The legacy /api/v1 endpoint used by Airflow 2.x is not supported.

   After setting the following connection properties, you are ready to connect:

   • AuthScheme: Set this to APIKey.
   • APIKey: Set this to your Apache Airflow JWT Bearer token.
   • Server: Set this to the base URL of your Airflow instance (e.g. http://localhost:8080).

   
   
   <configuration>
      ... 
     <connectionStrings>
       <add name="APIContext" connectionString="Offline=False;Profile=C:\profiles\ApacheAirflow.apip;AuthScheme=APIKey;ProfileSettings='APIKey=your_jwt_token;Server=http://localhost:8080';" providerName="System.Data.CData.API" />
     </connectionStrings>
     <entityFramework>
       <providers>
          ... 
         <provider invariantName="System.Data.CData.API" type="System.Data.CData.API.APIProviderServices, System.Data.CData.API.Entities.EF6" />
       </providers>
     <entityFramework>
   </configuration>
   </code> 
   

4. Add a reference to System.Data.CData.API.Entities.EF6.dll, located in the lib -> 4.0 subfolder in the installation directory.
5. Build the project at this point to ensure everything is working correctly. Once that's done, you can start coding using Entity Framework.
6. Add a new .cs file to the project and add a class to it. This will be your database context, and it will extend the DbContext class. In the example, this class is named APIContext. The following code example overrides the OnModelCreating method to make the following changes:
   • Remove PluralizingTableNameConvention from the ModelBuilder Conventions.
   • Remove requests to the MigrationHistory table.

   
   
   using System.Data.Entity;
   using System.Data.Entity.Infrastructure;
   using System.Data.Entity.ModelConfiguration.Conventions;
   
   class APIContext : DbContext {
     public APIContext() { }
   
     protected override void OnModelCreating(DbModelBuilder modelBuilder)
     {
       // To remove the requests to the Migration History table
       Database.SetInitializer<APIContext>(null);  
       // To remove the plural names    
       modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
     }  
   }
   

7. Create another .cs file and name it after the Apache Airflow entity you are retrieving, for example, DagRuns. In this file, define both the Entity and the Entity Configuration, which will resemble the example below:

   
   
   using System.Data.Entity.ModelConfiguration;
   using System.ComponentModel.DataAnnotations.Schema;
   
   [System.ComponentModel.DataAnnotations.Schema.Table("DagRuns")]
   public class DagRuns {
     [System.ComponentModel.DataAnnotations.Key] 
     public System.String DagRunId { get; set; }
     public System.String State { get; set; }
   }
       
   

8. Now that you have created an entity, add the entity to your context class:

   
   
   public DbSet<DagRuns> DagRuns { set; get; }
   

9. With the context and entity finished, you are now ready to query the data in a separate class. For example:

   
   APIContext context = new APIContext();
   context.Configuration.UseDatabaseNullSemantics = true;
   var query = from line in context.DagRuns select line;