Access Kafka Data with Entity Framework 6

Ready to get started?

Download for a free trial:

Download Now

Learn more:

Apache Kafka ADO.NET Provider

Rapidly create and deploy powerful .NET applications that integrate with Apache Kafka.



This article shows how to access Kafka data using an Entity Framework code-first approach. Entity Framework 6 is available in .NET 4.5 and above.

Entity Framework is an object-relational mapping framework that can be used to work with data as objects. While you can run the ADO.NET Entity Data Model wizard in Visual Studio to handle generating the Entity Model, this approach, the model-first approach, can put you at a disadvantage if there are changes in your data source or if you want more control over how the entities operate. In this article you will complete the code-first approach to accessing Kafka data using the CData ADO.NET Provider.

  1. Open Visual Studio and create a new Windows Form Application. This article uses a C# project with .NET 4.5.
  2. Run the command 'Install-Package EntityFramework' in the Package Manger Console in Visual Studio to install the latest release of Entity Framework.
  3. Modify the App.config file in the project to add a reference to the Kafka Entity Framework 6 assembly and the connection string.

    Set BootstrapServers and the Topic properties to specify the address of your Apache Kafka server, as well as the topic you would like to interact with.

    Authorization Mechanisms

    • SASL Plain: The User and Password properties should be specified. AuthScheme should be set to 'Plain'.
    • SASL SSL: The User and Password properties should be specified. AuthScheme should be set to 'Scram'. UseSSL should be set to true.
    • SSL: The SSLCert and SSLCertPassword properties should be specified. UseSSL should be set to true.
    • Kerberos: The User and Password properties should be specified. AuthScheme should be set to 'Kerberos'.

    You may be required to trust the server certificate. In such cases, specify the TrustStorePath and the TrustStorePassword if necessary.

    <configuration> ... <connectionStrings> <add name="ApacheKafkaContext" connectionString="Offline=False;User=admin;Password=pass;BootStrapServers=https://localhost:9091;Topic=MyTopic;" providerName="System.Data.CData.ApacheKafka" /> </connectionStrings> <entityFramework> <providers> ... <provider invariantName="System.Data.CData.ApacheKafka" type="System.Data.CData.ApacheKafka.ApacheKafkaProviderServices, System.Data.CData.ApacheKafka.Entities.EF6" /> </providers> <entityFramework> </configuration> </code>
  4. Add a reference to System.Data.CData.ApacheKafka.Entities.EF6.dll, located in the lib -> 4.0 subfolder in the installation directory.
  5. Build the project at this point to ensure everything is working correctly. Once that's done, you can start coding using Entity Framework.
  6. Add a new .cs file to the project and add a class to it. This will be your database context, and it will extend the DbContext class. In the example, this class is named ApacheKafkaContext. The following code example overrides the OnModelCreating method to make the following changes:
    • Remove PluralizingTableNameConvention from the ModelBuilder Conventions.
    • Remove requests to the MigrationHistory table.
    using System.Data.Entity; using System.Data.Entity.Infrastructure; using System.Data.Entity.ModelConfiguration.Conventions; class ApacheKafkaContext : DbContext { public ApacheKafkaContext() { } protected override void OnModelCreating(DbModelBuilder modelBuilder) { // To remove the requests to the Migration History table Database.SetInitializer<ApacheKafkaContext>(null); // To remove the plural names modelBuilder.Conventions.Remove<PluralizingTableNameConvention>(); } }
  7. Create another .cs file and name it after the Kafka entity you are retrieving, for example, SampleTable_1. In this file, define both the Entity and the Entity Configuration, which will resemble the example below: using System.Data.Entity.ModelConfiguration; using System.ComponentModel.DataAnnotations.Schema; [System.ComponentModel.DataAnnotations.Schema.Table("SampleTable_1")] public class SampleTable_1 { [System.ComponentModel.DataAnnotations.Key] public System.String Id { get; set; } public System.String Column1 { get; set; } }
  8. Now that you have created an entity, add the entity to your context class: public DbSet<SampleTable_1> SampleTable_1 { set; get; }
  9. With the context and entity finished, you are now ready to query the data in a separate class. For example: ApacheKafkaContext context = new ApacheKafkaContext(); context.Configuration.UseDatabaseNullSemantics = true; var query = from line in context.SampleTable_1 select line;