Read the blog
 

Automate Kafka Integration Tasks from PowerShell

Ready to get started?

Download a free trial:

Download Now

Learn more:

Apache Kafka ADO.NET Provider

Rapidly create and deploy powerful .NET applications that integrate with Apache Kafka.

In this article

Related articles


Are you looking for a quick and easy way to access Kafka data from PowerShell? We show how to use the Cmdlets for Kafka and the CData ADO.NET Provider for Kafka to connect to Kafka data and synchronize, automate, download, and more.

The CData Cmdlets for Kafka are standard PowerShell cmdlets that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time and bidirectional access to Kafka.

Cmdlets or ADO.NET?

The cmdlets are not only a PowerShell interface to the Kafka API, but also an SQL interface; this tutorial shows how to use both to create, retrieve, update, and delete Kafka data. We also show examples of the ADO.NET equivalent, which is possible with the CData ADO.NET Provider for Kafka. To access Kafka data from other .NET applications, like LINQPad, use the CData ADO.NET Provider for Kafka.

After obtaining the needed connection properties, accessing Kafka data in PowerShell consists of three basic steps.

Set BootstrapServers and the Topic properties to specify the address of your Apache Kafka server, as well as the topic you would like to interact with.

Authorization Mechanisms

  • SASL Plain: The User and Password properties should be specified. AuthScheme should be set to 'Plain'.
  • SASL SSL: The User and Password properties should be specified. AuthScheme should be set to 'Scram'. UseSSL should be set to true.
  • SSL: The SSLCert and SSLCertPassword properties should be specified. UseSSL should be set to true.
  • Kerberos: The User and Password properties should be specified. AuthScheme should be set to 'Kerberos'.

You may be required to trust the server certificate. In such cases, specify the TrustStorePath and the TrustStorePassword if necessary.

PowerShell

  1. Install the module:

    Install-Module ApacheKafkaCmdlets

  2. Connect:

    $apachekafka = Connect-ApacheKafka -User "$User" -Password "$Password" -BootStrapServers "$BootStrapServers" -Topic "$Topic"

  3. Search for and retrieve data:

    $column2 = "100" $sampletable_1 = Select-ApacheKafka -Connection $apachekafka -Table "SampleTable_1" -Where "Column2 = `'$Column2`'" $sampletable_1

    You can also use the Invoke-ApacheKafka cmdlet to execute SQL commands:

    $sampletable_1 = Invoke-ApacheKafka -Connection $apachekafka -Query 'SELECT * FROM SampleTable_1 WHERE Column2 = @Column2' -Params @{'@Column2'='100'}

ADO.NET

  1. Load the provider's assembly:

    [Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for Kafka\lib\System.Data.CData.ApacheKafka.dll")

  2. Connect to Kafka:

    $conn= New-Object System.Data.CData.ApacheKafka.ApacheKafkaConnection("User=admin;Password=pass;BootStrapServers=https://localhost:9091;Topic=MyTopic;") $conn.Open()

  3. Instantiate the ApacheKafkaDataAdapter, execute an SQL query, and output the results:

    $sql="SELECT Id, Column1 from SampleTable_1" $da= New-Object System.Data.CData.ApacheKafka.ApacheKafkaDataAdapter($sql, $conn) $dt= New-Object System.Data.DataTable $da.Fill($dt) $dt.Rows | foreach { Write-Host $_.id $_.column1 }

Update Kafka Data

PowerShell

Update-ApacheKafka -Connection $ApacheKafka -Columns @('Id','Column1') -Values @('MyId', 'MyColumn1') -Table SampleTable_1 -Id "MyId"

ADO.NET

$cmd = New-Object System.Data.CData.ApacheKafka.ApacheKafkaCommand("UPDATE SampleTable_1 SET Column2='100' WHERE Id = @myId", $conn) $cmd.Parameters.Add(new System.Data.CData.ApacheKafka.ApacheKafkaParameter("@myId","10456255-0015501366")) $cmd.ExecuteNonQuery()

Insert Kafka Data

PowerShell

Add-ApacheKafka -Connection $ApacheKafka -Table SampleTable_1 -Columns @("Id", "Column1") -Values @("MyId", "MyColumn1")

ADO.NET

$cmd = New-Object System.Data.CData.ApacheKafka.ApacheKafkaCommand("INSERT INTO SampleTable_1 (Column2) VALUES (@myColumn2)", $conn) $cmd.Parameters.Add(new System.Data.CData.ApacheKafka.ApacheKafkaParameter("@myColumn2","100")) $cmd.ExecuteNonQuery()

Delete Kafka Data

PowerShell

Remove-ApacheKafka -Connection $ApacheKafka -Table "SampleTable_1" -Id "MyId"

ADO.NET

$cmd = New-Object System.Data.CData.ApacheKafka.ApacheKafkaCommand("DELETE FROM SampleTable_1 WHERE Id=@myId", $conn) $cmd.Parameters.Add(new System.Data.CData.ApacheKafka.ApacheKafkaParameter("@myId","001d000000YBRseAAH")) $cmd.ExecuteNonQuery()