Discover how a bimodal integration strategy can address the major data management challenges facing your organization today.
Get the Report →Automate Avro Integration Tasks from PowerShell
Are you in search of a quick and easy way to access Avro data from PowerShell? This article demonstrates how to utilize the Avro Cmdlets for tasks like connecting to Avro data, automating operations, downloading data, and more.
The CData Cmdlets for Avro are standard PowerShell cmdlets that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time access to Avro.
PowerShell Cmdlets or ADO.NET Provider?
The Cmdlets are not only a PowerShell interface to Avro, but also an SQL interface; this tutorial shows how to use both to retrieve Avro data. We also show examples of the ADO.NET equivalent, which is possible with the CData ADO.NET Provider for Avro. To access Avro data from other .NET applications, like LINQPad, use the CData ADO.NET Provider for Avro.
Once you have acquired the necessary connection properties, accessing Avro data in PowerShell can be enabled in three steps.
Connect to your local Avro file(s) by setting the URI connection property to the location of the Avro file.
PowerShell
-
Install the module:
Install-Module AvroCmdlets
-
Connect:
$avro = Connect-Avro -URI "$URI"
-
Search for and retrieve data:
$column2 = "value_2" $sampletable_1 = Select-Avro -Connection $avro -Table "SampleTable_1" -Where "Column2 = `'$Column2`'" $sampletable_1
You can also use the Invoke-Avro cmdlet to execute SQL commands:
$sampletable_1 = Invoke-Avro -Connection $avro -Query 'SELECT * FROM SampleTable_1 WHERE Column2 = @Column2' -Params @{'@Column2'='value_2'}
ADO.NET
-
Load the provider's assembly:
[Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for Avro\lib\System.Data.CData.Avro.dll")
-
Connect to Avro:
$conn= New-Object System.Data.CData.Avro.AvroConnection("URI=C:/folder/table.avroInitiateOAuth=GETANDREFRESH") $conn.Open()
-
Instantiate the AvroDataAdapter, execute an SQL query, and output the results:
$sql="SELECT Id, Column1 from SampleTable_1" $da= New-Object System.Data.CData.Avro.AvroDataAdapter($sql, $conn) $dt= New-Object System.Data.DataTable $da.Fill($dt) $dt.Rows | foreach { Write-Host $_.id $_.column1 }