Automate BigQuery Integration Tasks from PowerShell

The CData Cmdlets for BigQuery are standard PowerShell cmdlets that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time and bidirectional access to BigQuery.

PowerShell Cmdlets or ADO.NET Provider?

The Cmdlets are not only a PowerShell interface to BigQuery, but also an SQL interface; this tutorial shows how to use both to create, retrieve, update, and delete BigQuery data. We also show examples of the ADO.NET equivalent, which is possible with the CData ADO.NET Provider for Google BigQuery. To access BigQuery data from other .NET applications, like LINQPad, use the CData ADO.NET Provider for Google BigQuery.

Once you have acquired the necessary connection properties, accessing BigQuery data in PowerShell can be enabled in three steps.

Google uses the OAuth authentication standard. To access Google APIs on behalf of individual users, you can use the embedded credentials or you can register your own OAuth app.

OAuth also enables you to use a service account to connect on behalf of users in a Google Apps domain. To authenticate with a service account, you will need to register an application to obtain the OAuth JWT values.

In addition to the OAuth values, you will need to specify the DatasetId and ProjectId. See the "Getting Started" chapter of the help documentation for a guide to using OAuth.

PowerShell

1. Install the module:

   Install-Module GoogleBigQueryCmdlets

2. Connect:

   $googlebigquery = Connect-GoogleBigQuery -DataSetId "$DataSetId" -ProjectId "$ProjectId"

3. Search for and retrieve data:

   $shipcity = "New York" $orders = Select-GoogleBigQuery -Connection $googlebigquery -Table "Orders" -Where "ShipCity = `'$ShipCity`'" $orders

   You can also use the Invoke-GoogleBigQuery cmdlet to execute SQL commands:

   $orders = Invoke-GoogleBigQuery -Connection $googlebigquery -Query 'SELECT * FROM Orders WHERE ShipCity = @ShipCity' -Params @{'@ShipCity'='New York'}

ADO.NET

1. Load the provider's assembly:

   [Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for Google BigQuery\lib\System.Data.CData.GoogleBigQuery.dll")

2. Connect to BigQuery:

   $conn= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryConnection("DataSetId=MyDataSetId;ProjectId=MyProjectId;InitiateOAuth=GETANDREFRESH") $conn.Open()

3. Instantiate the GoogleBigQueryDataAdapter, execute an SQL query, and output the results:

   $sql="SELECT OrderName, Freight from Orders" $da= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryDataAdapter($sql, $conn) $dt= New-Object System.Data.DataTable $da.Fill($dt) $dt.Rows | foreach { Write-Host $_.ordername $_.freight }

Update BigQuery Data

PowerShell

ADO.NET

Insert BigQuery Data

PowerShell

ADO.NET

Delete BigQuery Data

PowerShell

ADO.NET