Automate BigQuery Integration Tasks from PowerShell



Are you in search of a quick and easy way to access BigQuery data from PowerShell? This article demonstrates how to utilize the BigQuery Cmdlets for tasks like connecting to BigQuery data, automating operations, downloading data, and more.

The CData Cmdlets for BigQuery are standard PowerShell cmdlets that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time and bidirectional access to BigQuery.

PowerShell Cmdlets or ADO.NET Provider?

The Cmdlets are not only a PowerShell interface to BigQuery, but also an SQL interface; this tutorial shows how to use both to create, retrieve, update, and delete BigQuery data. We also show examples of the ADO.NET equivalent, which is possible with the CData ADO.NET Provider for Google BigQuery. To access BigQuery data from other .NET applications, like LINQPad, use the CData ADO.NET Provider for Google BigQuery.

Once you have acquired the necessary connection properties, accessing BigQuery data in PowerShell can be enabled in three steps.

Google uses the OAuth authentication standard. To access Google APIs on behalf of individual users, you can use the embedded credentials or you can register your own OAuth app.

OAuth also enables you to use a service account to connect on behalf of users in a Google Apps domain. To authenticate with a service account, you will need to register an application to obtain the OAuth JWT values.

In addition to the OAuth values, you will need to specify the DatasetId and ProjectId. See the "Getting Started" chapter of the help documentation for a guide to using OAuth.

PowerShell

  1. Install the module:

    Install-Module GoogleBigQueryCmdlets
  2. Connect:

    $googlebigquery = Connect-GoogleBigQuery -DataSetId "$DataSetId" -ProjectId "$ProjectId"
  3. Search for and retrieve data:

    $shipcity = "New York" $orders = Select-GoogleBigQuery -Connection $googlebigquery -Table "Orders" -Where "ShipCity = `'$ShipCity`'" $orders

    You can also use the Invoke-GoogleBigQuery cmdlet to execute SQL commands:

    $orders = Invoke-GoogleBigQuery -Connection $googlebigquery -Query 'SELECT * FROM Orders WHERE ShipCity = @ShipCity' -Params @{'@ShipCity'='New York'}

ADO.NET

  1. Load the provider's assembly:

    [Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for Google BigQuery\lib\System.Data.CData.GoogleBigQuery.dll")
  2. Connect to BigQuery:

    $conn= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryConnection("DataSetId=MyDataSetId;ProjectId=MyProjectId;InitiateOAuth=GETANDREFRESH") $conn.Open()
  3. Instantiate the GoogleBigQueryDataAdapter, execute an SQL query, and output the results:

    $sql="SELECT OrderName, Freight from Orders" $da= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryDataAdapter($sql, $conn) $dt= New-Object System.Data.DataTable $da.Fill($dt) $dt.Rows | foreach { Write-Host $_.ordername $_.freight }

Update BigQuery Data

PowerShell

Update-GoogleBigQuery -Connection $GoogleBigQuery -Columns @('OrderName','Freight') -Values @('MyOrderName', 'MyFreight') -Table Orders -Id "MyId"

ADO.NET

$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("UPDATE Orders SET ShipCity='New York' WHERE Id = @myId", $conn) $cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myId","10456255-0015501366"))) $cmd.ExecuteNonQuery()

Insert BigQuery Data

PowerShell

Add-GoogleBigQuery -Connection $GoogleBigQuery -Table Orders -Columns @("OrderName", "Freight") -Values @("MyOrderName", "MyFreight")

ADO.NET

$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("INSERT INTO Orders (ShipCity) VALUES (@myShipCity)", $conn) $cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myShipCity","New York"))) $cmd.ExecuteNonQuery()

Delete BigQuery Data

PowerShell

Remove-GoogleBigQuery -Connection $GoogleBigQuery -Table "Orders" -Id "MyId"

ADO.NET

$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("DELETE FROM Orders WHERE Id=@myId", $conn) $cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myId","001d000000YBRseAAH"))) $cmd.ExecuteNonQuery()

Ready to get started?

Download a free trial of the Google BigQuery Data Provider to get started:

 Download Now

Learn more:

Google BigQuery Icon Google BigQuery ADO.NET Provider

Rapidly create and deploy powerful .NET applications that integrate with Google BigQuery data including Tables and Datasets.