Discover how a bimodal integration strategy can address the major data management challenges facing your organization today.
Get the Report →Automate BigQuery Integration Tasks from PowerShell
Are you in search of a quick and easy way to access BigQuery data from PowerShell? This article demonstrates how to utilize the BigQuery Cmdlets for tasks like connecting to BigQuery data, automating operations, downloading data, and more.
The CData Cmdlets for BigQuery are standard PowerShell cmdlets that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time and bidirectional access to BigQuery.
PowerShell Cmdlets or ADO.NET Provider?
The Cmdlets are not only a PowerShell interface to BigQuery, but also an SQL interface; this tutorial shows how to use both to create, retrieve, update, and delete BigQuery data. We also show examples of the ADO.NET equivalent, which is possible with the CData ADO.NET Provider for Google BigQuery. To access BigQuery data from other .NET applications, like LINQPad, use the CData ADO.NET Provider for Google BigQuery.
Once you have acquired the necessary connection properties, accessing BigQuery data in PowerShell can be enabled in three steps.
Google uses the OAuth authentication standard. To access Google APIs on behalf of individual users, you can use the embedded credentials or you can register your own OAuth app.
OAuth also enables you to use a service account to connect on behalf of users in a Google Apps domain. To authenticate with a service account, you will need to register an application to obtain the OAuth JWT values.
In addition to the OAuth values, you will need to specify the DatasetId and ProjectId. See the "Getting Started" chapter of the help documentation for a guide to using OAuth.
PowerShell
-
Install the module:
Install-Module GoogleBigQueryCmdlets
-
Connect:
$googlebigquery = Connect-GoogleBigQuery -DataSetId "$DataSetId" -ProjectId "$ProjectId"
-
Search for and retrieve data:
$shipcity = "New York" $orders = Select-GoogleBigQuery -Connection $googlebigquery -Table "Orders" -Where "ShipCity = `'$ShipCity`'" $orders
You can also use the Invoke-GoogleBigQuery cmdlet to execute SQL commands:
$orders = Invoke-GoogleBigQuery -Connection $googlebigquery -Query 'SELECT * FROM Orders WHERE ShipCity = @ShipCity' -Params @{'@ShipCity'='New York'}
ADO.NET
-
Load the provider's assembly:
[Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for Google BigQuery\lib\System.Data.CData.GoogleBigQuery.dll")
-
Connect to BigQuery:
$conn= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryConnection("DataSetId=MyDataSetId;ProjectId=MyProjectId;InitiateOAuth=GETANDREFRESH") $conn.Open()
-
Instantiate the GoogleBigQueryDataAdapter, execute an SQL query, and output the results:
$sql="SELECT OrderName, Freight from Orders" $da= New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryDataAdapter($sql, $conn) $dt= New-Object System.Data.DataTable $da.Fill($dt) $dt.Rows | foreach { Write-Host $_.ordername $_.freight }
Update BigQuery Data
PowerShell
Update-GoogleBigQuery -Connection $GoogleBigQuery -Columns @('OrderName','Freight') -Values @('MyOrderName', 'MyFreight') -Table Orders -Id "MyId"
ADO.NET
$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("UPDATE Orders SET ShipCity='New York' WHERE Id = @myId", $conn)
$cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myId","10456255-0015501366")))
$cmd.ExecuteNonQuery()
Insert BigQuery Data
PowerShell
Add-GoogleBigQuery -Connection $GoogleBigQuery -Table Orders -Columns @("OrderName", "Freight") -Values @("MyOrderName", "MyFreight")
ADO.NET
$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("INSERT INTO Orders (ShipCity) VALUES (@myShipCity)", $conn)
$cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myShipCity","New York")))
$cmd.ExecuteNonQuery()
Delete BigQuery Data
PowerShell
Remove-GoogleBigQuery -Connection $GoogleBigQuery -Table "Orders" -Id "MyId"
ADO.NET
$cmd = New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryCommand("DELETE FROM Orders WHERE Id=@myId", $conn)
$cmd.Parameters.Add((New-Object System.Data.CData.GoogleBigQuery.GoogleBigQueryParameter("@myId","001d000000YBRseAAH")))
$cmd.ExecuteNonQuery()
CodeProject