Automate HDFS Integration Tasks from PowerShell



Are you in search of a quick and easy way to access HDFS data from PowerShell? This article demonstrates how to utilize the HDFS Cmdlets for tasks like connecting to HDFS data, automating operations, downloading data, and more.

The CData Cmdlets for HDFS are standard PowerShell cmdlets that make it easy to accomplish data cleansing, normalization, backup, and other integration tasks by enabling real-time access to HDFS.

PowerShell Cmdlets or ADO.NET Provider?

The Cmdlets are not only a PowerShell interface to HDFS, but also an SQL interface; this tutorial shows how to use both to retrieve HDFS data. We also show examples of the ADO.NET equivalent, which is possible with the CData ADO.NET Provider for HDFS. To access HDFS data from other .NET applications, like LINQPad, use the CData ADO.NET Provider for HDFS.

Once you have acquired the necessary connection properties, accessing HDFS data in PowerShell can be enabled in three steps.

In order to authenticate, set the following connection properties:

  • Host: Set this value to the host of your HDFS installation.
  • Port: Set this value to the port of your HDFS installation. Default port: 50070

PowerShell

  1. Install the module:

    Install-Module HDFSCmdlets
  2. Connect:

    $hdfs = Connect-HDFS -Host "$Host" -Port "$Port" -Path "$Path" -User "$User"
  3. Search for and retrieve data:

    $fileid = "119116" $files = Select-HDFS -Connection $hdfs -Table "Files" -Where "FileId = `'$FileId`'" $files

    You can also use the Invoke-HDFS cmdlet to execute SQL commands:

    $files = Invoke-HDFS -Connection $hdfs -Query 'SELECT * FROM Files WHERE FileId = @FileId' -Params @{'@FileId'='119116'}

ADO.NET

  1. Load the provider's assembly:

    [Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for HDFS\lib\System.Data.CData.HDFS.dll")
  2. Connect to HDFS:

    $conn= New-Object System.Data.CData.HDFS.HDFSConnection("Host=sandbox-hdp.hortonworks.com;Port=50070;Path=/user/root;User=root;") $conn.Open()
  3. Instantiate the HDFSDataAdapter, execute an SQL query, and output the results:

    $sql="SELECT FileId, ChildrenNum from Files" $da= New-Object System.Data.CData.HDFS.HDFSDataAdapter($sql, $conn) $dt= New-Object System.Data.DataTable $da.Fill($dt) $dt.Rows | foreach { Write-Host $_.fileid $_.childrennum }

Ready to get started?

Download a free trial of the HDFS Data Provider to get started:

 Download Now

Learn more:

HDFS Icon HDFS ADO.NET Provider

Rapidly create and deploy powerful .NET applications that integrate with HDFS.