Powershell からの Azure Data Lake Storage データ連携タスクを自動化

CData ADO.NET Provider for ADLS は、ADO.NET 標準インターフェースへ統合し、PowerShell のような. NET アプリケーションからAzure Data Lake Storage API へのデータ連携を可能にします。このプロバイダーは、Azure Data Lake Storage の認証および相互作用を簡単にします。このチュートリアルでは、PowerShell から直接SQL クエリを実行するための、いくつかの一般的なADO.NET オブジェクトの使い方を説明します。

CRUD コマンドの実行

次の3つのステップに従って SELECT クエリ PowerShell からリアルタイムAzure Data Lake Storage に実行：

1. プロバイダーのアセンブリをロード：

   [Reflection.Assembly]::LoadFile("C:\Program Files\CData\CData ADO.NET Provider for ADLS\lib\System.Data.CData.ADLS.dll")

2. Azure Data Lake Storage に接続：

   Authenticating to a Gen 1 DataLakeStore Account

   Gen 1 uses OAuth 2.0 in Azure AD for authentication.

   For this, an Active Directory web application is required. You can create one as follows:

   1. Sign in to your Azure Account through the .
   2. Select "Azure Active Directory".
   3. Select "App registrations".
   4. Select "New application registration".
   5. Provide a name and URL for the application. Select Web app for the type of application you want to create.
   6. Select "Required permissions" and change the required permissions for this app. At a minimum, "Azure Data Lake" and "Windows Azure Service Management API" are required.
   7. Select "Key" and generate a new key. Add a description, a duration, and take note of the generated key. You won't be able to see it again.

   To authenticate against a Gen 1 DataLakeStore account, the following properties are required:

   • Schema: Set this to ADLSGen1.
   • Account: Set this to the name of the account.
   • OAuthClientId: Set this to the application Id of the app you created.
   • OAuthClientSecret: Set this to the key generated for the app you created.
   • TenantId: Set this to the tenant Id. See the property for more information on how to acquire this.
   • Directory: Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.

   Authenticating to a Gen 2 DataLakeStore Account

   To authenticate against a Gen 2 DataLakeStore account, the following properties are required:

   • Schema: Set this to ADLSGen2.
   • Account: Set this to the name of the account.
   • FileSystem: Set this to the file system which will be used for this account.
   • AccessKey: Set this to the access key which will be used to authenticate the calls to the API. See the property for more information on how to acquire this.
   • Directory: Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.
   $constr = "Schema=ADLSGen2;Account=myAccount;FileSystem=myFileSystem;AccessKey=myAccessKey;" $conn= New-Object System.Data.CData.ADLS.ADLSConnection($constr) $conn.Open()

3. ADLSDataAdapter のインスタンスを生成してSQL クエリを実行し、結果を出力：

   $sql="SELECT FullPath, Permission from Resources" $da= New-Object System.Data.CData.ADLS.ADLSDataAdapter($sql, $conn) $dt= New-Object System.Data.DataTable $da.Fill($dt) $dt.Rows | foreach { Write-Host $_.fullpath $_.permission }