Back Up Azure Data Lake Storage data to SQL Server through SSIS

Ready to get started?

Download for a free trial:

Download Now

Learn more:

Azure Data Lake Storage ADO.NET Provider

Rapidly create and deploy powerful .NET applications that integrate with Azure Data Lake Storage.



Easily back up data to SQL Server using the CData ADO.NET Provider for Azure Data Lake Storage. This example uses an SSIS workflow to populate a database with Azure Data Lake Storage data.

The CData ADO.NET Provider for Azure Data Lake Storage connects Azure Data Lake Storage data to applications that provide backup, reporting, full-text search, analytics, and more.

This example demonstrates how to use the provider for Azure Data Lake Storage inside of a SQL Server SSIS workflow to directly transfer Azure Data Lake Storage data into a Microsoft SQL Server database. The exact same procedure outlined below can be used with any CData ADO.NET Data Providers to connect SQL Server directly with remote data via SSIS.

  1. Open Visual Studio and create a new Integration Services project.
  2. Add a new Data Flow task from the toolbox onto the Control Flow screen.
  3. In the Data Flow screen, add an ADO.NET Source and an OLE DB Destination from the toolbox.

  4. Add a new connection and select .NET Providers\CData ADO.NET Provider for Azure Data Lake Storage.
  5. In the connection manager, enter the connection details for Azure Data Lake Storage data.

    Authenticating to a Gen 1 DataLakeStore Account

    Gen 1 uses OAuth 2.0 in Azure AD for authentication.

    For this, an Active Directory web application is required. You can create one as follows:

    1. Sign in to your Azure Account through the .
    2. Select "Azure Active Directory".
    3. Select "App registrations".
    4. Select "New application registration".
    5. Provide a name and URL for the application. Select Web app for the type of application you want to create.
    6. Select "Required permissions" and change the required permissions for this app. At a minimum, "Azure Data Lake" and "Windows Azure Service Management API" are required.
    7. Select "Key" and generate a new key. Add a description, a duration, and take note of the generated key. You won't be able to see it again.

    To authenticate against a Gen 1 DataLakeStore account, the following properties are required:

    • Schema: Set this to ADLSGen1.
    • Account: Set this to the name of the account.
    • OAuthClientId: Set this to the application Id of the app you created.
    • OAuthClientSecret: Set this to the key generated for the app you created.
    • TenantId: Set this to the tenant Id. See the property for more information on how to acquire this.
    • Directory: Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.

    Authenticating to a Gen 2 DataLakeStore Account

    To authenticate against a Gen 2 DataLakeStore account, the following properties are required:

    • Schema: Set this to ADLSGen2.
    • Account: Set this to the name of the account.
    • FileSystem: Set this to the file system which will be used for this account.
    • AccessKey: Set this to the access key which will be used to authenticate the calls to the API. See the property for more information on how to acquire this.
    • Directory: Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.

  6. Open the DataReader editor and set the following information:

    • ADO.NET connection manager: In the Connection Managers menu, select the Data Connection you just created.
    • Data access mode: Select 'SQL command'.
    • SQL command text: In the DataReader Source editor, open the Component Properties tab and enter a SELECT command, such as the one below:

      SELECT FullPath, Permission FROM Resources WHERE Type = 'FILE'

  7. Close the DataReader editor and drag the arrow below the DataReader Source to connect it to the OLE DB Destination.
  8. Open the OLE DB Destination and enter the following information in the Destination Component Editor.

    • Connection manager: Add a new connection. Enter your server and database information here. In this example, SQLExpress is running on a separate machine.
    • Data access mode: Set your data access mode to "table or view" and select the table or view to populate in your database.
  9. Configure any properties you wish on the Mappings screen.

  10. Close the OLE DB Destination Editor and run the project. After the SSIS task has finished executing, your database will be populated with data obtained from Azure Data Lake Storage data.