Integrate Dify with Live Azure Data Lake Storage Data via CData Connect AI

Dify is an open source platform for building production-ready agentic workflows, chatbots, and other LLM applications. It includes built-in, two-way support for the model context protocol (MCP), allowing you to register remote MCP servers as tools in the platform to add external data sources and give your agents access to live data.

By integrating Dify with CData Connect AI through the built-in MCP Server, Dify gains governed, real-time access to live Azure Data Lake Storage data. You can list catalogs, explore schemas, and query records from Azure Data Lake Storage data using natural language prompts, with all data access running securely against authorized sources.

This article explains how to configure Azure Data Lake Storage connectivity in Connect AI, generate the required personal access token, register the Connect AI MCP Server as a tool in Dify, add it to an agent application, and verify the integration by querying live Azure Data Lake Storage data from Dify.

Step 1: Configure Azure Data Lake Storage connectivity for Dify

Connectivity to Azure Data Lake Storage from Dify is made possible through Connect AI's Remote MCP Server. To interact with Azure Data Lake Storage data from Dify, start by creating and configuring a Azure Data Lake Storage connection in Connect AI.

1. Log into Connect AI, click Sources, and then click Add Connection

2. Select Azure Data Lake Storage from the Add Connection panel

3. Enter the necessary authentication properties to connect to Azure Data Lake Storage.

   Authenticating to a Gen 1 DataLakeStore Account

   Gen 1 uses OAuth 2.0 in Entra ID (formerly Azure AD) for authentication.

   For this, an Active Directory web application is required. You can create one as follows:

   1. Sign in to your Azure Account through the .
   2. Select "Entra ID" (formerly Azure AD).
   3. Select "App registrations".
   4. Select "New application registration".
   5. Provide a name and URL for the application. Select Web app for the type of application you want to create.
   6. Select "Required permissions" and change the required permissions for this app. At a minimum, "Azure Data Lake" and "Windows Azure Service Management API" are required.
   7. Select "Key" and generate a new key. Add a description, a duration, and take note of the generated key. You won't be able to see it again.

   To authenticate against a Gen 1 DataLakeStore account, the following properties are required:

   • Schema: Set this to ADLSGen1.
   • Account: Set this to the name of the account.
   • OAuthClientId: Set this to the application Id of the app you created.
   • OAuthClientSecret: Set this to the key generated for the app you created.
   • TenantId: Set this to the tenant Id. See the property for more information on how to acquire this.
   • Directory: Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.

   Authenticating to a Gen 2 DataLakeStore Account

   To authenticate against a Gen 2 DataLakeStore account, the following properties are required:

   • Schema: Set this to ADLSGen2.
   • Account: Set this to the name of the account.
   • FileSystem: Set this to the file system which will be used for this account.
   • AccessKey: Set this to the access key which will be used to authenticate the calls to the API. See the property for more information on how to acquire this.
   • Directory: Set this to the path which will be used to store the replicated file. If not specified, the root directory will be used.
   
4. Click Save & Test
5. Navigate to the Permissions tab and update user-based permissions

Add a Personal Access Token

A Personal Access Token (PAT) is used to authenticate the connection to Connect AI from Dify. It is best practice to create a separate PAT for each integration to maintain granular access control.

1. Click the gear icon () at the top right of the Connect AI app to open Settings
2. On the Settings page, go to the Access Tokens section and click Create PAT
3. Give the PAT a descriptive name and click Create

4. Copy the token when displayed and store it securely. It will not be shown again

With the Azure Data Lake Storage connection configured and a PAT generated, Dify can now connect to Azure Data Lake Storage data through Connect AI.

Step 2: Register the Connect AI MCP Server in Dify

Next, register the Connect AI Remote MCP Server as a tool in Dify so your agents and workflows can discover and call live data tools through Connect AI.

1. Log into Dify, or open your self-hosted Dify instance (version 1.6.0 or later, which includes built-in MCP support)
2. Navigate to the Integrations page, select Tools and click MCP tab

3. Click Add MCP Server (HTTP) and enter the following details:
   • Server URL: https://mcp.cloud.cdata.com/mcp
   • Name Icon: Give a descriptive name, for example, CData Connect AI
   • Server Identifier: A unique identifier, for example, cdata-connect-ai
   • Headers: Add an Authorization header with the value Basic your_base64_encoded_email_PAT

   Note: Dify will use Basic authentication with Connect AI. Combine your Connect AI user email and the PAT you created earlier in the format email:PAT, base64 encode the combined string, and prefix it with Basic.

4. Click Add & Authorize. Dify connects to the Connect AI MCP Server and lists the available tools

With the MCP server registered, the Connect AI tools are available to any agent application or workflow in your Dify workspace.

Step 3: Query live Azure Data Lake Storage data from Dify

With the integration complete, build an agent application in Dify and interact with live Azure Data Lake Storage data through natural language prompts.

1. From the Dify Studio page, click Create from Blank and select Agent as the application type
2. In the agent configuration, click Add under the Tools section and select the Connect AI MCP tools registered in Step 2
3. Select an LLM provider and model for the agent so it can interpret prompts and call MCP tools
4. In the preview panel, type a prompt in the chat input, for example:
   • Use the cdata-connect-ai tools to list all available catalogs
   • Show the available schemas and tables for Azure Data Lake Storage
   • Query the top 5 records from a table in Azure Data Lake Storage
5. The Dify agent calls the Connect AI MCP Server and returns live results from Azure Data Lake Storage data

At this point, Dify communicates with the Connect AI MCP Server and retrieves live Azure Data Lake Storage data through remote MCP tools directly from your agentic workflows.

Get started with CData Connect AI

To access hundreds of SaaS, big data, and NoSQL sources directly from your cloud applications, try CData Connect AI today. Download a free 14-day trial of CData Connect AI, and our Support Team is available to help with any questions you have.