Connect and Query Live XML Data in Databricks with CData Connect AI

Databricks is a leading AI cloud-native platform that unifies data engineering, machine learning, and analytics at scale. Its powerful data lakehouse architecture combines the performance of data warehouses with the flexibility of data lakes. Integrating Databricks with CData Connect AI gives organizations live, real-time access to XML data without the need for complex ETL pipelines or data duplication—streamlining operations and reducing time-to-insights.

In this article, we'll walk through how to configure a secure, live connection from Databricks to XML using CData Connect AI. Once configured, you'll be able to access XML data directly from Databricks notebooks using standard SQL—enabling unified, real-time analytics across your data ecosystem.

Overview

Here is an overview of the simple steps:

1. Step 1 — Connect and Configure: In CData Connect AI, create a connection to your XML source, configure user permissions, and generate a Personal Access Token (PAT).
2. Step 2 — Query from Databricks: Install the CData JDBC driver in Databricks, configure your notebook with the connection details, and run SQL queries to access live XML data.

Prerequisites

Before you begin, make sure you have the following:

1. An active XML account.
2. A CData Connect AI account. You can log in or sign up for a free trial here.
3. A Databricks account. Sign up or log in here.

Step 1: Connect and Configure a XML Connection in CData Connect AI

1.1 Add a Connection to XML

CData Connect AI uses a straightforward, point-and-click interface to connect to available data sources.

1. Log into Connect AI, click Sources on the left, and then click Add Connection in the top-right.

2. Select "XML" from the Add Connection panel.

3. Enter the necessary authentication properties to connect to XML.

   Connecting to Local or Cloud-Stored (Box, Google Drive, Amazon S3, SharePoint) XML Files

   CData Drivers let you work with XML files stored locally and stored in cloud storage services like Box, Amazon S3, Google Drive, or SharePoint, right where they are.

   Setting connection properties for local files

   Set the URI property to local folder path.

   Setting connection properties for files stored in Amazon S3

   To connect to XML file(s) within Amazon S3, set the URI property to the URI of the Bucket and Folder where the intended XML files exist. In addition, at least set these properties:

   • AWSAccessKey: AWS Access Key (username)
   • AWSSecretKey: AWS Secret Key

   Setting connection properties for files stored in Box

   To connect to XML file(s) within Box, set the URI property to the URI of the folder that includes the intended XML file(s). Use the OAuth authentication method to connect to Box.

   Dropbox

   To connect to XML file(s) within Dropbox, set the URI proprerty to the URI of the folder that includes the intended XML file(s). Use the OAuth authentication method to connect to Dropbox. Either User Account or Service Account can be used to authenticate.

   SharePoint Online (SOAP)

   To connect to XML file(s) within SharePoint with SOAP Schema, set the URI proprerty to the URI of the document library that includes the intended XML file. Set User, Password, and StorageBaseURL.

   SharePoint Online REST

   To connect to XML file(s) within SharePoint with REST Schema, set the URI proprerty to the URI of the document library that includes the intended XML file. StorageBaseURL is optional. If not set, the driver will use the root drive. OAuth is used to authenticate.

   Google Drive

   To connect to XML file(s) within Google Drive, set the URI property to the URI of the folder that includes the intended XML file(s). Use the OAuth authentication method to connect and set InitiateOAuth to GETANDREFRESH.

   The DataModel property is the controlling property over how your data is represented into tables and toggles the following basic configurations.

   • Document (default): Model a top-level, document view of your XML data. The data provider returns nested elements as aggregates of data.
   • FlattenedDocuments: Implicitly join nested documents and their parents into a single table.
   • Relational: Return individual, related tables from hierarchical data. The tables contain a primary key and a foreign key that links to the parent document.

   See the Modeling XML Data chapter for more information on configuring the relational representation. You will also find the sample data used in the following examples. The data includes entries for people, the cars they own, and various maintenance services performed on those cars.

   
4. Click Save & Test in the top-right.
5. Navigate to the Permissions tab on the XML Connection page and update the user-based permissions based on your preferences.

1.2 Generate a Personal Access Token (PAT)

When connecting to Connect AI through the REST API, the OData API, or the Virtual SQL Server, a Personal Access Token (PAT) is used to authenticate the connection to Connect AI. PAT functions as an alternative to your login credentials for secure, token-based authentication. It is a best practice to create a separate PAT for each service to maintain granularity of access.

1. Click on the Gear icon () at the top right of the Connect AI app to open the settings page.
2. On the Settings page, go to the Access Tokens section and click Create PAT.
3. Give the PAT a name and click Create.
4. Note: The personal access token is only visible at creation, so be sure to copy it and store it securely for future use.

Step 2: Connect and Query XML Data in Databricks

Follow these steps to establish a connection from Databricks to XML. You'll install the CData JDBC Driver for Connect AI, add the JAR file to your cluster, configure your notebooks, and run SQL queries to access live XML data data.

2.1 Install the CData JDBC Driver for Connect AI

1. In CData Connect AI, click the Integrations page on the left. Search for JDBC or Databricks, click Download, and select the installer for your operating system.
2. Once downloaded, run the installer and follow the instructions:
   • For Windows: Run the setup file and follow the installation wizard.
   • For Mac/Linux: Unpack the archive and move the folder to /opt or /Applications. Make sure you have execute permissions.
3. After installation, locate the JAR file in the installation directory:
   • Windows:

     C:\Program Files\CData\CData JDBC Driver for Connect AI\lib\cdata.jdbc.connect.jar

   • Mac/Linux:

     /Applications/CData/CData JDBC Driver for Connect AI/lib/cdata.jdbc.connect.jar

2.2 Install the JAR File on Databricks

1. Log in to Databricks. In the navigation pane, click Compute on the left. Start or create a compute cluster.
2. Click on the running cluster, go to the Libraries tab, and click Install New at the top right.
3. In the Install Library dialog, select DBFS, and drag and drop the cdata.jdbc.connect.jar file. Click Install.

2.3 Query XML Data in a Databricks Notebook

Notebook Script 1 — Define JDBC Connection:

1. Paste the following script into the notebook cell:

driver = "cdata.jdbc.connect.ConnectDriver"
url = "jdbc:connect:AuthScheme=Basic;User=your_username;Password=your_pat;URL=https://cloud.cdata.com/api/;DefaultCatalog=Your_Connection_Name;"

2. Replace:
   • your_username - With your CData Connect AI username
   • your_pat - With your CData Connect AI Personal Access Token (PAT)
   • Your_Connection_Name - With the name of your Connect AI data source, from the Sources page
3. Run the script.

Notebook Script 2 — Load DataFrame from XML data:

1. Add a new cell for this second script. From the menu on the right side of your notebook, click Add cell below.
2. Paste the following script into the new cell:

remote_table = spark.read.format("jdbc") \
  .option("driver", "cdata.jdbc.connect.ConnectDriver") \
  .option("url", "jdbc:connect:AuthScheme=Basic;User=your_username;Password=your_pat;URL=https://cloud.cdata.com/api/;DefaultCatalog=Your_Connection_Name;") \
  .option("dbtable", "YOUR_SCHEMA.YOUR_TABLE") \
  .load()

3. Replace:
   • your_username - With your CData Connect AI username
   • your_pat - With your CData Connect AI Personal Access Token (PAT)
   • Your_Connection_Name - With the name of your Connect AI data source, from the Sources page
   • YOUR_SCHEMA.YOUR_TABLE - With your schema and table, for example, XML.people
4. Run the script.

Notebook Script 3 — Preview Columns:

1. Similarly, add a new cell for this third script.
2. Paste the following script into the new cell:

display(remote_table.select("ColumnName1", "ColumnName2"))

3. Replace ColumnName1 and ColumnName2 with the actual columns from your XML structure (e.g. [ personal.name.first ], [ personal.name.last ], etc.).
4. Run the script.

You can now explore, join, and analyze live XML data directly within Databricks notebooks—without needing to know the complexities of the back-end API and without replicating XML data.

Try CData Connect AI Free for 14 Days

Ready to simplify real-time access to XML data? Start your free 14-day trial of CData Connect AI today and experience seamless, live connectivity from Databricks to XML.

Low code, zero infrastructure, zero replication — just seamless, secure access to your most critical data and insights.