How to Build an ETL App for Sage X3 Cloud Data in Python with CData

The rich ecosystem of Python modules lets you get to work quickly and integrate your systems more effectively. With the CData Python Connector for Sage X3 Cloud and the petl framework, you can build Sage X3 Cloud-connected applications and pipelines for extracting, transforming, and loading Sage X3 Cloud data. This article shows how to connect to Sage X3 Cloud with the CData Python Connector and use petl and pandas to extract, transform, and load Sage X3 Cloud data.

With built-in, optimized data processing, the CData Python Connector offers unmatched performance for interacting with live Sage X3 Cloud data in Python. When you issue complex SQL queries from Sage X3 Cloud, the driver pushes supported SQL operations, like filters and aggregations, directly to Sage X3 Cloud and utilizes the embedded SQL engine to process unsupported operations client-side (often SQL functions and JOIN operations).

Connecting to Sage X3 Cloud Data

Connecting to Sage X3 Cloud data looks just like connecting to any relational data source. Create a connection string using the required connection properties. For this article, you will pass the connection string as a parameter to the create_engine function.

Sage X3 Cloud uses the OAuth 2.0 Client Credentials flow, and an X-API-Key is also required for API access. Set AuthScheme to OAuth and specify the following connection properties:

• URL: The base URL of your Sage X3 Cloud instance.
• OAuthAccessTokenUrl: The OAuth token endpoint (e.g., https://your-auth-domain/oauth/token).
• OAuthClientId: Your OAuth application client ID.
• OAuthClientSecret: Your OAuth application client secret.
• Audience: The API audience value for the token request.
• XAPIKey: The X-API-Key provided by your Sage X3 Cloud administrator.
• Folder: The Sage X3 folder name (e.g., SEED). This folder is used as the default schema.
• Folders (optional): A comma-separated list of Sage X3 folders (e.g., SEED,PERF). Each folder is exposed as a separate schema, so you can query across folders with the Schema.Table syntax.

The driver obtains an access token with the Client Credentials flow and sends it with the X-API-Key on every API request. With InitiateOAuth set to GETANDREFRESH (the default), the driver acquires and refreshes the token automatically.

After installing the CData Sage X3 Cloud Connector, follow the procedure below to install the other required modules and start accessing Sage X3 Cloud through Python objects.

Install Required Modules

Use the pip utility to install the required modules and frameworks:

pip install petl
pip install pandas

Build an ETL App for Sage X3 Cloud Data in Python

Once the required modules and frameworks are installed, we are ready to build our ETL app. Code snippets follow, but the full source code is available at the end of the article.

First, be sure to import the modules (including the CData Connector) with the following:

import petl as etl
import pandas as pd
import cdata.sagex3cloud as mod

You can now connect with a connection string. Use the connect function for the CData Sage X3 Cloud Connector to create a connection for working with Sage X3 Cloud data.

cnxn = mod.connect("AuthScheme=OAuth;URL=https://x3server/;OAuthAccessTokenUrl=https://auth-domain/oauth/token;OAuthClientId=your_client_id;OAuthClientSecret=your_client_secret;Audience=https://api-audience;XAPIKey=your_api_key;Folder=SEED;")

Create a SQL Statement to Query Sage X3 Cloud

Use SQL to create a statement for querying Sage X3 Cloud. In this article, we read data from the BPCUSTOMER entity.

sql = "SELECT BPCNUM, BPCNAM FROM BPCUSTOMER WHERE BPCNUM = 'MARTIN'"

Extract, Transform, and Load the Sage X3 Cloud Data

With the query results stored in a DataFrame, we can use petl to extract, transform, and load the Sage X3 Cloud data. In this example, we extract Sage X3 Cloud data, sort the data by the BPCNAM column, and load the data into a CSV file.

Loading Sage X3 Cloud Data into a CSV File

table1 = etl.fromdb(cnxn,sql)

table2 = etl.sort(table1,'BPCNAM')

etl.tocsv(table2,'bpcustomer_data.csv')

With the CData Python Connector for Sage X3 Cloud, you can work with Sage X3 Cloud data just like you would with any database, including direct access to data in ETL packages like petl.

Free Trial & More Information

Download a free, 30-day trial of the CData Python Connector for Sage X3 Cloud to start building Python apps and scripts with connectivity to Sage X3 Cloud data. Reach out to our Support Team if you have any questions.

Full Source Code

import petl as etl
import pandas as pd
import cdata.sagex3cloud as mod

cnxn = mod.connect("AuthScheme=OAuth;URL=https://x3server/;OAuthAccessTokenUrl=https://auth-domain/oauth/token;OAuthClientId=your_client_id;OAuthClientSecret=your_client_secret;Audience=https://api-audience;XAPIKey=your_api_key;Folder=SEED;")

sql = "SELECT BPCNUM, BPCNAM FROM BPCUSTOMER WHERE BPCNUM = 'MARTIN'"

table1 = etl.fromdb(cnxn,sql)

table2 = etl.sort(table1,'BPCNAM')

etl.tocsv(table2,'bpcustomer_data.csv')