How to Build an ETL App for Certinia Data in Python with CData Connect AI

The rich ecosystem of Python modules lets you get to work quickly and integrate your systems more effectively. With the CData Connect AI Python SDK and the petl framework, you can build Certinia-connected applications and pipelines for extracting, transforming, and loading Certinia data. This article shows how to connect to Connect AI and use petl to extract, transform, and load Certinia data.

The Connect AI Python SDK (cdata-connect-ai) is a DB-API 2.0 (PEP 249) compliant client, so petl can read directly from the SDK connection with etl.fromdb. There is no driver to install per source: connect with a Personal Access Token and build your pipeline.

Connect to Certinia in Connect AI

CData Connect AI uses a straightforward, point-and-click interface to connect to data sources.

1. Log into Connect AI, click Sources, and then click Add Connection

2. Select "Certinia" from the Add Connection panel

3. Enter the necessary authentication properties to connect to Certinia.

   There are several authentication methods available for connecting to Certinia: login credentials, SSO, and OAuth.

   Authenticating with a Login and Token

   Set the User and Password to your login credentials. Additionally, set the SecurityToken. By default, the SecurityToken is required, but you can make it optional by allowing a range of trusted IP addresses.

   To disable the security token:

   1. Log in to Certinia and enter "Network Access" in the Quick Find box in the setup section.
   2. Add your IP address to the list of trusted IP addresses.

   To obtain the security token:

   1. Open the personal information page on certinia.com.
   2. Click the link to reset your security token. The token will be emailed to you.
   3. Specify the security token in the SecurityToken connection property or append it to the Password.

   Authenticating with OAuth

   If you do not have access to the user name and password or do not want to require them, use the OAuth user consent flow. See the OAuth section in the Help for an authentication guide.

   Connecting to Certinia Sandbox Accounts

   Set UseSandbox to true (false by default) to use a Certinia sandbox account. Ensure that you specify a sandbox user name in User.

   
4. Click Save & Test
5. Navigate to the Permissions tab and update the user-based permissions.

Generate a Personal Access Token (PAT)

The Python SDK authenticates to Connect AI with your account email and a Personal Access Token (PAT). It is best practice to create a separate PAT for each application to maintain granularity of access.

1. Click the Gear icon () at the top right of the Connect AI app to open the Settings page.
2. On the Settings page, go to the Access Tokens section and click Create PAT.
3. Give the PAT a name and click Create.
4. The PAT is only visible at creation, so copy it and store it securely.

Install Required Modules

Install the SDK and the petl framework using the pip utility:

pip install cdata-connect-ai
pip install petl

Build an ETL App for Certinia Data in Python

Once the required modules are installed, you are ready to build the ETL app. Code snippets follow, but the full source code is available at the end of the article.

First, import the modules and connect to Connect AI with your account email and PAT:

import petl as etl
import cdata_connect_ai

conn = cdata_connect_ai.connect(
    username="[email protected]",
    password="<your_pat>",
)

Create a SQL Statement to Query Certinia

Use SQL to create a statement for querying Certinia. In this article, we read data from the Account entity. Identifiers are three-part: <Connection>.<Schema>.<Table>, where the connection name defaults to the source name (for example, Certinia1).

sql = (
    "SELECT BillingState, Name "
    "FROM [Certinia1].[Certinia].[Account] "
    "WHERE Industry = 'Floppy Disks'"
)

Extract, Transform, and Load the Certinia Data

With a connection and query in hand, use petl to extract, transform, and load the Certinia data. In this example, we extract Certinia data, sort the data by the Name column, and load the data into a CSV file.

table1 = etl.fromdb(conn, sql)

table2 = etl.sort(table1, 'Name')

etl.tocsv(table2, 'account_data.csv')

Load New Rows Back into Certinia

When Certinia supports writes, load rows back with a batch INSERT. The SDK's executemany takes @name placeholders and a list of parameter dictionaries, one per row.

cur = conn.cursor()
cur.executemany(
    "INSERT INTO [Certinia1].[Certinia].[Account] (BillingState, Name) "
    "VALUES (@val1, @val2)",
    [
        {"@val1": "New value 1", "@val2": "New value 1"},
        {"@val1": "New value 2", "@val2": "New value 2"},
    ],
)
print(f"Rows inserted: {cur.rowcount}")

conn.close()

Note: Even for writable sources, a read-only PAT or connection permission will reject write operations.

With the CData Connect AI Python SDK, you can work with Certinia data just like you would with any database, including direct access to data in ETL packages like petl.

More Information and Free Trial

Now you can pipe live Certinia data through petl using the CData Connect AI Python SDK. For more information on connecting to Certinia (and hundreds of other data sources), visit the Connect AI page. Sign up for a free trial and start building data pipelines for live Certinia data in Python.

Full Source Code

import petl as etl
import cdata_connect_ai

conn = cdata_connect_ai.connect(
    username="[email protected]",
    password="<your_pat>",
)

sql = (
    "SELECT BillingState, Name "
    "FROM [Certinia1].[Certinia].[Account] "
    "WHERE Industry = 'Floppy Disks'"
)

table1 = etl.fromdb(conn, sql)

table2 = etl.sort(table1, 'Name')

etl.tocsv(table2, 'account_data.csv')

cur = conn.cursor()
cur.executemany(
    "INSERT INTO [Certinia1].[Certinia].[Account] (BillingState, Name) "
    "VALUES (@val1, @val2)",
    [
        {"@val1": "New value 1", "@val2": "New value 1"},
        {"@val1": "New value 2", "@val2": "New value 2"},
    ],
)
print(f"Rows inserted: {cur.rowcount}")
conn.close()