How to Build an ETL App for Trello Data in Python with CData



Create ETL applications and real-time data pipelines for Trello data in Python with petl.

The rich ecosystem of Python modules lets you get to work quickly and integrate your systems more effectively. With the CData Python Connector for Trello and the petl framework, you can build Trello-connected applications and pipelines for extracting, transforming, and loading Trello data. This article shows how to connect to Trello with the CData Python Connector and use petl and pandas to extract, transform, and load Trello data.

With built-in, optimized data processing, the CData Python Connector offers unmatched performance for interacting with live Trello data in Python. When you issue complex SQL queries from Trello, the driver pushes supported SQL operations, like filters and aggregations, directly to Trello and utilizes the embedded SQL engine to process unsupported operations client-side (often SQL functions and JOIN operations).

Connecting to Trello Data

Connecting to Trello data looks just like connecting to any relational data source. Create a connection string using the required connection properties. For this article, you will pass the connection string as a parameter to the create_engine function.

Trello uses token-based authentication to grant third-party applications access to their API. When a user has granted an application access to their data, the application is given a token that can be used to make requests to Trello's API.

Trello's API can be accessed in 2 different ways. The first is using Trello's own Authorization Route, and the second is using OAuth1.0.

  • Authorization Route: At the moment of registration, Trello assigns an API key and Token to the account. See the Help documentation for information on how to connect via the Authorization route.
  • OAuth Route: Similar to using Authorization, OAuth creates an Application Id and Secret when you create your account. See the Help documentation for information on how to to connect.

After installing the CData Trello Connector, follow the procedure below to install the other required modules and start accessing Trello through Python objects.

Install Required Modules

Use the pip utility to install the required modules and frameworks:

pip install petl
pip install pandas

Build an ETL App for Trello Data in Python

Once the required modules and frameworks are installed, we are ready to build our ETL app. Code snippets follow, but the full source code is available at the end of the article.

First, be sure to import the modules (including the CData Connector) with the following:

import petl as etl
import pandas as pd
import cdata.trello as mod

You can now connect with a connection string. Use the connect function for the CData Trello Connector to create a connection for working with Trello data.

cnxn = mod.connect("APIKey=myApiKey;Token=myGeneratedToken;InitiateOAuth=GETANDREFRESH;OAuthSettingsLocation=/PATH/TO/OAuthSettings.txt")")

Create a SQL Statement to Query Trello

Use SQL to create a statement for querying Trello. In this article, we read data from the Boards entity.

sql = "SELECT BoardId, Name FROM Boards WHERE Name = 'Public Board'"

Extract, Transform, and Load the Trello Data

With the query results stored in a DataFrame, we can use petl to extract, transform, and load the Trello data. In this example, we extract Trello data, sort the data by the Name column, and load the data into a CSV file.

Loading Trello Data into a CSV File

table1 = etl.fromdb(cnxn,sql)

table2 = etl.sort(table1,'Name')

etl.tocsv(table2,'boards_data.csv')

With the CData Python Connector for Trello, you can work with Trello data just like you would with any database, including direct access to data in ETL packages like petl.

Free Trial & More Information

Download a free, 30-day trial of the CData Python Connector for Trello to start building Python apps and scripts with connectivity to Trello data. Reach out to our Support Team if you have any questions.



Full Source Code


import petl as etl
import pandas as pd
import cdata.trello as mod

cnxn = mod.connect("APIKey=myApiKey;Token=myGeneratedToken;InitiateOAuth=GETANDREFRESH;OAuthSettingsLocation=/PATH/TO/OAuthSettings.txt")")

sql = "SELECT BoardId, Name FROM Boards WHERE Name = 'Public Board'"

table1 = etl.fromdb(cnxn,sql)

table2 = etl.sort(table1,'Name')

etl.tocsv(table2,'boards_data.csv')

Ready to get started?

Download a free trial of the Trello Connector to get started:

 Download Now

Learn more:

Trello Icon Trello Python Connector

Python Connector Libraries for Trello Data Connectivity. Integrate Trello with popular Python tools like Pandas, SQLAlchemy, Dash & petl.