Connect to Google Data Catalog Data in CloverDX (formerly CloverETL)

The CData JDBC Driver for Google Data Catalog enables you to use the data transformation components in CloverDX (formerly CloverETL) to work with Google Data Catalog as sources. In this article, you will use the JDBC Driver for Google Data Catalog to set up a simple transfer into a flat file. The CData JDBC Driver for Google Data Catalog enables you to use the data transformation components in CloverDX (formerly CloverETL) to work with Google Data Catalog as sources. In this article, you will use the JDBC Driver for Google Data Catalog to set up a simple transfer into a flat file.

Connect to Google Data Catalog as a JDBC Data Source

1. Create the connection to Google Data Catalog data. In a new CloverDX graph, right-click the Connections node in the Outline pane and click Connections -> Create Connection. The Database Connection wizard is displayed.
2. Click the plus icon to load a driver from a JAR. Browse to the lib subfolder of the installation directory and select the cdata.jdbc.googledatacatalog.jar file.
3. Enter the JDBC URL.

   Google Data Catalog uses the OAuth authentication standard. Authorize access to Google APIs on behalf on individual users or on behalf of users in a domain.

   Before connecting, specify the following to identify the organization and project you would like to connect to:

   • OrganizationId: The ID associated with the Google Cloud Platform organization resource you would like to connect to. Find this by navigating to the cloud console.

     Click the project selection drop-down, and select your organization from the list. Then, click More -> Settings. The organization ID is displayed on this page.

   • ProjectId: The ID associated with the Google Cloud Platform project resource you would like to connect to.

     Find this by navigating to the cloud console dashboard and selecting your project from the Select from drop-down. The project ID will be present in the Project info card.

   When you connect, the OAuth endpoint opens in your default browser. Log in and grant permissions to the application to completes the OAuth process. For more information, refer to the OAuth section in the Help documentation.

   Built-in Connection String Designer

   For assistance in constructing the JDBC URL, use the connection string designer built into the Google Data Catalog JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

   java -jar cdata.jdbc.googledatacatalog.jar

   Fill in the connection properties and copy the connection string to the clipboard.

   A typical JDBC URL is below:

   jdbc:googledatacatalog:ProjectId=YourProjectId;InitiateOAuth=GETANDREFRESH

Query Google Data Catalog Data with the DBInputTable Component

1. Drag a DBInputTable from the Readers selection of the Palette onto the job flow and double-click it to open the configuration editor.
2. In the DB connection property, select the Google Data Catalog JDBC data source from the drop-down menu.
3. Enter the SQL query. For example: SELECT Type, DatasetName FROM Schemas WHERE ProjectId = 'bigquery-public-data'

Write the Output of the Query to a UniversalDataWriter

1. Drag a UniversalDataWriter from the Writers selection onto the jobflow.
2. Double-click the UniversalDataWriter to open the configuration editor and add a file URL.
3. Right-click the DBInputTable and then click Extract Metadata.
4. Connect the output port of the DBInputTable to the UniversalDataWriter.
5. In the resulting Select Metadata menu for the UniversalDataWriter, choose the Schemas table. (You can also open this menu by right-clicking the input port for the UniversalDataWriter.)
6. Click Run to write to the file.