Build AlloyDB-Connected ETL Processes in Google Data Fusion

Google Data Fusion allows users to perform self-service data integration to consolidate disparate data. Uploading the CData JDBC Driver for AlloyDB enables users to access live AlloyDB data from within their Google Data Fusion pipelines. While the CData JDBC Driver enables piping AlloyDB data to any data source natively supported in Google Data Fusion, this article walks through piping data from AlloyDB to Google BigQuery,

Upload the CData JDBC Driver for AlloyDB to Google Data Fusion

Upload the CData JDBC Driver for AlloyDB to your Google Data Fusion instance to work with live AlloyDB data. Due to the naming restrictions for JDBC drivers in Google Data Fusion, create a copy or rename the JAR file to match the following format driver-version.jar. For example: cdataalloydb-2020.jar

1. Open your Google Data Fusion instance
2. Click the to add an entity and upload a driver
3. On the "Upload driver" tab, drag or browse to the renamed JAR file.
4. On the "Driver configuration" tab:
   • Name: Create a name for the driver (cdata.jdbc.alloydb) and make note of the name
   • Class name: Set the JDBC class name: (cdata.jdbc.alloydb.AlloyDBDriver)
5. Click "Finish"

Connect to AlloyDB Data in Google Data Fusion

With the JDBC Driver uploaded, you are ready to work with live AlloyDB data in Google Data Fusion Pipelines.

1. Navigate to the Pipeline Studio to create a new Pipeline
2. From the "Source" options, click "Database" to add a source for the JDBC Driver
3. Click "Properties" on the Database source to edit the properties

   NOTE: To use the JDBC Driver in Google Data Fusion, you will need a license (full or trial) and a Runtime Key (RTK). For more information on obtaining this license (or a trial), contact our sales team.

   • Set the Label
   • Set Reference Name to a value for any future references (i.e.: cdata-alloydb)
   • Set Plugin Type to "jdbc"
   • Set Connection String to the JDBC URL for AlloyDB. For example:
     
     jdbc:alloydb:RTK=5246...;User=alloydb;Password=admin;Database=alloydb;Server=127.0.0.1;Port=5432

     The following connection properties are usually required in order to connect to AlloyDB.

     • Server: The host name or IP of the server hosting the AlloyDB database.
     • User: The user which will be used to authenticate with the AlloyDB server.
     • Password: The password which will be used to authenticate with the AlloyDB server.

     You can also optionally set the following:

     • Database: The database to connect to when connecting to the AlloyDB Server. If this is not set, the user's default database will be used.
     • Port: The port of the server hosting the AlloyDB database. This property is set to 5432 by default.

     Authenticating with Standard Authentication

     Standard authentication (using the user/password combination supplied earlier) is the default form of authentication.

     No further action is required to leverage Standard Authentication to connect.

     Authenticating with pg_hba.conf Auth Schemes

     There are additional methods of authentication available which must be enabled in the pg_hba.conf file on the AlloyDB server.

     Find instructions about authentication setup on the AlloyDB Server here.

     Authenticating with MD5 Authentication

     This authentication method must be enabled by setting the auth-method in the pg_hba.conf file to md5.

     Authenticating with SASL Authentication

     This authentication method must be enabled by setting the auth-method in the pg_hba.conf file to scram-sha-256.

     Authenticating with Kerberos

     The authentication with Kerberos is initiated by AlloyDB Server when the ∏ is trying to connect to it. You should set up Kerberos on the AlloyDB Server to activate this authentication method. Once you have Kerberos authentication set up on the AlloyDB Server, see the Kerberos section of the help documentation for details on how to authenticate with Kerberos.

     Built-in Connection String Designer

     For assistance in constructing the JDBC URL, use the connection string designer built into the AlloyDB JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

     java -jar cdata.jdbc.alloydb.jar

     Fill in the connection properties and copy the connection string to the clipboard.

   • Set Import Query to a SQL query that will extract the data you want from AlloyDB, i.e.:
     SELECT * FROM Orders
4. From the "Sink" tab, click to add a destination sink (we use Google BigQuery in this example)
5. Click "Properties" on the BigQuery sink to edit the properties
   • Set the Label
   • Set Reference Name to a value like alloydb-bigquery
   • Set Project ID to a specific Google BigQuery Project ID (or leave as the default, "auto-detect")
   • Set Dataset to a specific Google BigQuery dataset
   • Set Table to the name of the table you wish to insert AlloyDB data into

With the Source and Sink configured, you are ready to pipe AlloyDB data into Google BigQuery. Save and deploy the pipeline. When you run the pipeline, Google Data Fusion will request live data from AlloyDB and import it into Google BigQuery.

While this is a simple pipeline, you can create more complex AlloyDB pipelines with transforms, analytics, conditions, and more. Download a free, 30-day trial of the CData JDBC Driver for AlloyDB and start working with your live AlloyDB data in Google Data Fusion today.