How to load PingOne data into Elasticsearch via Logstash

Elasticsearch is a popular distributed full-text search engine. By centrally storing data, you can perform ultra-fast searches, fine-tuning relevance, and powerful analytics with ease. Elasticsearch has a pipeline tool for loading data called "Logstash". You can use CData JDBC Drivers to easily import data from any data source into Elasticsearch for search and analysis.

This article explains how to use the CData JDBC Driver for PingOne to load data from PingOne into Elasticsearch via Logstash.

Using CData JDBC Driver for PingOne with Elasticsearch Logstash

• Install the CData JDBC Driver for PingOne on the machine where Logstash is running.
• The JDBC Driver will be installed at the following path (the year part, e.g. 20XX, will vary depending on the product version you are using). You will use this path later. Place this .jar file (and the .lic file if it's a licensed version) in Logstash.
  C:\Program Files\CData\CData JDBC Driver for PingOne 20XX\lib\cdata.jdbc.pingone.jar
  
• Next, install the JDBC Input Plugin, which connects Logstash to the CData JDBC driver. The JDBC Plugin comes by default with the latest version of Logstash, but depending on the version, you may need to add it.
  https://www.elastic.co/guide/en/logstash/5.4/plugins-inputs-jdbc.html
• Move the CData JDBC Driver’s .jar file and .lic file to Logstash's "/logstash-core/lib/jars/".

Sending PingOne data to Elasticsearch with Logstash

Now, let's create a configuration file for Logstash to transfer PingOne data to Elasticsearch.

• Write the process to retrieve PingOne data in the logstash.conf file, which defines data processing in Logstash. The input will be JDBC, and the output will be Elasticsearch. The data loading job is set to run at 30-second intervals.
• Set the CData JDBC Driver's .jar file as the JDBC driver library, configure the class name, and set the connection properties to PingOne in the form of a JDBC URL. The JDBC URL allows detailed configuration, so please refer to the product documentation for more specifics.

To connect to PingOne, configure these properties:

• Region: The region where the data for your PingOne organization is being hosted.
• AuthScheme: The type of authentication to use when connecting to PingOne.
• Either WorkerAppEnvironmentId (required when using the default PingOne domain) or AuthorizationServerURL, configured as described below.

Configuring WorkerAppEnvironmentId

WorkerAppEnvironmentId is the ID of the PingOne environment in which your Worker application resides. This parameter is used only when the environment is using the default PingOne domain (auth.pingone). It is configured after you have created the custom OAuth application you will use to authenticate to PingOne, as described in Creating a Custom OAuth Application in the Help documentation.

First, find the value for this property:

1. From the home page of your PingOne organization, move to the navigation sidebar and click Environments.
2. Find the environment in which you have created your custom OAuth/Worker application (usually Administrators), and click Manage Environment. The environment's home page displays.
3. In the environment's home page navigation sidebar, click Applications.
4. Find your OAuth or Worker application details in the list.
5. Copy the value in the Environment ID field. It should look similar to:
   WorkerAppEnvironmentId='11e96fc7-aa4d-4a60-8196-9acf91424eca'

Now set WorkerAppEnvironmentId to the value of the Environment ID field.

Configuring AuthorizationServerURL

AuthorizationServerURL is the base URL of the PingOne authorization server for the environment where your application is located. This property is only used when you have set up a custom domain for the environment, as described in the PingOne platform API documentation. See Custom Domains.

Authenticating to PingOne with OAuth

PingOne supports both OAuth and OAuthClient authentication. In addition to performing the configuration steps described above, there are two more steps to complete to support OAuth or OAuthCliet authentication:

• Create and configure a custom OAuth application, as described in Creating a Custom OAuth Application in the Help documentation.
• To ensure that the driver can access the entities in Data Model, confirm that you have configured the correct roles for the admin user/worker application you will be using, as described in Administrator Roles in the Help documentation.
• Set the appropriate properties for the authscheme and authflow of your choice, as described in the following subsections.

OAuth (Authorization Code grant)

Set AuthScheme to OAuth.

Desktop Applications

Get and Refresh the OAuth Access Token

After setting the following, you are ready to connect:

• InitiateOAuth: GETANDREFRESH. To avoid the need to repeat the OAuth exchange and manually setting the OAuthAccessToken each time you connect, use InitiateOAuth.
• OAuthClientId: The Client ID you obtained when you created your custom OAuth application.
• OAuthClientSecret: The Client Secret you obtained when you created your custom OAuth application.
• CallbackURL: The redirect URI you defined when you registered your custom OAuth application. For example: https://localhost:3333

When you connect, the driver opens PingOne's OAuth endpoint in your default browser. Log in and grant permissions to the application. The driver then completes the OAuth process:

1. The driver obtains an access token from PingOne and uses it to request data.
2. The OAuth values are saved in the location specified in OAuthSettingsLocation, to be persisted across connections.

The driver refreshes the access token automatically when it expires.

For other OAuth methods, including Web Applications, Headless Machines, or Client Credentials Grant, refer to the Help documentation.

Executing data movement with Logstash

Now let's run Logstash using the created "logstash.conf" file.

A log indicating success will appear. This means the PingOne data has been loaded into Elasticsearch.

For example, let's view the data transferred to Elasticsearch in Kibana.

    GET pingone_table/_search
    {
        "query": {
            "match_all": {}
        }
    }

We have confirmed that the data is stored in Elasticsearch.

By using the CData JDBC Driver for PingOne with Logstash, it functions as a PingOne connector, making it easy to load data into Elasticsearch. Please try the 30-day free trial.