How to load SAP SuccessFactors data into Elasticsearch via Logstash

Elasticsearch is a popular distributed full-text search engine. By centrally storing data, you can perform ultra-fast searches, fine-tuning relevance, and powerful analytics with ease. Elasticsearch has a pipeline tool for loading data called "Logstash". You can use CData JDBC Drivers to easily import data from any data source into Elasticsearch for search and analysis.

This article explains how to use the CData JDBC Driver for SAP SuccessFactors to load data from SAP SuccessFactors into Elasticsearch via Logstash.

Using CData JDBC Driver for SAP SuccessFactors with Elasticsearch Logstash

• Install the CData JDBC Driver for SAP SuccessFactors on the machine where Logstash is running.
• The JDBC Driver will be installed at the following path (the year part, e.g. 20XX, will vary depending on the product version you are using). You will use this path later. Place this .jar file (and the .lic file if it's a licensed version) in Logstash.
  C:\Program Files\CData\CData JDBC Driver for SAPSuccessFactors 20XX\lib\cdata.jdbc.sapsuccessfactors.jar
  
• Next, install the JDBC Input Plugin, which connects Logstash to the CData JDBC driver. The JDBC Plugin comes by default with the latest version of Logstash, but depending on the version, you may need to add it.
  https://www.elastic.co/guide/en/logstash/5.4/plugins-inputs-jdbc.html
• Move the CData JDBC Driver’s .jar file and .lic file to Logstash's "/logstash-core/lib/jars/".

Sending SAP SuccessFactors data to Elasticsearch with Logstash

Now, let's create a configuration file for Logstash to transfer SAP SuccessFactors data to Elasticsearch.

• Write the process to retrieve SAP SuccessFactors data in the logstash.conf file, which defines data processing in Logstash. The input will be JDBC, and the output will be Elasticsearch. The data loading job is set to run at 30-second intervals.
• Set the CData JDBC Driver's .jar file as the JDBC driver library, configure the class name, and set the connection properties to SAP SuccessFactors in the form of a JDBC URL. The JDBC URL allows detailed configuration, so please refer to the product documentation for more specifics.

You can authenticate to SAP Success Factors using Basic authentication or OAuth with SAML assertion.

Basic Authentication

You must provide values for the following properties to successfully authenticate to SAP Success Factors. Note that the provider will reuse the session opened by SAP Success Factors using cookies. Which means that your credentials will be used only on the first request to open the session. After that, cookies returned from SAP Success Factors will be used for authentication.

• Url: set this to the URL of the server hosting Success Factors. Some of the servers are listed in the SAP support documentation (external link).
• User: set this to the username of your account.
• Password: set this to the password of your account.
• CompanyId: set this to the unique identifier of your company.

OAuth Authentication

You must provide values for the following properties, which will be used to get the access token.

• Url: set this to the URL of the server hosting Success Factors. Some of the servers are listed in the SAP support documentation (external link).
• User: set this to the username of your account.
• CompanyId: set this to the unique identifier of your company.
• OAuthClientId: set this to the API Key that was generated in API Center.
• OAuthClientSecret: the X.509 private key used to sign SAML assertion. The private key can be found in the certificate you downloaded in Registering your OAuth Client Application.
• InitiateOAuth: set this to GETANDREFRESH.

Executing data movement with Logstash

Now let's run Logstash using the created "logstash.conf" file.

A log indicating success will appear. This means the SAP SuccessFactors data has been loaded into Elasticsearch.

For example, let's view the data transferred to Elasticsearch in Kibana.

    GET sapsuccessfactors_table/_search
    {
        "query": {
            "match_all": {}
        }
    }

We have confirmed that the data is stored in Elasticsearch.

By using the CData JDBC Driver for SAP SuccessFactors with Logstash, it functions as a SAP SuccessFactors connector, making it easy to load data into Elasticsearch. Please try the 30-day free trial.