We are proud to share our inclusion in the 2024 Gartner Magic Quadrant for Data Integration Tools. We believe this recognition reflects the differentiated business outcomes CData delivers to our customers.
Get the Report →Creating Spark-Connected Applications in Unqork
You can use CData Connect Cloud to feed Spark data to Unqork and build custom business apps.
Overview
You can use CData Connect Cloud to feed Spark data to Unqork. From there you can build custom business applications in a visual, drag-and-drop interface. So, how does this process work?
Unqork is a completely visual, no-code application development platform that helps large organizations build complex custom software. CData Connect Cloud exposes Spark as a REST endpoint, which you can then query from Unqork with simple HTTP requests. This enables you to create your own custom application with Spark data.
What You'll Learn
In this article, you'll learn how to:
- Add a CData Connect Cloud user
- Add a Personal Access Token for the Connect Cloud user
- Connect to Spark from Connect Cloud
- Add Spark OData endpoints in Connect Cloud
- Register Spark (via Connect Cloud) as on Unqork service
- Create a simple Unqork application to display Spark data
Connect to Spark from Unqork
To work with live Spark data in Unqork, you need to:
- Connect to Spark from CData Connect Cloud
- Provide user access to the new connection in CData Connect Cloud
- Create OData endpoints for Spark data
(Optional) Add a New Connect Cloud User
As needed, create Users to connect to Spark through Connect Cloud.
- Navigate to the Users page and click Invite Users
- Enter the new user's email address and click Send to invite the user
- You can review and edit users from the Users page
Add a Personal Access Token
If you are connecting from a service, application, platform, or framework that does not support OAuth authentication, you can create a Personal Access Token (PAT) to use for authentication. Best practices would dictate that you create a separate PAT for each service, to maintain granularity of access.
- Click on your username at the top right of the Connect Cloud app and click User Profile.
- On the User Profile page, scroll down to the Personal Access Tokens section and click Create PAT.
- Give your PAT a name and click Create.
- The personal access token is only visible at creation, so be sure to copy it and store it securely for future use.
Connect to Spark from Connect Cloud
CData Connect Cloud uses a straightforward, point-and-click interface to connect to data sources.
- Log into Connect Cloud, click Connections and click Add Connection
- Select "Spark" from the Add Connection panel
-
Enter the necessary authentication properties to connect to Spark.
Set the Server, Database, User, and Password connection properties to connect to SparkSQL.
- Click Create & Test
- Navigate to the Permissions tab in the Add Spark Connection page and update the User-based permissions.
Configure Spark Endpoints for Unqork
After connecting to Spark, create a workspace and virtual dataset for your desired table(s).
- Navigate to the Virtual Datasets page and click Add to create a new Workspace (or select an existing workspace).
- Click Add to add new assets to the Workspace.
- Select the Spark connection (e.g. SparkSQL1) and click Next.
- Select the table(s) you wish to work with and click Confirm.
- Make note of the OData Service URL for your workspace, e.g. https://cloud.cdata.com/api/odata/{workspace_name}
Sample URLs for Workspaces
Once you have configured a connection to Spark, created a user, and created Workspace(s) in Connect Cloud, you can access OData feeds for Spark data. Below, you will see the URLs to access tables and the list of tables. For information on accessing the tables, you can navigate to the OData page for Connect Cloud. For the URLs, you will need the URL of Connect Cloud: https://cloud.cdata.com/api/odata/{workspace_name}.
Table | URL | |
---|---|---|
Entity (table) List | https://cloud.cdata.com/api/odata/{workspace_name}/ | |
Metadata | https://cloud.cdata.com/api/odata/{workspace_name}/$metadata | |
Customers | https://cloud.cdata.com/api/odata/{workspace_name}/Customers |
As with standard OData feeds, if you wish to limit the fields returned, you can add a $select parameter to the query, along with other standard OData URL parameters, such as $filter, $orderby, $skip, and $top. See the help documentation for more information on supported OData queries.
With the connection and OData endpoints configured, you are ready to connect to Spark data from Unqork.
Create a Simple Application for Spark Data
With the REST API configured for Spark through Connect Cloud, you are ready to create a Spark-connected application in Unqork. Here are the basic steps to this process:
- Register Spark (via Connect Cloud) as an Unqork service
- Create a workspace in Unqork
- Create a module-type application
Register Spark (via Connect Cloud) as an Unqork Service
To connect to Connect Cloud, you need to give your environment permission to connect with an external service. Do this through Services Administration.
- Click the Settings drop-down at the top right corner of the Unqork Designer Platform.
- Click Administration.
- Under Integrations, click Services Administration.
- Enter a Service Title, for example, Connect Cloud.
- Enter a Service Name, for example, cdataConnect.
- Enter a Service protocol + host. This is the API endpoint you want to reference. For example, https://cloud.cdata.com/api/odata/{workspace_name}.
- Select Basic Auth from the Type of Authentications drop-down.
- Enter the Connect Cloud user ID (e.g. [email protected]) in the User Name field.
- Enter the Connect Cloud user PAT in the Password field.
- Click Add Service.
Create a Workspace
Unqork lets you create your own workspace, which is a virtual container for you and your team to store and organize your applications. Once you build a workspace, you can add applications to it.
To create a workspace:
- Click Create Workspace
- Enter a unique Workspace Name
- Click Create
Your workspace tile appears under the Workspaces tab.
Create a Module-Type Application
With your workspace added, you are ready to create your application. You have the choice between a module-type and workflow-type application. For this use case, make a module-type application.
- Navigate to the Apps tab from your open workspace.
- Click Create App.
- Enter a unique App Name.
- Select a preview style from the Style drop-down. A style specifies your application's appearance to your and-users. If left blank, your application uses the environment's default style.
- Select Module from the App Type drop-down.
- Select New Entrypoint Module from the App Entrypoint Module drop-down.
- Click Create.
You will land on the Modules tab of your new application. It should look like the following:
Click the title to name the module in the Module Editor.
To complete your application, you need the following components:
- Hidden component: stores the data retrieved by the Plug-In
- Initializer component: executes the Plug-In when displaying tables
- Plug-In component: retrieves Spark data through Connect Cloud and stores it in the Hidden component
- ViewGrid component: creates on HTML table from the data stored in the Hidden component
Configure the Hidden Component
This Hidden component stores the Spark data retrieved through Connect Cloud.
- Drag and drop a Hidden component onto your canvas.
- Enter a Property ID and Canvas Label Text.
NOTE: Property IDs must be in Camel Case (stylized as camelCase). For example, hiddenComponent. Since the Hidden component is not visible to and-users, use your Property ID as the Canvas Label Text.
- Click Save.
Configure the Plug-in Component
Next, add a Plug-In component to request data from Connect Cloud. This component connects with the service you set up in Services Administration.
- Drag and drop a Plug-In component onto your canvas, placing it below the Hidden component.
- Enter a Property ID and Canvas Label Text.
NOTE: You must use Camel Case (stylized as camelCase). For naming best practices, start your Plug-In's Property ID with plugin. For example, pluginGetAWProducts.
- Configure the Outputs table as follows:
- Enter the Property ID of your Hidden component in the Property ID column.
- Enter value in the Mapping column.
- Select External as the Service Type
- From the External Services drop-down, select your Connect Cloud service. This is the external service you set up in Services Administration.
- Add the specific endpoint for your Connect Cloud resources to the and of the Data Sources URL. For example, AdventureWorks2012_Production_Product.
NOTE: Your Data Sources URL will have a value similar to: https://cloud.cdata.com/api/odata/{workspace_name}/AdventureWorks2012_Production_Product.
- Click Save.
Configure the Initializer Component
The Initializer component is what fires the Plug-In to retrieve your Spark data.
- Drag and drop on Initializer component onto your canvas, placing it above the Plug-In component.
- Enter a Property ID and Canvas Label Text.
NOTE: You must use Camel Case (stylized as camelCase). For naming best practices, start your Initializer's Property Name with init. For example, initGetAWProducts.
- Set New Submission as the Trigger Type.
- In the Outputs table, enter the following:
- Enter the Property ID of your Plug-In in the Property ID column's field.
- Enter trigger in the Type column's field.
- Enter GO in the Value column's field.
- Click Save.
Configure the ViewGrid Component
The ViewGrid component references the data stored in your Hidden component. Then, it displays the Spark data to your and-users in a dashboard (or table).
- Drag and drop a ViewGrid component onto your canvas, placing it below the Plug-In component.
- Enter a Label and Property Name.
NOTE: You must use Camel Case (stylized as camelCase). For naming best practices, start your ViewGrid's Property Name with vg. For example, vgAWProductView.
- In the Inputs table, set id to the Property ID of the hidden component.
- In the Display table:
- In the id column: enter the IDs of the fields you want to display.
- In the handing column: enter the title you want for each field. This is how the field will display to your and-users.
- Click Save.
- Save your module.
Once you add and configure the above components, you can preview how your application looks to and-users. In the Module Editor, click Preview Module to see the front-facing view (or "Express View") of the module.
Your Express View module should look something like this:
More Information & Free Trial
With Connect Cloud and Unqork, you can easily build Spark-connected applications. Request a free trial of Connect Cloud and start working with Spark data in Unqork today.