How to work with Spotify Data in Apache Spark using SQL

Apache Spark is a fast and general engine for large-scale data processing. When paired with the CData JDBC Driver for Spotify, Spark can work with live Spotify data. This article describes how to connect to and query Spotify data from a Spark shell.

The CData JDBC Driver offers unmatched performance for interacting with live Spotify data due to optimized data processing built into the driver. When you issue complex SQL queries to Spotify, the driver pushes supported SQL operations, like filters and aggregations, directly to Spotify and utilizes the embedded SQL engine to process unsupported operations (often SQL functions and JOIN operations) client-side. With built-in dynamic metadata querying, you can work with and analyze Spotify data using native data types.

Install the CData JDBC Driver for Spotify

Download the CData JDBC Driver for Spotify installer, unzip the package, and run the JAR file to install the driver.

Start a Spark Shell and Connect to Spotify Data

1. Open a terminal and start the Spark shell with the CData JDBC Driver for Spotify JAR file as the jars parameter:

   $ spark-shell --jars /CData/CData JDBC Driver for Spotify/lib/cdata.jdbc.api.jar
   

2. With the shell running, you can connect to Spotify with a JDBC URL and use the SQL Context load() function to read a table.

   Using OAuth Authentication

   Spotify uses OAuth 2.0 for authentication. You will need to create an application in the Spotify Developer Dashboard to obtain your client credentials.

   Setting Up Your Spotify Application

   1. Visit the Spotify Developer Dashboard.
   2. Log in with your Spotify account and click Create app.
   3. Provide an app name, description, and set a Redirect URI (e.g.,

      http://localhost:33333

      for desktop applications).
   4. Copy your Client ID and Client Secret from the app settings.

   Connection Properties

   After setting the following connection properties, you are ready to connect:

   • AuthScheme: Set this to OAuth.
   • InitiateOAuth: Set this to GETANDREFRESH. You can use InitiateOAuth to manage the process to obtain the OAuthAccessToken.
   • OAuthClientId: Set this to your Spotify application's Client ID.
   • OAuthClientSecret: Set this to your Spotify application's Client Secret.
   • Scope: Set this to the required OAuth scopes (space-separated). The default includes all read scopes needed for the tables in this profile.
   • CallbackURL: Set this to the Redirect URI configured in your Spotify application (e.g., http://localhost:33333).

   Example Connection String

   Profile=C:\profiles\Spotify.apip;AuthScheme=OAuth;InitiateOAuth=GETANDREFRESH;OAuthClientId=your_client_id;OAuthClientSecret=your_client_secret;CallbackURL=http://localhost:33333;
   

   Available OAuth Scopes

   • user-read-private: Read access to user's subscription details and explicit content settings.
   • user-read-email: Read access to user's email address.
   • user-library-read: Read access to a user's saved tracks, albums, episodes, shows, and audiobooks.
   • playlist-read-private: Read access to user's private playlists.
   • playlist-read-collaborative: Read access to collaborative playlists the user follows.
   • user-follow-read: Read access to the list of artists the current user follows.
   • user-read-playback-state: Read access to a user's player state (device, current track, progress).
   • user-read-currently-playing: Read access to a user's currently playing content.
   • user-read-playback-history: Read access to a user's recently played tracks.
   • user-top-read: Read access to a user's top artists and tracks.

   Built-in Connection String Designer

   For assistance in constructing the JDBC URL, use the connection string designer built into the Spotify JDBC Driver. Either double-click the JAR file or execute the jar file from the command-line.

   java -jar cdata.jdbc.api.jar
   

   Fill in the connection properties and copy the connection string to the clipboard.

   

   Configure the connection to Spotify, using the connection string generated above.

   scala> val api_df = spark.sqlContext.read.format("jdbc").option("url", "jdbc:api:Profile=C:\profiles\Spotify.apip;AuthScheme=OAuth;InitiateOAuth=GETANDREFRESH;OAuthClientId=your_client_id;OAuthClientSecret=your_client_secret;CallbackURL=http://localhost:33333;").option("dbtable","Albums").option("driver","cdata.jdbc.api.APIDriver").load()
   

3. Once you connect and the data is loaded you will see the table schema displayed.

4. Register the Spotify data as a temporary table:

   scala> api_df.registerTable("albums")

5. Perform custom SQL queries against the Data using commands like the one below:

   scala> api_df.sqlContext.sql("SELECT ,  FROM Albums WHERE Id = 4aawyAB9vmqN3uQ7FjRGTy").collect.foreach(println)

   You will see the results displayed in the console, similar to the following:

   

Using the CData JDBC Driver for Spotify in Apache Spark, you are able to perform fast and complex analytics on Spotify data, combining the power and utility of Spark with your data. Download a free, 30 day trial of any of the hundreds of CData JDBC Drivers and get started today.