Using Databricks to Connect Data

Overview

Databricks is a cloud-based platform that lets you store, organize, and analyze your data using SQL and other tools. By connecting Databricks to SundaySky, you can use your existing viewer data to personalize videos for each video at scale.

When using Databricks, your data is never stored on SundaySky servers. Instead, each time a viewer clicks to watch a video, SundaySky retrieves the relevant records from Databricks in real time using SQL queries to generate a personalized video, ensuring the use of the most up-to-date data.

Each video link created for a viewer is unique and contains a "sticky" ID. This means that if you update the viewer data in Databricks or make changes to the video content, the video updates automatically. There's no need to resend links as viewers will always see the latest version when they revisit their original link.

In this article, you'll find complete guidelines for working with the Databricks data connector, from setting up a data library through to generating video links for your viewers.

Step 1: Select or Create a Data Library

Based on how you want to personalize your video, you'll need to connect the video to a data library that includes the data fields for creating the personalization. SundaySky supports three methods of personalization:

Before creating a new data library, review the existing data libraries in your account to determine whether one of them already meets your requirements. The same data library can be used for multiple videos. If you choose to use an existing data library, it must meet all of the following conditions:

• It contains the data fields you need for personalization.
• It is connected to the Databricks data connector.
• The Databricks integration points to the database containing your viewers' data.

If you find a data library with the required data fields but it is connected to a data source other than Databricks, do not change its data connector. Doing so could negatively impact any videos already using this library. In this case, you will need to create a new data library instead.

For instructions, see:
Selecting an Existing Data Library
Creating a New Data Library

Step 2: Set Up the Databricks Database

In this step, you'll set up—or make sure you have ready—the Databricks schema (database) and table that store the data used for personalization. Once these components are in place and your Databricks warehouse is running, you can connect the schema and table to SundaySky so it can access the data for video personalization.

Below is an example of the Databricks interface displaying the new_employees table in the my_database schema, showing all column headers with no data entries yet.
For your convenience, a file with this SQL code, My Database_New Employees.txt, is available for download at the end of the article.


 

Here are the key points to know about the table you create in Databricks:

• As a best practice, create a table exclusively for SundaySky. This keeps all fields needed for personalization in one place and avoids including viewer data that isn't required. This approach helps keep your data organized and focused, without exposing unnecessary information.
   
• Include a unique identifier for each viewer. The table must have a field that uniquely identifies each viewer. In the example above, the field employee_id serves this purpose, but any field can be used as long as each value is unique. SundaySky uses this identifier to match each viewer to their record and personalize the video accordingly. A unique video link is generated for each record.
   
• All other fields (except the unique identifier) are optional. If a value is missing in any of them, SundaySky will use the default value for the corresponding content in the video.

Step 3: Select the Databricks Data Connector

The connector is what enables the connection between the data fields defined in the data library and the actual viewer data stored in Databricks. In simple terms, the connector is what determines how data flows from Databricks to SundaySky.

a.	Open the relevant video in the Studio.
b.	Select Data in the sidebar.
c.	Click Select in the data connector section.
d.	Select the Databricks data connector and then click Select at the bottom of the window.
	A dialog window is displayed, prompting you to enter your Databricks database details. ▶ If you haven't yet integrated Databricks with your SundaySky account, you'll need to do so now. See Step 4: Integrate Databricks with SundaySky for step-by-step instructions.
f.	From the drop-down lists in the dialog window, select the Databricks warehouse, catalog, schema, and table that contain the viewer data you want to use for personalization. ▶ Make sure you select the correct combination so SundaySky can access the appropriate records.
g.	In the ID field, select the column in the Databricks table that serves as the unique identifier.
h.	Click Create & map Data Fields to continue directly to field mapping. ▶ If you prefer mapping at a later time, click Complete authentication and map Data Fields later to return to the data library.

Step 4: Integrate Databricks with SundaySky

Here are the essential things to know about integrating Databricks with SundaySky:

• Before you begin the authentication process, make sure the secret associated with your service principal is available. To avoid disruptions when integrating with SundaySky, we recommend setting the secret's lifetime to the maximum of 730 days.
  
  Also, make a note of the secret's expiration date. Before it expires, generate a new secret and reauthenticate from SundaySky using the new secret to ensure there's no interruption in video personalization.
   
  Show me where to generate the secret

  
• The Databricks integration only needs to be configured once. After it's set up, you can use Databricks as a data source with any data library in your account.
   
• Each SundaySky account can be integrated with only one Databricks account.
   
• Integrating with SundaySky involves connecting to a specific database. If you need to change it later, see Changing the Databricks Database.
   
• If you need to reauthenticate or remove the Databricks connector, see Data Connector Management for instructions. Keep in mind that reauthenticating will break the current connection to SundaySky in all videos (draft, approved, and live), and those videos will not be personalized until you integrate Databricks again.

To integrate Databricks with SundaySky:

a.	Select the Databricks data connector and then click Select at the bottom of the window.
b.	Click Log in.
c.	Click Begin authentication.
d.	In the Databricks instance field, paste the instance URL you copied from Databricks.
	Show me where to find my Databricks instance After signing in to your Databricks account, copy the section of the URL shown below. This section represents your Databricks instance.
e.	In the Client Id field, paste the Application (client) ID from your Databricks Service principals page.
	Show me where to find my Client Id
f.	In the Client Secret field, enter the secret from your Databricks account.
g.	Click Create.
	A green check mark indicates that you are successfully logged in to your Databricks account.
h.	From the drop-down lists in the dialog window, select the Databricks warehouse, catalog, schema, and table that contain the viewer data you want to use for personalization. ▶ Make sure you select the correct combination so SundaySky can access the appropriate records.
i.	In the ID field, select the column in the Databricks table that serves as the unique identifier.
j.	Click Create & map Data Fields to continue directly to field mapping. ▶ If you prefer mapping at a later time, click Complete authentication and map Data Fields later to return to the data library.

Step 5: Map the Data Fields

The mapping process defines which field in your Databricks table corresponds to each data field in the SundaySky data library. Accurate mapping ensures that the correct data is pulled from Databricks to populate each personalized video.

Keep in mind:

• If you add a new data field after completing the initial mapping, you'll need to map it as well.
• Renaming a data field that has already been mapped does not affect the existing mapping.

a.	Open the relevant video in the Studio.
b.	Select Data in the sidebar.
c.	Click Map Fields in the data connector section.
	In the mapping window, the data fields created for the data library are displayed on the right. For each of these data fields, you'll need to select the Databricks field to which it should be mapped.
d.	For each data field, open the Databricks Field drop-down list and select the field to map it to. ▶ The drop-down list also includes the identifier field you created in your Databricks table (for example, employee_id). This field does not need to be mapped.
e.	After all the fields are mapped, click Save at the bottom of the window.

Step 6: Configure the Content

Now that you've set up your data library, you can configure the content in the video's scenes using the data fields.

The on-screen text, narration, and media can be personalized using either a personalization token or audience messaging. In addition, you can also define that a specific scene will be shown or skipped according to an audience segment.

In the example below, a personalization token is used to customize the text according to the first name of the viewer.

In the following articles, you'll find step-by-step instructions on how to configure the different types of content:

Setting Up Personalized Text Using a Personalization Token
Text: Message by Audience

Setting Up Personalized Narration Using a Personalization Token
Narration: Message by Audience

Configuring Media According to Data Feed
Media: Message by Audience

Showing or Skipping a Scene by Audience

Step 7: Approve the Video

After you finish creating the video—including configuring all personalized content—we recommend previewing it with viewer profiles to ensure everything looks as intended.

Once you're satisfied with the result, approve the video to lock in your edits. Approval is always required before you can share the video with viewers.

Keep in mind that you'll need to reapprove the video each time you make changes so viewers can see the updated content.

Step 8: Get Video Links

In this final step, you're ready to generate the video links so that each viewer can watch a personalized version. The actions in this step vary depending on whether the video will play on a SundaySky landing page or elsewhere.

Generating Video Links for a SundaySky Landing Page
Generating Video Links for a Non-SundaySky Landing Page

As mentioned earlier, each video link is created with a "sticky" ID that is unique to each viewer. Because of this, you don't need to resend links after updating the video content or the viewer's data. Viewers can simply revisit the original link to see the updated content.

Generating Video Links for a SundaySky Landing Page

a.	On the Video Page, select the Share tab.
b.	Make sure that the status of the video is Approved for sharing.
c.	On the right side of the page, click the Share icon next to the Landing Page option. ▶ If you haven't created the landing page yet, click the plus sign (+) to create the landing page.
d.	Copy and paste the landing page URL into your distribution platform.
e.	Configure your distribution platform to replace INSERT_VALUE with the appropriate unique identifier value used for personalization. For example: https://myvideo.sundaysky.com/?programId=8cea01ea-9192-4977-a8e1-f869f348a15e&id=1

Generating Video Links for a Non-SundaySky Landing Page

a.	On the Video Page, select the Share tab.
b.	Make sure that the status of the video is Approved for sharing.
c.	On the right side of the page, click the Share icon next to the Embed Video option. ▶ If you haven't created the code snippet yet, click the plus sign (+) to create the snippet.
d.	Select the embed method you want to use, iframe or JavaScript. ▶ By default, the iframe option is selected. ▶ To learn more about iframe and JavaScript embed codes, see Embedding the SundaySky Player in a Webpage.
e.	Copy and paste the embed code snippet into the page code where you want the video player to appear.
f.	Copy the required parameters and add them to the end of the landing page URL.
g.	Configure your distribution platform to replace INSERT_VALUE with the appropriate unique identifier value used for personalization. For example: https://video.YourWebsite.com/?programId=8cea01ea-9192-4977-a8e1-f869f348a15e&id=1

Changing the Databricks Database

If you change the Databricks database (schema) connected to SundaySky, you'll also need to update the associated catalog, table, and ID field. These elements define how viewer data is accessed, so they must stay aligned with the new database.

Changing a database affects only draft videos. Videos that are already live will continue playing with the previous data connection until they are reapproved. Once reapproved, they will use the updated database settings.

After changing the database, you'll need to remap all the SundaySky data fields to the new Databricks fields.

To change the database:

a.	Open the relevant video in the Studio.
b.	Select Data in the sidebar.
c.	Click the Options menu in the upper-right corner of the window.
d.	Select Update Databricks database.
e.	In the dialog window, update any fields you want to change: Warehouse, Catalog, Schema (database), Table, and ID field.
f.	Click Save updates.
g.	Remap the data fields according to the new connection.