Databricks integration

This guide walks you through setting up a new Databricks integration.

What you need to know

To enable the Databricks integration in TheyDo, a Service Principal needs to be created. This requires either Account Admin or Workspace Admin privileges, depending on how the environment is set up.

Think of a Service Principal as a non-human, machine account, which we use for secure, programmatic access (M2M auth) More about this.

Authentication is based on OAuth 2.0

This connection uses OAuth 2.0b. Please ensure your Databricks account has access to sql and all-apis scopes.

• sql — to list and query warehouses

• all-apis — to support general API operations (authentication, metadata, etc.)

Make sure your Databricks role grants these permissions. If unsure, contact your workspace admin.

Creating metrics from Databricks

Now that your instance is connected, you can start creating metrics:

1. Go to Metrics in TheyDo or directly from your journey “Add a metric”

2. Choose Databricks as the data source.

3. Select the Warehouse to connect to

   

4. Choose the type of metric (e.g., Other).

5. Provide a name (e.g., Amount of Incoming Calls).

6. Enter your SQL query using the connected Databricks table.

7. Save and run the metric. (please note: if no data is available in the last 30days, you can click “Show latest data” to retrieve latest synchronized datapoints)

How to prepare your data in Databricks?

To ensure seamless data usage within TheyDo, it is important that data fetched and imported from Snowflake is formatted correctly. Each table queried must return specific values that correspond to the type of metric being analyzed. Below are the requirements for each metric type.

• CSAT

  Required columns: date, positives, negatives, respondents

• CES

  Required columns: date, respondents , value

• NPS

  Required columns: date, detractors, promoters, respondents

• RATIO

  Required columns: date, numerator, denominator

• OTHER

  Required columns:date, value

Data types for integration

For the integration to function correctly and for your data to be processed without errors, it is essential to adhere to specific data type requirements for each column in your Snowflake tables. Here’s a breakdown of the supported data types:

Date columns
Required data type: DATE

Metric Value columns
Required data type: INTEGER | FLOAT | NUMERIC

All other columns that represent metric values (such as positives, negatives, respondents, value, detractors, promoters, numerator, and denominator) must use one of the supported Databricks numeric data types.

Aliasing columns for compatibility

The original column names in your Databricks tables do not need to match the column names expected by TheyDo. You can use SQL aliasing to map your table columns to the required fields. For example, for a metric of type NPS, if your table has a column for detractors called det you can alias it in your query as follows:

SELECT det AS detractors, ... FROM project_id.dataset_id.your_table_name

This flexibility allows you to maintain your original database schema while ensuring compatibility with the TheyDo integration.