The Databricks Connection in BlueConic can enrich profiles and timeline events using data from Databricks, and it can also export BlueConic data into your Databricks workspace. With this release, BlueConic supports Databricks Delta Sharing as a direct, secure, and efficient mechanism for exchanging live tables without managing storage credentials, export jobs, or intermediary cloud storage. The integration uses Databricks Direct Shares and secure recipient access to provide a streamlined setup process and ongoing, governed data flow.
Note: This feature can be used for profile imports and exports and timeline data import and export. Profiles can be created or updated, and timeline events can be added (not updated)
The new connection bridges BlueConic’s rich, actionable customer profiles with Databricks’ computational power. This connection enables customers to:
Seamlessly sync profile and timeline data with Databricks (import and export).
Power advanced modeling, segmentation, and activation workflows.
Strengthen collaboration between marketing, data science, and data engineering teams.
Reduce custom integration work and improve data governance through secure Delta Sharing protocols.
Before you begin
Enable Delta Sharing:
In your Databricks UI, go to the admin console for your account and navigate to Catalog > Metastore. Enable Allow Delta Sharing with parties outside your organization.
Ensure that the tables you want to share in Databricks have Change Tracking enabled. Delta Sharing relies on change data capabilities to detect incremental updates.
In your Databricks workspace, open a table, navigate to the Data details, and confirm that Change Data Feed is turned on. If it is not enabled, set the property enableChangeDataFeed to true for each table you want to share:
ALTER TABLE <table_name> SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
Note: If you don't see the tables listed in the Import goal, double-check if Change tracking is enabled on each table and view.
Adding a Databricks Connection
Choose Connections in the Navigation bar.
Click Add Connection.
Search for "Databricks" in the Search bar and choose the Databricks connection.
The Databricks connection page opens.
You can expand or collapse metadata fields (favorite, labels, description) by clicking the gray chevron at the top of the page.
Enter a descriptive name for your connection.
Optional: Select the Email notifications box if you would like BlueConic to notify one or more email addresses if there's a problem running the connection or when it runs successfully. Enter one or more email addresses and click OK. You can choose whether to receive an email each time it runs or only if it fails.
Configure Delta Sharing
Prepare your Connection:
Open the Set up and run page of your connection.
Choose your Databricks cloud platform and geographic region in the BlueConic UI.
Note: Currently only AWS is supported, so you can select the appropriate AWS region from the dropdown.
Set up import using Delta Sharing
After the cloud platform and region are defined, BlueConic displays a Share Identifier and a BlueConic Metastore Identifier. You will use these values in Databricks when creating the share.
In your Databricks account, to create a share using Delta Sharing, navigate to Catalog > Delta Sharing > Shared by me > Shares > Share Data
In the next step, add the Share Identifier provided in the BlueConic UI to the share name.
Select a schema and the tables that you want to share with BlueConic.
Add Blueconic as a recipient to the share:
For the just-created share, add BlueConic as a recipient.
Navigation path to add a recipient to a share - Catalog > Delta Sharing > Add Recipient > Create New Recipient.
Copy BlueConic’s Account ID as shown in the BlueConic UI.
Add BlueConic account as a recipient to the share.
In BlueConic, click the Validate configuration button to test that everything works correctly.
When validated, the button will show a checkmark.
Note: If you make any changes to the configuration, the button will reactivate.
Set up export using Delta Sharing
Once the cloud platform and region are defined, if you would like to export data from BlueConic to your Databricks account, do the following steps:
Enter your Databricks Metastore ID in the “Your Databricks Account” textbox; the BlueConic share identifier is automatically populated.
To find you Metastore ID - In your Databricks workspace, you can use the SQL query:
SELECT CURRENT_METASTORE();
Click the “Validate configuration” button in the BlueConic UI.
Set up Import goals
Import goals allow you to import data from Databricks into BlueConic profile properties or timeline events, and /or create new profiles within BlueConic. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your import. Click away, and the name will update.
Select a BlueConic domain group (If applicable)
If you are using domain groups , you need to select a BlueConic domain group for the import and choose the appropriate domain group from the dropdown menu.
Note: This setting is visible only when additional domain groups exist beyond the default.Select Databricks data to import from
Select your schema and tables or views that you want to import data from. You can search for specific tables or views. Nested files are supported.Link identifiers between Databricks and BlueConic
Select the Databricks field and BlueConic profile property that will be used to match incoming Databricks records to existing BlueConic profiles.Determine whether creating new BlueConic profiles is allowed. Activate the checkbox to create a BlueConic profile when an incoming record is processed and no matching profile is found.
Map Databricks data to BlueConic
Determine how fields from Databricks are mapped to BlueConic profile properties. Add mapping rules by clicking Add mapping.
Configure each mapping by following the steps below:
Select the Databricks field.
Select the BlueConic profile property it should map to.
Choose how the value should be imported:
Set – always set the value
Set if empty – only set the value if the profile does not yet have a value
Add – add another value to a list property
Sum – add a numerical value to the existing value in the profile.
(Optional) Use Bulk mapping to add all detected fields at once.
Optional) Add a prefix to BlueConic properties for easier organization.
Optional: Import Timeline events into BlueConic.
You can import data from Databricks fields to enrich BlueConic timeline events. Map the correct Databricks fields into the corresponding BlueConic Timeline event properties.Save your settings and run the connection
Save your configuration and run the Databricks connection to start importing data into BlueConic.
Set up export goals
Export goals allow you to export profile property and timeline event data from BlueConic into Databricks. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your export. Click away and the name will update.Select a BlueConic segment to export profile data from
Select the BlueConic segment that contains the profiles that are allowed to be exported. Only identifiable profiles in this segment will be exported. In the top right-hand corner, the number of profiles available for export appears. This number shows how many profiles meet the conditions for the selected segment (e.g., they have the required identifier and the appropriate consent) and are therefore available for export.Select the Databricks location to export to
BlueConic automatically creates a database for the share. Select the schema and table for the exported files within this share.Link identifiers between BlueConic and Databricks
The connection uses the BlueConic Profile Identifier as the predefined identifier on both sides to match BlueConic profiles with records in Databricks. This identifier is fixed and cannot be changed.
You can optionally allow the creation of new records in Databricks when no matching identifier is found in Databricks.Map BlueConic data to Databricks
Determine how BlueConic profile properties will be mapped into Databricks.
Add as many mapping rules as needed by clicking Add mapping. For each mapping Select the type(s) of information to match and export:Profile property: The value of a specific profile property.
BlueConic profile identifier: The unique identifier for a profile.
Associated segments: All segments, or a selection of one or more specific segments that the profile is associated with.
Associated lifecycles: All lifecycles this profile is associated with.
All viewed interactions (all time): The interactions that the profile has seen.
Permission level: The permission level set in the profile.
Text value: Enter any static text.
Save your settings and run the connection
Save your configuration and run the Databricks connection to export profiles.
Note: To see the data exported, note the region displayed in the 'Import Data' section of the Setup and Run page in BlueConic, switch your location to that same region in Databricks and search for the BlueConic Share Identifier/table name created while configuring the export in BlueConic.
Run the connection
Activate the goal
Check the goal you want to activate. We recommend testing import and export goals separately.
The green arrow indicates the active direction for the connection.
Click Save.
Run on a Schedule
Click the Settings icon in the run history step.
Choose an option from the drop-down list to configure by:
Number of times per day
Days of the week
Days of the month
Weekday of the month
Select the time of day if required.
Click OK.
Click Save.
Make sure the connection is on.
Use Databricks Connections with BlueConic Lifecycles
You can add Databricks export goals as touchpoints in BlueConic Lifecycles. Here are some key points to consider:
Exporting Lifecycle stage information to Databricks: When you add a Databricks export goal to a BlueConic Lifecycle, you can use that export goal to send a list of profiles that attain a certain lifecycle stage along their customer journeys and use this information for campaigns.
Setting Lifecycle stage restrictions: You can restrict a connection goal to export only the profiles that are currently part of a certain lifecycle stage.
Privacy management
Connections can be added to Objectives, allowing for privacy management of the information being collected. A connection will only process the profiles of visitors who have consented to at least one objective linked to it.