The Databricks Connection in BlueConic can enrich profiles, timeline events and/or groups using data from Databricks, and it can also export BlueConic data into your Databricks workspace. With this release, BlueConic supports Databricks Delta Sharing as a direct, secure, and efficient mechanism for exchanging live tables without managing storage credentials, export jobs, or intermediary cloud storage. The integration uses Databricks Direct Shares and secure recipient access to provide a streamlined setup process and ongoing, governed data flow.
This feature supports
Batch profile import and export (Up to 6 times per day)
Batch group import and export (Up to 6 times per day)
Batch timeline event import and export (Up to 6 times per day)
Delta-based data processing
The new connection bridges BlueConic’s rich, actionable customer profiles with Databricks’ computational power. This connection enables customers to:
Seamlessly sync profile and timeline data with Databricks (import and export).
Power advanced modeling, segmentation, and activation workflows.
Strengthen collaboration between marketing, data science, and data engineering teams.
Reduce custom integration work and improve data governance through secure Delta Sharing protocols.
Note: To enable Delta Sharing for Databricks, Databricks has been added as a subprocessor under your existing BlueConic agreement. For more details, you can review the full explanation click here.
Before you begin
Databricks Delta Sharing is a paid upgrade.
Interested? Contact your BlueConic Customer Success Manager, Account Manager, or BlueConic Support team to enable Databricks Delta Sharing and begin activating your customer data more securely and efficiently. Learn more at this link.To use the Databricks Connection, a feature flag must be enabled for your tenant. Your CSM can enable this for you, allowing you to set up and run the connection.
Enable Delta Sharing:
In your Databricks UI, go to the admin console for your account and navigate to Catalog > Metastore. Enable Allow Delta Sharing with parties outside your organization.
Ensure that the tables you want to share in Databricks have Change Tracking enabled. Delta Sharing relies on change data capabilities to detect incremental updates.
In your Databricks workspace, open a table, navigate to the Data details, and confirm that Change Data Feed is turned on. If it is not enabled, set the property enableChangeDataFeed to true for each table you want to share:
ALTER TABLE <table_name> SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
Note: If you don't see the tables listed in the Import goal, double-check if Change tracking is enabled on each table and view.
Adding a Databricks Connection
Choose Connections in the Navigation bar.
Click Add Connection.
Search for "Databricks" in the Search bar and choose the Databricks connection.
The Databricks connection page opens.
You can expand or collapse metadata fields (favorite, labels, description) by clicking the gray chevron at the top of the page.
Enter a descriptive name for your connection.
Optional: Select the Email notifications box if you would like BlueConic to notify one or more email addresses if there's a problem running the connection or when it runs successfully. Enter one or more email addresses and click OK. You can choose whether to receive an email each time it runs or only if it fails.
Configure Delta Sharing
Open the Set up and run page of your connection.
BlueConic currently uses AWS as its cloud platform for Delta Sharing, but import and export are supported for AWS, Azure, and Google Cloud. Choose the AWS region where you want BlueConic to host data, ideally matching your Databricks workspace region to avoid data egress charges.
Set up import using Delta Sharing
After the region is defined, BlueConic displays a Share Identifier and a BlueConic Metastore Identifier. You will use these values in Databricks when creating the share.
In your Databricks account, to create a share using Delta Sharing, navigate to Catalog > Delta Sharing > Shared by me > Shares > Share Data
In the next step, add the Share Identifier provided in the BlueConic UI to the share name.
Select a schema and the tables that you want to share with BlueConic.
Add BlueConic as a recipient to the share:
For the just-created share, add BlueConic as a recipient.
Navigation path to add a recipient to a share - Catalog > Delta Sharing > Add Recipient > Create New Recipient.
Copy BlueConic’s Account ID as shown in the BlueConic UI.
Add BlueConic account as a recipient to the share.
In BlueConic, click the Validate configuration button to test that everything works correctly.
When validated, the button will show a checkmark.
Note: If you make any changes to the configuration, the button will reactivate.
Configure batch group import
Import group goals allow you to import data from Databricks into BlueConic group properties and/or create new groups To do this, complete the steps that are outlined in the goal.
1. Optional: Name your goal
You can edit the name to describe your import. Click away, and the name will update.
2. Select Databricks data to import from
Select your schema and tables or views that you want to import data from. You can search for specific tables or views. Nested files are supported.
3. Link identifiers between Databricks and BlueConic.
To match group data between the two systems, enter the identifier used in Databricks. Use the "Allow creation..." checkbox to allow BlueConic to create new groups for data that does not match an existing BlueConic group.
4. Map Databricks data into BlueConic group properties.
Match Databricks fields on the left with BlueConic group properties on the right. Use the dropdown at the right to select how imported content should be added to existing values:
Set
Set if empty (if the group property is empty)
Set or clear (if the Databricks field is empty, allow to clear)
Add the data field to the list of existing values
Sum a number with the existing values (if the data field is a number).
Use the Add mapping button to create additional mapping rows. You can add multiple mapping rows at once with the "Add remaining fields" option. BlueConic detects the remaining Databricks fields and matches them with BlueConic group properties. You can set the merge strategy for imported field (set, set if empty, set or clear, add, sum) and also provide a custom prefix to the names of BlueConic profile properties (optional).
5. Save your settings and run the connection
Save your configuration and run the Databricks connection to start importing data into BlueConic.
Configure batch group share
Share group goals allow you to export group data from BlueConic into Databricks. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your export. Click away and the name will update.Select the Databricks location to export to
BlueConic automatically creates a database for the share. Select the schema and table for the exported files within this share.
Link identifiers between BlueConic and Databricks
Here, the identifiers between BlueConic groups and Databricks fields are matched. It is not editable. You can optionally choose to let BlueConic create new records in Databricks when no existing match is found.
Map BlueConic data to Databricks
Determine how BlueConic group properties will be mapped into Databricks.Save your settings and run the connection
Save your configuration and run the Databricks connection to export groups.
Note: To see the data exported, note the region displayed in the 'Import Data' section of the Setup and Run page in BlueConic, switch your location to that same region in Databricks and search for the BlueConic Share Identifier/table name created while configuring the export in BlueConic.
Set up export using Delta Sharing
Once the cloud platform and region are defined, if you would like to export data from BlueConic to your Databricks account, do the following steps:
Enter your Databricks Metastore ID in the “Your Databricks Account” textbox; the BlueConic share identifier is automatically populated.
To find your Metastore ID - In your Databricks workspace, you can use the SQL query:
SELECT CURRENT_METASTORE();
Click the “Validate configuration” button in the BlueConic UI.
Configure batch profile import
Import profile goals allow you to import data from Databricks into BlueConic profile properties and/or create new profiles. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your import. Click away, and the name will update.
Select a BlueConic domain group (If applicable)
If you are using domain groups , you need to select a BlueConic domain group for the import and choose the appropriate domain group from the dropdown menu.
Note: This setting is visible only when additional domain groups exist beyond the default.Select Databricks data to import from
Select your schema and tables or views that you want to import data from. You can search for specific tables or views. Nested files are supported.Link identifiers between Databricks and BlueConic
Select the Databricks field and BlueConic profile property that will be used to match incoming Databricks records to existing BlueConic profiles.Determine whether creating new BlueConic profiles is allowed. Activate the checkbox to create a BlueConic profile when an incoming record is processed and no matching profile is found.
Map Databricks data to BlueConic
Determine how fields from Databricks are mapped to BlueConic profile properties. Add mapping rules by clicking Add mapping.
Configure each mapping by following the steps below:
Select the Databricks field.
Select the BlueConic profile property it should map to.
Choose how the value should be imported:
Set – always set the value
Set if empty – only set the value if the profile does not yet have a value
Add – add another value to a list property
Sum – add a numerical value to the existing value in the profile.
(Optional) Use Bulk mapping to add all detected fields at once.
Optional) Add a prefix to BlueConic properties for easier organization.
Optional: Import Timeline events into BlueConic.
You can import data from Databricks fields to enrich BlueConic timeline events. Map the correct Databricks fields into the corresponding BlueConic Timeline event properties.Save your settings and run the connection
Save your configuration and run the Databricks connection to start importing data into BlueConic.
Configure batch profile share
Share profile goals allow you to export profile property and timeline event data from BlueConic into Databricks. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your export. Click away and the name will update.Select a BlueConic segment to export profile data from
Select the BlueConic segment that contains the profiles that are allowed to be exported. Only identifiable profiles in this segment will be exported. In the top right-hand corner, the number of profiles available for export appears. This number shows how many profiles meet the conditions for the selected segment (e.g., they have the required identifier and the appropriate consent) and are therefore available for export.Select the Databricks location to export to
BlueConic automatically creates a database for the share. Select the schema and table for the exported files within this share.Link identifiers between BlueConic and Databricks
The connection uses the BlueConic Profile Identifier as the predefined identifier on both sides to match BlueConic profiles with records in Databricks. This identifier is fixed and cannot be changed.
You can optionally allow the creation of new records in Databricks when no matching identifier is found in Databricks.Map BlueConic data to Databricks
Determine how BlueConic profile properties will be mapped into Databricks.
Add as many mapping rules as needed by clicking Add mapping. For each mapping Select the type(s) of information to match and export:Profile property: The value of a specific profile property.
BlueConic profile identifier: The unique identifier for a profile.
Associated segments: All segments, or a selection of one or more specific segments that the profile is associated with.
Associated lifecycles: All lifecycles this profile is associated with.
All viewed interactions (all time): The interactions that the profile has seen.
Permission level: The permission level set in the profile.
Text value: Enter any static text.
Save your settings and run the connection
Save your configuration and run the Databricks connection to export profiles.
Note: To see the data exported, note the region displayed in the 'Import Data' section of the Setup and Run page in BlueConic, switch your location to that same region in Databricks and search for the BlueConic Share Identifier/table name created while configuring the export in BlueConic.
Run the connection
Activate the goal
Check the goal you want to activate. We recommend testing import and export goals separately.
The green arrow indicates the active direction for the connection.
Click Save.
Run on a Schedule
Click the Settings icon in the run history step.
Choose an option from the drop-down list to configure by:
Number of times per day
Days of the week
Days of the month
Weekday of the month
Select the time of day if required.
Click OK.
Click Save.
Make sure the connection is on.
Use Databricks connections with BlueConic Lifecycles
You can add Databricks profile share goals as touchpoints in BlueConic Lifecycles. Here are some key points to consider:
Exporting Lifecycle stage information to Databricks: When you add a Databricks export goal to a BlueConic Lifecycle, you can use that export goal to send a list of profiles that attain a certain lifecycle stage along their customer journeys and use this information for campaigns.
Setting Lifecycle stage restrictions: You can restrict a connection goal to export only the profiles that are currently part of a certain lifecycle stage.