The Databricks Connection in BlueConic can enrich profiles, timeline events and/or groups using data from Databricks, and it can also export BlueConic data into your Databricks workspace. With this connection BlueConic currently uses AWS. BlueConic supports Databricks Delta Sharing as a direct, secure, and efficient mechanism for exchanging live tables without managing storage credentials, export jobs, or intermediary cloud storage. The integration uses Databricks Direct Shares and secure recipient access to provide a streamlined setup process and ongoing, governed data flow.
This feature supports
Batch profile import and export (Up to 6 times per day)
Batch group import and export (Up to 6 times per day)
Batch timeline event import and export (Up to 6 times per day)
Delta-based data processing
The new connection bridges BlueConic’s rich, actionable customer profiles with Databricks’ computational power. This connection enables customers to:
Seamlessly sync profile and timeline data with Databricks (import and export).
Power advanced modeling, segmentation, and activation workflows.
Strengthen collaboration between marketing, data science, and data engineering teams.
Reduce custom integration work and improve data governance through secure Delta Sharing protocols.
Note: To enable Delta Sharing for Databricks, Databricks has been added as a subprocessor under your existing BlueConic agreement. For more details, you can review the full explanation click here.
Before you begin
Databricks Delta Sharing is a paid upgrade.
Interested? Contact your BlueConic Customer Success Manager, Account Manager, or BlueConic Support team to enable Databricks Delta Sharing and begin activating your customer data more securely and efficiently. Learn more at this link.To use the Databricks Connection, a feature flag must be enabled for your tenant. Your CSM can enable this for you, allowing you to set up and run the connection.
Enable Delta Sharing:
In your Databricks UI, go to the admin console for your account and navigate to Catalog > Metastore. Enable Allow Delta Sharing with parties outside your organization.
Ensure that the tables you want to share in Databricks have Change Tracking enabled. Delta Sharing relies on change data capabilities to detect incremental updates.
In your Databricks workspace, open a table, navigate to the Data details, and confirm that Change Data Feed is turned on. If it is not enabled, set the property enableChangeDataFeed to true for each table you want to share:
ALTER TABLE <table_name> SET TBLPROPERTIES (delta.enableChangeDataFeed = true)
Databricks Delta tables use two properties to control how long the history needed for change tracking is kept:
delta.deletedFileRetentionDuration(default: 7 days) — how far back Change Data Feed can lookdelta.logRetentionDuration(default: 30 days) — how long the transaction log is kept
If your connection runs infrequently: for example, weekly or monthly, then the default retention can expire before the next run. When that happens, BlueConic can no longer read the change history and will perform a full run instead of an incremental one.
To avoid this, increase both properties to match or exceed your run interval:
ALTER TABLE <Schema>.<Table>SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 30 days','delta.logRetentionDuration' = 'interval 30 days');Check current values with:SHOW TBLPROPERTIES <Schema>.<Table>;
Note: Longer retention increases storage costs, since Databricks keeps more historical file versions and log entries.
If you don't see the tables listed in the Import goal, double-check if Change tracking is enabled on each table and view.
Adding a Databricks Connection
Choose Connections in the Navigation bar.
Click Add Connection.
Search for "Databricks" in the Search bar and choose the Databricks connection.
The Databricks connection page opens.
You can expand or collapse metadata fields (favorite, labels, description) by clicking the gray chevron at the top of the page.
Enter a descriptive name for your connection.
Optional: Select the Email notifications box if you would like BlueConic to notify one or more email addresses if there's a problem running the connection or when it runs successfully. Enter one or more email addresses and click OK. You can choose whether to receive an email each time it runs or only if it fails.
Configure Delta Sharing
Open the Set up and run page of your connection.
Choose the AWS region for BlueConic's Databricks environment, which acts as the pass-through that makes your data available between BlueConic and your Databricks workspace. This is independent of the cloud platform your own workspace runs on (AWS, Azure, or Google Cloud). Choosing a region close to — or matching — your workspace's region can help minimize data-transfer latency.
Explained
BlueConic hosts its Delta Sharing service on AWS. This is fixed — it's not something you configure. What you do choose is which AWS region BlueConic's Databricks environment uses as the pass-through for your shared data.
Your own Databricks workspace can run on any cloud platform — AWS, Azure, or Google Cloud. Delta Sharing works across cloud providers, so this choice is independent of BlueConic's AWS hosting.
For best performance, we recommend selecting the AWS region closest to (or matching, if you're also on AWS) your Databricks workspace's region.
For example, if your Databricks workspace runs on Azure West Europe, choose an AWS region in the same geography, such as eu-west-1 (Ireland) or eu-central-1 (Frankfurt), rather than a US-based AWS region.
Set up import using Delta Sharing
Once the cloud platform and region are configured, if you want to import data, follow these steps:
After the region is defined, BlueConic displays a Share Identifier and a BlueConic Metastore Identifier. You will use these values in Databricks when creating the share.
In your Databricks account, to create a share using Delta Sharing, navigate to Catalog > Delta Sharing > Shared by me > Shares > Share Data.
In the next step, add the Share Identifier provided in the BlueConic UI to the share name.
Select a schema and the tables that you want to share with BlueConic.
Add BlueConic as a recipient to the share:
For the just-created share, add BlueConic as a recipient.
In Databricks, go to: Catalog > Delta Sharing > Add Recipient > Create New Recipient.
Copy BlueConic’s Account ID as shown in the BlueConic UI.
Add BlueConic account as a recipient to the share.
In BlueConic, click the Validate import configuration button to test that everything works correctly.
When validated, the button will show a check mark.
Note: If you make any changes to the Delta Sharing configuration on this connection's Set up and run page, the Validate Configuration button will reactivate — you'll need to validate the configuration again before the connection can run.
Set up export using Delta Sharing
Once the cloud platform and region are configured, if you want to export data, follow these steps:
Enter your Databricks Metastore ID in the “Your Databricks Account” text box; the BlueConic share identifier is automatically populated.
To find your Metastore ID - In your Databricks workspace, you can use the SQL query:
SELECT CURRENT_METASTORE();
Click the “Validate share configuration” button in the BlueConic UI.
Configure batch group import
Import group goals allow you to import data from Databricks into BlueConic group properties and/or create new groups. To do this, complete the steps that are outlined in the goal.
1. Optional: Name your goal
You can edit the name to describe your import. Click away, and the name will update.
2. Select Databricks data to import from
Select your schema and tables or views that you want to import data from. You can search for specific tables or views. Nested files are supported.
3. Link identifiers between Databricks and BlueConic.
To match group data between the two systems, enter the identifier used in Databricks. Use the "Allow creation..." checkbox to allow BlueConic to create new groups for data that does not match an existing BlueConic group.
4. Map Databricks data into BlueConic group properties.
Match Databricks fields on the left with BlueConic group properties on the right. Use the dropdown at the right to select how imported content should be added to existing values:
Set
Set if empty (if the group property is empty)
Set or clear (if the Databricks field is empty, allow to clear)
Add the data field to the list of existing values
Sum a number with the existing values (if the data field is a number).
Use the Add mapping button to create additional mapping rows. You can add multiple mapping rows at once with the "Add remaining fields" option. BlueConic detects the remaining Databricks fields and matches them with BlueConic group properties. You can set the merge strategy for imported field (set, set if empty, set or clear, add, sum) and also provide a custom prefix to the names of BlueConic profile properties (optional).
5. Save your settings and run the connection
Save your configuration and run the Databricks connection to start importing data into BlueConic.
Configure batch profile import
Import profile goals allow you to import data from Databricks into BlueConic profile properties and/or create new profiles. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your import. Click away, and the name will update.
Select a BlueConic domain group (If applicable)
If you are using domain groups , you need to select a BlueConic domain group for the import and choose the appropriate domain group from the dropdown menu.
Note: This setting is visible only when additional domain groups exist beyond the default.Select Databricks data to import from
Select your schema and tables or views that you want to import data from. You can search for specific tables or views. Nested files are supported.Link identifiers between Databricks and BlueConic
Select the Databricks field and BlueConic profile property that will be used to match incoming Databricks records to existing BlueConic profiles.Determine whether creating new BlueConic profiles is allowed. Activate the checkbox to create a BlueConic profile when an incoming record is processed and no matching profile is found.
Map Databricks data to BlueConic
Determine how fields from Databricks are mapped to BlueConic profile properties. Add mapping rules by clicking Add mapping.
Configure each mapping by following the steps below:
Select the Databricks field.
Select the BlueConic profile property it should map to.
Choose how the value should be imported:
Set – always set the value
Set if empty – only set the value if the profile does not yet have a value
Add – add another value to a list property
Sum – add a numerical value to the existing value in the profile.
(Optional) Use Bulk mapping to add all detected fields at once.
Optional) Add a prefix to BlueConic properties for easier organization.
Optional: Import Timeline events into BlueConic.
You can import data from Databricks fields to enrich BlueConic timeline events. Map the correct Databricks fields into the corresponding BlueConic Timeline event properties.Save your settings and run the connection
Save your configuration and run the Databricks connection to start importing data into BlueConic.
Configure batch group share
Share group goals allow you to export group data from BlueConic into Databricks. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your export. Click away and the name will update.Select the Databricks location to export to
BlueConic automatically creates a database for the share. Select the schema and table for the exported files within this share.
Link identifiers between BlueConic and Databricks
Here, the identifiers between BlueConic groups and Databricks fields are matched. It is not editable. You can optionally choose to let BlueConic create new records in Databricks when no existing match is found.Map BlueConic data to Databricks
Determine how BlueConic group properties will be mapped into Databricks.Save your settings and run the connection
Save your configuration and run the Databricks connection to export groups.
Note: To see the data exported, note the region displayed in the 'Import Data' section of the Setup and Run page in BlueConic, switch your location to that same region in Databricks and search for the BlueConic Share Identifier/table name created while configuring the export in BlueConic.
Configure batch group export
Share profile goals allow you to export profile property and timeline event data from BlueConic into Databricks. To do this, complete the steps that are outlined in the goal.
Optional: Name your goal
You can edit the name to describe your export. Click away and the name will update.Select a BlueConic segment to export profile data from
Select the BlueConic segment that contains the profiles that are allowed to be exported. Only identifiable profiles in this segment will be exported. In the top right-hand corner, the number of profiles available for export appears. This number shows how many profiles meet the conditions for the selected segment (e.g., they have the required identifier and the appropriate consent) and are therefore available for export.Select the Databricks location to export to
BlueConic automatically creates a database for the share. Select the schema and table for the exported files within this share.Link identifiers between BlueConic and Databricks
The connection uses the BlueConic Profile Identifier as the predefined identifier on both sides to match BlueConic profiles with records in Databricks. This identifier is fixed and cannot be changed.
You can optionally allow the creation of new records in Databricks when no matching identifier is found in Databricks.Map BlueConic data to Databricks
Determine how BlueConic profile properties will be mapped into Databricks.
Add as many mapping rules as needed by clicking Add mapping. For each mapping Select the type(s) of information to match and export:Profile property: The value of a specific profile property.
BlueConic profile identifier: The unique identifier for a profile.
Associated segments: All segments, or a selection of one or more specific segments that the profile is associated with.
Associated lifecycles: All lifecycles this profile is associated with.
All viewed interactions (all time): The interactions that the profile has seen.
Permission level: The permission level set in the profile.
Text value: Enter any static text.
Save your settings and run the connection
Save your configuration and run the Databricks connection to export profiles.
Note: To see the data exported, note the region displayed in the 'Import Data' section of the Setup and Run page in BlueConic, switch your location to that same region in Databricks and search for the BlueConic Share Identifier/table name created while configuring the export in BlueConic.
Run the connection
Activate the goal
Make sure the goals you want to run are enabled (checked). We recommend testing import and export goals separately.
The green arrow indicates the active direction for the connection.
Click Save.
Run on a Schedule
Click the Settings icon in the run history step.
Choose an option from the drop-down list to configure by:
Number of times per day
Days of the week
Days of the month
Weekday of the month
Select the time of day if required.
Click OK.
Click Save.
Make sure the connection is on.
Use Databricks connections with BlueConic Lifecycles
You can add Databricks profile share goals as touchpoints in BlueConic Lifecycles. Here are some key points to consider:
Exporting Lifecycle stage information to Databricks: When you add a Databricks export goal to a BlueConic Lifecycle, you can use that export goal to send a list of profiles that attain a certain lifecycle stage along their customer journeys and use this information for campaigns.
Setting Lifecycle stage restrictions: You can restrict a connection goal to export only the profiles that are currently part of a certain lifecycle stage.
Privacy management
Connections can be added to Objectives, allowing for privacy management of the information that is being picked up. A connection will only process the profiles of visitors who have consented to at least one of the objectives that the connection is linked to.
FAQ
Is Delta Sharing mandatory?
Yes. BlueConic supports Databricks through Delta Sharing only.
How does Delta Sharing work?
BlueConic reads the Databricks datasets you authorize through a governed, zero-copy share between your Databricks environment and a Databricks instance used by BlueConic. No credentials, pipelines, or file transfers are required at that layer. BlueConic then ingests the data into your tenant for activation and analysis. Read more about Delta Sharing on the Databricks website.
Why do I need to enable Change Data Feed on my Databricks tables?
Delta Sharing relies on change tracking to detect incremental updates. If Change Data Feed isn't enabled on a table, BlueConic can't read its change history, and the table won't show up as available in your import goal.
Why is Delta Sharing a paid feature?
When you use Delta Sharing, BlueConic runs and pays for the Databricks compute needed to process your import and export runs — this isn't compute drawn from your own Databricks environment or billed to your Databricks account. Since BlueConic covers those compute costs on its side, Delta Sharing is offered as a paid add-on rather than being included by default.
Is a free trial available?
Yes, a free trial of Databricks Delta Sharing is available before committing to a paid plan.
Do I need anything enabled before I can set up the connection?
Yes — a feature flag must be turned on for your tenant. Your Customer Success Manager can manage this for you.
Are real-time imports supported?
No, the connection support batch runs, up to 6 times a day.
What happens if my connection runs less often than my table's retention settings allow?
Databricks Delta tables retain the history required for incremental processing for a limited time. By default, obsolete data files become eligible for deletion after 7 days, while transaction-log history is retained for 30 days.
If the time between successful connection runs exceeds the available history—for example, if the connection runs monthly with the default retention settings—BlueConic might no longer be able to read all changes since the previous run. In that case, BlueConic performs a full run instead of an incremental run.
To avoid this, configure both retention settings to exceed the longest expected interval between successful runs, including enough time for delays and retries:
1. ALTERTABLE <schema>.<table>
2. SET TBLPROPERTIES (
3. 'delta.deletedFileRetentionDuration' = 'interval 35 days',
4. 'delta.logRetentionDuration' = 'interval 35 days'
5 );
Increasing retention can increase storage usage.
Why does cross-cloud Delta Sharing work between BlueConic (AWS) and my Databricks environment (Azure or Google Cloud)?
Delta Sharing is an open, cloud-independent protocol designed to share data across platforms and cloud providers. The provider and recipient therefore do not need to use the same cloud.
For data shared by BlueConic:
BlueConic makes the selected tables available through Delta Sharing from its AWS environment.
Your Databricks environment receives access to the share using the configured recipient and metastore identifiers.
When Databricks reads the shared tables, the Delta Sharing service authorizes the request and provides temporary access to the required data files.
The data is transferred securely to the recipient’s compute environment without requiring it to be permanently re-hosted in an intermediate cloud location.
Cross-cloud data transfer and associated egress charges can still apply. Where possible, select a BlueConic AWS region close to your Databricks environment. For more information see Delta Sharing protocol documentation.
Is Delta Sharing secure when BlueConic and my Databricks environment are on different clouds?
Yes. Delta Sharing applies the same authorization and transport-security model across cloud providers:
Controlled access: Only configured recipients can access the tables included in a share.
Temporary data access: Authorized reads use short-lived, scoped access to the required data files rather than permanent public URLs.
Revocable sharing: BlueConic can change or revoke access to the share.
Encryption in transit: Requests and data transfers use HTTPS.
No required intermediate copy: Data can be read directly from the provider’s storage without first copying it to a shared staging location.
Cloud-independent security: The authorization model remains the same whether the provider and recipient use the same cloud or different clouds.
Once data has been read into the recipient’s environment, the recipient may cache, process, or persist it. Those copies are governed by the recipient’s Databricks and cloud-security controls.
How is my data kept separate from other customers' data in BlueConic's Databricks environment?
BlueConic runs its Databricks environment under one overall Databricks account. Within that account, each customer's data is kept separate:
Dedicated environments: If your BlueConic tenant runs on a dedicated environment, you get your own dedicated Databricks workspace.
Shared environments: If your BlueConic tenant runs on a shared environment, you share a Databricks workspace with other customers on that environment. Your data is still securely isolated at database level and logically separated from other customers' data.
Either way, data is isolated per tenant. Other customers can't see or reach your shared data.
Can I control which profiles get exported to Databricks?
Yes. When exporting profile data, you select a BlueConic segment — only profiles in that segment (that also have the required identifier and consent) are exported.
Does the connection respect visitor privacy and consent settings?
Yes. Like other BlueConic connections, it can be linked to Objectives, so it only processes profiles that have consented to at least one linked objective.
Are data processors supported for imports or exports?
No, processor support for profile and group import/export is not currently supported.