What: The Kafka Connection lets you listen to Kafka topics and store the results in BlueConic for segmentation, targeting, reporting, or data activation.
About BlueConic: The BlueConic customer data platform harnesses the data required to power the recognition of an individual at each interaction, and then synchronizes their intent across the marketing ecosystem.
Why: By importing data from Kafka topics, you can update BlueConic profiles with data from Apache Kafka.
How BlueConic connects to Kafka
Apache Kafka is a streaming platform, and messages can be posted to a topic. For example, for a topic called "subscriptions," if a subscription changes, a message is posted to the Kafka topic about the change. The BlueConic connection can listen to the message of this topic, with the message serving as input for an import, based on the import mapping and matching rules you specify in an import goal. A message can also be translated to a BlueConic Timeline event (for example, adding a subscription event to the BlueConic timeline).
Importing message from Kafka: During configuration, you can add an example message to let BlueConic know upfront the message format to expect from Kafka. Based on the example message you add, dropdown menus in the mapping and matching steps pre-populate with your format. You can optionally use a restriction step to clarify how to treat different messages posted from the topic: for example: message 1 to event of type X, message 2 to event of type Y. The BlueConic Kafka connection runs as a batch connection, which can run as often as every 3 minutes.
Creating a BlueConic Kafka Connection
- Click Connections in the BlueConic navigation bar.
- Click Add Connection.
- A pop-up window appears. Enter "Kafka" in the Search bar.
- Click Kafka connection.
The Kafka connection page opens. You can expand or collapse the metadata fields (to mark it as a favorite, add labels, or write a description) by clicking the gray chevron at the top of the page. - Enter a name for your connection at the top of the page, and save your settings.
Configuring a Kafka Connection in BlueConic
To set up a connection between BlueConic and Kafka, you need to add your Kafka account credentials you're connecting to. Then you create at least one import goal, turn the connection on, and either schedule or run the connection. Contact your Kafka account team for the information to enter in the steps outlined below.
Setting up a Kafka connection in BlueConic
- Select Set up and run in the left panel.
- Enter your Kafka account credentials: brokers, group ID, and mechanism.
Contact your Kafka account team if you need help gathering this information. - Optional: Use a self-signed certificate for Kafka authentication. When this option is enabled, you will have to upload an RSA private key and certificate key and set your passphrase.
- Specify the date format and time zone used in the Kafka messages.
- Save your connection settings at the top of the page.
Your connection should not show an error, which means that BlueConic can now import customer data from Kafka.
To make the connection work, you have to add at least one import goal, save your settings, and switch the connection on at the top of the page before you run it. In the import goal, you specify how data from BlueConic should be linked and mapped to Kafka topics.
After you run the connection, the Data exchange section shows the number of unique profiles exchanging data with Kafka. See Running the connection for details.
Importing groups into BlueConic
In the bottom left panel, click Add goal > Import groups to create an import goal or click Import groups into BlueConic on the left panel to set import goals for the connection.
- Select a BlueConic group type to import data into.
- Select the Kafka topic name you're importing messages from.
- Add an example JSON message to be sent from Kafka to BlueConic.
- Optionally add a data processor to transform or filter data being imported.
Contact your Customer Success Manager if you need data processors installed.
Learn more about BlueConic data processors. - Link identifiers between Kafka messages and BlueConic groups.
Select a Kafka value on the left and link it to a BlueConic profile identifiers. Data without matching identifiers will not be imported. - Map Kafka data fields to BlueConic groups.
Use the Add mapping button to add additional import fields.
You can select how to import the data from the Set drop-down menu:- Set
- Set if empty
- Set or clear the value in the profile property if the message contains no value for this field.
- Add the data field to the list of existing values.
- Sum a number with the existing values (if the data field is a number).
-
Run the connection.
Be sure to turn the connection on and save your settings before running the connection.
See Running the connection for details.
Importing profiles into BlueConic
In the bottom left panel, click Add goal > Import profiles to create an import goal or click Import profiles into BlueConic on the left panel to set import goals for the connection.
- Select the Kafka topic name you're importing messages from.
- Add an example JSON message to be sent from Kafka to BlueConic.
- Process Kafka data (optional).
Select a data processor to transform or filter the Kafka data during the import.
For example, you can use a processor to normalize email addresses, filter data fields, or transform data during the import, before it is matched and mapped to BlueConic properties. To add data processors to your BlueConic environment, contact your Customer Success Manager. Learn more about using data processors to filter or transform data imports. - Define the restriction to be used when mapping data with this import goal.
This optional step is necessary if multiple types of messages are posted to the same topic. - Link identifiers between Kafka messages and BlueConic profiles properties.
Select Kafka values on the left and link them to BlueConic profile properties on the right. Data without matching identifiers will not be imported.
You can optionally choose to create a new BlueConic profile when a matching profile is not found. - Map Kafka data fields to BlueConic profile properties.
Use the Add mapping button to add additional import fields.
You can select how to import the data from the Set drop-down menu:- Set
- Set if empty
- Set or clear the value in the profile property if the message contains no value for this field.
- Add the data field to the list of existing values.
- Sum a number with the existing values (if the data field is a number).
- Optional: Map Kafka fields to import data into BlueConic Timeline events.
You can choose to import customer order data to BlueConic using either the default BlueConic Order event type, or a custom Timeline event type you have already created with a plugin XML file. (Contact your BlueConic Customer Success Manager at support@blueconic.com for details.) You can map order details in the left column to order event properties, event date/time, and the event ID.
Use the Add mapping button if you want to extend the mapping for each order to include additional, nested Order event details, such as additional products.
Use the Add a Timeline event type button to add additional event mappings.
Learn more about BlueConic Timeline events. - Run the connection.
Be sure to turn the connection on and save your settings before running the connection.
See Running the connection for details.
Running the connection
To run the connection, turn the connection on at the top of the page and save your changes. Use the settings in the Run history section of the Set up and run page. See Scheduling connections for details on running connections.
Connection status
Once you've set up and saved your connection, the icon at the top of the window reflects the status of the connection.
An arrow runs between BlueConic to Kafka, signifying the direction of the data import. When the arrow is green, the connection is ready to be used.
Privacy management
Connections can be added to Objectives, allowing for privacy and consent management of the information that is being picked up. A connection will only process the profiles of visitors who have consented to at least one of the objectives that the connection is linked to.