Kafka Connection

How to connect BlueConic customer profile data with Apache Kafka topics and messagesWhat: The BlueConic Kafka Connection lets you exchange customer profile data between BlueConic profiles and Apache Kafka topics. 

About BlueConic: The BlueConic customer data platform harnesses the data required to power the recognition of an individual at each interaction, and then synchronizes their intent across the marketing ecosystem.

Why: By exchanging profile data and timeline events from BlueConic to Kafka topics, you can update BlueConic profiles with data from Apache Kafka.

For information on accessing the BlueConic Kafka connection, contact your BlueConic Customer Success Manager.

How BlueConic connects to Kafka

Apache Kafka is a streaming platform, and messages can be posted to a topic. For example, for a topic called "subscriptions," if a subscription changes, a message is posted to the Kafka topic about the change. The BlueConic connection can listen to the message of this topic, with the message serving as input for an import, based on the import mapping and matching rules you specify in an import goal. A message can also be translated to a BlueConic Timeline event (for example, adding a subscription event to the BlueConic timeline).

Importing message from Kafka: During configuration, you can add an example message to let BlueConic know upfront the message format to expect from Kafka. Based on the example message you add, dropdown menus in the mapping and matching steps pre-populate with your format. You can optionally use a restriction step to clarify how to treat different messages posted from the topic: for example: message 1 to event of type X, message 2 to event of type Y. The BlueConic Kafka connection runs as a batch connection, which can run as often as every 3 minutes.

Exporting profile data to Kafka: Exports from BlueConic to Kafka can run either in batch via the Run command, or in real time. For real-time exports, BlueConic can send a message to a Kafka topic when a profile property in BlueConic changes (as defined in the export goal's mappings).

Creating a BlueConic Kafka Connection

For information on accessing the Kafka connection, contact your BlueConic Customer Success Manager.

  1. Click Connections in the BlueConic navigation bar.
  2. Click Add Connection.
  3. A pop-up window appears. Enter "Kafka" in the Search bar.
  4. Click Kafka connection.
    The Kafka connection page opens. You can expand or collapse the metadata fields (to mark it as a favorite, add labels, or write a description) by clicking the gray + or -.
  5. Enter a name for your connection at the top of the page, and save your settings.

Configuring the Kafka Connection in BlueConic

To set up a connection between BlueConic and Kafka, you need to add your Kafka account credentials you're connecting to. Then you create at least one import or export goal, turn the connection on, and either schedule or run the connection. Contact your Kafka account team for the information to enter in the steps outlined below.

Setting up a Kafka connection in BlueConic

  1. Select Set up and run in the left panel.
  2. Enter your Kafka account credentials: brokers, group ID, mechanism, username, and password.
    Contact your Kafka account team if you need help gathering this information.
  3. Save your connection settings at the top of the page.

Your connection should not show an error, which means that BlueConic can now exchange customer data with Kafka.

To make the connection work, you have to add at least one import or export goal, save your settings, and switch the connection on at the top of the page before you run it. In the goals, you specify how data from BlueConic should be linked and mapped to Kafka topics.

After you run the connection, the Data exchange section shows the number of unique profiles exchanging data with Kafka. See Running the connection for details.

Importing Kafka messages to BlueConic

In the bottom left panel, click Add goal > Import profiles to create an import goal or click Import data into BlueConic on the left panel to set import goals for the connection.

  1. Select the Kafka topic name you're importing messages from.
  2. Add an example JSON message to be sent from Kafka to BlueConic.
  3. Process Kafka data (optional).
    BlueConic-data-processors.png
    Select a data processor to transform or filter the Kafka data during the import.
    For example, you can use a processor to normalize email addresses, filter data fields, or transform data during the import, before it is matched and mapped to BlueConic properties. To add data processors to your BlueConic environment, contact your Customer Success Manager. Learn more about using data processors to filter or transform data imports.
  4. Define the restriction to be used when mapping data with this import goal.
    This optional step is necessary if multiple types of messages are posted to the same topic. 
  5. Link identifiers between Kafka messages and BlueConic profiles properties.
    Select a Kafka value on the left and link it to a BlueConic profile identifiers. Data without matching identifiers will not be imported.
    Link-profile-identifiers-between-Kafka-BlueConic.png
    You can optionally choose to create a new BlueConic profile when a matching profile is not found.
  6. Map Kafka data fields to BlueConic profile properties.
    Use the Add mapping button to add additional import fields.
    You can select how to import the data from the Set drop-down menu:
    • Set
    • Set if empty
    • Set or clear the value in the profile property if the CSV contains no value for this field.
    • Add the data field to the list of existing values.
    • Sum a number with the existing values (if the data field is a number).Import Timeline events into BlueConic.
  7. Optional: Map Kafka fields to import data into BlueConic Timeline events.
    Kafka-import-timeline-event-type.png

    You can choose to import customer order data to BlueConic using either the default BlueConic Order event type, or a custom Timeline event type you have already created with a plugin XML file. (Contact your BlueConic Customer Success Manager at support@blueconic.com for details.) You can map order details in the left column to order event properties, event date/time, and the event ID. 
    Use the Add mapping button if you want to extend the mapping for each order to include additional, nested Order event details, such as additional products.
    Use the Add a Timeline event type button to add additional event mappings.
    Learn more about BlueConic Timeline events.
  8. Run the connection.
    Be sure to turn the connection on and save your settings before running the connection.
    See Running the connection for details.

Exporting data from BlueConic to Kafka topics

In the bottom left panel, click Add goal > Export profiles to create an export goal or click Export profile data to Kafka on the left panel to set export goals for the connection.

  1. Select a BlueConic customer segment to export profile data from.
    Only profiles from this segment will have data exported.
  2. Select the Kafka topic name you're exporting data to.
  3. Map BlueConic data to the corresponding Kafka fields.
    You can use the Add mapping button to add additional data mappings.
    Mapping_BlueConic_exported_data_to_Kafka.png
  4. Preview messages BlueConic will send to Kafka.
    Be sure to turn the connection on and save your settings for the export to be active.

Running the connection

To run the connection, turn the connection on at the top of the page and save your changes. Use the settings in the Run history section of the Set up and run page. See Scheduling connections for details on running connections.

Connection status

Once you've set up and saved your connection, the icon at the top of the window reflects the status of the connection.

Two arrows run between BlueConic to Kafka, signifying the direction of the data import and export. When the arrows are green, the connection is ready to be used.

Privacy management

Connections can be added to Objectives, allowing for privacy and consent management of the information that is being picked up. A connection will only process the profiles of visitors who have consented to at least one of the objectives that the connection is linked to.