Help Center

Adding an Azure Data Lake Storage connection

1. Click Connections in the Navigation bar.
2. Click Add Connection.
3. A pop-up window appears. Check the Show all box. Enter “Azure Data Lake Storage” in the Search bar. Click on the Azure Data Lake Storage Connection.
4. The Azure Data Lake Storage Connection page opens.

   You can expand or collapse metadata fields (mark this connection as a favorite, add labels, provide a custom description, or change edit rights) by clicking the gray chevron at the top of the page.

5. Enter a descriptive name for your connection.

6. Optional: Select the Email notifications box if you would like BlueConic to notify one or more email addresses if there's a problem running the connection or when it runs successfully. Enter one or more email addresses and click OK. You can select whether to receive emails each time it runs or only if the connection fails to run successfully.

Configuring an Azure Data Lake Storage connection

After you create a connection in BlueConic, you need to authenticate the connection to the Azure server and set up at least one import or export goal. In the connection goals, you specify the data you are exchanging and how it maps to BlueConic profiles. You can also prioritize the order of your goals by dragging and dropping them to your desired sequence by using the drag handle that appears at the left-hand side of a goal when hovering over it.

The connection will then import and export data based on the order of goals within each category. Categories will always run in the order listed - import groups, import profiles, export profiles.

Authenticating the Azure Data Lake Storage connection

1. Click Set up and run in the left panel.
2. In the Set up and run page, enter your Azure Data Lake Storage credentials, including your:
   • Tenant ID - Enter your Azure tenant ID that can be found in your Azure portal.
   • Client ID - Enter your Azure client ID. To receive a client ID, you must register your client application with an Azure AD tenant in your Azure portal. Once you use Azure AD to authorize access to storage resources, you will be provided with a client ID. For more information, refer to Azure documentation.
   • Client secret - Enter your Azure client secret. You can create a new access key in the Azure portal by going to Certificates & secrets > Client secrets > and clicking the “New client secret” button. For more information, refer to Azure documentation.
   • Storage account - Enter your Azure storage account name. 
3. Save your settings.

Using PGP encryption for file transfers

For added security while transferring customer data with Azure Data Lake Storage, you can use PGP encryption for file imports and exports. To exchange data with PGP, you need to provide a private key to be able to import the data and a public key to export data to the Azure server. The keys are securely stored.

1. In the setup and authentication section, in the Encryption field, enable the Use PGP encryption option. A settings menu opens below for Decryption and/or encryption.
2. Enter the Private key (for decrypting files), Public key (for encrypting files), and optionally a Passphrase.
   

How selecting PGP encryption affects the connection setup process

When you enable PGP encryption, files will be encrypted securely, so you may notice several subtle changes to the connection setup process:

• Source preview files will not be available.
• Fields for the mappings will not be found.
• Bulk mapping will not be available.
• You cannot prefill mappings in the import and export goals.

If you prefer to use the bulk mapping feature or field selection feature, you can set up the import and export goals first using a file with identical fields that is not encrypted, before you enable PGP encryption, so mappings are found and filled more easily.

Setting up a Group Import goal

To import data into BlueConic groups, such as households, accounts, companies, etc., click Add goal > Import groups and follow the steps outlined in the goal. The groups import option is available in the connection once you've set up a BlueConic group.

1. Select a group type to import data to a BlueConic group.
   Use the drop-down menu to select from the group types defined for your BlueConic tenant.
   
2. Select Azure Data Lake Storage source files to be imported.
   
   

   Enter the directory that holds the files or click to navigate the file structure.
   Optional: Specify the files to be imported. You can use regular expressions or wildcards to choose the files.

   Note: If you use a wildcard, make sure the value of that wildcard matches your root file and all nested files. For instance, if example_course_20*.csv is used for your root file, and this results in example_course_20244.csv being selected, then your nested files must contain the wildcard value of 244 in their filenames to avoid an error.
3. Optional: Correct your files before the import.

   

   Optionally add one or more data processors to transform or filter the data in your CSV file, to ensure that BlueConic imports the data correctly. To add data processors to your BlueConic environment, contact your Customer Success Manager. Read more about the available data processors.
4. Define file and field handling.
   Here you can customize how your CSV files and fields should be handled during the import. See the options for configuring CSV file and field imports for details.
5. Link identifiers between Azure Data Lake Storage files and BlueConic groups.
   Data that has no matching identifier will be excluded from the import.
   
6. Map imported fields to BlueConic group properties.
   Here, you can configure which values should be mapped to which BlueConic group properties:
   • Add mapping: Select a data field from the drop-down list, and select the BlueConic group property it populates by entering a search term.
   • Bulk mapping: If there are many fields to be mapped, BlueConic detects these fields and offers the option to "Add all detected fields." This also allows you to set a merge rule for these fields simultaneously.
   • Add a prefix: You can optionally add a prefix to the BlueConic properties. If you set the prefix and then add an empty mapping, the right side automatically adds the prefix as well.
     
7. Run the connection.
   Be sure to check the box next to Import groups into BlueConic in the left panel, and save your settings first. A green arrow appears, going from the Azure Data Lake Storage icon to the BlueConic icon. See the Running an Azure Data Lake Storage connection section for more information.

Setting up a Profile Import goal

Click Add goal > Import profiles to create an import goal or click Import profiles into BlueConic in the left panel to set import goals for adding profile and event data.

1. Select a BlueConic domain group for the import.
   BlueConic supports domain groups to ensure profiles are exchanged between the right domains. Select a domain group for the import to ensure that the correct profiles are updated or added.
2. Select the Azure Data Lake Storage files to import data from.
   Here you select the file(s) that contain your source data for the import. Use the green plus icon to expand file selection blocks.
   

   
   Tip: When you are choosing which file to designate as the root file, choose the one containing customer or profile data first. Then, you can link it to additional CSV files, such as event or product files, using the Connect (chain) icon. Use the Connect icon to choose how data in CSV columns should be linked together. For example, you might match customer profiles in the profiles file with the products file using matching order IDs.
   Note about unique file names: The name of the root file (Orders.csv in the example shown here) must be unique. The root file cannot have the same filename as the files linked to it, or the files will not be imported. 
   
   Optional: You can also use regular expressions or wildcards to choose the files. Click OK. Continue adding files and linking their data columns as needed. Note that the Connect icon is red until linking is established correctly, and then turns clear. If you select one of the files, a lightbox opens with a preview of the file columns and contents.

   
3. Optional: Correct your files before the import.

   

   Optionally add one or more data processors to transform or filter the data in your CSV file, to ensure that BlueConic imports the data correctly. To add data processors to your BlueConic environment, contact your Customer Success Manager. Read more about the available data processors.
4. Define file and field handling.
   Here you can customize how your CSV files and fields should be handled during the import. See the options for configuring CSV file and field imports for details.
5. Link identifiers between the Azure Data Lake Storage files and BlueConic. 

   Link identifiers from the Azure Data Lake Storage import to BlueConic profiles. Data without matching identifiers will be excluded from the import.

   
   Click Add identifier to create additional links.

   Check the Allow the creation of new profiles in BlueConic box to create a new BlueConic profile if the data field doesn't match an existing BlueConic profile when the import runs. If the box isn't checked, only existing profiles are updated, and data that doesn't match BlueConic profiles will be ignored.

6. Map the Azure Data Lake Storage data to BlueConic profile properties.

   Here, you can configure which values should be mapped to which BlueConic profile properties. Click Add mapping, select a data field from the drop-down list, and select the BlueConic profile property it populates by entering a search term.
   Bulk mapping: If there are many fields to be mapped, BlueConic detects these fields and offers the option to "Add all detected fields." This also allows you to set a merge rule for these fields simultaneously.
   Add a prefix: You can optionally add a prefix to the BlueConic properties. If you set the prefix and then add an empty mapping, the right side automatically adds the prefix as well.
   

   
   You can select how to import the data from the drop-down menu on the right of each property:

   • Set: Always overwrite the profile property value in BlueConic.
   • Set if empty: Import the value in BlueConic only if the profile property does not contain a value yet.
   • Set or clear: Clear the value in the profile property if the CSV contains no value for the field.
   • Add: Add the imported value to the list of values for the profile property. The list will only contain unique values, doubles are not imported.
   • Sum: Numerically add the imported value to the value of the profile property in BlueConic. If the profile property was empty, it will be treated as 0. If the imported value is not a number, it is not imported.

7. Optional: Import Timeline events into BlueConic.
   You can choose to import customer order data to BlueConic using either the default BlueConic Order event type, or a custom Timeline event type you have already created with a plugin XML file. (Contact your BlueConic Customer Success Manager at support@blueconic.com for details.) You can map order details in the left column to order event properties, event date/time, and the event ID.
   Use the Add mapping button if you want to extend the mapping for each order to include additional, nested Order event details, such as additional products.
   Use the Add a Timeline event type button to add additional event mappings. Learn more about using Timeline events in BlueConic.

8. Run the Azure Data Lake Storage connection.
   Be sure to check the box next to Import data into BlueConic in the left panel and save your settings. A green arrow appears, going from the Azure Data Lake Storage icon to the BlueConic icon. See the Running an Azure Data Lake Storage connection section for more information.

Setting up a Profile Export goal

Export customer profile data as well as Timeline event data from BlueConic profiles to a CSV file through Azure Data Lake Storage. Click Add goal > Export profiles to create the export goal.

1. Select a BlueConic segment to export profile data from.
   Only profiles in this segment with a value for at least one of the selected profile properties from Step 4 are exported.
   
2. Select an Azure Data Lake Storage directory.

   Select the Azure Data Lake Storage directory to export your BlueConic profiles to.

3. Define file and field handling.

   Here you can customize how your CSV files and fields should be handled during the import. See the options for configuring CSV file and field imports for details.

4. Map the BlueConic data to Azure Data Lake Storage.
   
   

   Map the BlueConic data you want to export to the corresponding Azure Data Lake Storage fields.

5. Optional: Export Timeline events to Azure Data Lake Storage.
   
   If you have BlueConic Timeline events, you can choose whether to export events for profiles in the current export, for all event types or a single event type, for all time or a during a specific time period. When you run the connection, BlueConic creates a CSV file for each event type. If there are nested events, BlueConic creates additional CSV files in the same directory for each nested event. For example, if the export is written as "segment-202202092359.csv," additional CSV files for orders and products could be "order-202202092359.csv" and "products-202202092359.csv" Column headers in the CSV file are derived from the Event type definition. 

6. Run the connection.

   If you want to create another Export goal, click Add goal and then Export goal in the left panel. Before running the connection, you need to Save your settings and turn the connection On at the top of the Connection window. When you check the box next to Export data into BlueConic, a green arrow appears, going from the BlueConic icon to the Azure Data Lake Storage icon.

Running an Azure Data Lake Storage connection

Tip: Make sure you have added your import or export goals and saved your settings before running the connection.

1. Turn the Azure Data Lake Storage connection on.
2. Go to the Set up and run page. Scroll down to the Run history section. This table shows all past and scheduled runs for the connection. In the details column, it also lists the last five created or updated profiles and links to the profile identifier so you are able to see exactly how and whether profiles were created or updated correctly.
3. To start the import and export manually, click the Run now button.
4. To schedule the import and export for a future date, click the Settings icon. Select how to schedule the import by choosing an option from the drop-down list:
   • Every X minutes
   • Number of times per day
   • Days of the week
   • Days of the month
   • Weekday of the month

   Set a time for the import. Click OK.

Data exchange shows you the total number of unique profiles that have been imported or exported via the connection. Note that these are unique profiles. If the same profile is exported multiple times, it is counted as one.

Privacy management

Connections can be added to Objectives, allowing for privacy management of the information that is being picked up. A connection will only process the profiles of visitors who have consented to at least one of the objectives that the connection is linked to.