Amazon Web Services (S3) Connection

How to use the BlueConic connection to integrate customer data with Amazon Web Services S3 via CSV

What: The Amazon Web Services (S3) connection allows you to synchronize data from various marketing platforms and databases into BlueConic via delimited flat files on servers. User profiles can be created or enhanced with this data and BlueConic information can be shared with marketing platforms. This connection supports scheduled batch import and export, as well as importing groups such as households, companies, or accounts.

About BlueConic: BlueConic is a Customer Data Platform that harnesses the data required to power the recognition of an individual at each interaction, and then synchronizes their intent across the marketing ecosystem. Learn more about the BlueConic platform.

Why: Transfer customer data from and to internal data warehouses, CRM databases, analytics platforms and other marketing systems to enrich BlueConic profiles with purchase history, predictive analytics segmentation (e.g. propensity modeling and lifecycle signals). This data can then be utilized directly within BlueConic or synchronized out to other marketing platforms to interact with users with relevancy and enable right-time messaging, or to enrich data and enhance capabilities with behavioral data collected by BlueConic and profile data integrated from other systems.

Concepts

CSV file format

Before we delve into the CSV file and field handling settings, let's first have a look at the actual text format of the file that we're about to import. Let's assume we're about to import the following data:

How to exchange customer profile data using CSV files in BlueConic

This would be exported as the following CSV text file:

id,ib_avg,ib_ddg,ib_z1,ib_z2
aghvww32,4123551,GK,open,3
agkkemw3,4233215,GK,open,0
alfk3oor,4135111,GR,open,
bezz5las,4009852,GR,closed,0
c9uuz5g0,4823507,K,closed,1
cc2rkes6,4222941,G,open,1

The first line of the CSV file is expected to contain column names, separated by a comma. Each following line is expected to contain values with the same number of columns.

See the best practices for exchanging data with BlueConic via CSV files.

Adding an Amazon Web Services (S3) connection

  1. Choose Connections in the Navigation bar.
  2. Click Add Connection.
  3. Search for "S3" in the Search bar and choose Amazon Web Services (S3) connection.
  4. The Amazon Web Services (S3) connection page opens.

    You can expand or collapse metadata fields (favorite, labels, description) by clicking the gray + or .

Configuring an Amazon S3 connection

In the side bar on the left are a number of tabs. There is a setup section and there are profile import and export goals, and group import goals. You can activate or deactivate goals by using the checkbox in front of the name. Multiple goals can be added by using the [Add goal] button at the bottom of the page and you can delete or copy goals by hovering over them and selecting the "x" respectively copy icon. 

Amazon-S3-Connection-Left-Panel.png 

Select a goal to open its tab and see that goal's configuration options. You can change the name of the goal at the top of the goal configurations.

Set up and run

This is where you set up the authentication that allows BlueConic to connect to the Amazon Web Services (S3) platform. Look up and enter the configuration parameters:

Access key ID
Enter the Amazon Web Services (S3) access key ID that you can retrieve via the security credentials page.
Secret access key
Enter the Amazon Web Services (S3) secret access key. You can create a new access key for your account by going to the security credentials page. In the "Access Key" section, choose "Create New Access Key". After saving the connection, the secret key will be hidden. You can choose "Change secret access key" to change the existing key.

For more information on working with credentials, read AWS Security Credentials.

How to configure a connection between the BlueConic customer data platform and Amazon S3 platform

Optional: Add email addresses for email notifications. Check the Notifications settings box if you would like BlueConic to send an email if an error occurs. You can then also check the box for BlueConic to send emails on successful runs.

Setting up profile import goals

Use the profile import goal to import profile data and events such as transactions or orders (via CSV files) from Amazon Web Services (S3) into BlueConic. To do this, choose Add goal > Import profiles and complete the steps outlined in the goal. 

1. Select a BlueConic domain group for the import.
BlueConic supports domain groups to ensure profiles are exchanged between the right domains. Select a domain group for the import to ensure that the correct profiles are updated or added.

2. Select the S3 source files to be imported.
Start by entering or selecting the S3 bucket where the files are stored that will be imported.

In order for BlueConic to be able to scan the buckets, the S3 account needs to have the "listAllBuckets" permission enabled.

Your S3 bucket can contain many files. You can specify exactly which file (or files) will be imported from by either entering a name or using wildcards to match multiple names, e.g. 10-import.csv, *.csv or  *import*.

How to synchronize customer data between the BlueConic CDP and Amazon S3 platform

Tip: If you are importing events, such as order data, contained in several CSV files (for example, orders, products, profile data, etc.), select the file with Order event info first, and then link it to additional CSV files using the Connect (chain) icon.

Use the Connect icon to choose how data in CSV columns should be linked together. For example, you might match customer profiles in the profiles file with the products file using matching order IDs.

How to exchange customer transaction data from Amazon S3 platform to BlueConic customer data platform

Optional: You can also use regular expressions or wildcards to choose the files.
Click OK. Continue adding files and linking their data columns as needed. Note that the Connect icon is red S3-red-chain.pnguntil linking is established correctly, and then turns clear. In this example, we are importing product order data from an Orders.csv file that is linked to Products.csv, Profiles.csv, and Postal Codes.csv. 

How to set up a connection to add customer orders from Amazon S3 web services platform and BlueConic

If you select one of the files a lightbox opens with a preview of the file columns and contents.

How to syncronize customer order data between Amazon S3 platform and the BlueConic CDP

 

3. Define file and field handling.

Here you can specify how the files' fields should be handled by entering a field separator. This is usually ,;, or \t (for tab-separated files).
Choose whether to use a point . or a comma , as a decimal separator for decimal or currency values.

Optional: Click Show advanced settings for additional options. Here, you can enter a multi-value separator, and choose a quote character (either ", ', or empty) in case your imported data contains quote characters around text. You can also set a custom date format, and select your time zone. Use the "Add .done suffix" option to append the suffix .done to the filename of files successfully imported, to help you know which files have been imported.
You can also choose Click Preview date and time to test if the date values in the detected files match your selected date format.
See the best practices for exchanging data with BlueConic via CSV files.

Optional: Use the Rename files option to add a ".done" suffix at the end of the file when the file is imported.

Optional: Use the Process CSV data field to add a data processor to filter or transform the data during the import. For example, you can create a processor to normalize email addresses, filter data fields, or transform data during the import, before it is matched and mapped to BlueConic properties. To add data processors to your BlueConic environment, contact your Customer Success Manager. Learn more about using data processors to filter or transform data imports.

How to I use preprocessors in BlueConic to transform customer data during an import from Amazon S3 web services platform?

4. Link identifiers between S3 files and BlueConic.

Here you link profile identifiers from the S3 import to BlueConic profiles. Data without matching identifiers will be excluded from the import.

How to link customer identifiers to match customer data between Amazon Web Services S3 platform and BlueConic

Select a column name on the left and select either the BlueConic profile identifier or a profile property that is a unique identifier on the right hand side for matching.

Check the Allow the creation of new profiles in BlueConic box to create a new BlueConic profile if the data field doesn't match an existing BlueConic profile when the import runs. If the box isn't checked, only existing profiles are updated, and data that doesn't match BlueConic profiles will be ignored.

5. Map S3 data to BlueConic profile properties.

Choose Add mapping to configure which CSV values should be mapped to which BlueConic profile properties. If you have multiple fields to map, BlueConic detects the fields and offers the option to map them all at once.

Bulk mapping: Select "Add all detected fields" to set up a bulk mapping between Amazon S3 and BlueConic.
Add a prefix: You can optionally add a prefix for the BlueConic profile properties. If you set the prefix and then add an empty mapping, the right side automatically adds the prefix as well. 

How do I import customer data from Amazon Web Services S3 to BlueConic

A warning message saying "Existing property" appears if the BlueConic profile property you're mapping to already exists and the import would save data into an existing profile property.

You can use the dropdown menu to specify how the value(s) should be imported:

  • Set: Always overwrite the profile property value in BlueConic.
  • Set if empty: Import the value in BlueConic only if the profile property does not contain a value yet.
  • Set or clear: Clear the value in the profile property if the CSV contains no value for the field.
  • Add: Add the imported value to the list of values for the profile property. The list will only contain unique values, doubles are not imported.
  • Sum: Numerically add the imported value to the value of the profile property in BlueConic. If the profile property was empty, it will be treated as 0. If the imported value is not a number, it is not imported.

You do not have to create a mapping and import every field of the CSV into BlueConic; there is no need to clutter BlueConic's data storage with values that serve no purpose in BlueConic or other connected systems. It is a best practice to only import values that are of interest to BlueConic use cases.

6. Optional: Import Timeline events into BlueConic.

You can choose to import customer order data to BlueConic using either the default BlueConic Order event type, or using a custom Timeline event type you have already created with a plugin XML file. (Contact your BlueConic Customer Success Manager at support@blueconic.com for details.) You can map order details in the left column with order event properties, event date/time, and the event ID. 
Learn more about BlueConic Timeline events.

Use the checkbox to say whether order event details should be added to a Product store, which collects data to feed product recommendations. Select this box if you plan to create product recommendations based on which products have been bought most often or most recently.
Learn more about BlueConic product recommendations.

How to import BlueConic Timeline events from Amazon Web Services to BlueConic customer data platform

Use the Add mapping button if you want to extend the mapping for each order to include additional, nested Order event details, such as additional products.

How do map customer data from BlueConic to Amazon Web Services S3 data

Click the Add Timeline event type button to add additional Timeline event types to the data import. Learn more about using Timeline events in BlueConic.

Click Save on the top right corner of the screen to save your import goal.
Optional: In the left panel, click Add goal and then Import goal to create another Import goal.

7. Run the connection.

To run the connection, you need to Save your changes and turn the connection On. See Running the Amazon Web Services (S3) Connection below for details.

Check the box for Import data into BlueConic in the left panel. A green arrow appears, going from the S3 icon to the BlueConic icon.

How to run the Amazon Web Services S3 connection to exchange customer profile and order data with BlueConic

If this is the first time you are running the connection, it is a good idea to do a test run with a small example file, to see if everything works as intended. If you are satisfied with the result, you change the import to the actual, full size file.

Setting up profile export goals for an Amazon S3 connection

Export goals enable you to send profile data from BlueConic to Amazon Web Services (S3). To do this, choose Add goal > Export profiles and complete the steps outlined in the goal.

1. Select a BlueConic segment to export profile data from.

Using the Select segment Search bar, choose a BlueConic segment. Only profiles in this segment with a value for at least one of the selected profile properties (from Step 4) are exported.

In the top right-hand corner, the number of profiles available for the export appears.

How to select BlueConic customer profile data and behavioral segments of customer data to export to Amazon Web Services S3 platform

This number shows how many profiles meet the conditions for the selected segment (for example, they have an email address and gave required consent for an objective) and therefore are available for the export. This number of profiles exported may be lower than the number displayed here. 

2. Set up the directory to export to.

Enter the Amazon Web Services (S3) directory for the exported files, or choose the folder icon to select a directory. View the files already in the directory in the Files detected table.

Optional: Choose Scan now to rescan the directory to retrieve updated files and data fields.

3. Determine file and field handling.

Select how to name the exported files. Select Segment name and timestamp to add a new file every time the export runs. Select Custom name to replace the file with a newer version every time the export is run.

Specify how the files' fields should be handled by entering a field separator. This is usually ,;, or \t (for tab-separated files).

For the decimal separator, specify whether your files use a point or a comma as a decimal separator.

Optional: Select Show advanced settings for additional options. Here, you can enter a multi-value separator, replace BlueConic IDs with the file name, compress the exported files, set a custom date format, and select your time zone. Select Date and time preview to test if the date values in the detected files match your selected date format.

4. Map BlueConic data to S3.

Select the BlueConic data you want to export into Amazon Web Services (S3).
How to export customer profile data from BlueConic to the Amazon Web Services S3 platfrom

Select a BlueConic profile property by entering a search term and enter the data field value it should populate. Order mappings by grabbing the drag bar on the left side; this will be reflected in the order of the fields in the exported file.
Select the BlueConic icon to open a drop-down menu where you can switch between the type of information you want to export. Pick one of:

  • BlueConic profile identifier: The unique identifier for a profile
  • Associated segments: All segments, or a selection of one or more specific segments that the profile is associated with
  • Profile property: The value of a specific profile property
  • All viewed interactions (all time): The interactions that the profile has seen
  • Permission level: The permission level set in the profile
  • Text value: Enter any static text
    How to authenticate a connnection between BlueConic and Amazon Web Services S3 platform

5. Optional: Export Timeline events to S3.

If you have BlueConic Timeline events, you can choose whether to export events for profiles in the current export, for all event types or a single event type, for all time or a during a specific time period.
When you run the connection, BlueConic creates a CSV file for each event type (for example, order.csv for the event type orders). If there are nested events (for example, products), BlueConic creates additional CSV files in the same directory for each nested event. Column headers in the CSV file are derived from the Event type definition. 

6. Run the connection.

Choose Save on the top right corner of the screen to save your Export goal and turn the connection On at the top of the Connection window.

To create another Export goal, choose Add goal > Export profiles in the left panel. 

In the left panel, check the box next to Export profiles. A green arrow appears, going from the BlueConic icon to the Amazon Web Services (S3) icon.

How to runn a connection to export customer profile data and behavioral customer segments from BlueConic to Amazon Web Services S3 

Setting up a Group import goal

To import data from Amazon S3 into BlueConic groups, such as households, accounts, companies, etc., click Add goal > Import groups and follow the steps outlined in the goal. The groups import option is available in the connection once you've set up a BlueConic group. If you want to use this capability, contact your BlueConic account manager.

Note about resource usage: Existing BlueConic customers can use the Group feature free of charge as long as your total data storage remains smaller than the amount that is allotted to your subscription. Please contact your Customer Success Manager at support@blueconic.com if you intend to use this feature, to ensure your subscription can accommodate your intended usage. We'll discuss your requirements and upgrade your subscription as necessary.

1. Select a group type to import data to a BlueConic group.
Use the drop-down menu to select from the group types defined for your BlueConic tenant.
How to import data to BlueConic groups of profiles

2. Select your S3 directory and files.
How to select customer data in Amazon Web Services S3 platform to add to BlueConic profiles

Enter the directory that holds the files.

Optional: Specify the files to be imported. You can use regular expressions or wildcards to choose the files.
Optional: Click Scan now to rescan the directory. This retrieves updated files and data fields.

3. Define file and field handling.
Here you can specify how the files' fields should be handled by entering a field separator. This is usually ,, ;, or \t (for tab-separated files).
Choose whether to use a point . or a comma , as a decimal separator for decimal or currency values.
Optional: Clicking Show advanced settings lets you enter a multi-value separator, set a custom date format, and select your time zone. Click Preview date and time to test if the date values in the detected files match your selected date format.

4. Link identifiers between S3 files and BlueConic groups.
Data that has no matching identifier will be excluded from the import.
How to use BlueConic to synchronize data into unified customer profiles from my martech stack

5. Map imported fields to BlueConic group properties.
Here, you can configure which values should be mapped to which BlueConic group properties. Click Add mapping, select a data field from the drop-down list, and select the BlueConic group property it populates by entering a search term.
Bulk mapping: If there are many fields to be mapped, BlueConic detects these fields and offers the option to "Add all detected fields."
Add a prefix: You can optionally add a prefix to the BlueConic properties. If you set the prefix and then add an empty mapping, the right side automatically adds the prefix as well.
How to create a 360 view of my customers in unified profiles in the BlueConic customer data platform

6. Run the connection.
Check the box next to Import group data into BlueConic in the left panel, and save your settings. A green arrow appears, going from the S3 icon to the BlueConic icon. Follow the steps in the next section to run the import.

Running an Amazon Web Services (S3) connection

Tip: Make sure you have added Import and Export goals before running the connection.

  1. Turn the Amazon Web Services (S3) connection on.
    How to create unified customer profiles using data from my marketing technology stack using the BlueConic customer data platform (CDP)
  2. Go to the Authentication and run page. Scroll down to the Run history section.
  3. To start the import and export manually, select Run now
  4. To schedule the import and export for a future date, select the Settings icon How to schdule and run a connection between the BlueConic CDP and my martech stack and marketing technologies on Amazon Web Services. Choose how to schedule the import by choosing an option from the drop-down list:
    • Every X minutes
    • Number of times per day
    • Days of the week
    • Days of the month
    • Weekday of the month

    Set a time for the import. Choose OK.

Data exchange shows you the total number of unique profiles that have been imported or exported via the connection. Note that these are unique profiles. If the same profile is exported multiple times, it is counted as 1.

How to exchange customer data between BlueConic and the Amazon Web Services S3 platform in my martech stack

Privacy management

Connections can be added to Objectives, allowing for privacy management of the information that is being picked up. A connection will only process the profiles of visitors who have consented to at least one of the objectives that the connection is linked to.

Frequently asked questions:

If a file has already been imported, will it be imported again? 

If the connection configuration has not been changed, and a file has already been imported, it will not be imported again. If the file has changed, or the configuration has changed, files will be reimported.