Firehose connections

What: Firehose connections allow you to stream BlueConic data and events to other systems in real-time.

About BlueConic: BlueConic is a Customer Data Platform that harnesses the data required to power the recognition of an individual at each interaction, and then synchronizes their intent across the marketing ecosystem. Click here to learn more.

Why: Your business intelligence team already works on deriving strategic information from your data store. BlueConic features a unique data collection engine that captures data and events at the customer level for known and anonymous users. Handing over this information to your data store in real time will gives your BI team complete freedom in how to process, enrich and improve existing reports and applications and invent new ones.

Concepts

Firehose

Your visitors generate a lot of events across all of your channels. BlueConic comes out of the box with dashboards that provide insights in all this data. Additionally, you may want to activate this data even further by handing it off to other systems for processing and analyzing in real-time.

Google, Amazon and Microsoft all have event queueing systems that provide key capabilities like many-to-many, asynchronous messaging that decouple senders and receivers:

BlueConic can log data and events through a so-called firehose connection and stream it into an event queueing system. These systems all work at scale and can handle the potentially millions of events coming their way in real-time via a BlueConic firehose streaming connection. The systems enable you to process and analyze data as it arrives, so you can respond instantly instead of having to wait until all your data is collected before the processing can begin.

Technology stack

An example technology stack could look like this: 

firehose-g.png

This example uses Google cloud systems as an example, but logos and names can easily be exchanged for their Amazon or Microsoft counterparts. The event queueing system delivers low-latency, durable messaging that allows developers to quickly integrate systems for machine learning, analytics, and other applications. 

Marketers use BlueConic to interact with visitors and engage with them. This results in your visitors producing events and data. These are being collected in BlueConic and can be handed off via a firehose connection.

The BlueConic firehose connection connects to the event queueing system (Cloud Pub/Sub, Kinesis or Event Hubs). From here, the data can be transformed by all sorts of systems into data that is usable for presentation and action.

Real-time

The term "real-time" can mean different things in different contexts, so let's be specific about the timing details:

  • Changes to profile properties are sent every 2 minutes.
  • Events are sent every 10 seconds.
  • The segments a particular profile belongs to are evaluated every 2 minutes. Only profiles "in rest", i.e. profiles that did not have any profile property value changes in the last 2 minutes, are evaluated.

Use cases

BlueConic customers use big cloud vendor firehose technologies to send data to:

  • Business Intelligence tools
  • Data lakes
  • Data warehouses (e.g. Google BigQuery, or Amazon Redshift)