Content Collector Connection

The Content Collector Connection collects data about your content and stores it in a BlueConic content store that can be accessed by dialogues making use of Content Recommendations.


To set up the connection, click "Collect data from your channels."


1. Select the channel(s) to collect content data from.
Optionally, define URL rules to specify which areas of the sites to collect content data from.

2. Manage which data is collected.
Next, paste an article URL into the Test URL field, and click 'Test' to review the metadata that would be collected for your content:


BlueConic requires a number of meta data fields out of the box. Select the checkbox for other meta data fields that are also required.

If required meta data fields are not populated, the webpage may not surface content information in a supported format. The default BlueConic selector will automatically detect:

Providing meta data in one of these standard ways is not only good for the content collector, it is good for Google, Facebook, Twitter and countless other platforms, as it allows these platforms to understand your content!

Click Default to select an alternative method of retrieving meta data. You can retrieve information from meta data in the HTML, or retrieve it from the HTML using a selector, or using JavaScript.

Click Add data field to add custom metadata fields. For example, if your content has a sponsorship association, or is tagged based on overarching stories or topics, influencers, or sources, you can use this data in recommendation placement filtering.

3. Set the algorithm timeframe.
Some recommendation algorithms are based on a look-back timeframe (for example, Viral articles or recent high CTR). You can configure that timeframe here, based on hours or days:


 4. Set request headers (optional).
To enable content recommendations to be collected from webpages that are under development or behind a login or paywall, you can add HTTP request headers here. Click "Add request header," choose a channel, and add the custom header name and value used to access the content.




  1. Make sure images are being scraped properly.
  2. Review article names to be sure appendages do not exist, for example " |" may be appended to every title depending where it is being scraped from.