In BlueConic, a Product or Content Collector can be used to add items to a product or content store that can then be used for creating personalized recommendations. Since the principles of these two connections are almost the same, they will be both covered in this article.
What to do if items are not being added to a content or product store
If you notice that products or articles are not being added to your product or content stores, there are a few areas to check within the collector connections. To troubleshoot, first ask yourself the following questions:
Is the connection set to “On”?
Have the correct channels been added to the connection?
Is the “Collect data from your channels” goal enabled?
Are all of your values present in your page source?
If the answers are “yes” and your product or content stores still aren’t being populated, the next step would be to visit a page on your channel that you would expect data to be collected from. Open the console. If the console contains a message along the lines of “Item will not be scraped,” you need to adjust the settings so the values for the required fields are scraped. If this message is not shown, you should see a recommendation request (/rest/v2/recommendations/stats) to BlueConic on the network tab. Verify that the recommendation request contains the correct URL and a unique itemId.
If the recommendation request is correct, the client-side part of the flow is handled correctly. However, the server-side part could still result in the URL not being added to the store. Test this by pasting the same URL of the article or the product page in the “test URL” section of the collector.
If pressing the “Test” button does not return anything, the URL is probably blocked by the server of the website. In this case, check with your web or infrastructure team to allow BlueConic to get the content server-to-server. This could be passed on IP or an extra header can be defined in the collector, so the website knows it has to allow it. For example, in the screenshot below, BlueConic sets a “X-Web-Access” header when requesting the page, which is checked on the website side for validity.
If pressing the “Test” button does return values, here are some steps you can take:
Check that the correct types are being returned for the metadata.
This should be “article” for the Content Collector and “product” for the Product Collector.
Check that all required fields have a value.
If none of these troubleshooting tips work, please contact BlueConic Support.
Frequently asked questions
Can collectors scrape data from locked or gated pages?
If you are using the HTML metadata selector and the metadata is present in the page source, there should be no issue collecting the content.
Where do collectors scrape data from?
BlueConic Content and Product Collectors rely on the data available directly in the page source code. If the metadata is being dynamically inserted or modified by JavaScript after the initial page load, our scraper will not be able to detect and collect the required metadata.
How can I use a JavaScript expression with my collector?
The 'JavaScript expression' option in the collector configuration is meant to retrieve values from the page source. For example, it can be used to apply a method such as .split() to only retrieve partial values.
What does a 5XX error code mean and how can I resolve it?
In product or content collectors, 500-level error codes are related to your website. The first step in attempting to resolve these errors is to verify that BlueConic IPs are added to your website's allowlist. If this does not fix the error, it is likely coming from a cybersecurity or firewall service on your website, so double check those settings.
How can I test which items are in a product or content collector?
In Step 2 of the Collector, you can test to see which values are being returned. Paste a content item's URL into the Test URL field, click 'Test' to review the metadata that would be collected for its content, and then confirm whether all 'Required' metadata fields are returning values.
If values are not being returned for required fields, then items are not being collected successfully.