The Advanced A/B Testing notebook in AI Workbench analyzes a selected BlueConic dialogue and its variants using Bayesian statistical analysis. It requires a control group and helps determine the best-performing version and next steps. When run, the notebook evaluates how variant dialogues compare to the original.
Add an Advanced A/B testing notebook
Navigate to More > AI Workbench > Add notebook.
Choose Advanced A/B testing notebook from the pop-up window.
Give your notebook a name.
Save your settings.
Set the Advanced A/B testing notebook parameters
Select the Parameters tab.
In the Dialogue parameter, select an existing dialogue that has variants that are shown to a subset of customers and visitors.
Save your settings.
Run the Advanced A/B testing notebook
Select the Schedule and run history tab.
Click Run now to run the notebook analysis manually.
To schedule the import and export for a future date, activate Enable scheduling.
Click the Settings icon to select how to schedule the notebook by choosing an option from the drop-down list.
Set a time for the import. Click OK.
Save your settings.
View your results
After running the notebook, click Preview to view the output. In the Management Summary section, you'll find A/B test results and recommendations. Below, a graphical representation of the analysis illustrates how dialogue variants perform against the original.
The sample graph shows a posterior distribution, identifying the best-performing dialogue using Bayesian statistics. In this example, the original dialogue (blue, furthest right) outperforms others but has fewer results than the purple variant. The notebook assesses overlap among dialogues to predict the top performer.
Further down, the expected lift distribution compares a variant to the original, highlighting the region of practical equivalence between similarly performing dialogues. While the original is expected to win, more data is needed for confirmation.
FAQs
What is required for the notebook's analysis to run?
The selected dialogue must have a control group for the notebook's analysis to function properly.
What can I do with Notebook editor permissions?
With Notebook editor permissions, you can view the notebook's Python code, access detailed documentation, and understand how the machine learning model uses customer data and dialogue metrics for A/B modeling and analysis.