Reporting on Campaigns

You are able to see how campaigns perform for both Experiences and Experiments.

Within reporting, you are able to see the campaigns impact on the Campaign Goal, and Intent Metric.

Campaign Goal; You can currently only choose Transactions as your campaign goal. However, we are working on allowing you to build custom goals here that are specific to the campaign you are firing. For example, if the campaign is for email captures, this could be number of email submissions.

Intent Metric; You are able to see the following Intent metrics within campaign reporting;

Average xC; the average expected intent to purchase for visitors who have seen an experiment or experience vs. the variant or holdback group.
- This average is calculated as an Exponentially Weighted Moving Average (EWMA), and updates on each event we receive from a visitor to reflect evolving intent, resulting in a final average at the journey's end.
Max xC; the average maximum expected intent to purchase for visitors who have seen an experiment or experience vs. the variant or holdback group
Performance Score; the average Intent Performance Score for visitors who have seen an experiment or experience vs. the variant or holdback group

Reporting on Experiments

Experiments have variants, and controls.

You are able to look at reporting for experiments at a Global level, or an individual experiment level (if you have multiple variants within one campaign)

Global Level; this will show you all experiment results grouped together vs the control (when matching the global rules)

Individual Experiment Level; this will show you the results of each individual experiment separated, vs the control. In this case, the control group are users who match the global rules of the campaign, and the experience segmentation

Within the reporting section for an experiment you are able to see;

The number of users within the variant & control
The selected Intent metric
The Intent metric performance change vs the control
Total conversions
Conversion rate
Conversion uplift vs the control
Probability to beat baseline (when looking at experiments from an individual level, this baseline can be the control, or another variant)
Probability to be best

If you remove the control within an experiment, a new baseline would be set.

Campaigns have experiences, and a holdback group (more information on this can be found here).

You are able to look at reporting for experiences at a Global level, or an individual experience level (if you have multiple experiences within one campaign)

Global Level; this will show you all experience results grouped together vs the holdback group (when matching the global rules)

Individual Experience Level; this will show you the results of each individual experience separated. In this case, the holdback group are users who match the global rules of the campaign, and the experience segmentation

Within the reporting section for an experience you are able to see;

The number of users within the experience & holdback group
The selected Intent metric
The Intent metric performance change vs the holdback group
Total conversions
Conversion rate
Conversion uplift vs the holdback group
Probability to beat baseline (when looking at campaigns from an individual experience level, this baseline can be the holdback group, or another experience)
Probability to be best

What does 'Probability to be best' mean?

The Probability of Being Best estimates how likely it is that a given variant, or experience is the top performer among all tested options, including the control.

Rather than just comparing average success rates, it accounts for uncertainty in the data.

This metric is calculated using Bayesian simulation methods.

For each variant, we generate many possible outcomes based on its observed performance. We then measure how often each variant comes out ahead in these simulated scenarios.

If a variant has a Probability of Being Best of 75%, it means that in 75% of the simulations, it performed better than all other variants.

In essence, it answers:
"How confident are we that this variant, or experience, is the overall best?"