Understand Klaviyo pop-up A/B testing results

Question

Hi community! I’d be happy if someone can help me with a doubt related to pop-ups in Klaviyo.

I ran an A/B testing to see if my exit-intent pop-up with an image was more effective than the version just with text. The experiment was active for almost a month (around 25 days), but Klavyio says the results are not statistically significant:

Do you know how much time is need for the results to be conclusive? Or is more related to the traffic on my website? I’m having trouble understanding the results.

Spark Bridge Digital LLC · Accepted Answer

When I’m A/B testing, I find a month to be plenty. Especially with the amount of views/submissions you have within that time span.

With that summary from Klaviyo, I take from it that the two A/B tests aren’t so different in opt-in/engagement to say one is better than the other but A just has that slight lead. Basically, your visitors aren’t drawn to one or the other more clearly for Klaviyo to say one is preferred over the other.

If I was in your shoes, I’d select A as the Winner. And if you want to keep testing, develop another new B Test that is testing another new look/copy change from your current winner.

sofijap · Answer

Statistical significance is when Klaviyo is mathematically able to determine whether a variation will produce improved performance. It observes both the number of people and the win probability, which is how likely a variation will yield better results based on how well it outperforms the other variation(s).

The statistically significant tag on your A/B test means that a certain variation of your test is highly likely to win over the other option(s).

A test has reached statistical significance if all of the following are true:

Win probability >= 90
Minimum number of views in any variation >= 2000

Anything before that would be marked as statistically insignificant.

Sign up

Use your Klaviyo credentials

Log in to the Community

Use your Klaviyo credentials