Solved

Understand Klaviyo pop-up A/B testing results


Userlevel 1
Badge +6

Hi community! I’d be happy if someone can help me with a doubt related to pop-ups in Klaviyo.

 

I ran an A/B testing to see if my exit-intent pop-up with an image was more effective than the version just with text. The experiment was active for almost a month (around 25 days), but Klavyio says the results are not statistically significant: 


 

Do you know how much time is need for the results to be conclusive? Or is more related to the traffic on my website? I’m having trouble understanding the results. 

icon

Best answer by Spark Bridge Digital LLC 27 May 2022, 14:43

View original

3 replies

Userlevel 6
Badge +30

When I’m A/B testing, I find a month to be plenty. Especially with the amount of views/submissions you have within that time span.

With that summary from Klaviyo, I take from it that the two A/B tests aren’t so different in opt-in/engagement to say one is better than the other but A just has that slight lead. Basically, your visitors aren’t drawn to one or the other more clearly for Klaviyo to say one is preferred over the other. 

If I was in your shoes, I’d select A as the Winner. And if you want to keep testing, develop another new B Test that is testing another new look/copy change from your current winner. 

 

Badge +3

Statistical significance is when Klaviyo is mathematically able to determine whether a variation will produce improved performance. It observes both the number of people and the win probability, which is how likely a variation will yield better results based on how well it outperforms the other variation(s).

The statistically significant tag on your A/B test means that a certain variation of your test is highly likely to win over the other option(s).

A test has reached statistical significance if all of the following are true:

  • Win probability >= 90

  • Minimum number of views in any variation >= 2000

Anything before that would be marked as statistically insignificant.

Userlevel 7
Badge +43

In short, the change (you made) and as a result the change in conversion rate is too small to be able to say with confidence that the change in conversion has been caused by changing the popup.

 

The smaller the change the bigger amount of users needed to confirm if that change actually made a difference. The bigger the change the lesser amount of people are needed.

Reply