When I’m A/B testing, I find a month to be plenty. Especially with the amount of views/submissions you have within that time span.
With that summary from Klaviyo, I take from it that the two A/B tests aren’t so different in opt-in/engagement to say one is better than the other but A just has that slight lead. Basically, your visitors aren’t drawn to one or the other more clearly for Klaviyo to say one is preferred over the other.
If I was in your shoes, I’d select A as the Winner. And if you want to keep testing, develop another new B Test that is testing another new look/copy change from your current winner.
Statistical significance is when Klaviyo is mathematically able to determine whether a variation will produce improved performance. It observes both the number of people and the win probability, which is how likely a variation will yield better results based on how well it outperforms the other variation(s).
The statistically significant tag on your A/B test means that a certain variation of your test is highly likely to win over the other option(s).
A test has reached statistical significance if all of the following are true:
Anything before that would be marked as statistically insignificant.
In short, the change (you made) and as a result the change in conversion rate is too small to be able to say with confidence that the change in conversion has been caused by changing the popup.
The smaller the change the bigger amount of users needed to confirm if that change actually made a difference. The bigger the change the lesser amount of people are needed.
