The question about A/B test rules

  • 16 August 2022
  • 4 replies

Badge +1
  • Contributor I
  • 3 replies

Hello , i  have the following doubts about AB test. I hope to get answers

1.when test size is 100% flow ,there is no test duration,I want to know how long it will take to reach an experimental conclusion


2、i did a test, with duration  lasting 1 hour,but  there is no experimental conclusion,Is there a requirement for the minimum number of samples in the experimental group

3.I found that the data in the following figure were not significant at 95% confidence level by chi square test, but it tells me there is 97.7% win confidence variation B is winner, I want to know what our A/B test rules are

looking  forward to your reply ,  thanks ! 


Best answer by stephen.trumble 16 August 2022, 19:25

View original

4 replies

Userlevel 7
Badge +60

Hey @ZCL 

Welcome to the Klaviyo Community! Thank you for reaching out about how statistical significance in Klaviyo is determined.

  1. To answer your first question, the A/B test test result is deemed statistically significant when:
    1. 50 people have received each variation
    2. The win probability is at least 90%

This ensures that a large enough sample size of recipients have seen the A/B test, and that the winning variation largely outperformed the other(s) for the chosen winning metric (which, for campaigns, is either open or click rate). Here is a visual to help the            community understand the decision for statistical significance:  

  1. There is no A/B test results for the experiment with only an hour duration time because not enough customers received the email to determine statistical significance. The article on best practices for A/B testing has more information on audience size. Imagine you have a great idea for an email, you A/B test it, and, on the first day, the first 10 people who see the email click through and make a purchase. Over the course of a week, the next 2000 people don’t open your email. If you had ended the test after the first day, you would think that your email was great, even though a larger audience showed otherwise. By waiting until your results are statistically significant or until you have a good sample size of viewers, you’ll ensure that you know which email is truly better for your brand.
  2. Under the win confidence, you’ll see whether or not the variation shown is statistically significant. If it is deemed significant, this means that the variation has a high win probability. For Klaviyo campaigns, this means the variation has a 90% chance or more of winning. In this case, a green tag saying Statistically significant will appear. On the other hand, if the variation is not statistically significant, you will see a gray tag saying Not statistically significant. Nothing will appear if the results are inconclusive. The article I linked above goes over in detail how Klaviyo determines statistical significance and there for how the win confidence is determined.

Hope this helps answer your questions! Thanks again for being part of the Klaviyo Community!


Badge +1


thanks for reply, but i think there is something wrong with the logic

I think the actual logic should be that Test size has no more than (1,800X  the number of variation) recipients.——yes/no

   yes: not statistically significant

   no: next step logic



Badge +1


1.Whichs status  has the A/B Test

2.What is the logic of the test duration

looking  forward to your reply ,  thanks ! 

Badge +1


How is Win probability calculated , Chi-Squared Test or something else?