Rules of A/B test

  • 18 August 2022
  • 1 reply

Userlevel 1
Badge +4

1.How many status  does the A/B Test has , whichs are they

2.What is the logic of the test duration , When can i get the ruslut

3.How is Win probability calculated , Chi-Squared Test or something else?


Looking  forward to your reply ,  thanks ! 


Best answer by David To 18 August 2022, 20:40

View original

1 reply

Userlevel 7
Badge +60

Hey @kimmia,

Great question!

Most of those questions are actually answered in our How Statistical Significance in Klaviyo Is Determined Help Center article. For example, the statuses you’ll see for your test results are listed under the Categories of Statistical Significance subsection which are: 

  • Statistically significant
  • Promising
  • Not statistically significant
  • Inconclusive

Overall, how long or how many test you run to get your results will vary from business to business, brand to brand and will be impacted by a number of variables such as your brand/business, your sample size, your sample behaviors, your marketing strategy/test strategy, etc. There’s just no clear answer to this without running through your very own tests. So, like I always like to say, test, test, and test some more!

Although we can’t share the model and algorithm we use to help determine win probability of your tests, one of our Data Scientists, Michael Lawson actually has a great Medium blog article they shared to help users further understand our data philosophy: Ensuring efficiency and accuracy in signup form A/B testing. If you’re interested, Michael also runs our Klaviyo Data Science Podcast which dives into a number of different topics, including A/B testing. 

I hope this helps!