Skip to main content
Solved

Rules of A/B test

  • August 18, 2022
  • 1 reply
  • 119 views

Forum|alt.badge.img+4

1.How many status  does the A/B Test has , whichs are they

2.What is the logic of the test duration , When can i get the ruslut

3.How is Win probability calculated , Chi-Squared Test or something else?

 

Looking  forward to your reply ,  thanks ! 

Best answer by David To

Hey @kimmia,

Great question!

Most of those questions are actually answered in our How Statistical Significance in Klaviyo Is Determined Help Center article. For example, the statuses you’ll see for your test results are listed under the Categories of Statistical Significance subsection which are: 

  • Statistically significant
  • Promising
  • Not statistically significant
  • Inconclusive

Overall, how long or how many test you run to get your results will vary from business to business, brand to brand and will be impacted by a number of variables such as your brand/business, your sample size, your sample behaviors, your marketing strategy/test strategy, etc. There’s just no clear answer to this without running through your very own tests. So, like I always like to say, test, test, and test some more!

Although we can’t share the model and algorithm we use to help determine win probability of your tests, one of our Data Scientists, Michael Lawson actually has a great Medium blog article they shared to help users further understand our data philosophy: Ensuring efficiency and accuracy in signup form A/B testing. If you’re interested, Michael also runs our Klaviyo Data Science Podcast which dives into a number of different topics, including A/B testing. 

I hope this helps!

David

View original
Did this topic or the replies in the thread help you find an answer to your question?

1 reply

David To
Klaviyo Employee
Forum|alt.badge.img+60
  • Klaviyo Employee
  • 2456 replies
  • Answer
  • August 18, 2022

Hey @kimmia,

Great question!

Most of those questions are actually answered in our How Statistical Significance in Klaviyo Is Determined Help Center article. For example, the statuses you’ll see for your test results are listed under the Categories of Statistical Significance subsection which are: 

  • Statistically significant
  • Promising
  • Not statistically significant
  • Inconclusive

Overall, how long or how many test you run to get your results will vary from business to business, brand to brand and will be impacted by a number of variables such as your brand/business, your sample size, your sample behaviors, your marketing strategy/test strategy, etc. There’s just no clear answer to this without running through your very own tests. So, like I always like to say, test, test, and test some more!

Although we can’t share the model and algorithm we use to help determine win probability of your tests, one of our Data Scientists, Michael Lawson actually has a great Medium blog article they shared to help users further understand our data philosophy: Ensuring efficiency and accuracy in signup form A/B testing. If you’re interested, Michael also runs our Klaviyo Data Science Podcast which dives into a number of different topics, including A/B testing. 

I hope this helps!

David