Skip to main content
Contributor IV
August 16, 2022
Solved

About A/B test

  • August 16, 2022
  • 4 replies
  • 195 views

Hello , i  have the following doubts about AB test. I hope to get answers

1.when test size is 100% flow ,there is no test duration,I want to know how long it will take to reach an experimental conclusion

 

2、i did a test, with duration  lasting 1 hour,but  there is no experimental conclusion,Is there a requirement for the minimum number of samples in the experimental group

3.I found that the data in the following figure were not significant at 95% confidence level by chi square test, but it tells me there is 97.7% win confidence variation B is winner, I want to know what our A/B test rules are

looking  forward to your reply ,  thanks ! 

This topic has been closed for replies.
Best answer by alex.hong

Hello @kimmia,

I see my colleague has responded to another one of your posts so I will redirect you to this post:

Thanks,

Alex

4 replies

alex.hong
Klaviyo Alum
August 16, 2022

Hello @kimmia,

Thanks for sharing to the Community.

For your first question, I believe this would depend on how long it would take for a user to complete your flow correct?  It would also depend on the # of contacts you are sending the flow variations too. In the same vein, while still testing one variable at a time, the more variations you add, the bigger the audience you’ll need to make sure you get reliable results. To get your results in a timely manner so you can perform more tests, use no more than four variations of one variable per test.

 

Imagine you have a great idea for a form, you A/B test it, and, on the first day, the first 10 people who see the form click through and make a purchase. Over the course of a week, the next 2000 people exit out of the form immediately. If you had ended the test after the first day, you would think that your form was great, even though a larger audience showed otherwise. By waiting until your results are statistically significant or until you have a good sample size of viewers, you’ll ensure that you know which form is truly better for your brand. Although we do not have a minimum # of people for a test, the more involved the better your data will be (that is true for anything).

 

For further information regarding your A/B test, I would recommend reaching out to Support to see if they can provide more detailed info into your account that we cannot see from screenshots. 

 

Thanks,
Alex

kimmiaAuthor
Contributor IV
August 17, 2022

@alex.hong 

1.How many status  does the A/B Test has , whichs are they

2.What is the logic of the test duration

 

Looking  forward to your reply ,  thanks ! 

kimmiaAuthor
Contributor IV
August 17, 2022

@alex.hong 

How is Win probability calculated , Chi-Squared Test or something else?

alex.hong
alex.hongAnswer
Klaviyo Alum
August 19, 2022

Hello @kimmia,

I see my colleague has responded to another one of your posts so I will redirect you to this post:

Thanks,

Alex