Hey @ZCL 

Welcome to the Klaviyo Community! Thank you for reaching out about how statistical significance in Klaviyo is determined.

1. To answer your first question, the A/B test test result is deemed statistically significant when:
   1. 50 people have received each variation
   2. The win probability is at least 90%

This ensures that a large enough sample size of recipients have seen the A/B test, and that the winning variation largely outperformed the other(s) for the chosen winning metric (which, for campaigns, is either open or click rate). Here is a visual to help the            community understand the decision for statistical significance:  

1. There is no A/B test results for the experiment with only an hour duration time because not enough customers received the email to determine statistical significance. The article on best practices for A/B testing has more information on audience size. Imagine you have a great idea for an email, you A/B test it, and, on the first day, the first 10 people who see the email click through and make a purchase. Over the course of a week, the next 2000 people don’t open your email. If you had ended the test after the first day, you would think that your email was great, even though a larger audience showed otherwise. By waiting until your results are statistically significant or until you have a good sample size of viewers, you’ll ensure that you know which email is truly better for your brand.
2. Under the win confidence, you’ll see whether or not the variation shown is statistically significant. If it is deemed significant, this means that the variation has a high win probability. For Klaviyo campaigns, this means the variation has a 90% chance or more of winning. In this case, a green tag saying Statistically significant will appear. On the other hand, if the variation is not statistically significant, you will see a gray tag saying Not statistically significant. Nothing will appear if the results are inconclusive. The article I linked above goes over in detail how Klaviyo determines statistical significance and there for how the win confidence is determined.

Hope this helps answer your questions! Thanks again for being part of the Klaviyo Community!

@stephen.trumble 

thanks for reply, but i think there is something wrong with the logic

I think the actual logic should be that Test size has no more than (1,800X  the number of variation) recipients.——yes/no

   yes: not statistically significant

   no: next step logic

@stephen.trumble 

1.Whichs status  has the A/B Test

2.What is the logic of the test duration

looking  forward to your reply ,  thanks ! 

@stephen.trumble 

How is Win probability calculated , Chi-Squared Test or something else?