Automatic selection of A/B test winner on flows

Question

Hi,I’ve had an A/B test running on a flow and I was under the impression that the test would end automatically once statistical significance was reached, which according to this article is defined as:At least 500 recipients have received each variation.	A variation has at least 90% win probability.As far as I can tell, this test (pictured below) satisfied both criteria, but it still says it’s not statistically significant. In the end I just manually chose the winner, but could someone explain why this wasn’t classed as statistically significant?

Brian Turcotte · Accepted Answer

Hi @kfodor!That is interesting - at first glance, it seems it should have deemed it statistically significant.Did you change any of the default definitions cited here?- Brian

kfodor · Answer

Hi @Brian TurcotteThanks, I’m glad I’m not the only one that thought so! I don’t think I changed any of the default settings when I set the test up, so it could bea bug...or I’m forgetting something.I have another test running on a different flow that only needs ~50 more recipients on one of the versions, so I’ll see what happens there. If it again doesn’t end automatically, I might report it as an issue!Thanks,Katherine

Sign up

Use your Klaviyo credentials

Log in to the Community

Use your Klaviyo credentials