Skip to main content
Solved

Automatic selection of A/B test winner on flows

  • April 22, 2024
  • 2 replies
  • 46 views

Forum|alt.badge.img+3
  • Problem Solver I
  • 7 replies

Hi,

I’ve had an A/B test running on a flow and I was under the impression that the test would end automatically once statistical significance was reached, which according to this article is defined as:

  • At least 500 recipients have received each variation.
  • A variation has at least 90% win probability.

As far as I can tell, this test (pictured below) satisfied both criteria, but it still says it’s not statistically significant. In the end I just manually chose the winner, but could someone explain why this wasn’t classed as statistically significant? 

 

Best answer by Brian Turcotte

Hi @kfodor!

That is interesting - at first glance, it seems it should have deemed it statistically significant. 

 

Did you change any of the default definitions cited here?

- Brian

View original
Did this topic or the replies in the thread help you find an answer to your question?

2 replies

Brian Turcotte
Forum|alt.badge.img+37
  • Klaviyo Alum
  • 1393 replies
  • Answer
  • April 23, 2024

Hi @kfodor!

That is interesting - at first glance, it seems it should have deemed it statistically significant. 

 

Did you change any of the default definitions cited here?

- Brian


Forum|alt.badge.img+3
  • Author
  • Problem Solver I
  • 7 replies
  • April 23, 2024

Hi @Brian Turcotte 

Thanks, I’m glad I’m not the only one that thought so! I don’t think I changed any of the default settings when I set the test up, so it could be a bug...or I’m forgetting something. 

I have another test running on a different flow that only needs ~50 more recipients on one of the versions, so I’ll see what happens there. If it again doesn’t end automatically, I might report it as an issue!

Thanks,

Katherine