Solved

A/B testing, what version does the receiver receive?

  • 16 January 2023
  • 2 replies
  • 88 views

Badge +1

I am currently working on several flows in Klaviyo and I would like to test what kind of e-mails work best with an A/B test. Now I was wondering how Klaviyo decides which of the e-mails from the a/B test the receiver receives. 

Obviously my flow consists of multiple emails and each email sent in the flow consists of an A/B test with an A and B version. if the flow starts, does Klaviyo choose that the recipient of the flow always receives only the A version or can this person in the flow also receive a B version in the next email? how does Klaviyo choose which email the recipient receives? 

Hope you guys understand what I mean, let me know! :) 

 

icon

Best answer by Omar 17 January 2023, 00:37

View original

2 replies

Userlevel 7
Badge +29

Hi @MariekeUNC 

Thank you for posting your question in the community. 

Generally, Klaviyo will randomly assign a subscriber one of the variants in each AB test you’ve set up. So, it is possible that someone would receive all versions As or all versions Bs. However, depending on how long you run your AB test, you will get an even distribution across all subscribers to determine which messages at each stage of your flow perform the best and Klaviyo will show you which version statistically performed the best. 

Now, you’ll want to configure your AB test to either let Klaviyo start weighting the winning variant or keep an even distribution until you decide the winner. If you let Klaviyo weight the winner, over time, the variant that performs the best will receive a higher percentage of users for that test. 

I hope this helps! Please let me know if you have any additional questions.

@In the Inbox 

Userlevel 7
Badge +44

Hi @MariekeUNC,

Depending on what settings you choose Klaviyo will either distribute based on the distribution you've set (i.e. 50%/50%) or an automatic distribution using a multi armed bandid to distribute more traffic to the winning variation to make sure you’re also capitalizing as much as possible.

Now as for your question it sounds like you want to both A/B test an email and also the path that people take depending on the first email they got (A or B).

I would make sure you have a good hypothesis you're testing first either it's the email itself or a sequence of emails (a path).

For a sequence of email you'll need to add a split somewhere, either at the beginning and add a 50/50 random sample conditional split and see which path works best

Or test the email first and have an a/b test running to see which works best and then follow up. You could however add a split after the email a/b test that creates 2 paths depending on which email someone received in the test. IF you want to do this please use, “what someone has done” in the conditional split and choose received email (and find the correct email or email attribute that distinguishes the emails in the a/b test.)

I hope this helps!

 

Omar Lovert // Polaris Growth // Klaviyo Master Platinum Partner

We help with e-commerce growth through CRO, Klaviyo and CVO

 

Reply