Hi Community, has anyone tried to conduct a hold out test on their Abandoned Cart email to measure effectiveness?
We are looking to utilize the “random sample” conditional split in the flow to conduct this but are uncertain for the best strategy on how to see the impact on those who don’t receive the email.
Thanks in advance,
Solved
Abandoned Cart - hold out test
+6Best answer by David To
Hello
Love the idea of wanting to compare how sending an abandoned cart email will impact you customer’s conversion rates!
I would be highly interested to hear what other strategies other Community members have come up with as well. For my recommendation, I would agree with using a random sample conditional split at the beginning of your flow to separate out traffic of those users who have triggered the flow. I would then suggest creating two segments; one to capture those customers that went through the abandoned cart flow experience, and another capturing those customers who have triggered the abandoned cart flow but was actually removed from the random sample.
You can create the first segment by using the rules of “What someone has Received email at least once in the last X-timeframe where flow equals X (Name of your abandoned cart flow)”.
For your second segment, you can use the rules “What someone has done, Checkout Started at least once in the last X-timeframe AND What someone has done, Placed Order zero times in the last X-timeframe where Attributed Flow equals X (Name of your abandoned cart flow) AND What someone has Received email at least once in the last X-timeframe where flow equals X (Name of your abandoned cart flow)”
When creating both of these segments, you’ll want to ensure the timeframe selected is the same for both segment and for all the rules.
With both of these segments created, you can run an engagement report which can offer you the average performance of both segments. Because each of these segments are capturing customers who have received the flow and a second group of users who abandoned their cart but did not receive the flow, we can broadly compare the performance of each segment’s engagement report to extrapolate how effective your abandoned cart flow is. You can learn more about running a list or segment’s engagement report from through the Understanding Engagement Reports for Lists and Segments Help Center Article.
Thanks for being a Klaviyo Community member!
David
+60Hello
Love the idea of wanting to compare how sending an abandoned cart email will impact you customer’s conversion rates!
I would be highly interested to hear what other strategies other Community members have come up with as well. For my recommendation, I would agree with using a random sample conditional split at the beginning of your flow to separate out traffic of those users who have triggered the flow. I would then suggest creating two segments; one to capture those customers that went through the abandoned cart flow experience, and another capturing those customers who have triggered the abandoned cart flow but was actually removed from the random sample.
You can create the first segment by using the rules of “What someone has Received email at least once in the last X-timeframe where flow equals X (Name of your abandoned cart flow)”.
For your second segment, you can use the rules “What someone has done, Checkout Started at least once in the last X-timeframe AND What someone has done, Placed Order zero times in the last X-timeframe where Attributed Flow equals X (Name of your abandoned cart flow) AND What someone has Received email at least once in the last X-timeframe where flow equals X (Name of your abandoned cart flow)”
When creating both of these segments, you’ll want to ensure the timeframe selected is the same for both segment and for all the rules.
With both of these segments created, you can run an engagement report which can offer you the average performance of both segments. Because each of these segments are capturing customers who have received the flow and a second group of users who abandoned their cart but did not receive the flow, we can broadly compare the performance of each segment’s engagement report to extrapolate how effective your abandoned cart flow is. You can learn more about running a list or segment’s engagement report from through the Understanding Engagement Reports for Lists and Segments Help Center Article.
Thanks for being a Klaviyo Community member!
David
+6Thank you for the detailed response
1 person likes this
+62However, I just wanted to verify with
To offer an alternative approach, you can also use the “Update Profile Property Action” and set a profile property for each path. For example, you can have the action update a property like “abandoned_cart_group = A” and “abandoned_cart_group = B.” Then, you can create a Segment where the profile property for “abandoned_cart_group” is A, and another Segment where “abandoned_cart_group” is B. Then use
Just to note a small edge case, since the Abandoned Cart Flow is typically a “Metric Triggered” (based on Checkout Started event), there’s a possibility that someone might abandon their checkout multiple times within your time window - so some people might start in one path, and then come back and be in the second path (your hold-out group) or vice versa. To avoid that, you can add a Message Filter on the Flow Message to skip anyone where their abandoned_cart_group IS NOT “B” - if “B” is your holdout group. That way, you can ensure your holdout group doesn’t accidentally receive the Cart Abandonment Flow Message on the off chance they abandon their cart a second or more times later.
Hopefully this helps!
+60Hey
Great call out! I had actually meant for that particular definition of the rule to read: Placed Order zero times in the last X-timeframe where Attributed Flow equals X (Name of your abandoned cart flow). I’ve since gone ahead and updated my comment and screenshot above!
Using the “Attributed to Flow” clarifying rule for that section will help in ensuring you are still capturing customers who are making a purchase; just not ones attributed to the Abandoned Cart flow. This would account for customers who may have remembered they had abandoned a product and actively decided to go to the site organically a second time to make a purchase or were swayed some other way such as through a campaign outreach.
Appreciate you also calling out the edge-case scenario as it certainly is a possibility due to how the Abandoned Cart flow is triggered by a metric!
David
Reply
Rich Text Editor, editor1
Editor toolbars
Press ALT 0 for help
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.