Klaviyo Forms AI - Fast track your list growth

  • 1 April 2024
  • 3 replies
  • 108 views

Userlevel 7
Badge +45
  • Champion & Partner
  • 489 replies

Klaviyo has recently introduced Klaviyo AI with many awesome features. 

One of the great features I've tested recently was the forms display optimization feature which allows you to cut down the time to test various variables in an automated way to find the best display timing. These are my first findings - so I'll try to update this topic as I've ran multiple of these tests.

Why I like this feature:

I'm was and still am very excited about this feature mainly because forms are the top of the funnel when it comes to email. It allows you to cut down precious thinking time and test various versions in an automated way.

 

Important things to know:

  1. You need an account with at least 400K profiles
    since it will be running various a/b tests at the same time - statistics is what it is - you need enough traffic.
  2. It will only test various display options / variations in an automated testing way
    No need to think about anything, tests will run automatically and winners will be tested against baseline.
  3. Since it will test various display (timing) options make sure you don't have multiple forms triggering.

How to set this up:

  1. Create a new test and choose the optimization test option - give it a name and click on Continue to test settings:
  2. Select the distribution of the traffic - the lower you set the percentage the longer it will take. Select any advanced settings of minimum required thesholds you'd like to use.
  3. The experiment will now run through various phases testing various settings and showing the end result.

 

First Results:

As you can see in the screenshot below we had a massive increase of almost 44% more signups in the winning variation. Believe me I had a very happy client!

This has cut down testing time by weeks if not more.

 

My first thoughts

While this takes down the thinking and increases speed on finding the right timing my first thoughts are - won't the shortest display time always win? How do we account for customer experience? 

I'll have to run some additional tests to see if this is the case indeed and also think about the experience of users and use the advanced sections to have a minimum threshold.

What I'd like to see next:

  • Not only automate testing of display timings but also with content options
  • Know what the bottom line effect is of the popup - does it also increase or decrease conversion or bounce rate? 
  • Be able to use this with less profile accounts

 

Further reading:

  • More information about Form display optimization in this Klaviyo help article
  • Two awesome community champions recently posted about some findings and learnings here:

 

 

Omar Lovert // Polaris Growth // Klaviyo Master Platinum Partner

We help with e-commerce growth through CRO, Klaviyo and CVO

Let's connect on linkedin


3 replies

Userlevel 5
Badge +26

This is an awesome breakdown, Omar! Thank you for sharing :)

Userlevel 7
Badge +45

Your welcome!

I'll try to update this post as I get new insights as well.

As for the question: 

won't the shortest display time always win? 

 

We've been testing a lot and noticed that the recommendations that this features comes up with are showing a complete different picture. Some strange combinations like 95% scroll or 32sec delay which show to be winning. So - shortest time doesn't always win and the recommendations it tests are so random where I would have thought these would be more rule based.

 

@Taylor Tarpley I'd love to hear more about the logic behind this. Any chance one of the engineers can share more about this?

 

Omar Lovert // Polaris Growth // Klaviyo Master Platinum Partner

We help with e-commerce growth through CRO, Klaviyo and CVO

Hi Omar,

I'm one of the engineers who worked on and support this feature.

 

How does Klaviyo AI figure out the best settings? We explore many different sets of display settings and observe if a given set performed better or worse than previous settings.

 

The first phase of a form optimization is unique. It uses the display settings configured for your form as the control, and compares that to up to six other variations with display settings that explore a wide range of settings all at once. You could consider this a "breadth-first" search.

 

After phase 1, the following phases narrow down on a particular set of settings to compare against the control. The algorithm looks at all previous test results and what hasn't been tested yet looking for better performance. This is why sometimes you may see strange combinations of settings - this is the optimization exploring the search space looking for better performance.

 

If the control settings are suboptimal, as in the example you provided, then the algorithm should be able to find a better alternative.

 

If you want to prevent show on page load or other behaviors because of what you already know about your form and its audience, you can limit the search space using the Advanced Settings to block certain combinations of settings from being selected.

Reply