How to A/B test Smart Opt-in versus other SMS opt-in methods

A/B test Smart Opt-in

Because the structure and functionality of the Smart Opt-in form is unique from other forms,  you’ll need to take an additional step to measure the impact of Smart Opt-in on your list growth. 

You can create an A/B test on a Smart Opt-in form to test it against a traditional SMS opt-in method, then create a segment for each variation and measure the segment growth per opt-in method. 

Configure a new A/B test

To do so, you’ll first need to create a new A/B test, and assign each form variation a new hidden profile property:

1. Open the form editor for your Smart Opt-in form.
2. In the top left corner, select Create A/B Test.
   
3. Name your test something descriptive (e.g., SMS Collection - Smart Opt-in vs. Traditional), then name each variation as the opt-in method that it will use (e.g., SMS Collection - Smart Opt-in and SMS Collection - Traditional opt-in).
   
4. Select Create test. From here, you’ll be directed back to the form editor where you can customize each variation to use the corresponding opt-in method. Select the variation name to edit it. Make sure that:
   1. The Smart Opt-in variation has Smart Opt-in configured.
   2. The Traditional opt-in variation has a traditional opt-in method (e.g., a phone number input field or a tap-to-text button) on the SMS Opt-in step, and does not include a one-time code step.

The steps in the following section will depend on if you’re testing Smart Opt-in against a form with a phone number input field, or against a tap-to-text form. 

Smart Opt-in versus phone number input block

If you’re testing a Smart Opt-in form against a form with a traditional phone number input field, add a hidden property on each variation so you can monitor each method’s performance. 

1. Within your A/B test, toggle to the Smart Opt-in variation.
2. Select the SMS Opt-in step, then click the submit button in the preview.
3. Under Submit Hidden Fields, click Add a property.
4. Create a new “Form Variation” property, and label its corresponding value as “SO.”
5. Toggle to the traditional opt-in variation, then repeat steps 2-4, but this time label the Form Variation value as “TO.”

    

6. Click Continue to Test Settings.
7. Configure your A/B test settings for weight and winner selection.
8. Select Publish A/B Test.
9. Select Publish.
10. Skip to the Create segments for each variation to monitor growth section for next steps.

Smart Opt-in versus tap-to-text

If you’re testing a Smart Opt-in against a tap-to-text form, create a custom keyword for the tap-to-text variation, and add a hidden profile property on the Smart Opt-in variation so you can monitor each method’s performance.

1. Once you have customized both variations in your A/B test, click Continue to Test Settings in the top right corner.
2. Select your organization name in the lower left corner. Note that all changes made to your A/B test will save as a draft when you navigate away from it.
3. Navigate to Settings > SMS > Automations.
4. Scroll to the Subscribe Keywords section, then click Add keyword.
   
5. In the Create Subscribe Keyword modal:
   1. Name your new custom keyword (e.g., TEXT).
   2. Choose the list that subscribers who use the keywords should be added to.
   3. Choose whether you want to use single opt-in or double opt-in. Note that Smart Opt-in, which you’re testing against, is a double opt-in method.
      
6. Click Save.
7. Select the Sign-up forms tab on the left-hand navigation.
8. Select your form with the draft A/B test to continue configuring the test settings.
   
9. Click Edit next to the traditional opt-in variation.
   
10. Click on the submit button in the preview of the SMS opt-in step.
11. Click the SMS Keyword dropdown and choose the new custom keyword you just created (e.g., TEXT).
    
12. Toggle to the Smart Opt-in variation.
    
13. Select the SMS Opt-in step, then click the submit button in the preview.
14. Under Submit Hidden Fields, click Add a property.
15. Create a new “Form Variation” property, and label its corresponding value as “SO.”

     

16. Click Continue to Test Settings.
17. Configure your A/B test settings for weight and winner selection.
18. Select Publish A/B Test.
19. Select Publish.

Create segments for each variation to monitor growth

Next, create a segment for each variation so you can monitor growth per opt-in method.
CALLOUT: Note that there must be at least 1 submit per form variation in the A/B test to create the following segments.

1. Navigate to Audience > List & segments.
2. Select Create New > Create segment.
3. Within the segment builder, add this definition:
   • Properties about someone > Form Variation > equals > SO
4. Click Create segment.
5. Repeat steps 1-4 to create another segment for the other variation in your A/B test. This time, depending on if you’re testing against a phone number input field form or a tap-to-text form, add this definition:
   • Phone number input field
     Properties about someone > Form Variation > equals > TO
   • Tap-to-text
     Properties about someone > $source > equals > rSubscribe keyword (e.g., TEXT)] 
6. Navigate back to the Segments page, then click the star icons next to the 2 segments you just created.
   

Once you’ve created segments for each variation, use the Segment growth tab for each one to monitor the corresponding form variation’s performance. 

Alternatively, you can also use segment growth cards on the overview dashboard. Learn more about the segment growth report.

Additional resources

Getting started with sign-up forms

Gettings started with SMS

Basics: SMS compliance