5 Rules for Smart Multivariate Testing

A campaign that doesn’t resonate with your customers is more than just a lost opportunity—it’s a problem. Too many irrelevant messages can lead to messaging opt-outs, uninstalls and just general bad feelings among your customers. Thankfully, there’s a simple way to improve the effectiveness of your messaging campaigns: multivariate testing.

With multivariate testing, you can compare multiple versions of the same message to see which one performs best among the customers you’re trying to reach. This lets you examine and fine-tune your outreach based on a number of different factors, including wording, image inclusion, message format and more.

The benefits of multivariate testing

The benefits of testing can be big—a recent Appboy study found that campaigns using the best-performing message variant of a multivariate tested message have conversion rates 40% higher than the control group. But if you aren’t thoughtful about your use of multivariate testing, you can end up making decisions based on incomplete or inaccurate data. So before you get started, make sure you’re keeping these five rules in mind.

The five rules of multivariate testing

1. Don’t include differences you’re not trying to test

When you’re creating a multivariate test, it’s important to make sure that the only differences you introduce between the variants are the differences you want to test. There’s nothing wrong with testing both different wording and the use of different pictures—multivariate testing is flexible enough to support both simple one-on-one comparisons and more complex assessments. But if you’re really only interested in knowing what effect the wording has, you’re introducing unhelpful variability by changing the pictures, too. Be clear about the purpose of each test from the beginning and don’t introduce variables that will distract from that focus.

2. Choose a sample size that fits

When you run a multivariate test with a sample size that’s too small, you can end up with results that aren’t statistically significant. Having a sample size that’s too big, on the other hand, can make it harder to run the test, drag out completion and reduce how actionable to results you receive are—after all, if you test your whole customer base, who are you going to send the winning variant to? While the sample size you use will differ from test to test, the goal should always be to have a sample that’s representative of the population you’re testing and allows for statistically significant results.

3. Randomize, randomize, randomize

If you carry out a multivariate test with a non-randomized sample, you’re running the risk that different variants are being received by populations that don’t resemble each other or the audience you’re trying to understand, whether that’s your whole customer base or just people who have opened your app in the last week. By randomizing your sample, you can ensure that the test population is a good representation of the population you’re examining, significantly reducing the chances that you get misleading results.

4. Set a testing period and stick to it

It’s tempting. You seen a variant getting great results early on and want to halt the test prematurely to make use of it. But if you haven’t gathered enough data for the result to be statistically significant, doing that can potentially skew your findings. Pick a testing period that allows you to connect enough data points to produce actionable results and then see it out. Doing otherwise can lead you to make marketing decisions based on bad information.

5. Use a control group if you can

By adding a control group to your multivariate testing, you can often get deeper insight into how each message variant is performing. A control group won’t receive any of the message variants, allowing it to serve as a baseline that each variant’s results can be compared against. That information can be quite valuable: after all, if your best-performing variant sees better results than the other versions of the message but underperforms the control group, you’re going to be better off not sending it. Not every type of multivariate test works with a control group—if you’re testing to determine the open rate of different variants, a control group won’t give you any useful information—but if it’s possible to use one, you should.

Getting your customer relationships right isn’t easy—and it only gets harder if your messaging campaigns aren’t everything they could be. So start sending better, more effective messages with the help of multivariate testing.