Your Tests Are a Waste of Time Without 1 Simple Thing

In an early episode of Friends, Joey and Chandler need to make a decision, so they decide to flip a coin. They flip the coin, which lands on heads, and the decision is made! Except it isn’t; neither of them had assigned a value to heads or tails. They were back at square one.

Marketers can run into this conundrum when conducting A/B or multivariate tests. You can set up as many tests as you’d like, gather as much information as you want, but if you don’t start out by defining what your success metric is, you might be left with a bunch of results you can’t actually use.

The first thing you should do before running an A/B or multivariate test is to explicitly outline your success metric for that specific test.

What is a success metric?

A success metric is a goal that lets you know whether or not a test has delivered useful insight that you’ll want to take action on. You might be testing two different versions of your new in-app message design, but how are you gauging if one design is performing better than the other? Is it total time in app? Is it the number of conversions to the coupon advertised on the message? That depends, of course, on what your broader goal is behind your test; it depends on your success metric.

Stating your success metric right off the bat will help you focus on making meaningful change in your business.

Identify which key business objective this test contributes to

To continue the example above, your ecommerce app could be testing the design of two “See the Sale” CTA buttons in an in-app message. But of course the goal isn’t to see which button results in the most clicks, just for fun. You’re probably trying to determine which design results in the most orders—meaning the key business objective it relates to is driving orders.

Make sure this test is aligned to deliver results around that goal

In the case of our sale message test, then, you’ll want to measure conversions achieved through each variable, versus the clicks on the CTA or any other metric. Clicks on the CTA may be the easiest and most obvious thing to track, but it won’t help answer your real question.

Now you’re ready to run a test

Now you can run your test with a goal in mind. You’re not trying to discover any old thing—you are after a solution that moves the needle on your conversions. This goal, whether it’s achieved or not by the end of the test, is what helps you make a useful test. You’re now better equipped to decide what your variant should look like, and you’re also ready to track the important metric—you can get any tracking codes or custom event tracking in place to capture what you really care about.

Essentially, to come up with your success metric, you’re coming up with an answer to the question, “What about this test would spell a win for my business?”