Using Global Control Groups to Measure the Impact of Customer Engagement

Let’s face it—as marketers, we’re all trying to prove our impact. With every audience we target, campaign we create, and lifecycle journey we launch, we’re hoping to achieve game-changing results.

But how can we know if our collective strategies were actually successful? While it can be tempting to report out on the typical engagement metrics (e.g. sends, opens, click-through rates), are those the ones that actually matter most? Are they going to make a case for that next big promotion or pique the interest of the leadership team? Are they going to impact the bottom line? According to our research: Not so much.

In the Braze 2021 Global Customer Engagement Review, we found that while 88% of marketing leaders believe their teams have “excellent” or “good” customer engagement practices, 74% still worry their metrics don’t translate to tangible business outcomes. But fear not, there’s an effective solution in sight.

Getting Started with Global Control Groups

Global Control Groups help you measure and showcase the impact of your customer engagement over time. By tying overarching campaign and customer journey performance to key user behaviors and business metrics, you can clearly see how you moved the needle on brand objectives and key results at a high level, not just campaign by campaign. Let’s take a look under the hood to explore the functionality, set-up, and helpful tips.

At a high-level, this feature works into two parts:

1. Creating a Global Control Group
2. Analyzing impact

Creating a Global Control Group

You’re likely familiar with the concept of control groups already. Thinking back to the days of science experiments, control groups help you understand if newly introduced variables (aka variants) yield a positive change compared to an unchanged baseline (aka control). Putting this in the context of customer engagement, Global Control Groups help you measure incremental uplift in conversions from users who have received messages sent via Braze, compared to users who have not received any messages at all.

Thankfully, setting up a Global Control Group is a breeze in Braze.

1. Choosing a Control Group Size

As part of Global Message Settings, you can assign between 1% and 15% of your user base to the control group. For the purpose of this walkthrough, let’s imagine you assign 4%. This means the moment you hit “Save”, a random 4%—or, in this case, ~40K—of your users will be assigned to the Global Control Group and will no longer receive messages via Braze campaigns or Canvas-powered customer journeys.

Pro Tip: Optimal Control Group Size

In general, we recommend your control group be no smaller than 1,000 users. From there, the optimal size will depend on how large your total audience is. When thinking about control group size, you will need a significantly large number of customers in your control so that the behavior analysis is trustworthy. The bigger the audience, the smaller your percentage can be in order to reach that statistical significance. As a bonus, it never hurts to bring in an expert opinion from your data team when it comes to choosing a control group size.

2. Carving Out Messaging Exceptions

Now, there will likely be certain situations, use cases, or times when you will want to bend the rules a bit to message users within a Global Control Group. To account for those situations, you can bake in exceptions using tags. Tags are not only a great way to organize your Braze instance, but they also help you experiment without sacrificing the customer experience. By adding exceptions for certain message types (e.g. transactional), you can ensure that any high-importance messages that absolutely must go out will indeed find their way to customers in your control group.

3. And Then You’re Good to Go

Set-up is really as simple as that. You can always come back to disable or reshuffle control group membership based on your specific use case or experiment.

Pro Tip: Reshuffling Membership

We recommend reshuffling control group membership on an ongoing basis. This ensures that no set of users is unwillingly and continually excluded from messaging. The specific length of time you leave between control group creation and reshuffling ultimately depends upon the elements that you’re looking to test. As a best practice, however, we recommend starting with a period of between one and three months.

Analyzing Impact

After creating a Global Control Group, you can get to the fun stuff: Measuring impact.

1. What’s Possible to Analyze with Global Control Groups

The Global Control Report allows you to track progress across a wide variety of custom conversion metrics.

For example, you could use this feature to understand:

• How your messages moved the needle on app/website sessions
• How your campaigns and Canvases impacted new user referrals
• How much incremental product/feature adoption you’ve driven through messaging
• How your marketing increased the average number of checkouts per user
• And so much more…

Pro Tip: Measure What Matters

You can measure uplift against any custom event you track within Braze. This is an especially helpful way to see how your team is making incremental progress towards identified high-value actions.

In this walkthrough, imagine your Global Control Group has been running for 3 months. Now that the quarter has ended, it’s time to report out on progress. The question you want to answer: “Did we meet our quarterly goal to increase checkouts by 10%”?

2. Identify the Key Insights That Matter to Your Brand

Using handy charts and metrics, you can immediately identify key insights. As seen below, in this example you exceeded the customer checkout goal. Your messaging drove a 14.59% incremental uplift in checkouts. In fact, you encouraged 645 people, who otherwise would not have completed a checkout, to actually checkout. For your unique conversion rate, your messaging drove a 17% higher change over the control.

Pro Tip: Quarterly Planning & Review

Global Control Groups are a great way to help set and showcase progress towards overarching OKRs or major department goals.

Final Thoughts

At the end of the day, marketers have to justify the value of their initiatives. Doing this successfully means putting the typical channel or campaign-specific results on the back burner, and bringing the most impactful conversion metrics to the forefront.

Global Control Groups offer an easy, intuitive way to tie Braze campaigns and customer journeys to the metrics that actually matter. Use this two-part feature to streamline testing and uncover noteworthy success stories—all without ever leaving the Braze platform.

To learn more about how Braze can help your team identify and act on data insights, check out our exclusive webinar on Data-Driven Customer Engagement.