How Braze Feature Flags Can Support A/B Testing and Experimentation

Consider Braze Feature Flags. This tool makes it possible for brands to easily toggle specific features or elements within their app or website, all without deploying extra code or having to submit an app store update. But feature flags can also empower Product and Engineering teams to roll out and test new features across various user cohorts, enabling data-driven decisions based on performance.

In particular, A/B testing with feature flags can play a crucial role in enhancing user experiences and achieving business goals. By integrating Braze Feature Flags for A/B testing with our core messaging channels and features, your team can leverage all available tools to create the best possible experience for end users.

Feature flags for A/B testing: How they support optimization

A/B testing with feature flags can support optimization in several ways. One powerful functionality is the ability to customize features according to user preferences or behaviors that update based on real-time feedback. For instance, if a user signs up on your website and indicates a love for fantasy novels, you might want to test a new website experience tailored to them, such as a dragon-themed side panel during their next visit.

This approach allows you to measure conversion rates, including checkouts and purchases, as well as track other relevant events, such as views of new fantasy releases, items added to a shopping cart, or time spent on the website. By leveraging Braze Feature Flag Experiments, you can easily implement and assess these changes across user segments, using control groups as a baseline to drive data accuracy.

Using feature flags for A/B testing in practice

To explore some of what’s possible when it comes to leveraging Braze Feature Flags to support A/B testing, let’s take a look at three different hypothetical scenarios where this tool could be used in your testing and optimization efforts:

1. Introducing gamification to boost checkouts

In a recent update, your app aimed to enhance user engagement and increase checkouts by introducing a new gamification feature that rewards users for completing specific actions (e.g. browsing products, adding items to their cart, making purchases). However, you wanted to ensure that this feature was appealing to your existing users who had upgraded to the latest version of the app. To test its impact, you set up an A/B test using Braze Feature Flags.

You created a segment of existing users who had updated their app and used a feature flag to control who would experience the gamification feature. Some users were shown the gamification elements (the test group), while others continued with the standard experience (the control group). To measure the success of the gamification feature, you tracked two key metrics: The number of add to cart events and total purchases.

After running the test for three weeks, the results were encouraging. The gamification feature led to a 25% increase in add to cart events and a 15% rise in purchases. With these positive outcomes, you decided to roll out the gamification feature to all users, enhancing their shopping experience and encouraging more purchases through engaging and rewarding interactions.

2. Testing a new checkout flow for an eCommerce website

Your website's checkout process currently requires users to click through eight steps to complete a purchase, which may be hurting customer satisfaction and sales. To improve this, your Product and Engineering teams developed two solutions: A three-click checkout process and a one-click checkout option using a third-party platform. To find out which option drives more purchases, you conducted an A/B test using Braze Feature Flags.

In the experiment, you set up two variants: Variant A was the three-click checkout, and Variant B was the one-click option. You also included a control group that used the existing eight-click process, targeting a segment of frequent users to minimize potential issues.

Running a feature flag experiment for a month during President’s Day, you observed that the one-click checkout initially performed well but quickly declined, while the three-click checkout gained traction and became the winning option. Although the one-click version generated 50% more purchases than the control group, its slower performance hurt its overall effectiveness.

Ultimately, you decided to implement the three-click checkout across your website, resulting in a 20% increase in purchases over the next six months!

3. Testing new visual effects as the seasons change

As winter approaches, your website can embrace the festive spirit by adding special features and interactive elements to it, such as animated snowflakes to the website's background. To ensure these enhancements are relevant to your audience, you can use Braze Feature Flags to A/B test these seasonal animations, targeting only users in the northern hemisphere where winter is in full swing.

You create two versions of your website: one with the animated snowflakes gently falling across the screen and another with a subtle twinkling lights animation. By gating these visual CSS and JavaScript components with Braze Feature Flags, you can easily control who sees these engaging features. This allows you to show the animations to Northern Hemisphere users while keeping the standard experience for those in the Southern Hemisphere, where it’s summer.

After running the experiment for a month, you notice impressive results. The animated snowflakes lead to a 10% increase in click through rates and an 8% rise in the time users spend on the page. These findings highlight how seasonal, interactive elements can enhance user engagement and create a more enjoyable browsing experience during the winter season!

4. Optimizing search within your app

Your music streaming service has a huge collection of songs, but users are finding it harder to search for music as the catalog has grown. This has led to a 12% drop in songs played over the last three months because users often leave without listening to anything.

To fix this, your engineering team developed two options to improve the search function. One option uses a method called an inverted index to help with searches, while the other uses a radix tree to make search suggestions more accurate. Since both methods can’t be used at the same time, you decided to test them using a feature flag in Braze.

You selected a small group of about 500,000 users to test these changes, splitting them into three groups: One using the inverted index, one using the radix tree, and a control group that continued with the current search. You tracked how many songs were listened to and also monitored if users were leaving the app due to poor experience.

After four weeks of testing, you found that the radix tree method led to 40% more songs played compared to the inverted index and 75% more than the control group. As a result, you decided to implement the radix tree across all your apps and websites, which boosted song listens by 25% over the next year.

Making informed business decisions using feature flag experiments

When running feature flag experiments, it’s important to start by identifying the desired business impact of the changes you are making. Working backwards from there will help you outline the right conversion events to measure in order to assess the success of your experiment. For instance, in the search optimization example, the drop in song listens related to a bad search engine made it clear that we needed a way to assess both song listens and app closures to get a full picture of the impact.

These conversion events help us measure not only how a change to our app or website matches up to other changes we are implementing (variants), but also how they match up to the baseline of what we currently have (control groups). In that example, none of the changes led to performance below the control, but that does happen—and can be a valuable warning sign before you implement a counterproductive approach more broadly.

Making informed decisions based on the data that we are presented with is essential, and feature flag experiments make it easier to control experiments and assess impact in clear, meaningful ways. By taking advantage of this tool, you can get a better picture of what is and isn’t connecting with your audience and then use those insights to improve things going forward.

Interested in learning more about Braze Feature Flags, check out “Braze Feature Flags: What They Are and How to Use Them.”

Forward-Looking Statements

This blog post contains “forward-looking statements” within the meaning of the “safe harbor” provisions of the Private Securities Litigation Reform Act of 1995, including but not limited to, statements regarding the performance of and expected benefits from Braze and its products and features. These forward-looking statements are based on the current assumptions, expectations and beliefs of Braze, and are subject to substantial risks, uncertainties and changes in circumstances that may cause actual results, performance or achievements to be materially different from any future results, performance or achievements expressed or implied by the forward-looking statements. Further information on potential factors that could affect Braze results are included in the Braze Annual Report on Form 10-K for the fiscal year ended January 31, 2025, filed with the U.S. Securities and Exchange Commission on March 31, 2025, and the other public filings of Braze with the U.S. Securities and Exchange Commission. The forward-looking statements included in this blog post represent the views of Braze only as of the date of this blog post, and Braze assumes no obligation, and does not intend to update these forward-looking statements, except as required by law.