Control groups in CRM — in plain language

1. A simple example

You want to send an offer to 100,000 customers. Instead of sending it to everyone:

1. 90,000 randomly selected customers receive the communication.
2. 10,000 similar customers do not — they are the control group.
3. After a predefined period, compare purchases, revenue or another primary metric across the two groups.

2. A control group for a single campaign

For most CRM teams, this is the clearest and most useful option. Before sending a particular campaign, randomly hold back part of the eligible audience from that communication.

• Works for email, push notifications, SMS, banners, personalised offers and other campaigns.
• Helps assess the incremental effect of a specific treatment.
• Especially useful when you need to understand not ‘how much people bought after the campaign’, but ‘how much they bought because of it’.

3. How to make a fair comparison

• Assign groups randomly BEFORE sending.
• Compare the same metric over the same period.
• Do not remove people who did not open the email or see the push notification from the test group: that makes the groups no longer fairly comparable.
• Check whether the control group received the same offer through another channel.
• If a discount is involved, look at profit / margin as well as revenue.

4. How many people should be in the control group?

There is no universal rule of ‘always 5%’ or ‘always 10%’. The absolute number of people and how rare an effect you want to detect matter more.

In a large database, 5% can be a huge and sufficient group. In a small database, even 20% may be too few. For important tests, calculate the size in advance, just as you would for an A/B test.

5. A global control group: a different question and a longer horizon

For a global control group, a small proportion of customers is excluded from a defined set of marketing CRM communications for an extended period. The rest of the audience follows the usual CRM programme. We then compare how the groups behave over time.

Here, it is especially important not to expect an answer after a week or two. The impact of CRM often accumulates: one communication has little effect, but dozens of touchpoints can change purchase frequency, retention and customer value over a longer period.

• Build this group by cohort: for example, separately for customers who first entered the database in the same month or quarter.
• Keep group membership stable long enough to observe the cumulative effect.
• Look at months, quarters and a full repeat purchase cycle, rather than just 7–14 days.
• Define in advance which marketing communications the control group does NOT receive. Service messages — receipts, order updates and security notifications — should generally remain.
• For a long-term comparison, purchases, frequency, retention, revenue per customer, margin and LTV are especially important, depending on the business.

6. A global control group example

Suppose a company has a new cohort of active customers every month. 95% enter the usual CRM programme, while 5% are randomly assigned to the global control group. After 3, 6 and 12 months, you can compare:

• the proportion of customers who returned;
• the number of purchases per average customer;
• revenue and profit per customer;
• how quickly customers stop buying;
• how customer value changes over time.

This approach is more complex than a campaign control group, but it reveals something a single campaign report cannot: whether CRM creates incremental value for the business as a whole.

7. Common mistakes

• Selecting the control group arbitrarily rather than randomly.
• Treating a single campaign’s control group as proof that the entire CRM programme works.
• Looking only at attributed revenue: ‘the person received the email and bought something’.
• Allowing the control group to receive the same discount or offer through another channel.
• Removing non-openers from the test after launch and comparing only active users.
• Assessing a global control group after a week and concluding there is no effect.
• Looking only at revenue and forgetting the cost of discounts and margin.

8. Which control group do you need?

Conclusion

If you want to assess CRM honestly, start with control groups for key campaigns. This alone will significantly improve the quality of decisions. Once the process is stable, add a long-term global control group by cohort — it will help you understand CRM’s overall contribution to retention, purchase frequency and unit economics.