Email A/B test sample size: how many emails before you trust a winner

The sample size for an email A/B test depends on two numbers: your current reply rate and the smallest lift you want to be able to detect. At a 3% reply rate, detecting a lift to 4.5% takes about 2,500 emails per version at the usual 95% confidence and 80% power. Detecting a doubling to 6% takes about 750. Most small cold email campaigns send far less than that, which is why a winner called on 200 emails per version is usually noise. When you cannot reach the sample, narrow your drafts first on simulated prospects, for example with an AI simulator that A/B tests cold emails on the same AI agents, and save live sends for confirming one strong version.

This guide gives you the table, the formula in plain words and the practical ways around small samples. It is part of our series on testing a cold email before you send it.

Key numbers

Number	Why it matters for sample size	Source
3.43%	Average cold email reply rate in 2025, the base rate most tests start from.	Instantly, Cold Email Benchmark Report 2026
449	Average number of recipients per cold email sequence in 2025. Split in two, that is about 225 per version.	Hunter, The State of Cold Email
About 2,500	Emails per version to detect a lift from 3% to 4.5% replies.	Our calculation, two-proportion test, 95% confidence, 80% power
About 750	Emails per version to detect a lift from 3% to 6%.	Our calculation, same settings
26.1%	False winner rate when you stop a no-difference test at the first significant result.	Evan Miller, How Not to Run an A/B Test

How many emails do you need for an A/B test?

Here is the number of emails you need per version, at 95% confidence and 80% power, for some common starting reply rates and the lift you want to detect. Each figure comes from the standard formula for comparing two proportions, the same one used by most online A/B test calculators.

Current reply rate	Lift you want to detect	Emails per version	Total for an A/B test
1%	to 1.5% (a 50% lift)	about 7,750	about 15,500
1%	to 2% (doubling)	about 2,300	about 4,600
3%	to 3.6% (a 20% lift)	about 13,900	about 27,800
3%	to 4% (a 33% lift)	about 5,300	about 10,600
3%	to 4.5% (a 50% lift)	about 2,500	about 5,000
3%	to 6% (doubling)	about 750	about 1,500
5%	to 7.5% (a 50% lift)	about 1,500	about 2,900
5%	to 10% (doubling)	about 440	about 870

Two patterns stand out. Small lifts need enormous samples, and low base rates make everything harder. If your campaigns get 1% replies, even a doubling takes thousands of emails to confirm.

The sample size formula, in plain words

You do not need to do the maths by hand, but it helps to know what drives it. Four inputs decide the sample size:

• The base rate. Your current reply rate. Rarer outcomes need bigger samples.
• The minimum detectable effect. The smallest lift you care about. Halve the lift and you need roughly four times the sample.
• Confidence (significance). How sure you want to be that a winner is not a fluke. 95% is the usual choice, which accepts a 5% chance of a false winner when there is no real difference.
• Power. How likely you are to spot a real lift of the size you chose. 80% is the usual choice, which means one in five real lifts will be missed.

Plug those into any calculator, such as Evan Miller's sample size calculator, and you get emails per version. Use prospects, not total emails including follow-ups, if you are measuring replies per prospect.

What is statistical significance in an email test?

A result is statistically significant when the gap between two versions is bigger than random variation would usually produce at that sample size. It does not mean the result is large or important, only that it is unlikely to be pure chance.

A worked example shows why small tests rarely get there. Say each version goes to 200 prospects. Version A gets 6 replies (3%) and version B gets 9 (4.5%). B looks 50% better. Run the numbers and the p-value is about 0.43: a gap like this would appear by chance almost half the time if the two emails were identical. Even 6 against 12 replies, a doubling, gives a p-value of about 0.15, still not significant. With 2,500 prospects per version, 75 against 113 replies gives a p-value under 0.005. That is a result.

We also simulated the 200 per version case directly: two identical emails, both with a true 3% reply rate. In about half of the runs, one of them came out at least 50% "better" than the other. At 1,000 per version that dropped to about one run in eight, and at 2,500 per version to about one in seventy.

How to run an email A/B test when your list is small

1. Test bigger differences. A different audience, offer or angle can plausibly double replies. A different adjective cannot. The table above shows how much easier a doubling is to detect.
2. Measure a more common outcome. Total replies happen more often than meetings. Use them to judge the copy, and watch positive replies as a check.
3. Pool tests over time. Run the same two versions across several campaigns to the same kind of audience, and add up the results before you decide.
4. Use fewer versions. Every extra version needs its own full sample. With a small list, two is the limit.
5. Fix the sample in advance and do not peek. Decide the number of prospects per version before sending and read the result once, after the follow-ups finish.
6. Screen drafts before sending. Use a copy review, the subject line tester and simulated prospects to throw out weak drafts, so the live test only compares strong ones.

If you need more sending capacity to reach a sample safely, the email infrastructure calculator shows how many mailboxes and domains a given daily volume needs, so you do not push a few mailboxes too hard and damage deliverability.

Common sample size mistakes

• Using total emails instead of prospects. A four step sequence to 500 people is 500 prospects, not 2,000 samples, when you measure replies per prospect.
• Calling the test as soon as it looks significant. Checking repeatedly and stopping at the first significant moment inflates the false winner rate, in Evan Miller's example from 5% to 26.1%.
• Choosing the lift after the test. Decide the minimum detectable effect first, or you will always find one that fits.
• Ignoring that lists differ. A new list can change reply rates more than any copy change. Compare versions within one random split.
• Treating "not significant" as "no difference". A small test that finds nothing has not proved the versions are equal. It just could not tell.

Our guide on why A/B tests mislead at low volume covers these traps in more detail, along with how many versions to test at once.

Where ZicReach helps

In the ZicReach AI Simulator you choose how many AI agents to test on: 10,000, 50,000, 100,000 or 1,000,000. Every version in an A/B test meets the same agents, so the sample is the same for each and the list is identical by design. The report names a winner only when the gap is unlikely to be chance, says how confident it is, and otherwise says the versions are too close to call and suggests testing a bigger change.

Use it to compare up to five drafts, keep the one or two that clearly win, and send those for real. Each version costs one test in credits, and the price for the size you choose is shown before you run it. It is a prediction for the audience you describe, not a measurement of real buyers, so confirm the winner with a live send.

Questions people ask

What is a good sample size for an email A/B test?

Enough emails per version to detect the lift you care about at your reply rate. At a 3% reply rate that is about 750 per version to detect a doubling and about 2,500 to detect a 50% lift, at 95% confidence and 80% power.

Is 100 emails enough for an A/B test?

Not for cold email. At typical reply rates, 100 emails produce three to five replies, and the gap between two versions at that size is almost always noise. Use 100 email batches to spot broken emails, not to pick winners.

How do I calculate statistical significance for an email test?

Use a two-proportion significance test, which any A/B test calculator runs. Enter the number of prospects and replies for each version. A p-value below 0.05 is the usual threshold, but only if you set the sample size in advance and did not stop early.

What is a minimum detectable effect?

The smallest improvement your test is designed to detect reliably. A smaller minimum detectable effect needs a much bigger sample: halving it roughly quadruples the number of emails you need.

Can I A/B test with a small list?

Yes, if you test large differences, use two versions only, pool results across campaigns and screen weak drafts before sending. For fine differences, a small list cannot give a reliable answer.

The short version

Work out the sample size before you send: your reply rate and the lift you care about decide it. At a 3% reply rate, plan on about 2,500 emails per version for a 50% lift and about 750 for a doubling. If you cannot reach that, test bigger changes, pool results and screen drafts on simulated prospects first.

To compare drafts on the same simulated prospects before you spend real sends, start ZicReach for free. The pricing page shows what each plan includes, and how to improve your cold email reply rate lists the changes big enough to test.

Cover photo by Aaron Lefler on Unsplash.