A/B test on proportions: sample size, duration and reading the result
In the “Plan” mode counts how many visitors each group needs and how many days the test must run to notice a given conversion lift with the required power; in the “Result” mode tests the significance of the difference, shows the confidence interval of the difference and the smallest noticeable difference.
At a glance
- What question does it answer?
- How many visitors and days are needed to notice a conversion lift, and whether to believe the difference the test showed.
- What data is needed?
- The baseline conversion and the smallest lift you want to notice.
- How to read the result?
- Example on the default values: to notice a conversion rise from 3% to 3.3% at 80% power and 5% significance you need 53,211 visitors per group, 106,422 in all, at 4,000 a day that is 27 days; half the lift needs 207,938 per group.
Why and how to reason
An A/B test compares two versions and answers whether a difference is real or chance. To avoid fooling yourself, three things are decided before the test: how small a difference you want to notice, how often you are ready to err toward “there is a difference” (significance level 5%) and how often you are ready to miss a real difference (power 80%). The smaller the difference, the more people are needed. For a base conversion of 3% and a lift of 10% (to 3.3%) you need 53,211 people per group, 106,422 in total. At a traffic of 4,000 a day the test runs 27 days, almost four weeks. If you look for a lift half as big (5%), you need 207,938 people, four times more. If twice as big (20%), only 13,914 in total. In “Result” mode the calculation checks whether the difference of your groups is significant and shows a confidence interval.
When to use it
It is useful before launching a test on a site or in a mailing to see whether it is worth the time, and after the test to read the result honestly. Do not use it if traffic is too small for a noticeable difference or the test cannot be completed to plan: better pick a bolder change. Assumptions: groups are independent, visitors are randomly assigned, the test runs in full weeks.
How to read the result
In “Plan” mode look at the duration: if the test runs more than two months, revisit the hypothesis, raise the minimum noticeable lift or collect more traffic. Do not stop a test early at the first significance: this raises the risk of a false win. In “Result” mode read the interval of the difference, not just the p-value: if the interval includes zero, no difference is proven, and if it lies above zero but wide, the effect may be small or large. Make the decision on the size of the effect in money, not just on significance. Write down in advance what you will do for each outcome: win, draw, loss. Then the result will not become a subject of argument.
Common mistakes
The usual mistakes: peeking at the result and stopping the test early, changing the hypothesis during the test, running a test over a change of season, comparing groups of different duration, treating significance as proof of importance. Write the plan in advance: the metric, the duration and the sample size. Do not run several tests on the same visitors: effects mix and significance loses its meaning.
What data is needed?
- The baseline conversion and the smallest lift you want to notice.
- Test traffic per day.
- After the test: visitors and conversions of each group.
Inputs
| Field | Unit | Value in the example | Meaning |
|---|---|---|---|
| What to count | Plan: how many visitors are needed | First the plan before the launch, then reading the result once the sample is collected. Peeking and stopping early is not allowed: false wins multiply. | |
| Significance level (two-sided) | 5% | The acceptable probability of a false win. Your choice; with several checks at once take a stricter one. | |
| How many variants you compare with the control | pcs | 1 | 1: one variant B, as before. With several variants (B, C, D) the significance level for one comparison is divided by their number (the Bonferroni correction): fewer false wins, a larger sample. |
| How many times the result was looked at before the end | times | 1 | 1: looked once by plan, as before. More: each look with a “stop if significant” decision raises the chance of a false win above the chosen level; the calculation shows an estimate. |
| What counts as success (the “Result” mode) | Conversion | In a price test conversion may fall while profit rises: then look at profit per visitor. Set the margin per order below; its spread between orders is not taken into account. | |
| Margin per order in group A | ₽ | 1,500 | For the “profit per visitor” mode. Excl. VAT, after cost and discounts. |
| Margin per order in group B | ₽ | 1,650 | For the “profit per visitor” mode: at the new price the margin per order differs. Excl. VAT. |
| Baseline conversion | % | 3 | The share of visitors taking the target action now: an order, add to cart. For footwear and children’s conversion by size is lower: take your own. The example value is illustrative. |
| Smallest lift you want to notice | % relative | 10 | Relative lift of conversion: 10% of 3% is 3.3%. The smaller the lift, the larger the sample. |
| Test power | 80% | The probability of noticing the lift if it exists. Your choice. | |
| Test traffic per day | visitors | 4,000 | How many visitors per day enter both variants together. For the test duration; may be left empty. |
| Visitors in group A | people | 20,000 | How many visitors are in group A (the control variant, the current page) over the test; in the read mode. |
| Conversions in group A | units | 600 | Not above the number of visitors. |
| Visitors in group B | people | 20,000 | How many visitors are in group B (the new variant) over the test; in the read mode. |
| Conversions in group B | units | 680 | Not above the number of visitors. |
How it is calculated
| Step | Formula |
|---|---|
| Conversion B | p2 = p1 × (1 + lift) |
| Sample per group | (z(α/2) × √(2 × p̄ × (1 − p̄)) + z(β) × √(p1(1 − p1) + p2(1 − p2)))² ÷ (p2 − p1)² |
| Days | 2 × sample ÷ traffic per day |
Worked numbers
Default values from the app; in the interactive version you enter your own.
- Conversion B to test
- 3.3 % · +0.3 pp
- Visitors per group
- 53,211
- Total visitors for the test
- 106,422
- Test days
- 27 daysRound up to whole weeks.
- Test weeks
- 4 weeks
- Sample at half the lift
- 207,938 per groupThe sample grows about four times.
- Sample at double the lift
- 13,914 per group
Data as a table
| Conversion lift, % | Visitors per group |
|---|---|
| 5 | 208k |
| 10 | 53.2k |
| 15 | 24.2k |
| 20 | 13.9k |
| 30 | 6,455 |
Sample at different lifts
| Lift, % | Conversion B | Per group | Total | Days |
|---|---|---|---|---|
| 5 | 3.15 % | 207,938 | 415,876 | 104 |
| 10 | 3.3 % | 53,211 | 106,422 | 27 |
| 15 | 3.45 % | 24,193 | 48,386 | 13 |
| 20 | 3.6 % | 13,914 | 27,828 | 7 |
| 30 | 3.9 % | 6,455 | 12,910 | 4 |
The sample size sets the power: the smaller the lift you want to notice, the longer the test. If traffic is low, test big changes and not small tweaks.
Round the duration up to whole weeks and avoid promotions and sales. For seasonal footwear and children’s categories demand shifts week by week: a one-week test does not show everything.
Nuances and limits
- Stopping the test the first time significance appears: false wins pile up.
- Treating “not significant” as proof of no effect: the sample may be small.
- Running a test during a promotion or on a partial week.