Syntha Lab RU

A/B tests for promotions and promos: hypothesis, sample, power, p-value, peeking

How to test fairly whether a promotion, a price or a site change works: state a hypothesis, calculate sample size and duration, judge the result by p-value and interval, and avoid being fooled by multiple comparisons and peeking. The outcome decides whether to roll the change out and with what confidence, and when the test must run longer.

Math and analytics Brand and retail Verified: 2026-11-13

Open interactively →

Why this document

The heads of e-commerce, marketing and analytics read this before rolling a promotion, price or new screen out to everyone. It matters because a difference in conversion between two groups is often pure noise, and without a sample calculation a test either shows nothing or gives a false success. Peeking at results and running a dozen comparisons at once greatly raise the chance of seeing an effect where none exists. A sample size, duration and minimum detectable effect set in advance protect against a decision that costs money.

At a glance

What is this document and why is it needed?
How to test fairly whether a promotion, a price or a site change works: state a hypothesis, calculate sample size and duration, judge the result by p-value and interval, and avoid being fooled by multiple comparisons and peeking. The outcome decides whether to roll the change out and with what confidence, and when the test must run longer.
Who owns the document and how often is it rebuilt?
The experiment analyst with the hypothesis owner (marketing, e-commerce, category manager); the test design is approved before launch and the result is recorded in a protocol; finance receives the effect estimate in money.
What mistakes are the most common?
A test is launched without a sample size calculation → not enough power, the effect is invisible → compute n and duration before launch. Peeking and stopping at the first p < 0.05 → false positives rise several-fold → one look or a sequential plan. Five variants are compared without a correction → one “win” is random → a Holm or Bonferroni correction.

For whom

Brand
A brand tests changes in own channels: pages, mailings, prices, D2C promotions; in wholesale and on marketplaces control is limited, so geo tests and before/after with a control group are used. The effect is converted into margin, not clicks.
Retail
A retailer tests promotions and displays on store groups (test and control with similar history) and online on part of the traffic; stores are paired by size, region and trend, otherwise group differences eat the effect.

What it consists of

Hypothesis and metric

  • The null hypothesis H₀: there is no effect; alternative H₁: there is. There is one primary metric (conversion, margin per visitor, average ticket), fixed in advance; secondary metrics (returns, revenue of neighboring goods) are guardrails and do not take part in the decision unless breached.
  • The minimum detectable effect (MDE) is set in money: the smallest improvement that repays the change and the effort. A test that cannot detect the MDE is not launched.

Sample size, power and duration

  • For two shares n per group = 2·(z_(α/2) + z_β)²·p̄(1 − p̄)/δ². At a baseline conversion of 9%, α = 5%, power 80% and an MDE of +10% relative (9 → 9.9%) you need 16,583 visitors per group; for an MDE of +5% already 64,915, for +20% 4,322. Size grows as 1/δ²: half the effect needs four times the sample.
  • With 18,400 visitors a week to the site (split equally into two groups) the required sample is collected in 1.8 weeks, and the duration is rounded to whole weeks (two) to cover all weekdays. Power of 80% is reached at 16,584 visitors per group, at 8,000 it is only 49%.

Result: p-value and interval

  • Example: control 1,530 purchases out of 17,000 (9%), variant 1,683 out of 17,000 (9.9%). Z = 2.84, p = 0.0046; the interval of the difference is 0.28…1.52 pp. The p-value is the probability of seeing such or a larger difference when there is no effect, not the probability that the hypothesis is true.
  • The decision is made on the interval and money: the lower bound of the interval converted into margin must repay the change; “significant but small” is no reason to roll out.

Multiple comparisons and peeking

  • Five variants against control at α = 5% give a probability of at least one false positive of 22.6%. The Bonferroni correction (α/m = 1%) rejects 1 of 5 hypotheses at p = 0.003; 0.012; 0.021; 0.04; 0.3, the Holm (step-down) correction rejects 2: Holm is more powerful with the same error control.
  • Peeking (stopping at the first p < 0.05) inflates false positives: with five equally spaced looks on a test without effect they occur in 14.2% of cases instead of 5%, with ten 19.3%, with twenty 24.8% (an exact calculation by numerical integration). Either one look on a pre-set day or a sequential plan with recomputed bounds.
  • Brand: in D2C peeking is the most common because of real-time dashboards; Retailer: in stores changing group composition during the test and overlapping promotions are dangerous. Stopping rules and guardrail metrics are written before launch.

Brand and retailer: how experiments are run

  • Brand: the brand randomization unit: a visitor or customer in D2C, a partner or region in wholesale; in wholesale there are few units so paired comparisons and differences-in-differences are used and conclusions are phrased cautiously.
  • Retailer: the retailer randomization unit: an online visitor or a store; for stores pairs are chosen by size, region and trend and test and control are then randomly assigned within pairs.
  • Brand: typical brand tests: pages and size hints in D2C, prices and bundles, pre-order terms, mailings; the primary metric is margin per visitor or customer, not clicks.
  • Retailer: typical retailer tests: promotions and discount mechanics, display and planogram, price thresholds, receipt formats and delivery; the primary metric is category or store margin including cannibalization.
  • Brand: the brand risk is flows between channels: a promotion test in D2C shifts partner sales; so the guardrail metric includes partner sell-out or total brand sales.
  • Retailer: the retailer risk is flows between stores and cannibalization within the category: a test on a store group is assessed on category sales and not only on the promoted item.
  • Brand: the brand test duration is set by the purchase cycle and D2C traffic: with low traffic the duration grows and the test becomes risky because of seasonal changes; a geo test on a marketplace is acceptable.
  • Retailer: the retailer test lasts at least two full weeks and covers all weekdays; for seasonal items the test runs in one window with a control so calendar effects act on both groups.
  • Brand: a brand converts the result into season margin and the line demand forecast: a win in D2C without partners can be a loss for the brand overall.
  • Retailer: a retailer converts the result into chain margin and promotion rules: scaling to all stores is checked with a pilot wave on part of the chain to confirm that the effect holds.

Where the data comes from

Data sourceFields
Events and ordersVisitors, sessions, orders, revenue and margin by test group and day; group identifier and assignment date.
Stores and historyStore sales and traffic for the preceding 8-12 weeks for pairing; promotion calendar.
Experiment protocolHypothesis, metric, MDE, α, power, duration, stopping rule, guardrail metrics, author and date.

How to build it step by step

  1. Step 1. State the hypothesis and one primary metric; write down which metric value repays the change (the MDE in money).
  2. Brand: Step 2. Choose the randomization unit (visitor, customer, partner, region); for wholesale and marketplaces use geo pairs and before/after with a control.
  3. Retailer: Step 3. Pair stores by size, region and trends over previous weeks and randomly assign test and control within pairs.
  4. Step 4. Compute sample size and duration from baseline conversion, MDE, α and power; round the duration to whole weeks.
  5. Step 5. Run an A/A check: two identical groups must not show a difference more often than α; otherwise there is an error in assignment or data.
  6. Step 6. Launch the test without composition changes, do not change other factors (prices, mailings) and do not peek before the deadline.
  7. Step 7. Compute the difference, interval and p-value by the pre-chosen criterion; for several variants apply a correction (Holm).
  8. Step 8. Check guardrail metrics (returns, revenue of neighboring goods, complaints) and segments but do not hunt for a “significant” slice after the test.
  9. Step 9. Convert the effect into money (margin, not clicks) with an interval and compare with the cost of the change.
  10. Step 10. Record the decision and protocol in the experiment register; failed results are kept too so they are not repeated.

Formulas

MeasureFormulasNote
Sample size per group (two proportions)n = 2·(z_(1−α/2) + z_(1−β))²·p̄·(1 − p̄) / (p₁ − p₀)², p̄ = (p₀ + p₁)/2For means: n = 2·σ²·(z_(1−α/2) + z_(1−β))² / δ².
Two-proportion z statisticz = (p_B − p_A) / √( 2·p̂·(1 − p̂)/n ), p̂ = (x_A + x_B)/(2n)Groups of equal size n; for unequal ones use the general formula.
p-valuep = 2·(1 − Φ(|z|))Two-sided; one-sided is acceptable only if chosen in advance.
Interval of the difference(p_B − p_A) ± 1.96·√( p_A(1 − p_A)/n + p_B(1 − p_B)/n )The interval matters more than the p-value: it shows the effect size.
Power1 − β = Φ( |p₁ − p₀| / √(2·p̄(1 − p̄)/n) − z_(1−α/2) )Power grows with n and with effect size.
Multiple comparison correctionsBonferroni: α_i = α/m; Holm: sort p_(1) ≤ … ≤ p_(m), reject while p_(i) ≤ α/(m − i + 1)Holm is never weaker than Bonferroni at the same type I error control.
False positives under peekingP(reject at least once) = 1 − P( |Z_k| ≤ z_(1−α/2) for all k = 1..K ) > αZ_k is the statistic at the k-th look; values are estimated by simulation or by a sequential boundary plan.

Formulas in notation

Sample size
n = 2·(z_{1−α/2} + z_{1−β})² · p̄(1 − p̄) / (p₁ − p₀)²
Per group; p̄ is the mean of the two proportions.
Z statistic
z = (p_B − p_A) / √[ 2·p̂(1 − p̂)/n ]
p̂ is the pooled proportion.
Difference interval
(p_B − p_A) ± 1,96·√[ p_A(1−p_A)/n + p_B(1−p_B)/n ]
95% confidence level.
Holm correction
reject H_(i) while p_(i) ≤ α / (m − i + 1)
p_(i) are the ordered p-values.
MDE at sample n
δ_MDE = (z_{1−α/2} + z_{1−β}) · √[ 2·p₀(1 − p₀)/n ]
The minimum absolute effect distinguishable at sample n per group.

Worked example

Visitors per group at different minimum effects (α = 5%, power 80%)

MDE (relative)Target conversion, %Visitors per groupWeeks at 18,400 a week
+5%9.4564,9157.1
+10%9.916,5831.8
+15%10.357,5270.8
+20%10.84,3220.5

Baseline conversion 9%. A test with MDE +10% collects the needed sample in two weeks; a test with MDE +5% needs about 7 weeks. The observed example (17,000 per group, +10%) gives p = 0.0046; with five parallel variants at p = 0.003; 0.012; 0.021; 0.04; 0.3 Holm rejects two. Peeking five times gives 14.2% false positives (an exact calculation). Figures are computed in code with a fixed seed.

Figures

Test power and sample size per group
Line chart: Power, %: 16.2 → 99.2; Target 80%: 80 → 800204060801002 0008 00016 58430 00040 000Power, %, 2 000: 16.2Power, %, 4 000: 28Power, %, 8 000: 49.4Power, %, 12 000: 66.4Power, %, 16 584: 80Power, %, 20 000: 86.8Power, %, 30 000: 96.5Power, %, 40 000: 99.2Target 80%, 2 000: 80Target 80%, 4 000: 80Target 80%, 8 000: 80Target 80%, 12 000: 80Target 80%, 16 584: 80Target 80%, 20 000: 80Target 80%, 30 000: 80Target 80%, 40 000: 80%
  • Power, %
  • Target 80%
Data as a table
MeasurePower, %Target 80%
2 00016.280
4 0002880
8 00049.480
12 00066.480
16 5848080
20 00086.880
30 00096.580
40 00099.280

Test power and sample size per group. Power of 80% is reached at about 16.6 thousand visitors per group for an MDE of +10%; at 8 thousand it is below 50%. Axis: visitors per group.

Which effect is visible in how many weeks
Bar chart: 1 13.1, 2 9.3, 3 7.6, 4 6.6, 6 5.4, 8 4.605101511MDE, % relative: 13.113.122MDE, % relative: 9.39.333MDE, % relative: 7.67.644MDE, % relative: 6.66.666MDE, % relative: 5.45.488MDE, % relative: 4.64.6%
Data as a table
MeasureMDE, % relative
113.1
29.3
37.6
46.6
65.4
84.6

Which effect is visible in how many weeks. In one week only an effect of about +13% is visible, in two about +9%: a short test cannot tell small improvements. Axis: test weeks at 18,400 visitors a week.

False positives under peeking
Bar chart: 1 5, 2 8.3, 3 10.7, 5 14.2, 10 19.3, 20 24.8051015202511Share of false positives, %: 55Nominal α, %: 5522Share of false positives, %: 8.38.3Nominal α, %: 5533Share of false positives, %: 10.710.7Nominal α, %: 5555Share of false positives, %: 14.214.2Nominal α, %: 551010Share of false positives, %: 19.319.3Nominal α, %: 552020Share of false positives, %: 24.824.8Nominal α, %: 55%
  • Share of false positives, %
  • Nominal α, %
Data as a table
MeasureShare of false positives, %Nominal α, %
155
28.35
310.75
514.25
1019.35
2024.85

False positives under peeking. With one look 5%, rising with each extra look: with 20 looks at a test without an effect a “win” is declared in about a quarter of cases (24.8%). Axis: number of equally spaced looks during the test; values are computed by numerical integration (not simulation), normal approximation of the statistic.

Five p-values and correction thresholds
Bar chart: H1 0.003, H2 0.012, H3 0.021, H4 0.04, H5 0.300.10.20.3H1H1p-value: 0.003Holm threshold: 0.01Bonferroni threshold: 0.01H2H2p-value: 0.012Holm threshold: 0.013Bonferroni threshold: 0.01H3H3p-value: 0.021Holm threshold: 0.017Bonferroni threshold: 0.01H4H4p-value: 0.04Holm threshold: 0.025Bonferroni threshold: 0.01H5H5p-value: 0.3Holm threshold: 0.05Bonferroni threshold: 0.01significance level
  • p-value
  • Holm threshold
  • Bonferroni threshold
Data as a table
Measurep-valueHolm thresholdBonferroni threshold
H10.0030.010.01
H20.0120.0130.01
H30.0210.0170.01
H40.040.0250.01
H50.30.050.01

Five p-values and correction thresholds. The p-value of 0.012 passes the Holm threshold (0.0125) and fails the Bonferroni threshold (0.01).

Specifics by category

Apparel
Season and weather change conversion within a test: test and control run simultaneously, not “before and after”; for promotions on seasonal items the test is short and its length is aligned with the life cycle.
Footwear
Conversion is lower so the sample is larger; the price effect is mixed with size availability, so check size availability in the groups and the returns guardrail.
Accessories
A low ticket and high frequency give a quick sample but gift peaks distort the test: avoid holiday periods and do not compare peak weeks with ordinary ones.
Kids
Purchases are tied to age and school; randomize by customer, not by order, otherwise one parent lands in both groups; the late-August peak is short.
Men
Conversion and ticket are steadier so the needed sample is smaller at the same relative effect; repeat purchases are large and the test effect is also measured by customer return.
Women
More promotions and variants so the multiple comparison risk is higher: limit the number of variants in advance and apply a correction; flows between models (cannibalization) must be measured at category level.

Common mistakes

  • A test is launched without a sample size calculation → not enough power, the effect is invisible → compute n and duration before launch.
  • Peeking and stopping at the first p < 0.05 → false positives rise several-fold → one look or a sequential plan.
  • Five variants are compared without a correction → one “win” is random → a Holm or Bonferroni correction.
  • A “significant” slice is hunted after the test → a random one will be found → segments are set in advance, the rest are hypotheses for the next test.
  • The p-value is taken for the probability of an effect and the interval is ignored → a small effect is rolled out → decide by the lower interval bound in money.

Russian specifics

  • Tests of promotions and prices on customers must not show buyers of the same item different prices without explanation; the tag and advertising state price and promotion terms correctly, and promo advertising is subject to the advertising law. Rule: Law 38-FZ of 13.03.2006: Federal Law of 13.03.2006 No. 38-FZ on advertising (rule check not completed).
  • A test in which some customers receive other terms than the public offer is agreed with a lawyer: retail sales terms and consumer rights must not be worsened within a test. Rule: Law 2300-1 of 07.02.1992: Law of 07.02.1992 No. 2300-1 on consumer rights protection (rule check not completed).
  • User data in tests is processed with consent and anonymously; group identifiers are stored separately from personal data. Rule: Law 152-FZ of 27.07.2006: Federal Law of 27.07.2006 No. 152-FZ on personal data (rule check not completed).

Open interactively →