A/B test
A comparison of two variants on randomly split visitor groups to tell an effect from chance.
Formula
Sample per group = (z(α/2) × √(2p̄(1 − p̄)) + z(β) × √(p1(1 − p1) + p2(1 − p2)))² ÷ (p2 − p1)²
How to read
The sample size is set beforehand and the test is not stopped early. Test in whole weeks and outside promotions.
Where it is discussed
Topics
- Cost and price
- Markdown and exit price
- Marketing budget and testing impact
- Site economics and the funnel
Calculations
- Elasticity from a price and sales table: log-log regression with an interval
- A/B test on proportions: sample size, duration and reading the result
- Price test: profit per visitor and significance
- Payment fees and instalment payback