p-value
The probability of seeing such a gap or a larger one between variants if there is in fact no effect.
Formula
p = 2 × (1 − Φ(|z|))
How to read
A small p does not prove the size of the effect and does not say B is better by so much: look at the confidence interval. Many checks in a row increase the share of false wins.
Where it is discussed
Topics
Calculations
- Weather adjustment of demand
- NPS confidence interval and significance
- A/B test on proportions: sample size, duration and reading the result
- Bayesian estimate of a rate: buyout or conversion of a new style from a small sample
- Price test: profit per visitor and significance
Documents
- Bayesian estimates of conversion and buyout: small samples, prior distributions, shrinkage and comparison with the frequentist approach
- Measuring marketing: path attribution, advertising carry-over, the budget optimum, the price test, NPS and the referral reward