Bayesian estimates of conversion and buyout: small samples, prior distributions, shrinkage and comparison with the frequentist approach
How to estimate a share (conversion, buyout, returns) and a sales rate when data is scarce: a new store, a new model, three days of sales. The Bayesian approach combines experience (the distribution across the chain or category) with new observations, gives an interval as a probability and a direct answer to “what is the probability that variant B is better than A”. The decision: what to treat as the estimate while data is scarce and when a decision can already be taken.
Why this document
The analyst and the manager read this when data is scarce: a new store, a new model, three days of sales. A share computed from ten orders jumps by chance, and a conclusion like a 30% conversion is worth nothing. A Bayesian estimate combines such observations with experience across the network or category and pulls extreme values toward the mean, so a small sample gives neither false records nor false failures. As data accumulates, the weight of observations grows and the estimate converges with the ordinary one. That way a decision to buy or close is not built on noise.
At a glance
- What is this document and why is it needed?
- How to estimate a share (conversion, buyout, returns) and a sales rate when data is scarce: a new store, a new model, three days of sales. The Bayesian approach combines experience (the distribution across the chain or category) with new observations, gives an interval as a probability and a direct answer to “what is the probability that variant B is better than A”. The decision: what to treat as the estimate while data is scarce and when a decision can already be taken.
- Who owns the document and how often is it rebuilt?
- A sales, e-commerce or CRM analyst; prior distributions are reviewed once a season, estimates are updated as data comes in; read by category managers, marketing and store managers.
- What mistakes are the most common?
- A store ranking by raw shares at different n → random small samples on top → shrunk estimates and intervals next to the ranking. Too strong a prior from another category → the estimate does not respond to data → a prior from the own group and a check on a weak prior. The prior and its strength are hidden → the estimate cannot be reproduced or challenged → the prior version and parameters are stored with the estimate.
For whom
- Brand
- A brand estimates site conversion and the buyout of a new collection from the first days and small partners, using the history of past collections as a prior.
- Retail
- A retailer estimates the conversion of new and small stores, returns of a new model and the sales rate of a new item from the first days, relying on the distribution across the chain and category.
What it consists of
When the frequentist estimate fails
- The share x/n of a small sample is noisy: in the example a store with 350 visitors showed 16.6% against a true 14.7%, and a store with 2200 visitors 9.5% against 7.9%. A ranking by raw shares can put chance successes of small samples on top.
- The Wilson confidence interval honestly shows uncertainty but does not use what we know in advance: a chain store’s conversion cannot be 40% or 0.1%. A Bayesian estimate adds that knowledge as a prior and gives a stable answer already on the first observations.
- A prior is not a guess but a distribution estimated from data: from chain stores, past collections, analogs. The less data and the narrower the prior the more it matters; with a large sample the prior ceases to matter.
The beta-binomial model
- Conversion p has a prior Beta(α, β) with mean α/(α + β) and strength κ = α + β (how many “virtual observations” it is worth). After x purchases out of n visitors the posterior is Beta(α + x, β + n − x) and its mean (x + α)/(n + κ) is a weighted average of the raw share x/n and the prior.
- The prior is estimated from chain stores by the method of moments with a correction for binomial noise: in the example average conversion is 8.5%, strength κ = 159 (α = 13.5, β = 145.8). The weight of the raw share n/(n + κ) is 69% for a store with 350 visitors and 99% for a store with 20,000.
- The result of shrinkage: the mean squared error of estimating true conversion is 0.65·10⁻⁴ for raw shares and 0.32·10⁻⁴ for Bayesian ones, and for the five smallest stores 1.37·10⁻⁴ against 0.62·10⁻⁴. Of the three best stores by true conversion the raw ranking finds 3, the Bayesian 3.
- Brand: the prior for a new partner or channel comes from the distribution over partners of the same type and size, not from the brand average across all sales; large partners make the average unrepresentative.
- Retailer: the prior for a new store comes from stores of the same format, city and area; a mall and a street give different distributions and must not be mixed.
A new store and intervals
- A new store: 7 purchases out of 120 visitors, a raw share of 5.8%. The Bayesian estimate with the chain prior is 7.3%, the 95% interval 4.6-10.7%; with a flat prior the interval is 2.9-11.6%, Wilson’s 2.9-11.6%. The informative prior narrows the interval by 30%.
- A credible interval is a direct statement: with 95% probability the true conversion lies within these limits under the chosen model. A frequentist confidence interval speaks of the share of such intervals covering the truth under repeated sampling; for decisions the first reading is handier but it holds only if the prior is justified.
- Prior sensitivity: at the same raw share of 5.8% and n = 20 the estimate ranges from 6.7% (a weak prior) to 9.8% (flat), at n = 2,500 the spread is 0.15 pp. Conclusion: at small n the prior must be shown and justified, and at large n its influence disappears.
Comparing variants A and B
- Example: A 40 of 500 (8%), B 55 of 520 (10.6%). The frequentist answer: p-value 0.157, so at α = 5% the difference is not proven. The Bayesian answer: the probability that B is better than A is 92%, the expected loss if B is chosen 0.07 pp of conversion.
- The two answers are consistent: there is too little data for “B is better with 95% certainty” but more than nine chances in ten point to B. The decision is taken by the cost of a mistake: if launching B is cheap and reversible this is enough, otherwise wait for data; the probability threshold and the expected loss threshold are set before the test.
- Brand: the roll-out threshold for a D2C page is low because rollback is cheap; for a price or partner terms change the threshold is higher because rollback is costly and visible to partners.
- Retailer: the threshold for a chain-wide promotion is higher than for a pilot in three stores: the cost of a mistake grows with reach, and the Bayesian estimate from the pilot is updated as sales come in.
- The Bayesian approach does not remove peeking and multiple comparisons problems entirely: the stopping rule and prior are fixed in advance and the number of variants is accounted for through the prior and thresholds, as in the frequentist approach (the A/B tests document).
Buyout of a new model and a new item forecast from the first days
- Returns of a new model: 14 of 40 (35%) with a category prior of mean 22% and strength 60. The posterior mean is 27.2%, the probability that returns exceed the 30% threshold is 26%. The decision is not “returns 35%, withdraw” but “the probability of exceeding the threshold is about a quarter, continue and recompute at 100 orders”.
- The sales rate of a new item (units a day) is modeled by a gamma distribution and the number of sales by a Poisson one. The analog prior Gamma(6, 1.5) with a mean of 4 units a day and an observation of 14 units in 3 days (a raw rate of 4.7) give a posterior Gamma(20, 4.5) with a mean of 4.44 units a day.
- The sales forecast for the next 14 days is built from the predictive distribution that accounts for both rate uncertainty and Poisson noise: a median of 61 units, an 80% interval of 43-84 units. The interval is what goes into the order and the reorder.
Brand and retailer: what is estimated in a Bayesian way
- Brand: conversion of new channels and pages, the buyout of a new collection from the first batch, the return rate of a new partner; the prior comes from past collections and channels.
- Retailer: conversion of new and small stores, the share of sales by discount card, returns of a new model and the rate of a new item; the prior comes from the distribution by chain, format and category.
- Brand: D2C page and price variants are compared by the probability of “B better than A” and the expected loss, and the roll-out decision is tied to the cost of a mistake.
- Retailer: a ranking of stores by conversion and receipt is built from shrunk estimates, not raw shares, so that bonuses and reviews are not given for the chance of a small sample.
- Brand: the decision to reorder fabric from the first weeks of sales relies on a predictive interval, not a point forecast.
- Retailer: the decision to reallocate a new item between stores from the first days relies on the posterior rate with an interval: one day without sales in a store is no reason to pull it.
Where the data comes from
| Data source | Fields |
|---|---|
| Counters and receipts | Visitors and purchases by store and week, orders and returns by model, daily sales for a new item; the number of observations n and events x. |
| History for the prior | Conversion and returns by chain stores, past collections and analogs, sales rate of analogs by day of life. |
| Decision thresholds | The acceptable return share, the “B better than A” probability threshold, the acceptable expected loss, the cost of a launch error and rollback. |
How to build it step by step
- Step 1. Define what is estimated (a share or a rate) and the model: beta-binomial for shares, gamma-Poisson for a sales rate.
- Brand: Step 2. Assemble the history of past collections and channels (buyout, conversion) and estimate the prior by the method of moments; for a new channel take the nearest by audience.
- Retailer: Step 3. Assemble chain store conversions and estimate the prior by format and region; the prior strength is no larger than the visitors of a typical store month.
- Step 4. Estimate the mean level m and the between-object variance τ² with a correction for binomial noise; obtain α and β.
- Step 5. For each object compute the posterior Beta(α + x, β + n − x), the mean and a 95% credible interval; show n and the raw share next to them.
- Step 6. Check the prior for stability: recompute with a flat and a weak prior; if the decision changes the data is insufficient and more must be collected.
- Brand: Step 7. To compare variants compute the probability of “B better than A” and the expected loss; take the roll-out decision by thresholds set in advance and the cost of rollback.
- Retailer: Step 8. Build a store ranking from posterior means and intervals; do not reward or penalize stores whose intervals overlap.
- Step 9. For a new item build a predictive sales interval for the coming weeks and hand it to the order and reorder with the re-estimation point.
- Step 10. Update the prior once a season from new data and store the prior version with the estimate so that decisions can be reproduced.
Formulas
| Measure | Formulas | Note |
|---|---|---|
| Posterior of a share | p | x, n ~ Beta(α + x, β + n − x); E[p | x, n] = (x + α) / (n + α + β) | Prior Beta(α, β) with mean α/(α + β) and strength κ = α + β. |
| Shrinkage | E[p | x, n] = w·(x/n) + (1 − w)·m, w = n / (n + κ), m = α / (α + β) | w grows with n: with a large sample the estimate equals x/n. |
| Empirical Bayes | m = Σx_i / Σn_i, τ̂² = max( Σn_i(p̂_i − m)²/Σn_i − m(1 − m)/n̄, ε ), κ̂ = m(1 − m)/τ̂² − 1 | Method of moments; n̄ is the average object sample size. |
| Credible interval | [ q_(0.025), q_(0.975) ] quantiles of Beta(α + x, β + n − x) | A probability of 95% under the chosen model; depends on the prior. |
| Probability B > A and loss | P(B > A) = P(p_B > p_A), L_B = E[ max(p_A − p_B, 0) ] | Computed by sampling from two beta distributions (simulation) or by an integral. |
| Gamma-Poisson model | λ ~ Gamma(a, r); x | λ ~ Poisson(λ·t); λ | x ~ Gamma(a + x, r + t) | Rate mean a/r; a prediction for horizon H is obtained by sampling λ and then the number of sales. |
Formulas in notation
- Posterior mean
E[p | x, n] = (x + α) / (n + α + β)
A weighted average of the share and the prior.- Weight of data
w = n / (n + κ), κ = α + β
Grows with sample size.- Prior strength
κ̂ = m(1 − m) / τ̂² − 1
An estimate from the spread between objects.- Probability B > A
P(p_B > p_A); L_B = E[ max(p_A − p_B, 0) ]
Sampling from two beta distributions.- Gamma-Poisson
λ | x ~ Gamma(a + x, r + t)
The sales rate of a new item.
Worked example
Twelve stores: raw share, Bayesian estimate and true conversion
| Store | Visitors | Raw, % | Bayesian, % | True, % |
|---|---|---|---|---|
| 1 | 350 | 16.6 | 14 | 14.7 |
| 2 | 600 | 3.5 | 4.5 | 3.9 |
| 3 | 900 | 5.8 | 6.2 | 6.1 |
| 4 | 1,500 | 15.8 | 15.1 | 15 |
| 5 | 2,200 | 9.5 | 9.4 | 7.9 |
| 6 | 3,000 | 13.4 | 13.1 | 12.9 |
| 7 | 4,500 | 12 | 11.9 | 12 |
| 8 | 6,000 | 7.2 | 7.2 | 7.6 |
| 9 | 8,000 | 9.7 | 9.7 | 9.3 |
| 10 | 11,000 | 8.6 | 8.6 | 9.1 |
| 11 | 14,000 | 6.2 | 6.2 | 6.2 |
| 12 | 20,000 | 7.8 | 7.8 | 8 |
Store conversions are generated in code (fixed seed) from Beta(7.2, 72.8); the prior is estimated from the same stores (m = 8.5%, κ = 159). For the five smallest stores the mean squared error of the Bayesian estimate is 0.62·10⁻⁴ against 1.37·10⁻⁴ for the raw share. The data is illustrative.
Figures
- Chain prior
- Posterior with chain prior
- Posterior with flat prior
Data as a table
| Measure | Chain prior | Posterior with chain prior | Posterior with flat prior |
|---|---|---|---|
| 0.1 | 0 | 0 | 0 |
| 0.5 | 0 | 0 | 0 |
| 1.1 | 0 | 0 | 0 |
| 1.6 | 0 | 0 | 0.3 |
| 2.1 | 0 | 0 | 1.1 |
| 2.6 | 0 | 0 | 2.7 |
| 3.1 | 0.2 | 0.1 | 5.3 |
| 3.6 | 0.7 | 0.6 | 8.6 |
| 4.1 | 1.6 | 1.9 | 12 |
| 4.6 | 3.3 | 4.8 | 15.1 |
| 5.1 | 5.7 | 9.4 | 17.3 |
| 5.6 | 8.6 | 15.2 | 18.4 |
| 6.1 | 11.7 | 20.8 | 18.5 |
| 6.6 | 14.6 | 24.7 | 17.6 |
| 7.1 | 16.9 | 26 | 16.1 |
| 7.6 | 18.2 | 24.6 | 14.1 |
| 8.1 | 18.5 | 21.2 | 12 |
| 8.6 | 17.8 | 16.8 | 9.9 |
| 9 | 16.4 | 12.4 | 7.9 |
| 9.6 | 14.4 | 8.5 | 6.2 |
| 10.1 | 12.2 | 5.5 | 4.7 |
| 10.5 | 10 | 3.4 | 3.5 |
| 11.1 | 7.9 | 2 | 2.6 |
| 11.6 | 6.1 | 1.1 | 1.9 |
| 12 | 4.6 | 0.6 | 1.3 |
| 12.6 | 3.3 | 0.3 | 0.9 |
| 13.1 | 2.4 | 0.1 | 0.6 |
| 13.6 | 1.6 | 0.1 | 0.4 |
| 14.1 | 1.1 | 0 | 0.3 |
| 14.6 | 0.7 | 0 | 0.2 |
| 15.1 | 0.5 | 0 | 0.1 |
| 15.6 | 0.3 | 0 | 0.1 |
| 16.1 | 0.2 | 0 | 0.1 |
| 16.6 | 0.1 | 0 | 0 |
| 17.1 | 0.1 | 0 | 0 |
| 17.6 | 0 | 0 | 0 |
| 18.1 | 0 | 0 | 0 |
| 18.6 | 0 | 0 | 0 |
| 19.1 | 0 | 0 | 0 |
| 19.6 | 0 | 0 | 0 |
| 20.1 | 0 | 0 | 0 |
| 20.6 | 0 | 0 | 0 |
| 21.1 | 0 | 0 | 0 |
| 21.6 | 0 | 0 | 0 |
| 22.1 | 0 | 0 | 0 |
| 22.6 | 0 | 0 | 0 |
| 23.1 | 0 | 0 | 0 |
| 23.6 | 0 | 0 | 0 |
| 24.1 | 0 | 0 | 0 |
| 24.6 | 0 | 0 | 0 |
| 25.1 | 0 | 0 | 0 |
A new store: prior and posterior of conversion. The chain prior is concentrated near 8.5%; after 120 visitors and 7 purchases the posterior is shifted to 7.3% and narrower than the flat one: an interval of 4.6-10.7%. A flat prior gives a wide curve peaking at the raw share of 5.8%. Axis: conversion, %.
- Raw share
- Bayesian estimate
- True
Data as a table
| Measure | Raw share | Bayesian estimate | True |
|---|---|---|---|
| 1 (350) | 16.6 | 14 | 14.7 |
| 2 (600) | 3.5 | 4.5 | 3.9 |
| 3 (900) | 5.8 | 6.2 | 6.1 |
| 4 (1,500) | 15.8 | 15.1 | 15 |
| 5 (2,200) | 9.5 | 9.4 | 7.9 |
| 6 (3,000) | 13.4 | 13.1 | 12.9 |
| 7 (4,500) | 12 | 11.9 | 12 |
| 8 (6,000) | 7.2 | 7.2 | 7.6 |
| 9 (8,000) | 9.7 | 9.7 | 9.3 |
| 10 (11,000) | 8.6 | 8.6 | 9.1 |
| 11 (14,000) | 6.2 | 6.2 | 6.2 |
| 12 (20,000) | 7.8 | 7.8 | 8 |
Twelve stores: raw share, Bayesian estimate and truth. For small stores the raw share departs widely from the truth, the Bayesian estimate is closer; the mean squared error over all stores is 0.65·10⁻⁴ against 0.32·10⁻⁴. Axis: store number (visitors).
- P(B better than A), %
- p-value, %
Data as a table
| Measure | P(B better than A), % | p-value, % |
|---|---|---|
| 100 | 72.2 | 52.7 |
| 200 | 81.3 | 37.1 |
| 300 | 86.5 | 27.3 |
| 500 | 91.7 | 15.7 |
| 800 | 96.5 | 7.3 |
| 1200 | 98.7 | 2.8 |
| 2000 | 99.9 | 0.5 |
Probability of “B better than A” and the p-value as the sample grows. At an observed difference of 8 → 10.6% and 500 visitors per variant the probability is 92% at p = 16%; as the sample grows at the same shares the probability tends to 100% and the p-value to zero. Axis: visitors per variant.
- Flat Beta(1, 1)
- Weak (strength 10)
- Chain (estimated from stores)
Data as a table
| Measure | Flat Beta(1, 1) | Weak (strength 10) | Chain (estimated from stores) |
|---|---|---|---|
| 20 | 9.85 | 6.71 | 8.18 |
| 100 | 6.7 | 6.07 | 7.45 |
| 500 | 6.01 | 5.89 | 6.47 |
| 2500 | 5.87 | 5.84 | 5.99 |
Estimate sensitivity to the prior at the same raw share. At n = 20 estimates with different priors differ by 3.1 pp, at n = 2,500 by 0.15 pp: the more data the less the prior matters. Axis: number of visitors; the raw share is 5.8%.
Data as a table
| Measure | Interval width |
|---|---|
| Chain prior | 6.1 |
| Flat prior | 8.7 |
| Wilson | 8.7 |
Width of the 95% interval for the new store. The informative prior gives 6.1 pp against 8.7 for the flat one and 8.7 for Wilson: the gain in precision is paid for by the assumption that the store resembles the chain.
Specifics by category
- Apparel
- Buyout and conversion differ across subgroups (outerwear, knitwear): the prior is built by subgroup, not for the whole category; seasonal models are estimated from analogs of the previous season.
- Footwear
- Returns by size and fit are high and vary widely across models: a new model has few orders, so shrinkage to the category prior is mandatory; sizes are estimated within a model.
- Accessories
- Few sales per SKU and many zeros: a gamma-Poisson model for the rate, a prior by group; the decision to delist an item relies on the posterior interval, not a couple of empty weeks.
- Kids
- A short season and size groups: little data per model, a prior by age group and last year’s season; an update after every week of sales.
- Men
- Demand is steadier, priors are narrower and more reliable so decisions are taken earlier; for basics shrinkage barely changes the estimate at a normal sample.
- Women
- Fast range turnover, little history per model and a high spread: Bayesian estimates and predictive intervals are the most useful; the prior is built by model group and color.
Common mistakes
- A store ranking by raw shares at different n → random small samples on top → shrunk estimates and intervals next to the ranking.
- Too strong a prior from another category → the estimate does not respond to data → a prior from the own group and a check on a weak prior.
- The prior and its strength are hidden → the estimate cannot be reproduced or challenged → the prior version and parameters are stored with the estimate.
- A “B better than A” probability of 92% is read as a proven difference → a launch ignoring the cost of a mistake → pre-set probability and loss thresholds.
- A test is stopped at the first favorable probability → inflated confidence → the stopping rule and prior are fixed before launch.
Russian specifics
- Retailer: data for a customer prior (cards, receipts with an identifier) is processed with a legal basis and in anonymized form; estimates by stores, models and days need no personal data. Rule: Law 152-FZ of 27.07.2006: Federal Law of 27.07.2006 No. 152-FZ on personal data (rule check not completed).
- Returns and refusal of goods in distance selling are governed by the consumer protection law; return thresholds in the model are management thresholds, not norms, and do not limit buyer rights. Rule: Law 2300-1 of 07.02.1992: Law of 07.02.1992 No. 2300-1 on consumer rights protection (rule check not completed).