Control group: promotion effect (difference-in-differences)
Compares stores or regions in a promotion with similar ones without it, before and after. Shows the effect of the promotion in pieces and percent, a rough interval from the pre-promotion spread, a “parallel trends” check before the promotion and the result in money.
At a glance
- What question does it answer?
- What effect the promotion had when compared with similar stores without it.
- What data is needed?
- Weekly sales of stores in the promotion and control stores for 4–8 weeks before.
- How to read the result?
- Before the promotion the test ÷ control sales ratio was about 0.50, during it 165 ÷ 208 = 0.79: an effect of +58.0% (rough interval 55.7–60.3%), that is 60.6 pcs a week. Difference-in-differences is 57.8 pcs a week. The required growth at a 20% discount is 50%: profit by the control +33,432 over two weeks. The slope of the ratio before the promotion is 0.2% a week: trends ran in parallel. Example on the default values: Promotion effect (relative) — 58%; Rough interval from the pre-promotion spread, from — 55.7%; Rough interval from the pre-promotion spread, to — 60.3%; Effect by ratio: pieces a week — 60.6 pcs.
Why and how to reason
To see what a promotion did, it is not enough to look at how much sales rose in promoted stores: they might have risen because of the season. So a control, similar stores without a promotion, is needed, and a before-and-after comparison in both groups, which is the difference of differences. First take the ratio of promoted-store sales to control sales before the promotion: 101.7 ÷ 202.5 = 0.502. During the promotion the same ratio is 165 ÷ 208 = 0.793. Effect = 0.793 ÷ 0.502 − 1 = 58%. In units: expected sales without the promotion 208 × 0.502 = 104.4, actual 165, gain 60.6 units a week, 121 over two weeks. The additive estimate (difference of gains in units) of 57.8 units gives a close number. A rough interval from the spread of pre-promotion weeks is 55.7–60.3%. Promotion profit by control is about 33 thousand ₽ at a 20% discount.
When to use it
Use it when the promotion did not run in all stores or regions and a similar control can be chosen. The main assumption: before the promotion both groups moved in parallel, and the tool checks this by the slope of the ratio. It does not fit if the promoted stores were not chosen at random (the best or worst), if the control was affected (customers traveled there), or with fewer than four pre-promotion weeks: the spread estimate is unreliable. For one promotion across the whole network use a baseline comparison.
How to read the result
First check the slope of the ratio before the promotion (0.21% against a 2% tolerance): if small, groups moved in parallel and the conclusion is reliable. If the slope exceeds tolerance, the effect is mixed with a different trend. Then read the effect and interval: 58% with an interval of 56–60% is robust, and if the interval crosses zero there is no effect. Profit by control turns gain into roubles with the discount included; compare it with costs and the post-promotion dip. Decision: scale up if the effect is stable and profit is positive. If the slope of the ratio before the promotion exceeds tolerance, choose other control stores or add more pre-promotion weeks: without parallel trends the conclusion is unreliable.
Common mistakes
Picking control stores similar by name rather than by dynamics; taking different weeks in the groups; ignoring that the control may also have received a competitor discount; treating the interval as exact (it is rough). A small number of stores makes the conclusion random. Do not use stores next to promoted ones as control: shoppers may travel between them.
What data is needed?
- Weekly sales of stores in the promotion and control stores for 4–8 weeks before.
- Average weekly sales of both groups during the promotion and the number of weeks.
- Optional: price, cost, discount and the slope threshold for the ratio.
Inputs
| Field | Unit | Value in the example | Meaning |
|---|---|---|---|
| Stores in the promotion: weekly sales | pcs a week | 165 | Total weekly sales of the stores in the promotion during the promotion. |
| Control stores: weekly sales | pcs a week | 208 | Total weekly sales of the control stores in the same weeks. |
| Weeks of the promotion | weeks | 2 | How many weeks the promotion lasted. |
| Price before discount, excl. VAT | ₽ | 5,000 | Empty price, cost or discount: money not calculated. |
| Unit cost excl. VAT | ₽ | 2,000 | Unit cost excl. VAT, delivered to the warehouse. |
| Discount depth | % | 20 | The discount as a percent of the price before the promotion. |
| Acceptable slope of the ratio before the promotion | % a week | 2 | A rule of thumb, replace with yours: a slope of the “test ÷ control” ratio over the weeks before above the threshold means trends diverged. |
How it is calculated
| Step | Formula |
|---|---|
| Difference-in-differences | (test after − test before) − (control after − control before) |
| Relative effect | (test after ÷ control after) ÷ (test before ÷ control before) − 1 |
| Expected without the promotion | control after × (test before ÷ control before) |
| Rough interval | effect ± 1.96 × ratio variation × √(1 ÷ promotion weeks + 1 ÷ weeks before) |
Worked numbers
Default values from the app; in the interactive version you enter your own.
- Promotion effect (relative)
- 58 %Test ÷ control after, relative to the same ratio before the promotion.
- Rough interval from the pre-promotion spread, from
- 55.7 %Not a confidence interval: it uses only the spread of the ratio in the weeks before the promotion, not the noise of the promotion weeks, so in reality it is wider.
- Rough interval from the pre-promotion spread, to
- 60.3 %
- Effect by ratio: pieces a week
- 60.6 pcsActual − expected without the promotion (control × the ratio before).
- Additive estimate, difference-in-differences: pieces a week
- 57.8 pcsThe control is not scaled: its change is subtracted as is. The two estimates differ when the groups differ in size; a robustness check, not a second answer: profit is computed from the ratio estimate. (test after − test before) − (control after − control before).
- Effect over the whole promotion
- 121.1 pcs
- Slope of the ratio before the promotion
- 0.2 % a week
- Promotion profit by the control
- 33,432 ₽ · required growth 50 %
Test and control before and after
| Group | Before, weekly | After, weekly | Change |
|---|---|---|---|
| Stores in the promotion | 101.7 | 165 | 63.3 |
| Control | 202.5 | 208 | 5.5 |
| Difference-in-differences | 57.8 |
The method requires both groups to move in parallel without a promotion. Self-selection (a promotion in the best stores) breaks this, and so does overlap with other promotions. The interval is rough, it uses only the pre-promotion spread and is no substitute for statistics: large decisions need a store-level analysis. When to use which: for advertising and channels “Channel incrementality: test against control”, for format and range “Chain pilot: the effect against control stores”.
Nuances and limits
- If trends diverged before the promotion the estimate is biased: pick other control stores.
- Self-selection: the promotion went to the best stores.
- Overlap with other promotions prevents separating the effect.
- The interval is rough; large decisions need a store-level analysis.