Syntha Lab RU

Chain pilot: the effect against control stores

Checks that a new display, seller uniform or price gave an effect rather than the season: compares the change in sales in pilot stores with the change in control stores. Shows the effect in percent, an interval from the spread across stores and the profit of the pilot.

Store format, rent and the fate of a store For whom: Chain directors and marketers introducing a change in stores.

Calculate on your data →

At a glance

What question does it answer?
Did the change give an effect beyond the season and is it worth rolling out to the chain.
What data is needed?
Weekly sales before and after for each pilot store.
How to read the result?
A pilot of four stores grew by 9.8% and the control by 1.8%: the effect is +7.8% (a rough interval from +5.9 to +9.9%, which does not include zero). A gain of about 244 thousand a week gives a pilot profit of 0.58m over 6 weeks with costs of 150 thousand. Example on the default values: Pilot effect (against control) — 7.8%; Interval (rough, 95%): from — 5.9%; Interval (rough, 95%): to — 9.9%; Weekly sales gain against control — RUB 243,817.

Why and how to reason

A pilot shows its effect only in comparison: sales in pilot stores might have risen because of the season, and then there is no effect. So we take control stores with no change and compare dynamics. Pilot: before 3,070 thousand, after 3,370 thousand (+9.77%). Control: before 3,060 thousand, after 3,116 thousand (+1.83%). Effect = 1.0977 ÷ 1.0183 − 1 = 7.8%: that is what the pilot added beyond what the season gave. Weekly gain against control: 3,370 − 3,070 × 1.0183 = 243.8 thousand ₽. Pilot profit over 6 weeks at a 50% margin and costs of 150 thousand: 243.8 × 6 × 0.5 − 150 = 581 thousand ₽. Stores differ from each other, so an interval from store spread is computed: 5.9–9.9%. It does not contain zero, so the effect is distinguishable from noise.

When to use it

It fits before rolling out a new display, staff scheme, price or service across the network: run it in several stores and compare with similar ones. You need at least 3–4 pilot and as many control stores. Assumption: pilot and control stores are comparable in size and season, and the control does not know about the pilot. It does not fit a single store per group, a pilot that coincided with a promotion, or a short run (under 4 weeks).

How to read the result

Look at effect and interval: if the interval is above zero, scale up. If it includes zero, the pilot did not prove benefit, and it is worth extending or strengthening. The size of groups before the pilot (ratio 1.003) shows groups are alike: strongly differing groups distort the conclusion. Compare pilot profit with the cost of rolling out to the whole network: 581 thousand over 6 weeks in four stores does not guarantee the same return in a network of 40. Decision: scale gradually. It is worth fixing the rule before the start: if effect and interval are above zero, the pilot scales to the next group of stores, otherwise the pilot turns into a discussion of tastes.

Common mistakes

Picking the best stores for the pilot; having no control; comparing different periods; using stores of different sizes; drawing a conclusion after one week. Nothing else should change in pilot stores: then you will not know what worked. Do not include in control stores that had a refit or a change of management.

What data is needed?

  • Weekly sales before and after for each pilot store.
  • The same for control stores that did not get the change.
  • Optional: margin, length and cost of the pilot.

Inputs

FieldUnitValue in the exampleMeaning
Margin on sales%50The share of revenue excl. VAT after the cost of goods. Empty: money not calculated.
Pilot lengthweeks6The weeks over which the gain is counted. Empty: money not calculated.
Pilot costs₽150,000Materials, training, time. Empty: zero.
Power of the test80%: the usual choiceThe chance to notice the effect if it exists; the significance level is 5%. The choice is yours: there is no single standard.
The effect that matters not to misspp—Optional. The pilot effect in percentage points of the sales change. Empty: only the minimum detectable effect for the current groups is shown.

How it is calculated

StepFormula
Group changesales after ÷ sales before
Effectpilot change ÷ control change − 1
Intervalfrom the spread of individual store changes (t distribution): (pilot mean ÷ control mean) ± t × standard error
Profitweekly gain × weeks × margin − pilot costs

Worked numbers

Default values from the app; in the interactive version you enter your own.

Pilot effect (against control)
7.8 %The pilot’s growth relative to the control’s growth.
Interval (rough, 95%): from
5.9 %
Interval (rough, 95%): to
9.9 %
Weekly sales gain against control
243,817 ₽Pilot sales after minus pilot sales before × control dynamics.
Dynamics: pilot and control
9.8 % · control 1.8 %
Pilot profit over the period
581,451 ₽
Group size before the pilot: pilot to control
1×
Difference in differences: 95% interval, from
6 p.p.The mean change of a pilot store minus a control store, an interval from the spread of stores (Welch).
Difference in differences: 95% interval, to
10.1 p.p.
Minimum detectable effect
2.2 p.p.A smaller effect cannot be told from random fluctuation with this number of stores and this spread. A normal-law approximation.

Groups before and after

GroupStoresWeekly sales beforeAfterChange
Pilot43.07 m ₽3.37 m ₽9.8 %
Control43.06 m ₽3.12 m ₽1.8 %

When to use which. This calculation: format, display, range in stores, a ratio before and after with an interval from the spread across stores. “Control group: promotion effect (difference-in-differences)”: a promotion, a series of weeks before it, parallel trends. “Channel incrementality: test against control”: advertising and channels, profit and return. All three compare a test with a control; choose by the subject of the test.

The method works if the stores moved similarly before the pilot and the control received no change. Account for the season, promotions and a competitor opening nearby separately. Choosing the “best” stores for the pilot overstates the effect.

Nuances and limits

  • Choose pilot stores before the pilot, not by the results.
  • The control should resemble the pilot in size, region and pre-pilot dynamics.
  • Account for the season, promotions and a competitor opening nearby separately.

Terms used

Calculate on your data →