Syntha Lab RU

Channel incrementality: test against control

Compares the revenue of a group with ads on against a similar group without them. Shows incremental revenue, incremental return and profit after spend, and compares with the ad platform’s last-click report.

Marketing budget and testing impact For whom: Marketers and owners who spend on ads and want to know their real return.

Calculate on your data →

At a glance

What question does it answer?
What the channel added to sales beyond what would have come without it and whether it pays.
What data is needed?
Revenue of the group with ads and a similar group without them for one period, excl. VAT; optionally for the period before the test as well.
How to read the result?
First profit after spend and incremental return against break-even. Then compare with the platform’s reported return: the gap shows how far last click overstates it. If the lift is below the spread, extend the test. Example on the default values: Incremental revenue — 2.1 m RUB; Lift over control — 17.4%; Incremental return (ROAS) — 1.4×; Break-even return — 1.92×.

Why and how to reason

An ad platform sees orders it was near: it will credit itself with a buyer who would have come without the ad. So real value shows only in comparison with a similar group that had no ads. Incremental revenue = revenue of the group with ads minus revenue of the control: 14.2 − 12.1 = RUB 2.1 million, lift over control 17.4%. Ad spend is 1.5 million, so incremental return (ROAS) is 2.1 ÷ 1.5 = 1.4. Advertising pays only if return exceeds break-even, which equals 1 ÷ margin = 1 ÷ 0.52 = 1.92: each rouble of ads must return RUB 1.92 of revenue to cover itself with margin. Here 1.4 is lower, profit −408 thousand ₽. The platform reports 6.8 million (return 4.5), 3.2 times more than incremental: the difference is buyers who came by themselves.

When to use it

It fits when you want the real value of a channel and can separate a control group (a region, part of the audience, a period without ads) similar in size and behavior. Assumption: groups are comparable and nothing but ads influenced them. It does not fit small volumes (noise hides the effect), groups in different seasons, or a channel working over a long period (brand): a short test will underestimate. If you have pre-test revenue, enter it: the tool will adjust for different starting levels.

How to read the result

Compare incremental return with break-even: above means the channel pays for itself, below means it loses money at your margin. Profit after costs turns this into roubles. The gap with the platform report (3.2 times) shows how much the dashboard overstates value and why you should not judge the budget by it. Compare the lift with the normal spread of groups: a 17% lift against 5% spread is a signal; 6% may be noise. Decision: cut budget in a loss-making channel, repeat the test, check in another period. If the effect is not confirmed, do not switch the channel off at once: repeat the test in another period and on another share of the audience, then decide whether to cut the budget.

Common mistakes

Comparing with an unsuitable control (another city, another size); counting revenue with VAT in one group and without in the other; ignoring that ads work with a lag; drawing a conclusion from one week. One test is not forever: revisit each season. Do not change creatives and prices in the test group during the test.

What data is needed?

  • Revenue of the group with ads and a similar group without them for one period, excl. VAT; optionally for the period before the test as well.
  • Test spend and margin.
  • Optionally: the platform report and the usual spread between groups.

Inputs

FieldUnitValue in the exampleMeaning
Revenue of the group with ads during the test, excl. VAT₽14,200,000Excl. VAT, like all amounts here: the margin is applied to revenue excl. VAT.
Revenue of a similar group without ads during the test, excl. VAT₽12,100,000The same period and a comparable size: cities, regions or random customer groups. Excl. VAT.
Ad spend in the test₽1,500,000Advertising spend in the channel during the test, excl. VAT.
Gross margin%52Gross margin of sales, as a percent of revenue excl. VAT.
Revenue the platform attributes to itself, excl. VAT₽6,800,000From the account, last click; if the account shows amounts incl. VAT, remove the VAT.
Revenue of the group with ads before the test, excl. VAT₽—Optional. A period of the same length before ads were switched on. If both “before” revenues are filled in, the effect is measured against each group’s own base (as in difference-in-differences), not as a difference of levels.
Revenue of the control before the test, excl. VAT₽—Optional, a pair to the previous field.
Usual revenue spread between groups without ads%5If the lift is below the spread it may be random.

How it is calculated

StepFormula
LiftRevenue of the group with ads − control revenue
Incremental returnIncremental revenue ÷ spend
ProfitIncremental revenue × margin − spend

Worked numbers

Default values from the app; in the interactive version you enter your own.

Incremental revenue
2.1 m ₽
Lift over control
17.4 %Without revenue before the test a difference of levels is used: it is biased if the groups differ in size to begin with.
Incremental return (ROAS)
1.4×
Break-even return
1.92×1 ÷ margin: a lower return does not cover the spend.
Profit after spend
−408,000 ₽
Platform-reported return
4.53×
The platform report is higher than incremental by
3.24×

There is a lift but it does not cover the spend: short by 408,000 ₽. A return of at least 1.92× is needed.

When to use which. This calculation: advertising or a channel, two groups, profit and return. “Control group: promotion effect (difference-in-differences)” (promotions): the promotion effect with a shifting control. “Chain pilot: the effect against control stores” (format, range): pilot stores against control stores, a ratio before and after. All three compare a test with a control; choose by the subject of the test; with revenue before the test this calculation uses the same adjustment as difference in differences.

The groups must be comparable in size and season. Advertising must be labelled (erid identifier); check the requirements with a lawyer.

Nuances and limits

  • Groups different in size and season.
  • A test too short and random jumps.
  • Believing the platform’s last-click report instead of a test.
  • Taking a difference of levels when the groups differ in size to begin with: enter revenue before the test. Promotions and store pilots have their own calculations with a control.

Terms used

Calculate on your data →