Syntha Lab RU

Checking an AI forecast against a naive baseline: WAPE, bias, improvement, share of periods and the gain net of costs

From a series of actual sales and an AI forecast for the same past periods it builds a naive forecast (the value of the period t − lag: last week or the same period of the last season) and compares with it. It counts the WAPE and the bias of the AI forecast and of the naive one, the improvement against the baseline, the share of periods where the AI is more accurate, whether the improvement threshold is passed, the saved error in units and the gain in money net of the cost of the tool. It answers whether the AI forecast deserves more trust than a simple rule. The forecast must have been made before the fact and not fitted; the default data are illustrative.

Season results: brand, category, SKU For whom: Analysts, planners, heads of data and owners who decide whether to introduce an AI demand forecast instead of a simple rule.

Calculate on your data →

At a glance

What question does it answer?
Whether the AI forecast is more accurate than the simple rule “as in the last period”, by how much, in how many periods and whether it pays off after the cost of the tool. WAPE and bias are counted by the definitions of the “Forecast accuracy by week” calculation (fcdash).
What data is needed?
The actual sales by period for the past.
How to read the result?
Example at the default values (12 weeks, a shirt-jacket): 11 weeks are compared; the WAPE of the AI is 12.1%, of the naive 14.4%; the improvement is 15.8% at a threshold of 10%; the bias of the AI is 0.3%; the AI is more accurate in 81.8% of the weeks; the saved error is 34 units, the gain RUB 20,400, the result after costs RUB 400. Example on the default values: Compared periods: 11; WAPE of the AI forecast: 12.1%; WAPE of the naive forecast: 14.4%; AI error to naive error: 0.84. The figures are illustrative: enter your own to see your result.

Why and how to reason

The simple rule “as last week” is free and often strong, so AI is judged not by absolute accuracy but by the gain against it. The calculation builds a naive forecast from the actuals (the value of the period t minus the lag) and compares it with the AI forecast over the same periods. WAPE is the sum of absolute errors divided by the sum of actual sales. At the default values over 11 compared weeks the WAPE of the AI is 12.1%, of the naive one 14.4%, the ratio of errors is 0.84, that is an improvement of 15.8%. The AI is more accurate in 82% of the weeks and its bias is 0.3%. The saved error of 34 units at a cost of an error of RUB 600 gives RUB 20,400, the cost of the tool is RUB 20,000, the result is RUB 400: the improvement threshold of 10% is passed but the gain almost equals the costs.

When to use it

Apply before introducing an AI forecast and in the regular check of one that already works: the forecast must be made before the period without a look into the future. At least a few periods after the lag are needed, preferably more. The calculation does not check the causes of an error and does not replace a check of the data. For footwear and accessories with small sales per SKU compare at the category level, for children’s take a lag equal to the length of the season.

How to read the result

Look at the improvement against the baseline and at the threshold: 15.8% is above 10%, so the AI beats the rule. Then at the share of periods (82%) and the bias (0.3%): a forecast without bias does not skew purchases. Then at the money: a result of RUB 400 over 12 weeks is not a reason to widen the use but a reason to check one more season and count the full cost, including preparing the data and a human check. If the improvement is below the threshold keep the simple rule and look for the cause in the data, the horizon and fitting and do not change the method at random.

Common mistakes

They check on the data the model was tuned on or fit the forecast to the fact. They take few periods: one lucky week changes the conclusion. They look at the average error and do not look at the bias. They do not count the cost of preparing data and a human check. They assess AI without a baseline and take a beautiful figure for a result.

What data is needed?

  • The actual sales by period for the past.
  • The AI forecast made before each period without fitting to the fact.
  • The cost of an error of one unit and the cost of the tool over the horizon.

Inputs

FieldUnitValue in the exampleMeaning
Lag of the naive forecastperiods1How many periods back the naive rule looks: 1 is last week; for a seasonal series take the length of the seasonal cycle. The example value is illustrative.
Required improvement against the baseline%10By how many percent the AI error must be smaller than the error of the naive rule to use the AI; the threshold is set by your policy. The example value is illustrative.
Cost of an error of one unit₽600How much one unit of forecast error costs on average: the lost margin on a shortage or the cost of holding and price reduction on an excess. Your estimate. The example value is illustrative.
Cost of the tool over the horizon₽20,000The subscription, the development and a specialist’s time over the checked horizon. The example value is illustrative.

How it is calculated

StepFormula
Naive forecastthe value of the period t − lag; the periods after the lag are compared
WAPEΣ|forecast − actual| ÷ Σ actual
BiasΣ(forecast − actual) ÷ Σ actual
Improvement1 − AI WAPE ÷ naive WAPE; compare with the threshold
Gain(naive error − AI error) × cost of an error − cost of the tool

Worked numbers

Default values from the app; in the interactive version you enter your own.

Compared periods
11
WAPE of the AI forecast
12.1 %
WAPE of the naive forecast
14.4 %
AI error to naive error
0.84Below one: the AI is more accurate than the simple rule; above: worse.
Improvement against the baseline
15.8 %
Bias of the AI
0.3 %A plus: the forecast overstates sales, a minus: understates.
Bias of the naive
−0.3 %
Periods where the AI is more accurate
81.8 %
Improvement threshold passed: yes (1) or no (0)
1
Saved error, units
34
Gain in money
20,400 ₽
The result net of the cost of the tool
400 ₽
Actual, the AI forecast and the naive forecast
Line chart: Actual: 95 → 85; AI forecast: 85 → 100; Naive: 80 → 10050100150200250Week 2Week 5Week 8Week 11Actual, Week 2: 95Actual, Week 3: 110Actual, Week 4: 140Actual, Week 5: 170Actual, Week 6: 190Actual, Week 7: 185Actual, Week 8: 160Actual, Week 9: 140Actual, Week 10: 120Actual, Week 11: 100Actual, Week 12: 85AI forecast, Week 2: 85AI forecast, Week 3: 98AI forecast, Week 4: 116AI forecast, Week 5: 146AI forecast, Week 6: 172AI forecast, Week 7: 201AI forecast, Week 8: 178AI forecast, Week 9: 156AI forecast, Week 10: 134AI forecast, Week 11: 114AI forecast, Week 12: 100Naive, Week 2: 80Naive, Week 3: 95Naive, Week 4: 110Naive, Week 5: 140Naive, Week 6: 170Naive, Week 7: 190Naive, Week 8: 185Naive, Week 9: 160Naive, Week 10: 140Naive, Week 11: 120Naive, Week 12: 100unitsPeriod
  • Actual
  • AI forecast
  • Naive
Data as a table
PeriodActualAI forecastNaive
Week 2958580
Week 31109895
Week 4140116110
Week 5170146140
Week 6190172170
Week 7185201190
Week 8160178185
Week 9140156160
Week 10120134140
Week 11100114120
Week 1285100100
WAPE error: AI and naive, %
Bar chart: AI 12.1, Naive 14.4051015AIAIWAPE, %: 12.112.1NaiveNaiveWAPE, %: 14.414.4%
Data as a table
MeasureWAPE, %
AI12.1
Naive14.4

Error by period

PeriodActualAI forecastNaiveAI errorNaive error
Week 29585801015
Week 311098951215
Week 41401161102430
Week 51701461402430
Week 61901721701820
Week 7185201190165
Week 81601781851825
Week 91401561601620
Week 101201341401420
Week 111001141201420
Week 12851001001515

The calculation: the naive rule is free and often strong, so the AI is judged not by absolute accuracy but by the improvement against it. The check is honest only if the forecast was made before the period, on data without a look into the future; a forecast fitted to the fact gives a beautiful and useless figure. Few periods give a noisy estimate: look at the share of periods and at the bias and not only at the average. For footwear and accessories with small sales per article compare at the category level, for children’s take the seasonal shift into account and a lag equal to the length of the season.

Nuances and limits

  • Judging the AI by an absolute error without comparison with a simple rule: a beautiful figure may be weaker than a free baseline.
  • Checking on the data the AI was trained or fitted on: the error is understated.
  • Taking few periods: the estimate is noisy and one lucky week changes the conclusion.

Terms used

Calculate on your data →