Syntha Lab RU

Forecast accuracy and the value of an AI forecast: WAPE, bias and FVA on a season assortment

The document shows how to measure the accuracy of a season demand forecast and understand what really works in it: a naive repeat of the last season, a statistical model or AI, manual adjustments at approvals. The decision it supports: whether to trust the model, where to keep a manual adjustment and how many roubles are lost to error in the form of markdowns on the excess and lost margin.

Math and analytics Brand and retail Verified: 2026-11-15

Open interactively →

Why this document

A forecast is often called “accurate” or “inaccurate” by one number, and nobody knows which step improved it and which spoiled it. The document is read by the head of planning, a category manager, the CFO and an analyst before the season plan. It splits the forecast into stages, compares them on the same groups and actuals and turns the error into roubles. This makes it possible to remove a step that only takes time and to keep a manual adjustment where new information stands behind it, and not a wish to play safe.

At a glance

What is this document and why is it needed?
The document shows how to measure the accuracy of a season demand forecast and understand what really works in it: a naive repeat of the last season, a statistical model or AI, manual adjustments at approvals. The decision it supports: whether to trust the model, where to keep a manual adjustment and how many roubles are lost to error in the form of markdowns on the excess and lost margin.
Who owns the document and how often is it rebuilt?
Brand: the head of planning or an analyst, after each season and every week in the season; the commercial director decides on the rule of adjustments. Retail: category analytics, weekly by categories and store clusters; the result goes to the monthly approval of the plan and the stock.
What mistakes are the most common?
Comparing the forecasts of a season with one recalculated after it → the model looks better than it was → fix the forecasts before the season and keep versions. Counting the error only overall → an adjustment saves one group and spoils five → count by groups and by stages. Measuring in pieces and not in roubles → a cheap error and an expensive one look the same → turn over- and under-forecast into money.

For whom

Brand
A brand forecasts the production volume months before sales, so the error is expensive: the excess turns into leftovers and markdowns, the shortage into lost orders with no chance to sew quickly. The forecast is built by lines and channels (wholesale, own sales, marketplaces), and the wholesale pre-order book often serves as the strongest signal.
Retail
A retailer forecasts demand by categories and stores weeks ahead and can replenish stock, but the lead time and the display limit the room for manoeuvre. For a retailer the error by store clusters and weeks matters more, the bias towards overstating (overloaded warehouses) and the value of the forecast on new items and promotions.

What it consists of

Metrics and how to read them

  • WAPE and bias are defined in the term “Forecast error (WAPE) and bias”: the metric is weighted by volume, so large groups count more than small ones; here it compares the stages of one season.
  • Bias shows the direction of the error: a plus means overstating, a minus understating; a small error at a large bias means purchases systematically go in one direction.
  • FVA (the term “Forecast value added”): a plus means the step added accuracy, a minus means it took it away; below it is counted for each step.
  • Brand: count the metrics by line and channel separately: wholesale, own sales and marketplaces have different errors, the overall figure hides the weak channel.
  • Retail: count the metrics by categories, store clusters and weeks; the error for the chain is smaller than the error for a store, so write the aggregation level next to the number.

Forecast stages and the value of each

  • Three stages on the same groups: naive (a repeat of the last season or the last week), a model (statistics or AI), the result after manual adjustments and approvals; forecasts are fixed before the season.
  • Brand: a manual adjustment is justified when there is new information: a large wholesale order, a new channel, a price change, the launch date of a collection; it is not made “just in case”.
  • Retail: a manual adjustment is justified for a promotion, the opening or closing of a store, weather and events; for the core assortment a model is usually more accurate than an expert estimate, and this is checked by FVA and not by opinion.
  • An AI forecast goes through the same check: if it is not better than the naive one on your data, there is nothing to complicate; on new items without history its work is beyond the method, and the rule there is separate.
  • An adjustment is made with an author and a reason in the log; after a season the log shows whose adjustments added value, and the rule is revised.

The money of error

  • Over-forecast costs the markdown on the excess and holding, under-forecast costs lost margin; so the stages must be compared both in percent and in roubles.
  • Brand: the price of the excess includes leftovers in the warehouse and a carry-over to the next season; the price of the shortage takes into account the impossibility of quickly sewing a batch and the loss of a partner’s pre-order.
  • Retail: the price of the excess includes money in stock, transfers between stores and a markdown; the price of the shortage is an empty shelf and the loss of a visitor who may go to another seller.
  • Brand: for wholesale count under-forecast also by pre-orders and not only by sales: an unfulfilled order spoils the terms of the next season.
  • Retail: for promotions count over-forecast separately: the slump after a promotion creates a leftover that is not visible in an ordinary week.

What to fix for the next season

  • Brand: set up a separate row for the wholesale pre-order book and compare it with the model forecast: if pre-orders are more accurate than the model, use them as a signal and not as a competitor.
  • Brand: fix the rule of redistributing stock between channels within the season: the error by channels rarely coincides in sign, and the excess of one covers the shortage of another.
  • Retail: set up separate rows for promotions and new stores: their forecast is built separately, otherwise the error of promotions distorts the assessment of the core assortment.
  • Retail: compare accuracy by store clusters and by weeks: if the error falls towards the middle of the season, replenishment works better than the initial order.

Where the data comes from

Data sourceFields
Forecasts fixed before the seasonnaive, model and final by groups and weeks, the date of fixing, the author of the adjustment and the reason
Actual sales and demandsales by the same groups and weeks, days out of stock, promotions and their dates
Price and costsprice excl. VAT, cost, the average sale markdown, holding, the share of lost demand

How to build it step by step

  1. Step 1: Fix three forecasts by groups before the season: naive, model, final; record the date and the author.
  2. Step 2: Brand: choose groups by lines and channels and not only by categories; mark the wholesale pre-orders in a separate row.
  3. Step 3: Retail: choose groups by categories and store clusters; for new stores and promotions set up separate rows.
  4. Step 4: After the season collect the actual; if goods ran out, add an estimate of demand, otherwise the under-forecast error is understated.
  5. Step 5: Count the WAPE and bias of each stage and the FVA of the steps; compare by groups and not only overall.
  6. Step 6: Turn the error into roubles: the price of a unit of over- and under-forecast; compare the stages by money.
  7. Step 7: Brand: decide the rule of adjustments for wholesale and marketplaces: who may change the model forecast and under what conditions.
  8. Step 8: Retail: decide the rule for the core assortment and promotions; for the core the model is the default, an adjustment needs a reason.
  9. Step 9: Repeat the comparison in the next season and record how the FVA of the steps changed.

Formulas

MeasureFormulasNote
WAPE, bias and FVA by stageCounted by the definitions in the term “Forecast error (WAPE) and bias”, the term “Forecast value added (FVA)” and the “Forecast accuracy by week” calculation (fcdash); here they are applied to three stages: naive, model, manual adjustmentsVolumes in pieces, sums over groups; FVA in percentage points: below zero means the step worsened the forecast.
Price of errorPrice = over-forecast × (price × markdown + cost × holding) + under-forecast × (price − cost) × share lostOver- and under-forecast are counted by groups separately and do not offset each other.

Worked example

Six groups of the season: three forecasts against an actual of 6,140 pieces; price RUB 6,000, cost RUB 2,400; all values illustrative

GroupActual, pcsNaive, %Model, %After adjustments, %Best stage
Dresses1,3209.16.113.6Model
Outerwear78015.45.110.3Model
Knitwear1,1806.86.83.4After adjustments
Trousers69015.94.313.0Model
Tees and tops1,72012.87.01.2After adjustments
Accessories45011.14.420.0Model
Total (WAPE)6,14011.46.08.1Model

The naive error is 11.4%, the model’s 6.0%, after adjustments 8.1%: the model added 5.4 pp, the adjustments took away 2.1 pp, the bias of the result is 7.5%. In roubles: the price of the naive error is RUB 1,811,040, of the model RUB 954,240, of the result RUB 1,340,640; the adjustments added losses of RUB 386,400 and in 4 groups of 6 the result turned out worse than the model. Calculation: “Forecast value added by stage and the price of error in roubles”.

Figures

Error and bias by forecast stage
Bar chart: Naive 11.4, Model 6, After adjustments 8.1024681012NaiveNaiveWAPE: 11.411.4Absolute bias: 2.32.3ModelModelWAPE: 66Absolute bias: 1.81.8After adjustmentsAfter adjustmentsWAPE: 8.18.1Absolute bias: 7.57.5%
  • WAPE
  • Absolute bias
Data as a table
MeasureWAPEAbsolute bias
Naive11.42.3
Model61.8
After adjustments8.17.5

Error and bias by forecast stage. The model is more accurate than the naive one, while the manual adjustments gave back part of the error and added an upward bias.

From the price of the naive error to the price of the final error
Waterfall: Naive error +1,811, What the model gave −857, What the adjustments added +387, Final error 1,34105001,0001,5002,000Naive errorNaive errorNaive error: +1,811+1,811What the model gaveWhat the model gaveWhat the model gave: −857−857What the adjustments addedWhat the adjustmentsaddedWhat the adjustments added: +387+387Final errorFinal errorFinal error: 1,3411,341RUB thousand
  • positive
  • negative
  • total
Data as a table
StepChangeLevel
Naive error+1,8111,811
What the model gave−857954
What the adjustments added+3871,341
Final error—1,341

From the price of the naive error to the price of the final error. The model removed part of the losses, the adjustments brought it back: the result costs more than the model.

Specifics by category

Apparel
Apparel: high seasonality and many colors and sizes; the forecast is easier to run by style-color groups and split into sizes by the size curve; the error by sizes is higher than by groups.
Footwear
Footwear: the size run and the last make the forecast by sizes decisive; under-forecast of extreme sizes costs more, over-forecast of middle ones too; count the error separately by sizes.
Accessories
Accessories: there are almost no sizes, demand is gift and impulse; the forecast is tied to the calendar of holidays; the error is lower and the excess is safer because the shelf life is longer.
Kids
Children’s: height-size and the school season; demand is tied to dates and the growth of children; an error in the size forecast quickly leads to dead stock, so the price of over-forecast is higher.
Men
Men’s: demand is steadier, core styles repeat from season to season; the naive forecast is often strong, and the value of a model must be checked especially carefully.
Women
Women’s: more novelties and trend items without history; forecasts by analogs and manual estimates are needed more often, so it is important to measure FVA separately for novelties and the core.

Common mistakes

  • Comparing the forecasts of a season with one recalculated after it → the model looks better than it was → fix the forecasts before the season and keep versions.
  • Counting the error only overall → an adjustment saves one group and spoils five → count by groups and by stages.
  • Measuring in pieces and not in roubles → a cheap error and an expensive one look the same → turn over- and under-forecast into money.
  • Taking the sales actual when goods ran out → under-forecast is understated and the model seems better → estimate demand by days in stock.
  • Trusting any “AI forecast” without comparing it with the naive one → an expensive tool is no better than a simple one → check FVA on your own data.

Russian specifics

  • Brand: there is no norm on the forecast, it is a management measure; for accounting the season result affects the reserve for impairment of stock (FSBU 5/2019): an excess not sold at a price above cost requires assessment. Rule: Order 180n of 15.11.2019: Ministry of Finance Order of 15.11.2019 No. 180n (FSBU 5/2019 Inventories).
  • Retail: VAT and marking do not change the metrics, but the price in the calculation is taken without VAT, and an excess of leftovers with marking codes must be sold within the leftovers circulation period (check the valid terms). Rule: Decree 2129 of 25.12.2025: Government Decree of 25.12.2025 No. 2129 (amendments to Decree No. 1956; fourth marking wave) (rule check not completed).
  • For AI services the personal data of buyers in training a model falls under the law on personal data; depersonalise sales before passing them to a service and check the terms of the contract (check). Rule: Law 152-FZ of 27.07.2006: Federal Law of 27.07.2006 No. 152-FZ on personal data (rule check not completed).

Open interactively →