Forecast value added by stage and the price of error in roubles: the season assortment
For groups of the assortment compares three forecasts of one season: naive (a repeat of the last season), a model (statistics or AI) and the result after manual adjustments. Counts the WAPE and bias of each stage, the value added of each step (FVA) and turns the error into roubles: over-forecast costs a markdown and holding, under-forecast costs lost margin. The definitions of WAPE, bias and FVA are given in the “Forecast accuracy by week” calculation (fcdash); here the point is the price of error and the stages.
At a glance
- What question does it answer?
- Which stage of the forecast adds accuracy (the model or manual adjustments), how biased it is and how many roubles the error costs in the form of markdowns on the excess and lost margin.
- What data is needed?
- Forecasts by assortment group fixed before the season: naive, model and final.
- How to read the result?
- Example at the default values (six groups, price RUB 6,000, cost RUB 2,400): WAPE of the naive 11.4%, of the model 6%, after adjustments 8.1%; FVA of the model 5.4 pp, of manual adjustments −2.1 pp; the model saved RUB 856,800, the adjustments added losses of RUB 386,400. Example on the default values: WAPE of the naive forecast: 11.4%; WAPE of the model: 6%; WAPE after adjustments: 8.1%; Bias of the naive: −2.3%. The figures are illustrative: enter your own to see your result.
Why and how to reason
Forecast accuracy is usually counted as one number and nobody knows where it came from. But a forecast is made in several steps: the last season as a starting point, a model (statistics or AI), then approvals and manual adjustments. Each step must make the forecast better than the previous one, otherwise it only takes time. So the stages are compared on the same groups and actuals: the WAPE error equals the sum of absolute deviations divided by the sum of sales, and the value of a step (FVA) equals the error before it minus the error after. The second question is what the error costs. The excess goes into a sale at a markdown and occupies the warehouse, the shortage loses the margin of lost sales. In the example the naive forecast is wrong by 11.4%, the model by 6%, the result after adjustments by 8.1%: the model added 5.4 pp, the adjustments took away 2.1 pp, and in roubles the model saved RUB 856,800, the adjustments added losses of RUB 386,400.
When to use it
Apply after the season, when there is an actual, and before the next plan: to decide what to trust in the new forecast and where manual adjustments are acceptable. Forecasts fixed before the season are needed. Not suitable for new styles without history and for groups with almost no sales. Count by groups of one level: SKUs and categories must not be mixed.
How to read the result
Look at three pairs of numbers. First: the WAPE and bias of each stage; a large bias at a small error means the forecast errs in one direction. Second: the FVA of the steps; plus means the step helps, minus means it should be removed or redone. Third: the price of the error in roubles. In the example the result after adjustments is biased by +7.5%: the adjustments systematically inflated the plan, and in four groups of six the result turned out worse than the model. The decision: keep an adjustment only where there is new information behind it (a contract, a promotion, a change of channel), and record who made it and why.
Common mistakes
The error is counted in pieces and not in roubles: over- and under-forecast cost differently. The comparison is with a forecast recalculated afterwards. The sales actual is taken when goods ran out: demand was in fact higher. A conclusion is drawn from one group. The illustrative price and markdown are taken as your own: substitute the data of past seasons.
What data is needed?
- Forecasts by assortment group fixed before the season: naive, model and final.
- Actual sales by the same groups over the season (better the demand with out-of-stock taken into account).
- The price, the cost, the average sale markdown, the holding cost and the share of irrecoverably lost demand.
Inputs
| Field | Unit | Value in the example | Meaning |
|---|---|---|---|
| Average price excl. VAT | ₽ | 6,000 | The unit price across the assortment excl. VAT: the cost of the excess markdown and the margin of a lost sale are counted from it. The example value is illustrative. |
| Average cost in the warehouse | ₽ | 2,400 | The full cost of a unit in the warehouse excl. VAT. The price must be above the cost, otherwise under-forecast does not cost margin. The example value is illustrative. |
| Markdown on the excess | % | 40 | By how many percent of the price the excess goes in the sale on average; take your own depth from past seasons. The example value is illustrative. |
| Holding and money on the excess | % of cost | 12 | Warehouse, frozen money and the risk of dead stock on the excess over the season, as a percentage of the cost. The example value is illustrative. |
| Demand that did not come back later | % | 70 | What part of the buyers lost because goods ran out goes for good: the rest wait or take a substitute. The example value is illustrative. |
How it is calculated
| Step | Formula |
|---|---|
| Stage WAPE | sum |forecast − actual| ÷ sum of actuals |
| Step FVA | WAPE before the step − WAPE after the step |
| Unit price of over-forecast | price × markdown + cost × holding = 2,688 RUB |
| Unit price of under-forecast | (price − cost) × share of lost demand = 2,520 RUB |
Worked numbers
Default values from the app; in the interactive version you enter your own.
- WAPE of the naive forecast
- 11.4 %The sum of |forecast − actual| ÷ the sum of actuals over all groups.
- WAPE of the model
- 6 %
- WAPE after adjustments
- 8.1 %
- Bias of the naive
- −2.3 %Plus: the forecast is above the actual, minus: below.
- Bias of the model
- −1.8 %
- Bias after adjustments
- 7.5 %
- FVA of the model over the naive, p.p.
- 5.4WAPE of the naive minus WAPE of the model: plus means the model added value.
- FVA of manual adjustments over the model, p.p.
- −2.1WAPE of the model minus WAPE of the result: minus means the adjustments worsened the forecast.
- Price of the naive error
- 1.81 m ₽
- Price of the model error
- 954,240 ₽
- Price of the error after adjustments
- 1.34 m ₽
- What the model gave over the naive
- 856,800 ₽
- What the adjustments cost against the model
- 386,400 ₽Plus means the adjustments added losses in roubles.
- Groups where adjustments worsened the model
- 4 of 6
- The best stage by WAPE
- Model
- WAPE
- Absolute bias
Data as a table
| Measure | WAPE | Absolute bias |
|---|---|---|
| Naive | 11.4 | 2.3 |
| Model | 6 | 1.8 |
| After adjustments | 8.1 | 7.5 |
- Over-forecast: markdown and holding
- Under-forecast: lost margin
Data as a table
| Measure | Over-forecast: markdown and holding | Under-forecast: lost margin |
|---|---|---|
| Naive | 753 | 1,058 |
| Model | 349 | 605 |
| After adjustments | 1,290 | 50 |
Error by group, % of the group actual
| Group | Actual, pcs | Naive | Model | Result | Best stage |
|---|---|---|---|---|---|
| Dresses | 1,320 | 9.1 % | 6.1 % | 13.6 % | Model |
| Outerwear | 780 | 15.4 % | 5.1 % | 10.3 % | Model |
| Knitwear | 1,180 | 6.8 % | 6.8 % | 3.4 % | After adjustments |
| Trousers | 690 | 15.9 % | 4.3 % | 13 % | Model |
| Tees and tops | 1,720 | 12.8 % | 7 % | 1.2 % | After adjustments |
| Accessories | 450 | 11.1 % | 4.4 % | 20 % | Model |
The manual adjustments worsened the forecast by 2.1 p.p. and added 386,400 ₽ of losses; in 4 of 6 groups the result is worse than the model. Keep an adjustment only where new information stands behind it.
The stages are compared on the same groups and actuals. The value of a step is the difference of errors before and after it: if it is below zero, the step is removed or replaced.
Nuances and limits
- Counting accuracy only overall: an adjustment may save one group and spoil the rest.
- Comparing with a forecast recalculated after the season: an honest comparative forecast is fixed in advance.
- Assessing the error in pieces and not in roubles: under- and over-forecast cost differently.