Do manual overrides of a statistical forecast improve accuracy?
Sometimes. In a study of more than 60,000 forecasts from four supply-chain companies, large adjustments helped most and small ones often made the forecast worse. The bigger problem is that, without a name and a recorded reason, a company cannot tell the planner who knew something from the one who was guessing, so it learns from neither.
How it works
An override can be the one thing nobody else in the room knew: a customer about to cancel, a promotion nobody loaded. Or it can be a mood with a keyboard. From the outside the two look the same: a change with no name and no reason, corrected or not in a generic account weeks later.
Two cheap rules change that. Make the statistical forecast the default, so that changing it takes an explicit request. And ask for a recorded reason before any change. A reason field that accepts “as discussed” is a checkbox, not a reason.
Evidence
Robert Fildes, Paul Goodwin, Michael Lawrence and Konstantinos Nikolopoulos examined more than 60,000 forecasts and outcomes from four supply-chain companies. In three of the four, judgmental adjustments improved accuracy on average. Large adjustments helped most; small ones often made the forecast worse.
In a forecasting experiment, Paul Goodwin tested cheap changes. Two of them worked: making the statistical forecast the default, so that changing it took an explicit request, and asking for a recorded reason before any change. Both improved accuracy.
What this does not prove
- Fildes’s data came from four companies’ forecasts, and Goodwin’s from an experiment. Neither is a warehouse after go-live.
- That a company learns from neither without a reason is my reading, not a finding of either study.
Sources
- Fildes, R., P. Goodwin, M. Lawrence, and K. Nikolopoulos. 2009. “Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supply-chain planning.” International Journal of Forecasting 25(1): 3-23. doi.org/10.1016/j.ijforecast.2008.11.010
- Goodwin, P. 2000. “Improving the voluntary integration of statistical forecasts and judgment.” International Journal of Forecasting 16(1): 85-99. doi.org/10.1016/S0169-2070(99)00026-6
From the field notes Nobody Pays for Being Wrong · PDF, 27 pages