Do manual overrides of a statistical forecast improve accuracy?

Sometimes. In a study of more than 60,000 forecasts from four supply-chain companies, large adjustments helped most and small ones often made the forecast worse. The bigger problem is that, without a name and a recorded reason, a company cannot tell the planner who knew something from the one who was guessing, so it learns from neither.

How it works

An override can be the one thing nobody else in the room knew: a customer about to cancel, a promotion nobody loaded. Or it can be a mood with a keyboard. From the outside the two look the same: a change with no name and no reason, corrected or not in a generic account weeks later.

Two cheap rules change that. Make the statistical forecast the default, so that changing it takes an explicit request. And ask for a recorded reason before any change. A reason field that accepts “as discussed” is a checkbox, not a reason.

Evidence

Robert Fildes, Paul Goodwin, Michael Lawrence and Konstantinos Nikolopoulos examined more than 60,000 forecasts and outcomes from four supply-chain companies. In three of the four, judgmental adjustments improved accuracy on average. Large adjustments helped most; small ones often made the forecast worse.

In a forecasting experiment, Paul Goodwin tested cheap changes. Two of them worked: making the statistical forecast the default, so that changing it took an explicit request, and asking for a recorded reason before any change. Both improved accuracy.

What this does not prove

Sources

From the field notes Nobody Pays for Being Wrong · PDF, 27 pages