Why do people follow the system even when it is wrong?
Because obeying is free today and questioning has a price. When a system is wrong after go-live, the cost usually lands in a generic account, weeks later, with nobody’s name on it. The person who questions it pays now, in delays and arguments. What reduces the error is a rule, not a sermon: make the person answer for the decision.
How it works
At one Tier-1 WMS go-live I heard the call: “The new system already has it calculated. Don’t question it.” The new rules began sending standard replenishment by air, for heavy parts, with freight well above the part’s margin. When I reported it, the instruction was to stop questioning the software and execute. It went on for about two weeks, until the carriers’ invoices reached senior management.
Obeying cost the supervisor nothing that day: the freight landed in a company account. Questioning had a price: a delay, an argument with the project, the part of the one who does not trust a very expensive system.
Evidence
Linda Skitka, Kathleen Mosier and Mark Burdick gave people an automated aid in simulated flight tasks. Participants with the aid made commission errors: they followed its wrong directives even when other instruments said otherwise.
In a follow-up study, the same authors made participants accountable for their overall performance or for their accuracy. Automation bias fell: fewer wrong directives followed, and fewer events missed because the aid stayed silent.
What this does not prove
- Skitka’s participants worked simulated flight tasks, not a warehouse after go-live.
- The air-freight scene is mine, written from memory. It illustrates; it does not measure.
- That obeying is cheap for the person doing it is my reading. What the studies support is narrower: accountability reduced the errors.
Sources
- Skitka, L. J., K. L. Mosier, and M. Burdick. 1999. “Does automation bias decision-making?” International Journal of Human-Computer Studies 51(5): 991-1006. doi.org/10.1006/ijhc.1999.0252
- Skitka, L. J., K. L. Mosier, and M. Burdick. 2000. “Accountability and automation bias.” International Journal of Human-Computer Studies 52(4): 701-717. doi.org/10.1006/ijhc.1999.0349
From the field notes Nobody Pays for Being Wrong · PDF, 27 pages