Why do people follow the system even when it is wrong?

Because obeying is free today and questioning has a price. When a system is wrong after go-live, the cost usually lands in a generic account, weeks later, with nobody’s name on it. The person who questions it pays now, in delays and arguments. What reduces the error is a rule, not a sermon: make the person answer for the decision.

How it works

At one Tier-1 WMS go-live I heard the call: “The new system already has it calculated. Don’t question it.” The new rules began sending standard replenishment by air, for heavy parts, with freight well above the part’s margin. When I reported it, the instruction was to stop questioning the software and execute. It went on for about two weeks, until the carriers’ invoices reached senior management.

Obeying cost the supervisor nothing that day: the freight landed in a company account. Questioning had a price: a delay, an argument with the project, the part of the one who does not trust a very expensive system.

Evidence

Linda Skitka, Kathleen Mosier and Mark Burdick gave people an automated aid in simulated flight tasks. Participants with the aid made commission errors: they followed its wrong directives even when other instruments said otherwise.

In a follow-up study, the same authors made participants accountable for their overall performance or for their accuracy. Automation bias fell: fewer wrong directives followed, and fewer events missed because the aid stayed silent.

What this does not prove

Sources

From the field notes Nobody Pays for Being Wrong · PDF, 27 pages