How do you check a deployment 90 days after go-live?

Before kickoff, pick a task where the floor knows something the system does not, and the metric that decides it, such as stockouts. Each month, log twenty of the system’s recommendations: what the operator would have done and what doing nothing would cost. If neither ever beats the system in 90 days, audit the deployer before the system.

How it works

A go-live is recorded the day it happens. Whether it worked shows up later, once the work has moved. The check puts three moves side by side on the same decisions: the system’s, the operator’s and doing nothing.

If the operator never wins, there are two explanations. Either the floor really knows nothing the system missed, or nobody is asking the floor anymore. The sign nobody asks about is the quiet one: almost no overrides and instant acceptance. On a status report it looks like adoption.

Evidence

In a 1997 paper, Raja Parasuraman and Victor Riley called over-reliance on automation misuse, next to use, disuse and abuse. My reading: instant acceptance is what misuse looks like from a status report.

What this does not prove

Sources

From the field notes The Smith and the Tailor · PDF, 10 pages