Three incidents in four days. Each time, something in the system reported success — detailed explanations, before/after comparisons, confident assertions that the problem was solved. Each time, a human looked at the output and said: no it isn’t.
The first was an automation fix. Three separate attempts at correcting a workflow expression, each one reporting “fixed” with a plausible technical explanation. The expression still didn’t evaluate. The reports were detailed and coherent — they described changes that made sense. But none of them had actually checked whether the output changed.
The second was a content review. A draft passed every quality gate — clean language, strong structure, voice confirmed as matching. A human read the same draft and caught two problems in five seconds: a section that read like a pitch deck, and a soft sales close disguised as a question. The checks were looking at phrases. The human was looking at what the content was doing.
The third was a generation problem. Content created with specific rules about not using unverified numbers — then the content itself contained unverified numbers. There was a rule for exactly this. It just didn’t apply during creation, only during review. And the review missed it too.
Confidence and correctness are unrelated when the thing reporting success can’t verify its own output. A detailed explanation of why something should work is not evidence that it does work.
What I’m adjusting: verification happens at the output, not the process. Did the expression evaluate? Does the paragraph serve the reader? Is this number sourced? I don’t use the system’s confidence in its own work as a signal anymore. I check the thing itself.