Drew T
2.0 KiB
L4 — verify-blast-radius-not-just-defect
id: L4 · group: - · status: active · tags: legacy,memory · origin: legacy memory verify-blast-radius-not-just-defect.md · added: 2026-09-29
Verify the BLAST RADIUS, not just the DEFECT. (Phase 26, 2026-07-14 — I got this wrong in front of Drew.)
A coverage audit reported that progress.py under-counted ~243k instructions because classify() reads a K&R
definition as a forward declaration. I did the R14 thing — reproduced the mechanism against the bytes, confirmed
it was real, measured 400 banked instances in that shape — and then told Drew our headline numbers had been
under-reporting our progress.
Wrong. The headline metrics come from weighted_metrics(), which never calls classify() at all: it tests
"is this function still an INCLUDE_ASM stub?", so it is structurally immune to the bug. The published
instruction-weighted and distinct-code numbers were correct all along; only a secondary function-count report was
wrong.
Why: "this tool is broken" and "this number is wrong" are different claims requiring different evidence. A confirmed mechanism proves nothing about consequence. Before reporting impact, trace the defect to the actual consumer and check whether that consumer is even on the affected path.
How to apply:
- After confirming a defect, ask "who consumes this, and does the consumer use this code path?" — then verify THAT, not the defect, before quoting an impact number to the owner.
- A null result where you predicted a large effect is a refutation — chase it, do not wave it off. The fix moved the numbers by +376 instructions when I had predicted +190,000. That gap was the whole story and it would have been trivially easy to dismiss as noise (or worse, to report as "the fix worked, the numbers moved").
- Do not amplify a sub-agent's impact claim (R14) — the auditor conflated "classify() is blind" with "the metrics are wrong", and I propagated it as fact while lecturing about unverified oracles.
Related: derive-from-invariants-not-reparsing.