BFM-decomp/phase-ends/PhaseEnd_Phase28.md

PhaseEnd — Phase 28: The Endgame Engine → the instrument-repair phase (every "wall" was our tooling)

Date: 2026-07-16 · Project Version: 1.27.0 · Phase Status: Complete (milestone confirmed by Drew, gate 2) · Generation: Gen2 (20th phase of the arc; Phase 14 public-flip deferred to Gen3+)

Gen2 phase 20 (8→…→27→28). Second phase off docs/roadmap-to-100.md (§3 P28). Ran across one long session + a resume after a terminal crash. The granular per-task crash-recovery trail (every probe, the wave verdicts, the two self-inflicted defects and their repair, the classified failure taxonomies) is preserved on-demand at phase-ends/logs/Phase28.md (R19 — NOT auto-loaded; consult only when researching a mechanism). This file is the synthesis. Owner decisions (Drew): re-scope on the Phase-Start finding; Fable5 = conditional slot (unused); the 39 type-1 modules = deferred/documented; expand T7 to the citizenship gate + fix the blind disc sweep; add R36; continue T6→T7→T3b to completion; close at Max.

Build Log

Files created/changed and complete — do not recreate (the durable Phase-28 output; the archived log has the exhaustive per-commit list):

New tooling (committed):

• tools/dedup_extend.py — the missing mode: EXTEND an existing macro-backed dedup group to a newly-onboarded binary (dedup_propagate only authors macros from inline defs; the ~1,600 shared bodies are already DEFINE_func_* macros). Owns only the set computation + a surgical text registry edit (never yaml.safe_dump — see Deviations); the splice + gate are harvest_verify verbatim. Carries the §53 jr-guard (with an R14 note on what it is not).
• tools/audit_binaries.py + make audit-binaries — the R36 citizenship gate (R32 enforcement): every onboarded binary (derived from config/splat.*.yaml, R33) is asserted present in dup_report.BINARIES, has a sig, and (overlays) includes ../shared/engine_core.h. Negative-control-proven to catch the exact SC07 bug; wired into tools-health.

Fixed tools (the phase's real substance — each fix CHANGED an answer the old tool gave):

• tools/family_remap.py — img_path derives the payload from the splat config's target_path (R33) and raises on absence (R32); was a hardcoded 0.4.dec that returned silent None for the 4 SC07 overlays → phantom LEN. Un-hid a 230,612-ins pool.
• tools/progress.py — #if 0-blindness (a dead body counted as REAL and stub) + the matchable = len()-sum (a fn in two buckets counted twice) → both fixed; added the OVER-coverage assertion (R32, both directions). Resident 123/146 → 122/145.
• tools/match_one.py — --work defaulted to a shared scratch dir while the docstring promised isolation; now a private <fn>.<pid> dir. Found by an agent mid-wave reading another agent's function out of its own t.c.
• tools/harvest_verify.py — --chunk 1 no longer double-builds every failing draft (~26% fewer builds on the gate's hot path).
• tools/disc_code_sweep.py — was structurally blind to compressed code (decoded only raw bytes → the type-4 row was vacuous for 138 known binaries); now decodes both the raw and the LZSS-decompressed layer. Could not have found a hidden type-4 overlay before; can now.
• tools/worklist.py — --assert-partition read source where the manifest key is source_overlay → the R32 assertion silently rode a hardcoded default; now reads the right key and fails loud if absent.
• tools/grinder.py — the poisoned 22-entry blacklist purged (8 had matched anyway; built from a gate that no longer exists); carries the R35 rule that a blacklist entry expires when its gate changes.

Config / source (the harvest — fleet 67.0% → 68.9% instr):

• The 4 SC07 overlays wired in (src/ov_SC07_{006,007,010,011}/*.c + config/dedup.us.yaml): 6,174 h_exact members banked (95.6%), C1 coverage 227,211 → 233,385, groups 134 → 138 binaries.
• B2 family (func_8017BEBC, 952 ins): 102/115 banked ×134-class via the jtbl carve path (src/ov_*/*_jr_8017BEBC.c + config/overlays.mk).
• Resident (src/resident/resident.c): 21 → 14 stubs (7 banked via the Ultracode wave); .run/resident_wave_verdicts.json preserved (R20).
• T3b + T3-A giants: 9 + 4 legacy/giant banks.

Docs / knowledge base (R16/R30/R31):

• docs/matching-cookbook.md §53 — sweep a family with the tool its exemplar needed (the carve law + the --raw rule + the symbol_map-jtbl trap + the "before a 0% retires a lever" test).
• docs/calibration.md — the "structural families ≈0%" table rewritten (the ≈0% row marked an artifact); the addressable pool tabulated; the legacy h_seq swing number recorded (~3% as-tooled, classified, ceiling-unknown).
• docs/decision-log.md (R31) — the B2 refutation; the T3b swing-number measurement + the named P29 disambiguating probe.
• docs/resident-dossier.md — new: the 14 remaining resident stubs, classed (5 PLUMBING / 4 DIFF / 5 jtbl-deferred) with each agent's byte-grounded residual analysis.
• docs/second-oracle.md (context), docs/SETUP.md (implicit via tool inventory), docs/disc-completeness.md (the fixed-sweep confirmation).
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase28.md (R19). phase-ends/PhaseEnd_Phase28.md — this file.

Local artifacts (gitignored / regenerable): .run/family_hseq.json (regenerated 138-overlay), .run/drafts-t5/*, .run/t5_verdicts.json, .run/t3b_*, .run/t7_*, build/, expected/, asm/, assets/. The resident Ghidra-C prefetch (.run/ghidra_c/func_800D*.c) is script-reproducible.

Tools/packages installed: None — the whole phase used the existing toolchain + venv. Fable5/Opus agents via the Agent tool; one Ultracode wave (worker_wave-style) via the Workflow tool. No Ghidra DB change (R23 no-op — the resident prefetch was read-only headless decompile; the db.*.gbf churn is restart-noise, not staged).

Verification results (literal) — the milestone proof:

• make clean && make extract-all && make check-all → 140 passed, 0 failed of 140 BYTE-IDENTICAL from a genuinely-clean tree (R22), reproduced after every banked batch.
• make tools-health → OK (exit 0): corpus(+resident) 0 PHANTOM + 0 TRUNCATED; audit-binaries: 140 onboarded, every one a full citizen; audit-cdecl green; dedup-check 1840 validated / 0 failed, C1 coverage 233,385/233,385; report(lint+dedup) green. All fail-closed.
• make report: FLEET instr-weighted 67.0% → 68.9% (9,017,678 / 13,081,451) · distinct-code 47.8% → 49.5% (2,757,556 / 5,574,674) · fn-count 82.16% → 83.94% · 0 NON_MATCHING in any default build (G4). Velocity: +1.9pp instr this phase.
• The disc is complete (fixed disc_code_sweep, coverage-asserted 1189/1189): 138 type-4 overlays all onboarded, 0 further hidden type-4; the 39 type-1 modules remain pending load-address RE (documented, deferred).
• git status: only config//tools//src//docs//.run/(allowlisted)/phase-ends/ tracked; the db.*.gbf churn is R23 restart-noise (not staged).

Milestone achieved (confirmed by Drew, gate 2): Phase 28 rebuilt the endgame plan on measured reality and, in doing so, banked a large harvest almost entirely by fixing instruments rather than out-muscling the compiler. The swing number was measured and classified (T3-A: the SC07 pool = h_exact, 95.6%; T3b: the legacy h_seq class = ~3% as-tooled, ceiling-unknown, 274 DIFF / 37 PLUMBING); the resident was resolved (90.34% + an honest classed dossier); the fleet rose 67.0% → 68.9% instr at 140/140 byte-identical / 0 NON_MATCHING; and the R36 citizenship gate now enforces that a newly-onboarded binary cannot hide from the tools again. The phase's decisive through-line: roadmap item B2 LIVES (the "structural families ≈0%" doctrine that reshaped two phases of strategy had no surviving post-fix evidence — its 0/8 was a missing carve), and every "compiler wall" probed this phase resolved to our own tooling.

Next: Phase 29 — The Family Campaign (roadmap §3 P29), opening with the T3b disambiguating probe (diff one PURE DIFF member's staged bytes region-by-region: reloc-position mismatch ⇒ fixable remap and the legacy h_seq ceiling is far above 3%; regalloc-away-from-relocs ⇒ a genuine TU wall) before scaling any "(cores)×(reach)" arithmetic on 3%. Carried tools: the burn-down velocity tracker (T7c), diff_regions.py, the grinder family-warm-start, and the same-binary shard farm (all deferred, none lost). Fresh session, plan mode, Max (Tier-1 Phase-Start).

Deviations

Item	Plan (roadmap P28)	Actual	Reason
The whole phase	build member_adapt.py + the delta engine around the swing number	RE-SCOPED at plan time — the swing number's evidence was untrustworthy (R35); the phase became instrument-repair + honest measurement	Phase-Start verification found the ≈0% doctrine rested on a broken-tool probe (B2's 0/8 was a missing carve); building on it would have been building on sand
member_adapt.py	THE deliverable	NOT built — the number said build nothing like it	T3-A: the SC07 pool is h_exact + unwired (95.6% via dedup_extend, the right tool), not the h_seq drift-adaptation member_adapt targets; the STRUCT/register-drift class it was scoped for is 1.8% of the population
The parallel gate farm	build it (throughput bottleneck)	only the free --chunk 1 fix (T6); the distinct-binary farm already exists in bulk_harvest; the same-binary shard farm deferred (conditional on P29 volume)	the premise was stale — the gate stopped being the bottleneck in Phase 23 (drafting is)
Fleet-wide Ghidra-C prefetch	import all 138 overlay programs + cache every stub	only the resident's 21 (the flag-plant needed them)	the SC07 harvest was h_exact (no drafting → no Ghidra-C needed); full-fleet prefetch is P29 fuel when the family campaign needs it
Grinder warm-start / diff_regions.py / burn-down tracker	build them	deferred/carried to P29 (T7c)	not load-bearing for this phase's actual work; genuine new tools better built with the campaign that uses them
The swing number	a measured close-rate	measured but CEILING-UNKNOWN (~3% as-tooled; the 274 DIFF is the tooling-vs-wall ambiguity, un-disambiguated)	refused to rush a clean number on tight context — that is how Phase 26 manufactured a wrong 0% (P9); a classified measurement + a named probe is the honest deliverable
Resident flag-plant	100%	90.34% (21→14) + an honest classed dossier	the "OR an honest wall dossier" branch; the 5 jtbl need the carve, the 5 PLUMBING need a def-side lever, the 4 DIFF are genuine residuals — none forced (forcing is how the ≈0% doctrine was made)
T7 scope	burn-down + stale labels	EXPANDED (Drew) to the audit-binaries R36 gate + fixing the blind disc_code_sweep	the SC07 evidence showed 4 consumers silently ignoring onboarded binaries — a gate, not a label sweep, is the fix
Fable5 slot	conditional	UNUSED	no new wall class appeared that cheap-Opus/known idioms couldn't handle; spending a discoverer on already-discovered idioms is the one thing the doctrine forbids
Three self-inflicted defects	(n/a)	the registry yaml.safe_dump (H5), two DIFF mis-reports (R14), the match_one shared-scratch — each caught + fixed forward	the phase's own thesis (instruments lie) applied to me; each is recorded honestly, not buried under the 95.6% headline

Commit Message

(Phase 28 landed as 16 commits 76db32455..6996d2537 + this PhaseEnd. Fleet 67.0 -> 68.9% instr,
140/140 byte-identical, 0 NON_MATCHING throughout. The granular trail -> logs/Phase28.md.)

feat(phase-28): the endgame engine -> the instrument-repair phase; B2 lives, SC07 wired, R36 (v1.27.0)

- RE-SCOPED at plan time (R35): the roadmap's swing number rested on a broken-tool probe. B2's
  "structural families bank ~0%" (0/8, which reshaped 2 phases of strategy) was a MISSING CARVE ->
  8/8, then 102/115 (88.7%). The ~0% doctrine has NO surviving post-fix evidence.
- T0 img_path derive-not-guess -> un-hid a 230,612-ins SC07 pool doubly hidden (P27 onboarded 4
  overlays, never regenerated the map; the hardcode would have called every member "LEN").
- T3-A: the SC07 pool is h_exact + UNWIRED, not h_seq. T4 dedup_extend (NEW) wired 6174/6457 (95.6%)
  -> groups 134->138 binaries, C1 coverage +6174. NOT member_adapt (the number said build nothing).
- T5: resident 21->14 (90.34%, 7 banked via an Ultracode wave) + honest dossier for the 14; fixed
  progress.py (#if 0 + len()-sum) and match_one's FAKE isolation (found by an agent mid-wave).
- T2 purged the poisoned grinder blacklist (8/22 matched anyway). T6 killed the --chunk 1 double-build.
- T7: make audit-binaries (the R36 citizenship gate, negative-control-proven) + fixed disc_code_sweep's
  BLINDNESS to compressed code (the type-4 row was vacuous for 138 known binaries) + the worklist key-bug.
- T3b: the legacy h_seq swing number = ~3% AS-TOOLED, CLASSIFIED (274 DIFF / 37 PLUMBING), ceiling
  UNKNOWN -- the 274 DIFF is byte-PURE members whose remapped bodies don't reproduce (the same
  tooling-vs-wall ambiguity that resolved to TOOLING twice this phase). P29 disambiguates before scaling.
- THREE SELF-INFLICTED DEFECTS fixed forward: the registry yaml.safe_dump (H5, destroyed 47 comments +
  1832 hex fields, invisible to every byte-gate), two DIFF mis-reports (R14), the match_one shared scratch.
- rule R36 (a newly-discovered binary is not real until every consumer knows it; enforced by
  audit-binaries). cookbook §53. R22 140/140 throughout; tools-health OK; dedup 1840/0.

Rules Added This Phase

Rule

Reason

R36 — A newly-discovered binary is not real until every consumer knows it. Onboarding a code-bearing payload produces a byte-clean binary that is not yet a citizen: the same change must wire it into every consumer that enumerates binaries — the sig set, the family map, the dedup registry (or an explicit exemption), the shared-header include, and the reports/labels — and that must be asserted by a gate (R32), not remembered. A binary a consumer silently ignores is invisible work, and the byte-gate is structurally blind to it (R34). Enforcement shipped: make audit-binaries (negative-control-proven, in tools-health).

P27 onboarded 4 SC07 overlays byte-clean; P28 found four consumers silently ignoring them (family_remap.img_path, .run/family_hseq.json, config/dedup.us.yaml, the overlays' own .c), hiding ~6,400 already-matched bodies. Every failure was silent. Drew-approved 2026-07-16.

(No OTHER new governance rules — per the Phase-8+ precedent, the phase's other lessons are techniques (→ cookbook §53) + findings (→ decision-log.md, calibration.md). The three self-inflicted defects reinforce existing rules rather than needing new ones: the registry-destroying yaml.safe_dump is H5 ("never silently drop comments on a rewrite") + the R34 corollary that a byte-gate is a null document oracle; the two DIFF mis-reports are R14 (verify counts against the bytes — I read head -6 and generalized); the match_one fake isolation is the docstring-vs-default gap R35 governs. The carve-law ("sweep a family with the tool its exemplar needed; a 0% from the wrong tool is not evidence") is cookbook §53, a direct application of R35/G3.)

PhaseEnd Changelog

v1.26.0 → v1.27.0 — Phase 28 complete (Gen2 phase 20; the instrument-repair phase). Chartered to measure "the swing number" and build a delta engine around it, the phase found in planning that the number's evidence was untrustworthy — the "structural families bank ≈0%" doctrine that had reshaped two phases of strategy rested on a broken-tool probe (B2's 0/8 was a missing jump-table carve, not a wall). Re-scoped on that finding, the phase became instrument-repair + honest measurement, and delivered: B2 LIVES (8/8 → 102/115, and the ≈0% doctrine has no surviving post-fix evidence); a doubly-hidden SC07 pool un-hid and wired (img_path derive-not-guess + the new dedup_extend → 6,174 h_exact members banked, 95.6%, groups 134→138 binaries); the resident resolved (21→14, 90.34%, + a classed dossier); and the R36 citizenship gate (make audit-binaries) that makes a newly-onboarded binary unable to hide from the tools again. The legacy h_seq swing number was measured and classified (~3% as-tooled, 274 DIFF / 37 PLUMBING) but honestly left ceiling-unknown — the 274 DIFFs are byte-PURE members whose remapped bodies don't reproduce, the same tooling-vs-wall ambiguity that resolved to tooling twice this phase, and the disambiguating probe is handed to P29 rather than rushed. Fleet 67.0% → 68.9% instr / 47.8% → 49.5% distinct at 140/140 byte-identical, 0 NON_MATCHING throughout; disc_code_sweep fixed (it was blind to compressed code, so it could not have found a hidden overlay — now it can). The phase's honesty ledger includes three self-inflicted defects, caught and fixed forward — a registry-destroying yaml.safe_dump (H5; invisible to every byte-gate because they measure bytes, not documents), two DIFF mis-reports (R14), and a match_one that faked the isolation its docstring promised (caught by an AI agent mid-wave). One rule (R36); techniques → cookbook §53; findings → decision-log/calibration. The decisive lesson, five times over: every "compiler wall" this phase probed was our own tooling — and the byte-gate, a perfect correctness oracle, is a null oracle for coverage (R34), for documents (H5), and for the iteration loops agents steer by.

Roadmap delta (roadmap §0 — what Phase 28 changed about the road ahead)

• B2 LIVES; the "structural families ≈0%" doctrine is RETIRED on no surviving evidence. The doctrine's one post-fix probe (B2) refuted it (0/8 was a missing carve → 102/115); its three Phase-26 corroborating probes all predate the _carry_macros fix that made such a probe honest. calibration.md's "decisive P28/P29 input" table is rewritten. P29's arithmetic is NOT settled — see the swing number.
• The swing number is ~3% AS-TOOLED, ceiling UNKNOWN (legacy h_seq; classified 274 DIFF / 37 PLUMBING). P29 MUST run the disambiguating probe first (reloc-position mismatch ⇒ fixable remap, ceiling far above 3%; regalloc-away-from-relocs ⇒ TU wall). Do not scale "(cores)×(reach)" on 3% until then.
• A doubly-hidden pool is now VISIBLE and WIRED: the SC07 harvest banked 6,174 h_exact members. The lesson generalizes — any future onboarding (the 39 type-1 modules) must be wired into every consumer, now enforced by R36 / make audit-binaries.
• member_adapt.py is VACATED for the h_exact case (dedup_extend is the right tool) and deferred for the h_seq case (pending the T3b disambiguation — if the residual is TU-context, a delta engine is the P29 tool; if it's incomplete remap, symbol_map is the fix).
• Binary count firm at 140 (main + resident + 138 overlays); the disc is byte-verified complete for type-4 (0 further hidden), 39 type-1 modules pending load-address RE. The completion contract's denominator is stable.
• Carried tools (none lost): the burn-down velocity tracker (T7c), diff_regions.py, the grinder family-warm-start, the same-binary shard farm. make audit-binaries is new and standing.

Plain-English Recap

This phase was supposed to measure one number — how often the "same function, slightly different per level" pattern can be rebuilt automatically — and build a tool around it. Instead, before writing a line, we found that the number the whole plan rested on was wrong, measured months ago with a broken tool: a "families can't be automated, ~0% success" verdict that had steered the project's strategy for two phases. Run correctly, that same family succeeded 89% of the time — the earlier failure was a missing build step, not a real wall. That set the tone for everything: almost every "the 1990s compiler beat us here" turned out to be our own tools lying.

Chasing it, we found four game levels that had been added last phase but never introduced to any of our tools — so about 6,400 functions already solved elsewhere were sitting unrecognized. One new command wired them in: 95.6% rebuilt, for essentially no cost. We drove the game's permanent engine chunk from 85% to 90% with a swarm of AI agents, and wrote an honest map of the 14 pieces left and exactly why each is hard. And we built a safety gate — the thing you asked for — that now refuses to let a newly-found level hide from our tools ever again; we proved it by deliberately breaking one level and watching the gate catch it.

The game went from 67% to 68.9% rebuilt bit-for-bit — but the real story is how: almost none of it came from beating the compiler, and a lot came from fixing our own instruments, including three mistakes that were mine — a tool that quietly destroyed a documentation file (caught and repaired), two miscounts I reported before checking, and a match-checker that faked the isolation it advertised (caught by one of the AI agents mid-run). On the headline question, we were deliberately honest: the leftover "harder" families rebuild at only ~3% with today's tools — but we did not call that a ceiling, because the way they fail is the exact same "is it the tool or the compiler?" ambiguity that kept turning out to be the tool this phase. So we wrote down precisely the one experiment that settles it, for next phase, rather than guess. Nothing broke; all 140 pieces of the game still rebuild perfectly. The lasting win isn't the percentage — it's that our instruments lie to us a lot less than they did, and there's now a gate that keeps it that way.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase28.md (R19, via git mv — left uncommitted for Drew's milestone-close commit). The Phase-28 work is committed across the session (76db32455..6996d2537); Drew commits AND pushes this PhaseEnd + the archived log + the two regenerated digests (docs/progress.fleet.md, .run/backlog.jsonl) (R6/R20 — the milestone-close is the human-controlled gate-2 sign-off). No Ghidra DB change this phase (R23 no-op — the db.*.gbf churn is restart-noise; do NOT stage). Gen2 continues — do NOT start Phase 29 here. Start a fresh session (effort Max, plan mode) for Phase 29 — The Family Campaign, opening with the T3b disambiguating probe before scaling any yield arithmetic. Keep this file forever.