BFM-decomp/phase-ends/PhaseEnd_Phase26.md

PhaseEnd — Phase 26: The Family-Endgame → the tooling-integrity audit + the §52 discovery-flywheel (mechanical harvest byte-proven exhausted)

Date: 2026-07-15 · Project Version: 1.25.0 · Phase Status: Complete (a PIVOT close — the mechanical/templating thesis reached its byte-gate ceiling; structural completion deferred to Phase 27; Phase-16 precedent) · Generation: Gen2 (18th phase of the arc; Phase 14 public-flip deferred to Gen3+)

Gen2 phase 18 (8→…→25→26). The longest phase, run across ~13 sessions (2026-07-11 → 07-15) and containing an inserted half-phase (26-A, the tooling-integrity audit) — the Phase-3.5 precedent. The granular per-task/per-session crash-recovery trail (Tasks 0–6, the A0–A11 audit ledger, the §52 flywheel waves) is preserved on-demand at phase-ends/logs/Phase26.md (R19 — NOT auto-loaded). This file is the synthesis. Owner decisions (Drew): open from the megaplan → insert the audit as a half-phase (gate-1 2026-07-14) → resume Task 7 at xHigh → single-Fable5 test → redirect to the §52 cheap-Opus flywheel → on the byte-proven exhaustion of the mechanical harvest, close the phase and open Phase 27 with fresh byte-gate-honest scans.

Build Log

Files created/changed and complete — do not recreate (the durable Phase-26 output; the archived log has the exhaustive per-commit list):

The mechanical family engine (Tasks 1–6, the phase's original substance):

• tools/family_remap.py — extended reloc_targets for the add/addu indexed-global idiom (the "reach-1 tail" was largely this tracker blind spot); backward-compatible cross-address to_addr + imm_map hook; single-pass simultaneous substitution (fixed the latent chained-rename bug); the shared word-diff classifier (stream_words/classify_member — PURE/IMM/STRUCT). norm_stream/h_norm UNTOUCHED.
• tools/family_hseq.py — full-frontier h_seq survey (the reframe engine) → .run/family_hseq.json + docs/family-hseq.md.
• tools/family_sweep.py — the two-phase stage-then-gate driver: hseq_sweep() (crack one exemplar → positionally remap per-overlay reloc/imm symbols → whole-binary byte-gate), --only, --band, --reconcile-raw (§41c per-sibling reconcile), the §42e static pin guard. This session: two fixes — the pin-guard now strips comments before the __asm__("$ check (a body that DOCUMENTS a removed pin is pin-free code, was false-skipping 133 members); and --allow-pins (bypass the guard, let the byte-gate arbitrate — added to TEST the func_8017A4AC pinned-×134 precedent).
• tools/family_manifest.py, tools/canon_sig_reconcile.py v3.2, tools/rtu_match.py, tools/t7_bank.py, tools/bank_exemplar.py — the curriculum + recovery + real-TU-faithful match tooling (Tasks 4–6).

The tooling-integrity audit (26-A — the inserted half-phase, A0–A11; the phase's most consequential work):

• tools/corpus.py — ONE derived corpus oracle (derives the binary's .c set + each function's .s home from the FILESYSTEM and matched = sig − stubs from the PROVEN INVARIANT, never re-parsed). Coverage-asserted (R32). Deleted ~4 decaying scanners.
• tools/cdecl.py — the ONE coverage-asserting C-declaration parser (recursive descent over C's total declarator grammar; ~250 lines replacing 15 divergent regexes). make audit-cdecl gate. gcc adjudicates its own coverage gap.
• make audit-corpus — the SECOND ORACLE (R34): cross-checks splat's boundaries against sig_image's independent ones; reproduced the A2 audit's 193-slice number from an independently-written tool.
• config/symbols.us.ram.txt — the listCdBuffer fix (A4): a symbol whose address falls inside another binary's vram window must never enter that binary's symbol stack (193 unmatchable slices → 0).
• tools/reconcile_tu.py (wired into the bank paths, retiring the fleet-majority oracle), tools/masked_diff.py (PC16 fix: 150 closeness lies → 4), build_engine_types (unblocked from 81% hard-exit), jr_isolate_all/overlay_src_split/jr_inventory (silent byte-changer + coverage fixes), lint_symbol_refs (green + wired), extract invalidates stale objects (a stale .o can produce a FALSE PASS). DELETED: census_conflict_callees, derive_canonical_sigs (R33). Full ledger: docs/tooling-audit.md (AUDIT-CLOSE LEDGER).

The §52 discovery-flywheel (Task 7, this session — the endgame lever):

• src/ov_SC01_077/** + the 134 overlay .c files + config/dedup.us.yaml — the banked matches. Task-7 this session banked 5 pin-free regalloc-order cores ×134 = 670 function-instances (func_80171FFC, func_801775E0, func_801379FC, func_801497A8, func_801495C4).
• docs/matching-cookbook.md §40–§52 + §52a + §52b — the mature family + gcc-2.7.2 regalloc toolkit. §52 (the walker-family skeleton, distilled from the Fable5 func_80178004 wall) + §52a/§52b (the sibling-wave de-pin levers + FOUR named intrinsic-wall classes: flagship $s0 local-alloc, caller-saved priority-first-fit, non-coalescing delay-slot copy, symbol-address-base wins-low-needs-high).
• docs/decision-log.md (R31 — the audit thesis, the single-Fable5 wall, the §52 flywheel, the mechanical-harvest-exhausted pivot), docs/tooling-audit.md, docs/family-endgame-megaplan.md.
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase26.md (R19). phase-ends/PhaseEnd_Phase26.md — this file.

Tools/packages installed: None — the whole phase used the existing Phase-4/6/…/25 toolchain + venv. Fable5/Opus agents via the Agent tool.

Verification results (literal) — the milestone proof:

• make clean && make extract-all && make check-all → 136 passed, 0 failed of 136 BYTE-IDENTICAL from a genuinely-clean tree (R22), reproduced after every banked batch this phase. The whole-binary byte-gate (harvest_verify, G3/P9) was the sole per-bank arbiter and never accepted a wrong match.
• Fleet: instr-weighted 58.2% → 68.9% (the phase arc: family engine + sessions 6–8 cracks 58.2→66.5; the 26-A audit fixes 66.5→68.6; the §52 flywheel this session 68.6→68.9) · distinct-code 30.3% → 49.2% (53,429 / 84,996 unique fns) · fn-count → 84.47%. 0 NON_MATCHING in any default build (G4); dedup 1840 validated / 0 failed; 53,371 INCLUDE_ASM stubs.
• make report / make audit-corpus (0 slices) / make audit-cdecl all green.
• The byte-proven pivot finding (R14/P9): the matched-sib mechanical harvest is exhausted — 3 byte-gate probes returned 0% (tiny-IMM 0/241, PURE reach-134 0/134, pinned-PURE-with-pins 0/133). The manifest's ~13,075 "templatable" member-slots are an h_seq prediction the whole-binary gate refuses (collision / register-drift / pin-crash). Phase 26's "crack ~986 exemplars → template ×120" thesis is spent at this ceiling.
• git status: only config//tools//src//docs//phase-ends/ tracked across the phase; zero ROM-derived/generated bulk staged; the db.*.gbf churn is R23 restart-noise (never staged); no Ghidra DB change (matching used cached asm).

Milestone (honest — a PIVOT, not the stated milestone): Phase 26's stated milestone was structural completion, and it is NOT reached (68.9% instr / 49.2% distinct). What the phase DID deliver: (1) the mechanical family engine + the h_seq reframe; (2) the tooling-integrity audit that found the tools were several of the walls, fixed ~15 of them, deleted decaying scanners, and built two derived oracles + a second disagreeing oracle (R32/R33/R34); (3) the §52 discovery-flywheel — a single Fable5 pass that failed to crack the flagship func_80178004 nonetheless distilled the walker-family idiom that cheap-Opus waves then applied to bank 670 instances and precisely characterize four intrinsic-wall classes; and (4) the byte-proven finding that the mechanical/templating harvest is exhausted at this ceiling. The structural remainder (hard-wall cracks + hand-decomp of the genuine unique residue) is a distinct phase, opened fresh in Phase 27 with byte-gate-honest re-scans. A Phase-16-style honest, finding-driven close.

Next: Phase 27 — the hard-remainder endgame, re-scanned honest. A fresh session, plan mode, Max (Tier-1). Open by re-scanning the frontier with the now-bugfree tools AND validating templatability against the byte-gate (not h_seq prediction), so the plan is built on reality, not a manifest that over-promises ~13k phantom members. Then scope the genuine remainder: the harder crack tail (Fable5 discovery → §52-flywheel application, diminishing), the documented giant walls (permanent INCLUDE_ASM per G4 unless a new idiom dissolves them), and hand-decomp of the true unique residue.

Deviations

Item	Plan	Actual	Reason
Phase shape	the megaplan: crack ~986 h_seq exemplars → template ×120	family engine built + a tooling audit inserted (26-A) + a §52 discovery-flywheel + the templating thesis byte-proven exhausted	the megaplan's scans were run on BUGGY pre-audit tools; the audit found the tools were the walls; the byte-gate then refuted the templatability the manifest predicted
26-A audit	(not in the original plan)	inserted as a half-phase (Drew, Phase-3.5 precedent)	seven silent-skip tool bugs in one session revealed a structural blind spot; a hole in a SELECTION tool makes work invisible to planning — fix before matching on top of it
Task 7 model tier	Fable5 per giant (window ~7/13)	window extended to 7/19; single-Fable5 test (xHigh) → wall → redirect to cheap-Opus §52 flywheel	the flagship is an intrinsic 3-integer regalloc wall (Fable5 at 477k tok couldn't crack it); but its idiom made cheap-Opus waves productive (Fable5 DISCOVERS, cheap-Opus APPLIES)
Task 8 (mechanical harvest)	the big deterministic lever (~2.6M templatable ins)	byte-proven TAPPED (3 probes, 0%)	A3h + the wave propagations banked everything cleanly templatable; the remaining manifest "templatable" is collision/drift the gate refuses (R14: verify predictions vs the bytes)
Tasks 9/11	mid-band harvest + residue map	folded into the Phase-27 fresh-scan re-plan	the mechanical bands are the same tapped thesis; an honest residue map needs the byte-gate-validated re-scan, so it belongs at the Phase-27 open
Milestone	structural completion	NOT reached (68.9%); honest PIVOT close	the mechanical thesis reached its byte-gate ceiling; structural completion is the distinct hard phase (Phase-16 precedent — a finding is a valid phase deliverable, P9)
Effort	Max throughout	Max (planning/audit/synthesis) · xHigh (the §52 flywheel orchestration + waves) · Fable5 (the single wall-breaker test) · Ultracode (the A2 audit fan-out)	R26/R27 — breadth under Ultracode, discovery under Fable5, deep synthesis at Max, routine orchestration at xHigh

Commit Message

(Phase 26 landed across ~13 sessions of checkpoint commits; the 26-A audit close = a33c6f85f;
this session's §52 flywheel = 8f3e4a12b..d40711fe5; this close = the archived log + this PhaseEnd.)

feat(phase-26): close — family engine + the tooling-integrity audit + the §52 discovery-flywheel;
mechanical harvest byte-proven exhausted; fleet 58.2->68.9% instr (v1.25.0)

- FAMILY ENGINE (Tasks 1-6): family_remap (extended reloc tracker + single-pass subst) +
  family_hseq (the h_seq reframe) + family_sweep (crack-one -> template-x134 -> byte-gate) +
  canon_sig_reconcile v3.2 + rtu_match. Sessions 6-8 cracked 13 cores (58.2->66.5% instr).
- 26-A TOOLING-INTEGRITY AUDIT (inserted half-phase, A0-A11): the tools WERE several of the
  walls. Fixed ~15; DELETED decaying scanners (R33); built corpus.py + cdecl.py (derived,
  coverage-asserted oracles) + make audit-corpus (a SECOND, disagreeing oracle, R34); the
  listCdBuffer 193-slice corpus defect -> 0; masked_diff 150 closeness-lies -> 4; the stale-
  object false-pass closed. Payoff 66.5->68.6% instr. docs/tooling-audit.md AUDIT-CLOSE LEDGER.
- 52 DISCOVERY-FLYWHEEL (Task 7): single Fable5 on func_80178004 = intrinsic 3-integer regalloc
  wall, BUT distilled the walker-family idiom (52); two cheap-Opus waves applied it -> 5 pin-free
  cores banked x134 = 670 instances (68.6->68.9%). 4 named wall classes; 52/52a/52b. pin-guard
  comment false-positive fixed; family_sweep --allow-pins.
- FINDING (R14/P9, 3 probes 0%): the matched-sib mechanical harvest is EXHAUSTED; the manifest's
  ~13k "templatable" members are an h_seq prediction the byte-gate refuses. Phase-26's templating
  thesis is spent -> close, open Phase 27 with byte-gate-honest re-scans.
- R22 clean-fleet 136/136 BYTE-IDENTICAL throughout; dedup 1840/0; 0 NON_MATCHING (G4).
- rules R32 (coverage assertion) / R33 (derive, don't re-derive) / R34 (a second, disagreeing
  oracle). worklog -> phase-ends/logs/Phase26.md (R19). bumps 1.24.0 -> 1.25.0.

Rules Added This Phase

Rule	Reason
R32 — Assert your COVERAGE. A tool that scans the corpus must compare what it found against an over-approximating candidate set and fail on the gap. A silent skip is a DEFECT, not a no-op — and note (the corrected form): a loud failure that nobody counts is exactly as invisible as a silent one (build_engine_types hard-exited on 81% of its corpus for four phases while "failing loud").	The audit's root cause: a scanner extracts N items, the true count is M>N, and nobody ever compared N to M. Seven silent-skip bugs in one session were one structural blind spot. (Drew: "agreed", 2026-07-14.)
R33 — Derive, don't re-derive. Where a proven invariant answers the question, derive the answer from it rather than re-parse the source. The best outcome is a DELETED SCANNER, not a fixed regex. Before adding a coverage assertion to a scanner, first ask whether the scanner should exist at all.	28 audit findings collapsed to one derived oracle (corpus.py) + ~10 deleted/retired scanners. The build invariant (INCLUDE_ASM pastes the ORIGINAL asm ⇒ a function not wrapped in it is byte-exact) makes most re-parsing scanners unnecessary.
R34 — A second, DISAGREEING oracle — not a better assertion. When an oracle is structurally blind to a class of error, no assertion inside it can help; add an INDEPENDENT oracle that can disagree with it, and make them argue.	The whole-binary byte-gate is a perfect CORRECTNESS oracle and a NULL COVERAGE oracle (green since Phase 5 at 0% decomp, because INCLUDE_ASM pastes the original). sig_image disagreeing with splat is the only reason the 193 listCdBuffer slices were ever visible — we had both oracles all along and never compared them.

(No OTHER new governance rules — per the Phase-8+ precedent, the phase's other lessons are techniques (→ cookbook §40–§52b) + findings (→ decision-log + the megaplan). The §52 discovery-flywheel ("a frontier model that FAILS to crack a wall still distills the idiom that lets a cheap tier crack its non-wall siblings; Fable5 DISCOVERS, cheap-Opus APPLIES") is captured as cookbook §52 + the matching-is-solved-integration-is-the-bottleneck / structural-family-mechanical-remap memories. The "a manifest's h_seq prediction is not a bank — validate templatability against the byte-gate" lesson is a direct application of R14/G3. G3/P9/R14/R16/R22/R26/R27/R30/R31 governed the phase.)

PhaseEnd Changelog

v1.24.0 → v1.25.0 — Phase 26 complete (Gen2 phase 18; the longest phase; a PIVOT close). Opened from the family-endgame megaplan ("crack ~986 h_seq exemplars → template ×120"), the phase built the mechanical family engine (family_remap/family_hseq/family_sweep, crack-one-then-template-×134-under-the-byte-gate) and, mid-arc, inserted the tooling-integrity audit (26-A) — the phase's most consequential work: the tools were several of the walls the project had "byte-proven" across 26 phases. The audit fixed ~15 tools, deleted decaying scanners (R33), built derived, coverage-asserted oracles (corpus.py, cdecl.py) and a second, disagreeing oracle (make audit-corpus, R34), and dissolved corpus defects (the 193 listCdBuffer slices; 150 masked_diff closeness-lies; the stale-object false-pass), banking 66.5→68.6% instr. Task 7 then delivered the §52 discovery-flywheel: a single Fable5 pass failed to crack the flagship func_80178004 (a genuine 3-integer regalloc wall) but distilled the walker-family idiom that two cheap-Opus waves applied to bank 5 pin-free cores ×134 = 670 instances (68.6→68.9%) and precisely characterize four intrinsic-wall classes — the flywheel principle Fable5 DISCOVERS, cheap-Opus APPLIES, proven end-to-end. The phase then byte-proved (3 probes, 0%) that the matched-sib mechanical harvest is exhausted — the manifest's ~13k "templatable" members are an h_seq prediction the whole-binary gate refuses. So Phase 26's templating thesis is spent at 68.9% instr / 49.2% distinct (from 58.2%/30.3% at Phase-25 close — a large arc), the structural completion milestone is honestly not reached, and the phase closes on that finding, handing the hard remainder (wall-cracks + hand-decomp) to a Phase 27 that re-scans the frontier honest (byte-gate-validated, not h_seq-predicted). Three governance rules (R32/R33/R34 — the audit's real deliverable, the derived-oracle discipline that makes the next 26 phases unable to manufacture a wall out of a lookup miss). 136/136 byte-identical, 0 NON_MATCHING, dedup 1840/0 throughout.

Plain-English Recap

This was the longest stretch of the project, and it ended by changing what we believe about the finish line. Three things happened. First, we built a "stamp machine" — reconstruct one copy of an engine function, and the tool stamps the matching copy into all ~130 game locations, each checked bit-for-bit. Second — the big one — we audited our own tools and discovered that many of the "unbeatable compiler walls" we'd been recording for months were actually bugs in our own scanners: tools that silently skipped work and made it look like there was nothing there. We fixed them, deleted the worst ones, and built new tools that derive the truth from the build instead of guessing, plus a second independent tool whose whole job is to disagree and catch the first one's blind spots. That alone recovered a couple percent of the game. Third, we hit a real wall on the single hardest function even with the best AI in the world — but that failed attempt taught us the compiler's exact behavior, which cheaper AI then used to reconstruct 670 more pieces and to explain precisely which remaining ones are genuinely impossible. Finally, we proved — by actually testing it — that the "stamp everything" approach has now given all it can (the leftover functions aren't clean copies of each other; they genuinely differ). So we're closing this phase honestly: we went from ~58% to ~69% of the game rebuilt bit-for-bit, we made our tools trustworthy, and we learned that the rest of the work is a different kind of work — hard, one-at-a-time reconstruction — which deserves a fresh start with clean tools and an honest re-scan of exactly what's left. The most important lasting win isn't the percentage; it's that our instruments no longer lie to us, and we wrote down three rules to keep it that way. Everything still rebuilds perfectly; nothing broke.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase26.md (R19, via git mv — left uncommitted for Drew's milestone-close commit). The Phase-26 matching + audit work is committed across the phase (26-A close a33c6f85f; this session 8f3e4a12b..d40711fe5); Drew commits AND pushes this PhaseEnd + the archived log (R6/R8 — the milestone-close is the human-controlled gate-2 sign-off). No Ghidra DB change this phase (R23 no-op — the db.*.gbf churn is restart-noise; do NOT stage). Gen2 continues — do NOT start Phase 27 here. Start a fresh session (effort Max, plan mode) for Phase 27 — the hard-remainder endgame, opening with a byte-gate-honest re-scan of the frontier (the now-bugfree tools + templatability validated against the gate, not h_seq prediction), then scope the wall-cracks + hand-decomp residue. Keep this file forever.