BFM-decomp/phase-ends/PhaseEnd_Phase20.md

PhaseEnd — Phase 20: Close the propagation cap, crack the gcc residual classes, then scale the waves

Date: 2026-06-21 · Project Version: 1.19.0 · Phase Status: Complete (open-ended phase, closed at a clean checkpoint — Drew, gate 2) · Generation: Gen2 (12th phase)

Gen2 phase 12 of the arc (8→9→10→11→12→13→15→16→17→18→19→20; 14 deferred to Gen3+). Ran across multiple sessions; the granular per-task/per-session trail (the T1 type-lift, the T6 wave, the cap-tool diagnosis, the -O0/data-cast byte-proofs, the T3a crack) is preserved on-demand at phase-ends/logs/Phase20.md (R19 — NOT auto-loaded; consult only when researching a mechanism). This file is the synthesis. Owner decisions (Drew): class-crack first + skip the permuter grind (this phase); then exhaust the cheap-recovery levers in order (cap-tool → -O0 rollout → data-cast → T3a); close Phase 20 at a clean checkpoint and do the automation architecture as a fresh-session, plan-mode Phase 21 (the approved design is the Phase-21 Backlog below).

Build Log

Files created/changed and complete — do not recreate:

New reusable tooling (the phase's durable output):

• tools/cast_call_sites.py — (THE cap tool, cookbook §20) the §17a-1 per-site function-pointer cast recovery for the loose-typing CALLEE-conflict class. Pure --in/--out draft transform (sibling of sig_unify / canon_resident_calls): per draft, for each callee whose canonical TU sig differs from the draft's intended sig, rewrite the decl line → canonical (kills the in-TU conflicting types, keeps the symbol in scope) AND cast each call site → the draft's intended sig ((ret(*)(args))func_X)(args) (decl lines never cast; gcc folds the cast of a known symbol → direct jal). Whole-binary gate is the sole arbiter (G3/P9). Pipeline: canon_resident_calls → cast_call_sites → sig_unify → harvest_verify --chunk 1.
• tools/build_engine_types.py — (T1, extended) added find_typedefs() (brace-aware anon-struct / fn-ptr / alias typedefs) + same-name-different-layout collision + tagged-struct-typedef overlap guards; emits typedefs after named structs in dep order. Closed the §19 type-blocked propagation cap.
• tools/exemplar_miner.py — (T2) residual router: wall_taxonomy.json + per-overlay reach → docs/exemplar_curriculum.md + .run/exemplar_routing.json (WAVE/STRUCT/PINS/STUB pools, reach-ranked).

Config / source (the matches):

• src/ov_SC01_077/ov_SC01_077.c + src/shared/engine_core.h + config/dedup.us.yaml + the 134 overlay .c files — T1's 9 typedef-freed fns ×134, T6 batch-1's 8 (7 ×134 + 1 local), the cap tool's 6 (5 ×134 + 1 local), all byte-gated + propagated. config/dedup.us.yaml 1485 → 1513 validated cross-binary groups (0 failed).
• src/ov_SC01_077/ov_SC01_077_o0.c — (T3a) func_8013B7AC matched via the array-of-struct %lo-fold idiom (×1 — the -O0 cluster is overlay-local).

Docs / knowledge base (R16 flywheel, R21):

• docs/matching-cookbook.md — §20 (NEW: the wave-at-scale GATE CAP = loose-typing call-graph wall; the cap tool RESOLUTION + the R14 re-diagnosis = DEF-side wall dominates, not callee-conflicts; the data-cast MOOT finding; the confirmed-irreducible classes store-vs-load / hoist-vs-remat / IV-combine / hoisted-invariant / the -O1 class; the stale-.o + args-as-array + --chunk 1 gotchas) + §18 (T3a %lo-fold CRACKED via array-of-struct + the -O0-cluster-is-overlay-local R14 correction) + §17a (the gate-cap framing).
• docs/SETUP.md — cast_call_sites.py tooling-inventory row.
• README.md / docs/{progress,difficulty,duplicates}.*.md — Phase-20 status + regenerated digests.
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase20.md (R19 archive). phase-ends/PhaseEnd_Phase20.md — this file.

Local artifacts (gitignored / regenerable): .run/drafts-t6-* (wave + cast + sig_unify recovery stages), .run/drafts-t3a/*, .run/ghidra_c/* (300 cached Ghidra-C), .run/exemplar_routing.json, .run/harvest_*.txt, build/, expected/, asm/, assets/. The 134 overlay Ghidra programs are script-reproducible (DB-commit deferred as regenerable).

Tools/packages installed: None — the whole phase used the existing Phase-4/6/10/11/12/13 toolchain + venv.

Verification results (literal) — the milestone proof:

• make check-all → 136 passed, 0 failed of 136 (main 143dbb89…, resident 8e17e02f…, ov_SC01_077 d19c9580…, all 134 overlays), reproduced from clean after every banked step (R22). A stale-artifact false failure (ov_SC01_005, from the reverted -O0 test) was caught by check-all and cleared by a clean re-extract — see Deviations.
• make report: FLEET byte-identical 202,350 / 344,010 = 58.82% (phase start 58.00% / 199,541 → +0.82%); LINKED 959; dedup 1485 → 1513 validated, 0 failed; 0 NON_MATCHING in any default build (G4).
• Phase arc (banked): T1 type-lift +0.35% (9 fns ×134) · T6 batch-1 +0.28% (8 matches) · the cap tool +0.19% (6 matches) · T3a func_8013B7AC (×1). The cracks/findings (below) are the durable non-% value.
• git status: only config//tools//src//docs//phase-ends/ tracked; zero ROM-derived/generated bulk staged. No Ghidra DB change this phase (matching used cached Ghidra-C + asm; the db.*.gbf churn is SessionStart-restart noise — R23 no-op, NOT staged).

Milestone achieved (confirmed by Drew, gate 2): the open-ended phase is closed at a clean checkpoint — fleet rose 58.00% → 58.82%, 136/136 binaries byte-identical, 0 NON_MATCHING, with three reusable tools (the cap tool, the typedef-lift, the residual router), the gcc-2.7.2 codegen map distilled (cookbook §20), the T3a %lo-fold class CRACKED (array-of-struct idiom — a long-"irreducible" residual, now a reusable cookbook idiom), and three byte-proven strategic findings (R14) that retire two dead levers and sharpen the roadmap: the cap is the DEF-side loose-typing wall (not callee-conflicts), the -O0 cluster is overlay-local (the ×134 rollout is invalid), and data-cast is moot (0 real data conflicts).

Next: Phase 21 — the automation architecture (the "automation manager" / supervised-autonomous grind). A fresh session, plan mode (Tier-1). The full approved design + the 6 build-pieces are the Phase-21 Backlog below — plan from it.

Deviations

Item	Plan	Actual	Reason
The cap "~33" recovery	the §20 cast tool recovers the batch's lost ~33	6 recovered	R14 re-diagnosis: the 33 are NOT mostly callee-conflicts (the cast tool's class); ~18 are the DEF-side loose-typing wall (byte-proven both ways), + a DATA tail. §20 corrected.
-O0 ×134 rollout	bank the 6 matched -O0 fns ×134 (~+0.6% "free")	INVALID — skipped	R14 byte-proof: only 1/134 overlays shares the cluster bytes; the fns reference per-overlay data → overlay-local. Built+reverted the split infra; cookbook §18 corrected.
data-cast extension	complete the cap tool with *(T*)&D_x	MOOT — not built	R14 byte-proof: 0 drafts have a real data-type conflict; the lone case was a typedef-redef masking the def-sig wall.
T3a %lo-folding	likely cite-irreducible (§18 wall)	CRACKED	the array-of-struct idiom folds %lo (the failing forms materialize &sym); a better-than-expected result.
T3b/T3c	crack-or-cite	cited (T3b R14-reframed strength-reduction; T3c store-vs-load confirmed unsteerable, sched.c)	genuine compiler residuals; byte-evidenced verdicts in §20.
Phase close	grind T5 waves + giants	closed at a clean checkpoint; automation → Phase 21	Drew: the cached ≤90-ins tail is byte-exhausted; the autonomy is a Tier-1 architecture deserving a fresh plan-mode phase.
ov_SC01_005 check-all	(n/a)	stale-artifact false-fail caught + fixed	the reverted -O0 test left a stale .ld/asm (3-way split); make check doesn't re-extract → linked a deleted .o. Clean re-extract restored 56760dbe. → the R22 corollary (a reverted config needs a re-extract).
Effort	Max	Max throughout (orchestration + the deep diagnosis/synthesis); no Ultracode needed (Workflow sets per-agent effort)	the cap diagnosis + T3a crack + the automation design are deep single-thread work.

Commit Message

(Phase 20 landed across multi-session commits; final session 6c5bd1c4a..f1a7e518f + this PhaseEnd.)

feat(phase-20): close — cap tool + T3a %lo crack + 3 byte-proven findings; fleet 58.00%->58.82% (v1.19.0)

- THE CAP TOOL tools/cast_call_sites.py (§17a-1 callee-cast): per draft, callee whose canonical
  TU sig != draft's intended sig -> decl->canonical + call-site cast ((ret(*)(args))func_X)(args)
  (codegen-neutral; gcc folds the cast of a known symbol -> direct jal). Recovered 6 of T6 batch-1's
  33 gate-fails byte-identical (5 x134 + 1 local). Pipeline canon_resident_calls -> cast_call_sites
  -> sig_unify -> harvest_verify --chunk 1.
- T3a CRACKED (not cited): the %lo-folding indexed-global -O0 residual falls to the array-of-STRUCT
  idiom (extern Struct base[], sizeof==stride, base[i].field; NOT *(T*)(&sym+i*stride) which
  materializes &sym). func_8013B7AC banked. Reusable idiom fleet-wide. cookbook §18/§20.
- THREE byte-proven findings (R14): (1) the wave cap is the DEF-side loose-typing wall, not
  callee-conflicts (the §20 "~33" projection was wrong); (2) the -O0 cluster is OVERLAY-LOCAL,
  not reach-134 -> the ×134 rollout is INVALID; (3) data-cast is MOOT (0 real data conflicts).
- T1 typedef type-lift (+0.35%, 9 fns x134); T2 exemplar_miner router; T4 codegen map (cookbook §20);
  T6 batch-1 (+0.28%, 8). dedup 1485->1513/0 failed. check-all 136/136 BYTE-IDENTICAL (R22);
  0 NON_MATCHING (G4). fleet 58.00% -> 58.82%.
- rule R30 (capture context-dependent knowledge before a fresh session). worklog -> logs/Phase20.md.
- bumps 1.18.0 -> 1.19.0

Rules Added This Phase

Rule

Reason

R30 — Capture context-dependent knowledge artifacts before a fresh session. Cookbook entries, byte-verified findings, codegen-map distillations, and doc corrections whose quality depends on the current session's full live context MUST be written DURING that session, before any fresh-session handoff (the fresh session inherits only compressed summaries and loses the detail / nuance / byte-evidence). Defer only mechanical / continuable work (build the next tool, run the next wave). Extends R16 (the flywheel — adds the timing constraint). Memory capture-knowledge-before-fresh-session.

Candidate since the multi-session handoff (Drew, 2026-06-21 — prompted by nearly deferring T4's §20 distillation to a fresh session that would have lost the byte-evidence). Exercised all session: §20 RESOLUTION, the §18 T3a crack, and the three R14 findings were all written during the producing session.

(Not elevated to rules — captured as cookbook techniques per the Phase-8+ precedent: the call-site-cast recovery (§20), the array-of-struct %lo-fold idiom (§18), the DEF-side-wall / overlay-local / data-cast-moot findings (§18/§20). The R22 corollary — a reverted config needs a make extract, not just make check, or it leaves a stale .ld/asm while git status looks clean — is a §13/§18 build-infra note, governed by the existing R22.)

PhaseEnd Changelog

v1.18.0 → v1.19.0 — Phase 20 complete (Gen2 phase 12; open-ended, closed at a clean checkpoint). The fleet rose 58.00% → 58.82% byte-identical-from-source (136/136 binaries byte-identical, 0 NON_MATCHING throughout), banked by the typedef type-lift (T1, +0.35%, closing the §19 propagation cap), an enriched wave (T6 batch-1, +0.28%), and the phase's headline tool — tools/cast_call_sites.py, the §17a-1 per-site function-pointer cast that recovers the loose-typing callee-conflict class (+0.19%, codegen-neutral, byte-gated). The T3a %lo-folding -O0 residual was CRACKED (not the expected cite-irreducible): the array-of-struct idiom (extern Struct base[], base[i].field) folds %lo where *(T*)(&sym+i*stride) materializes — a reusable fleet-wide idiom. The phase's most valuable output is three byte-proven (R14) strategic findings that retire two dead levers and sharpen the roadmap: the wave cap is the DEF-side loose-typing wall (a banked caller declares the draft's own function incompatibly; caller-side fix blocked because INCLUDE_ASM declares no symbol), not callee-conflicts as §20 first projected; the -O0 cluster is overlay-local (per-overlay data addresses), so the ×134 rollout is invalid (the "reach-134" label conflated function-present with byte-identical); and the data-cast is moot (0 real data conflicts). The gcc-2.7.2 codegen map is distilled (cookbook §20). Net read: the reach-134 ≤90-ins cached tail is byte-exhausted of cheap recoveries; the remaining yield is fresh targets, the giants, and the hard walls. One rule (R30, capture-knowledge-before-fresh-session). The phase closes deliberately so the automation architecture (the supervised-autonomous grind) gets its own fresh, plan-mode Phase 21 — designed and Drew-approved, captured as the Backlog below.

Plain-English Recap

This phase squeezed the remaining easy wins out of the game's shared engine code and, just as importantly, proved which "easy wins" were actually mirages. We built a tool that auto-fixes a common reason a correct function won't slot in (a function-declaration mismatch) — it recovered six functions cleanly. We then chased two more ideas and the automatic bit-for-bit referee proved both were dead ends (functions we thought were shared across all 134 levels turned out unique to each; "data conflicts" weren't real) — so we proved them dead, wrote down exactly why, and faked nothing. The happy surprise: a residual we'd long called "impossible" actually cracked with a clean C trick (treat the global as an array of structs), now a reusable recipe. We finished at ~58.8% of the game rebuilt bit-for-bit, with everything still byte-perfect. The big takeaway is honest: the leftover functions in this size-band are genuinely the hard wall now, not low-hanging fruit — so the next phase is a different shape entirely. We designed (and you approved) an automation manager setup for your 5 days away: I orchestrate; swarms of AI agents do the matching in isolated workflows (so my own memory stays clean enough to run for days); a background brute-forcer grinds the mechanical cases; and — the key deliverable — everything that gets close but doesn't finish is logged as a ranked backlog for you to hand-finish when you're back. That automation is its own fresh phase, planned from the design captured below.

Phase-21 Backlog — "The automation manager" (the approved design; nothing here is lost on close)

Phase 21 opens in plan mode (Tier-1 architecture). Drew-approved design: supervised-autonomous, verify-giants-first, build-now. I am the orchestrator (Max); workflows are the workers (xHigh via opts.effort — no global Ultracode needed). The goal: spend ~5 idle days grinding as much as possible and storing results — banking byte-matches AND logging every near-miss (closest-achieved) as a ranked backlog for Drew's hand-sessions. The byte-gate (G3/P9) prevents false matches; the ROI-gate prevents wall-grinding.

The 6 build-pieces (prioritised):

1. [FIRST, no MCP] Verify giants shared-ness + build the target-pool manifest. Byte-verify the 28 giants (reach-134 fns >150 ins) are genuinely shared ×134 (h_exact across overlays — the -O0 "reach-134" was a mislabel, so don't trust it). Assemble the fuel manifest: fresh tractable reach-134 ov_SC01_077 stubs + the SHARED giants + the 9 -O0 stubs (idiom-known), each tagged reach + class + size.
2. [needs /mcp once, up front] Fuel prefetch. Stop MCP (R23) → DecompileFunctions.java over the manifest → cache Ghidra-C to .run/ghidra_c/ (cache is 300; ov_SC01_077 has 668). CRITICAL: pre-load ALL fuel so the unattended run never needs live MCP (the run is cache-based — no /mcp reconnect mid-run).
3. Worker workflow + deterministic gate/bank/propagate stage. A Workflow: fan-out drafter agents (xHigh) over the pool using the full toolkit (§17 pins, §18/T3a array-of-struct, §19 recovery, §20 cast tool) → each returns {matched | near-miss + best-draft + diff + class | failed}. Deterministic stage: canon_resident_calls → cast_call_sites → sig_unify → harvest_verify --chunk 1 → dedup_propagate. Returns ONE compact summary (keeps the orchestrator's context lean). Validate on a small sample before scaling.
4. The backlog ledger (docs/backlog.md + .run/backlog.jsonl, committed) — the deliverable: every near-miss logged with closest-achieved draft, diff score, residual class, reach, and where it stuck — ranked for Drew's hand-sessions. Honest (P9): only byte-matches bank.
5. Grinder workflow — re-cast the Phase-16 auto_driver as a Workflow: decomp-permuter (CPU-bound, local, free) on the regalloc/schedule-class near-misses, banking through the same gate, logging near-misses. Runs alongside the worker waves (CPU budget vs token budget).
6. [LAST] ROI orchestrator loop + auto_supervisor keep-alive. The between-wave loop (Max): bank → route → assess close-rate; when a pool's close-rate drops below threshold for K rounds, switch pools (≤90-ins → giants → -O0) or pause. Deploy the Phase-16 auto_supervisor.sh (detect stalled/exited session → resume via continue; auto_stop.sh sentinel = Drew's kill switch). Sample-validate the full loop (a few hours) BEFORE the unattended stretch, then launch.

Carried context for the plan:

• The reusable wave recipe (proven Phase 20): gen_harvest_targets --min-reach 134 --max-nins 90 → filter to cached Ghidra-C → Workflow (one agent/target, args as a JSON array NOT a string — the gotcha) → canon_resident_calls → cast_call_sites → sig_unify → harvest_verify --chunk 1 → dedup_propagate --auto-from.
• Honest fuel/ROI caveat: the cached ≤90-ins tail is byte-exhausted; fresh targets are finite (~+a few %); the giants are high byte-weight but low auto-yield (Phase-16 ~3% wall) → they're primarily backlog fuel (workers get them close, log near-misses, Drew hand-finishes). 5 days of runtime ≠ 5 days of wins — the value is the harvested tractable % plus the ranked near-miss backlog.
• The 7 callee/data-plumbing capped fns (reach-134, matched-but-local): 0x80142B2C 0x801535F4 0x80155E30 0x80157580 0x801576A8 0x80168F40 0x80170B48 — fail compiles_standalone on undeclared callees/data; a macro-extern-injection (or canonical-callee-sig embed) frees them ×134 (~+0.3%). Worker-workflow fodder.
• The -O1 class (func_80161A90) — needs an -O1 split file (the §18 -O0 precedent); build-infra, low priority.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase20.md (R19). Drew commits AND pushes this PhaseEnd + the archived log + the final-session matching commits (6c5bd1c4a..f1a7e518f) (R6/R8 — Drew's commit is the gate-2 sign-off). No Ghidra DB change this phase (R23 no-op — the db.*.gbf churn is restart-noise, do NOT stage it). Gen2 continues — do NOT start Phase 21 here. Start a fresh session (effort Max, plan mode) for Phase 21 — the automation manager (the Backlog above; verify giants → prefetch fuel → worker/grinder/feeder workflows + backlog ledger → ROI loop + supervisor). Keep this file forever.