From ceecaeb2359550c3f2782aa37213e8ba2dcd46a5 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Sat, 20 Jun 2026 11:54:07 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-18):=20T5=20=E2=80=94=20cookbook=20?= =?UTF-8?q?=C2=A717=20(steerable-vs-not)=20+=20reconcile=20=C2=A710-vs-?= =?UTF-8?q?=C2=A716?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - cookbook §17 (NEW): the compiler-quirk wall, gcc-2.7.2-source + Xenogears confirmed: * UNSTEERABLE: call-crossing $s0/$s1 allocation ORDER (global.c allocno_compare density; exhaustive C/flag/cc1 sweep + Xenogears ships-as-asm corroboration) -> stub it (G4) * STEERABLE idioms: array-decay-forces-remat (NEW), for-loop-vs-do-while delay-slot scheduling (NEW), statement-order (§2-T2 ext); + reconfirmed branch-polarity/mask-local * pipeline gotcha: match_one masks relocs -> sig_unify/canonical-retype is MANDATORY before harvest_verify (the conflicting-types gate failure) * loose-typing narrow-param wall is REAL (func_80146A6C lhu/s16 vs canonical s32/lw) - §16 reconciled: the 'not source-steerable' line cross-refs §17; §10-vs-§16 tension resolved by naming the residual class precisely (both right about different residuals) - hand-matching-process.md §8e: Phase-18 outcome re-scopes the Phase-19 wave (triage-and-stub the unsteerable; spend on non-walled STRUCTURAL_MISS) --- docs/hand-matching-process.md | 15 ++++++++ docs/matching-cookbook.md | 67 ++++++++++++++++++++++++++++++++++- phase-ends/CURRENT_PHASE.md | 7 +++- 3 files changed, 87 insertions(+), 2 deletions(-) diff --git a/docs/hand-matching-process.md b/docs/hand-matching-process.md index eba8bb15d..469d95630 100644 --- a/docs/hand-matching-process.md +++ b/docs/hand-matching-process.md @@ -386,3 +386,18 @@ gcc-quirk tail, so the next lever is understanding gcc-2.7.2 (R17 research, Phas ### 8d. The deferred wave (staged, ready to resume after the compiler research) `.run/harvest_wave_s4.js` = the layer-aware probe (40 tractable reach-134, sig_unify-before-gate). Resume after Phase 18 lands new gcc-quirk idioms (which raise the close-rate above 33% and so the wave's yield). + +### 8e. Phase-18 OUTCOME (compiler-quirk research — the verdict that re-scopes the wave) +The "understand gcc-2.7.2 to crack the tail" research LANDED (cookbook §17, gcc source `tools/reference/ +gcc-papermario` + Xenogears mine). **Decisive finding: the highest-reach circular tail is the call-crossing +register-ALLOCATION-ORDER class, and it is NOT source-steerable** — `global.c:allocno_compare` density ordering, +confirmed unsteerable by an exhaustive C/flag/cc1 sweep AND independently corroborated (Xenogears, same +compiler, ships the class as INCLUDE_ASM). New *partial* idioms found (array-decay-remat, for-loop/statement- +order delay-slot scheduling) + one full byte-gated demonstration (func_801399A8, reach-134, fleet 55.51→55.55%). +**Re-scope:** the wave's yield will NOT rise by cracking the circular tail (it can't be cracked from C); the +yield comes from the **non-walled STRUCTURAL_MISS fns** (closeable via §17 idioms + mandatory sig_unify). Some +structurals are ALSO walled by the loose-typing/narrow-param dead-end (func_80146A6C). So Phase-19's wave should +**triage with `match_one` and stub-and-skip** any residue that is a call-crossing $s0/$s1 swap or a narrow-param +type conflict, and spend only on the genuinely-tractable remainder. Expect a close-rate in the calibration +range (~33% whole-binary), not higher — the research raised UNDERSTANDING (so we don't waste effort on the +unsteerable), not the per-function close-rate of the hard tail. diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index db3e5189e..65bb5efd4 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -1020,7 +1020,9 @@ the time; the work is byte-closing + sig reconciliation. **Full process: `docs/h `ret0: return 0;`. gcc then makes `ret0` a labeled block reached by branches (right polarity) + schedules the next test's constant into the delay slot. A lone `if(x)return 0;` inlines (wrong polarity/reg). - **v0↔v1 result/constant coalescing** + the §10 hoist-vs-remat / phantom-frame quirks = the residual hard - tail (not source-steerable; permuter only helps relocs=0, and slowly). Defer as `INCLUDE_ASM` stub. + tail (not source-steerable — **§17 confirms this for the call-crossing register-ORDER class via the gcc + source + Xenogears, and adds the array-decay-remat lever that DOES crack part of the hoist-vs-remat class**; + permuter only helps relocs=0, and slowly). Defer as `INCLUDE_ASM` stub. **Scaling = Ultracode wave (§12 pattern + §7):** Ghidra pre-pass (`DecompileFunctions.java`, headless batch, no /mcp) → parallel draft agents (m2c+Ghidra-C+asm+actor-struct+§3a, self-validate `match_one`) → whole-binary @@ -1042,3 +1044,66 @@ highest-reach circular targets are ALL §10-hoist / regalloc / layout-bound (0 c The layer makes a wave *sig-clean*; it does NOT unlock the quirk tail. **→ The match-% lever is understanding gcc-2.7.2 (R17 compiler-source research, Phase 18), not more brute waves.** Wave deferred; infra staged (`.run/harvest_wave_s4.js`, 40 tractable reach-134 targets). See `docs/hand-matching-process.md` §8. +**§17 resolves the "not source-steerable" question with the gcc source + Xenogears: confirmed for the +call-crossing register-ORDER class, refined for the rest.** + +## §17 The compiler-quirk wall — steerable vs not (Phase 18; gcc-2.7.2 source + Xenogears confirmed) +Phase 18 read the real gcc-2.7.2 source (`tools/reference/gcc-papermario`, SETUP §5.6) + mined Xenogears (our +EXACT compiler — `gcc-2.7.2-psx`+`-cdk`) to resolve §16's open item. **Verdict: the call-crossing +register-ALLOCATION-ORDER class is NOT source-steerable; several adjacent classes ARE. Name the residual class +precisely before deciding — that reconciles §10 ("steerable") with §16 ("not steerable"): both are right about +DIFFERENT residuals.** Triage every quirk residual with `match_one` (the floor-free oracle — NEVER the permuter +score on jtbl/rodata fns, §10) before investing. + +### The UNSTEERABLE class — call-crossing $s0/$s1 allocation ORDER (global.c, source-confirmed) +Two pseudos live across a call → both are **global** allocnos (NOT local-alloc) competing for callee-saved +$s0/$s1. `global.c:allocno_compare` sorts by **density** `floor_log2(n_refs)*n_refs/live_length*size` (tie = +allocno number); first-sorted gets the first free reg ($s0). A short-lived value (loaded just before the call, +consumed just after) = HIGH density → wins $s0; a whole-function-lived value (an arg captured pre-call, used at +a late op) = LOW density → $s1 — even when the original is the reverse, and the long-lived value's span is +structurally fixed so its density can't be raised from C. Confirmed unsteerable: statement order (no effect), +variable coupling (regressed), -O3 (identical alloc), `cdk` cc1 (worse), the Xenogears flag deltas +(`-funsigned-char`/`-fpcc-struct-return`/`-fpeephole`/`-ffunction-cse`/`-fcaller-saves` — none flip it), +`-fno-schedule-insns` (worse). **Independent corroboration (R14):** Xenogears, same toolchain, has NO C lever +for this class (no `register`, no `__asm__("$16")` pins, no permuter) and ships such functions as INCLUDE_ASM +(1174). **Policy: stub it (G4); don't burn time.** Exemplar func_8012B8E4 (75=75, 24→21 via branch-polarity; +the residue is the swap). + +### The STEERABLE idioms (byte-confirmed this phase) +- **array-decay forces rematerialization (NEW).** A stack buffer passed to a callee as `&struct` / `mtx.w` / + `*(T*)arr` (any address-taken form) is HOISTED into a callee-saved reg (needs an extra callee-saved → bigger + frame, more spills). Declare it a local **array** `T buf[N]` and pass it as `buf` (array-decay, never + address-taken) → gcc **rematerializes** `addiu $reg,$sp,off` per call instead (matches the original, frees + the reg). func_8012B4B8: 88→52 (the hard regalloc+remat half fixed). CAVEAT: an array can't take a struct + block-copy (`arr = STRUCT` needs a struct; element-copy constant-folds each global addr to its own `lui`, + +ins), so a fn that ALSO needs a load-base-once struct-copy has an unavoidable tension. +- **for-loop vs do-while controls delay-slot scheduling.** A counted scan as a `for` (init/cond/update) lets + gcc schedule the branch-taken return value into the loop test's delay slot; a `do-while` with increments in + the body fills that slot with an increment instead (+1 ins, wrong schedule). func_801399A8: do-while 7 + mismatch → for-loop 2 → MATCH. +- **statement order in the for-update = instruction order (§2-T2 extended).** Independent updates in + `for(...; ...; A, B)` emit in source order; swap to match. (func_801399A8 final 2.) +- (existing, reconfirmed) **§3-T4 branch-polarity invert** (func_8012B8E4 24→21), **§16 mask-local**, + **§16 shared-ret0 goto**. + +### The pipeline gotcha — match_one ≠ the gate; `sig_unify` is MANDATORY +`match_one` masks relocations → it MATCHES even when the draft's own def-signature or a data-extern TYPE +conflicts with the canonical decl in `engine_core.h` (`u8 *func(void)` vs canonical `s32 func(void)`; +`extern u8 D_x` vs `extern s32 D_x`). The whole-binary gate then fails `conflicting types`. ALWAYS retype the +draft to the canonical set (return + data-extern types; use integer address arithmetic `(s32)&sym`, +codegen-neutral): **draft → `sig_unify`/canonical-retype → `harvest_verify`.** + +### The LOOSE-TYPING wall is real for narrow params (Phase 16, reconfirmed) +Some STRUCTURAL_MISS fns are blocked by it: func_80146A6C needs an incoming arg as `lhu` (s16), but the shared +canonical sig declares it `s32` (→ `lw`); the byte-match needs s16, another call site needs s32, no single C +type satisfies both. No clean fix (the documented narrow-param dead-end). **Stub it.** *(Reconciles §16's +"Phase 17 disproves the loose-typing wall": disproven for pure-structure fns like func_801399A8; REAL for +narrow-param fns like func_80146A6C and inseparable from the regalloc tail.)* + +### Strategic conclusion — the match-% lever post-research +The high-reach **circular regalloc-order tail is unsteerable** (confirmed + corroborated) and some structurals +hit the loose-typing wall — both → INCLUDE_ASM (the Xenogears policy). The lever is the **STRUCTURAL_MISS fns +that AREN'T walled** (closeable via the steerable idioms above + mandatory sig_unify — proven on func_801399A8, +reach-134, fleet +134), i.e. the **tractable-247 wave (Phase 19)**, NOT cracking the circular tail. Each such +match is worth ×(overlay count). Don't spend on the unsteerable classes beyond a `match_one` triage: residue = +call-crossing $s0/$s1 swap OR narrow-param type conflict → stub it and move on. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index b0973f52a..9cd8a5429 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -58,7 +58,12 @@ source-steerable") contradicts §10 — Phase 18 reconciles it. So: **existing-k sig_unify → **byte-gated to a full MATCH** (func_801399A8, 136/136). branch-polarity (§3-T4): match_one byte-evidence (24→21). array-decay-forces-remat: match_one byte-evidence (88→52, fixes the hard regalloc+ remat). regalloc-order: byte-proven UNSTEERABLE (exhaustive sweep + Xenogears). Per-class verdicts set. -- [ ] **T5 — Distill into cookbook + reconcile §10-vs-§16.** +- [x] **T5 — Cookbook distilled + reconciled ✓ 2026-06-20.** New **§17** (steerable-vs-not: the UNSTEERABLE + call-crossing register-ORDER class with the global.c mechanism + Xenogears corroboration; the STEERABLE + idioms array-decay-remat / for-loop-delay-slot / statement-order; the sig_unify-MANDATORY gotcha; the + loose-typing narrow-param wall). **§16 reconciled** (the "not source-steerable" line now points to §17's + confirmation; the §10-vs-§16 tension resolved by naming the residual class). hand-matching §8e (Phase-18 + outcome re-scopes the Phase-19 wave: triage-and-stub the unsteerable, spend on the non-walled structurals). - [x] **T6 — Demonstration ✓ 2026-06-20.** **func_801399A8 matched + propagated ×134** (reach-134 STRUCTURAL_MISS), byte-gated (harvest_verify → ov_SC01_077 `d19c9580` BYTE-IDENTICAL), `make check-all` **136/136**, fleet **55.51% → 55.55%** (+134 instances). Idioms used: for-loop structure (delay-slot