diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 66648bcbe..ae6dd9f99 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -6447,3 +6447,70 @@ Two case residuals (6 and 11, 8 and 7 diffs) had been written off as "pure alloc the morph/lerp loop walks **copies** of its two input pointers (`pa = msa; pb = msb;`), not the originals — and spelling that took both cases to **zero**. **A residual class you assigned on an earlier base is a hypothesis, not a verdict** (§80). Re-run the negative list after the base moves. + +## §84 — The DERIVED-OFFSET remap bug: a hand-computed literal that encodes the DISTANCE between two per-overlay symbols, and why `match_one` is structurally blind to it (Phase 29 SESSION-20, `func_8013D53C`) + +A `family_remap` member draft reached **`match_one` MATCH (240 ins)** and **failed the whole-binary +gate**. The whole binary differed by **ONE BYTE**. + +### The construct +Cracking the exemplar produced a deliberate matching idiom — reach a symbol *via a different symbol +plus a literal offset*, so gcc cannot CSE the two `%hi/%lo` pairs into one: +```c +/* EXEMPLAR (ov_SC01_077): 0x801DA998 + 0x20 == 0x801DA9B8 ✓ */ +(*(S9 *)&D_801DAA78) = *(S9 *)(&D_801DA998 + 0x20); /* same addr as &D_801DA9B8; + distinct sym defeats cse, keeps %hi/%lo */ +``` +`family_remap` substitutes symbol NAMES correctly — `D_801DA998→D_801A5778`, `D_801DA9B8→D_801A5790` +— **and leaves the literal `0x20` alone.** But `0x20` is not a constant of the algorithm: it is the +**distance between two per-overlay symbols**, and that distance is per-overlay. +``` +exemplar: 0x801DA998 + 0x20 = 0x801DA9B8 ✓ +member: 0x801A5778 + 0x20 = 0x801A5798 ✗ (the real symbol is 0x801A5790) +member: 0x801A5778 + 0x18 = 0x801A5790 ✓ correct offset is 0x18, not 0x20 +``` +Changing `0x20`→`0x18` **+ running the full `gate_stage` ladder** banked it byte-identical. + +### Why it survived every candidate gate +**`match_one` masks HI16/LO16**, so a wrong `%lo` is invisible to it — it reports a clean MATCH. +Only the whole-binary gate sees the byte. This is the §81 blindness in its DATA form: §81 is +`match_one` blind to a jump table; §84 is `match_one` blind to a mis-derived data address. +**A `match_one` MATCH that fails the whole-binary gate by ONE BYTE is a masked-field bug — +diff the built image against the payload and read the differing word before assuming plumbing.** +The diagnostic that found it in one step: +``` +python3 - <<'PY' # built vs extracted payload, byte-diff, map file offset -> vram +a=open('build//','rb').read(); b=open('','rb').read() +d=[i for i in range(min(len(a),len(b))) if a[i]!=b[i]] +print(len(d), [hex(BASE+i) for i in d[:8]]) +PY +``` + +### THE FIX IS MECHANICAL — the tool already holds the answer +The remap knows both mappings, and the exemplar's own comment even names the aliased symbol (which +the remap has already substituted: *"same addr as `&D_801A5790`"*). So: +> **`correct_literal = mapped(aliased_sym) − mapped(base_sym)`** +Detect `&SYM + LITERAL` where `SYM_exemplar + LITERAL` equals another mapped symbol's exemplar +address, and recompute the literal from the member's own addresses. **Never carry the exemplar's +literal through a symbol substitution.** + +### Scope, measured (do not over-generalise — §80) +The idiom is **rare**: only **5 sites** across the whole matched corpus +(`ov_SC01_077.c` ×1, `ov_SC01_077_jr_8017AE2C.c` ×2, `engine_core.h` ×2). **But one of those five +gates an entire 123-member family** — 133 staged member drafts all carry the un-recomputed `+ 0x20` +with different per-overlay base symbols (`D_801F3058`, `D_801A5778`, `D_8018F9B8`, …), so the single +tool fix is worth ≈ **240 ins × 123 members ≈ 29,520 ins**. +**It does NOT explain the sibling pool generally:** the other three families probed the same day +(`func_80144090`, `func_8012CC88`, `func_8014D12C`) have **zero** derived-offset sites and fail for a +different, still-undiagnosed cause. + +### Two ladder lessons banked with it +1. **Bare `harvest_verify` is the LAST rung, not the ladder.** `gate_stage.py` runs + `canon_resident_calls → cast_call_sites → reconcile_tu → ARITY pre-pass → sig_unify → + harvest_verify`. A probe that calls `harvest_verify` directly measures the **un-recovered** rate + and will report PLUMBING for everything the ladder would have cleared. The byte fix AND the ladder + were each individually insufficient here; only together did it bank. +2. **`reconcile_decls.py` is RETIRED** (Phase 26-A, R33) — superseded by `reconcile_tu.py`, because + asking *"what does the FLEET call this symbol?"* is wrong by construction in a loosely-typed + engine (548 of its answers conflicted; it was rewriting 60 of 196 live drafts). Reach for + `reconcile_tu`, never `reconcile_decls`. diff --git a/docs/progress.fleet.md b/docs/progress.fleet.md index 64cec390c..803c7b514 100644 --- a/docs/progress.fleet.md +++ b/docs/progress.fleet.md @@ -4,23 +4,23 @@ # cross-binary collapsible-byte leverage: docs/duplicates.cross.md. # THREE progress metrics (all matter — see the labels): -FLEET fn-count byte-ident: 315467 / 353719 = 89.19% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay) -FLEET instr-weighted : 10589065 / 13141652 = 80.6% (shipped .text across main + resident + 138 overlays; the decomp.dev-DISPLAY number) -FLEET distinct-code(uniq): 3846416 / 5634875 = 68.3% (64911/87459 unique fns; the DISTINCT-RE number) +FLEET fn-count byte-ident: 315468 / 353719 = 89.19% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay) +FLEET instr-weighted : 10589305 / 13141652 = 80.6% (shipped .text across main + resident + 138 overlays; the decomp.dev-DISPLAY number) +FLEET distinct-code(uniq): 3846656 / 5634875 = 68.3% (64912/87459 unique fns; the DISTINCT-RE number) MAIN game-code weighted : 436 / 60201 = 0.7% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-06-14; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift) - (fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 10588629 / 13081451 = 80.9%) + (fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 10588869 / 13081451 = 80.9%) -FLEET REAL substantive : 313612 (of which dedup-shared 239530 via 1886 groups / 239604 instances) +FLEET REAL substantive : 313613 (of which dedup-shared 239530 via 1886 groups / 239604 instances) FLEET LINKED PsyQ objs : 959 FLEET NON_MATCHING : 7 (0 in any default build — G4) -FLEET INCLUDE_ASM stubs : 38245 +FLEET INCLUDE_ASM stubs : 38244 FLEET matchable : 353719 | binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % | |---|---:|---:|---:|---:|---:|---:| | main | 54 | 2 | 959 | 1055 | 2096 | 50.3% | | resident | 129 | 0 | 0 | 131 | 145 | 90.3% | -| ov_SC01_000 | 2250 | 1742 | 0 | 2250 | 2403 | 93.6% | +| ov_SC01_000 | 2251 | 1742 | 0 | 2251 | 2403 | 93.7% | | ov_SC01_001 | 2253 | 1743 | 0 | 2255 | 2466 | 91.4% | | ov_SC01_004 | 2242 | 1734 | 0 | 2243 | 2414 | 92.9% | | ov_SC01_005 | 2272 | 1757 | 0 | 2272 | 2503 | 90.8% | diff --git a/src/ov_SC01_000/ov_SC01_000_jr_801380E0.c b/src/ov_SC01_000/ov_SC01_000_jr_801380E0.c index 2cec8e39f..1e0562890 100644 --- a/src/ov_SC01_000/ov_SC01_000_jr_801380E0.c +++ b/src/ov_SC01_000/ov_SC01_000_jr_801380E0.c @@ -1891,7 +1891,7 @@ extern u8 D_8017F49C[]; extern u8 D_8017F51C[]; extern u8 D_8017F3F4[]; -extern void func_8013D53C(void); +extern void func_8013D53C(); extern void func_8013DD68(void); extern void func_8013D8FC(void); extern void func_8013CF68(void); @@ -1936,7 +1936,138 @@ void func_8013D3D4(int param_1, int param_2) } -INCLUDE_ASM("asm/ov_SC01_000/nonmatchings/ov_SC01_000_jr_801380E0", func_8013D53C); + + /* 9-byte, align-1 -> unaligned block copy */ + + +extern s32 D_801A44F8; +extern s32 D_801A44E0; + + +extern u8 D_801A5872; +extern u8 D_801A58E6; +extern u8 D_801A579A; +extern u8 D_801A5854; +extern u8 D_801A5778; +extern u8 D_801A5799; + + +extern s32 D_801A5858; +extern s32 D_801A44F4; +extern s32 D_801A44F0; +extern s32 D_801A44D8; +extern s32 D_801A44E4; +extern s32 D_801A44E8; + +extern void *D_801A44C8; +extern s32 D_801A4500; +extern void *D_801A44D0; + +void func_8013D53C(void *arg0v) { + + extern Rec9 D_8017F538[]; + extern Rec12 D_8017F570[]; + extern u8 D_80078EAF; + extern u8 D_801A5830; + extern s32 D_801A44EC; + Cmd_8013D53C *arg0 = arg0v; + + extern u8 D_8017F574[]; + extern u8 D_8017F578[]; + extern unsigned char D_801A5790; + extern s16 *D_801A44D4; + extern s32 D_801A4504; + extern s32 D_801A4508; + extern s32 D_801A450C; + s32 s0v; + s32 t9v; + s32 t8v; + u8 b0, b1, b2; + u8 pad[8]; /* dead BLKmode local: frame 0x10 -> 0x18, zero code */ + + if (!(D_801A44F8 & 1)) { + D_801A44E0 = 1; + } else { + D_801A44E0 = D_80078EAF; + } + + b0 = ((u8 *)D_8017F570)[D_801A44E0 * 12]; + D_801A5872 = b0; + D_801A58E6 = b0; + b1 = D_8017F574[D_801A44E0 * 12]; + D_801A579A = b1; + D_801A5854 = b1; + b2 = D_8017F578[D_801A44E0 * 12]; + D_801A5778 = b2; + D_801A5799 = b2; + + (*(S9 *)&D_801A5790) = ((S9 *)D_8017F538)[D_801A44E0]; + (*(S9 *)&D_801A5830) = *(S9 *)(&D_801A5778 + 0x18); /* same addr as (*(S9 *)&D_801A5790); distinct sym defeats cse, keeps %hi/%lo */ + + D_801A5858 = 1; + D_801A44F4 = -1; + D_801A44F0 = 0; + D_801A44EC = -1; + D_801A44D8 = 0; + D_801A44E4 = 0; + D_801A44E8 = 0; + + if ((D_801A44F8 & 2) && (D_801A44E0 == 4)) { + s0v = (*(s32 * *)&D_801A44C8)[18]; + t9v = (*(s32 * *)&D_801A44C8)[19]; + t8v = (*(s32 * *)&D_801A44C8)[20]; + } else { + s32 *p = (s32 *)(D_801A44E0 * 12 + (s32) (*(s32 * *)&D_801A44C8)); /* block-local: local-alloc ties sum into mul chain */ + s0v = p[0]; + t9v = p[1]; + t8v = p[2]; + } + + (*(Cmd_8013D53C * *)&D_801A44D4) = arg0; + if (arg0 != 0) { + if (D_801A4500 != 0) { + s32 *p = (s32 *)(D_801A44E0 * 12 + (s32) (*(s32 * *)&D_801A44D0)); + D_801A4504 = p[0]; + D_801A4508 = p[1]; + D_801A450C = p[2]; + } else { + while ((arg0->cmd & 0xFFFF) != 0xFF) { + if ((arg0->cmd & 0xFFFF) == 9) { + s32 n; + s32 i; + u16 *src; + u16 *dst; + n = arg0->w * arg0->h; + i = 0; + src = arg0->data; + __asm__("" :: "r"(src)); /* +2 refs on src (depth-2): keeps src above i, below the mfhi temp */ + dst = src + n; + if (n > 0) { + do { + u16 px; + s32 r, g, b, out; + __asm__("" :: "r"(i)); /* +3 refs on i (depth-3): lifts i over dst in the $t2 race */ + px = *src; + r = ((px & 0x1F) * s0v) / 2560; + g = (((px & 0x3E0) * t9v) / 2560) & 0x3E0; + b = (((px & 0x7C00) * t8v) / 2560) & 0x7C00; + out = r | g | b | (px & 0x8000); + if (out == 0 && px != 0) { + out = 0x8000; + } + *dst = out; + dst++; + i++; + src++; + } while (i < n); + } + } + arg0++; + } + } + } +} + diff --git a/src/ov_SC01_000/ov_SC01_000_jr_8013F350.c b/src/ov_SC01_000/ov_SC01_000_jr_8013F350.c index c8dbe9a50..8122d6c59 100644 --- a/src/ov_SC01_000/ov_SC01_000_jr_8013F350.c +++ b/src/ov_SC01_000/ov_SC01_000_jr_8013F350.c @@ -818,7 +818,7 @@ extern u8 D_8017F3A0[]; extern u8 D_8017F49C[]; extern u8 D_8017F51C[]; extern u8 D_8017F3F4[]; -extern void func_8013D53C(void); +extern void func_8013D53C(); extern void func_8013DD68(void); extern void func_8013D8FC(void); extern void func_8013CF68(void); diff --git a/src/ov_SC01_000/ov_SC01_000_jr_8013FFD8.c b/src/ov_SC01_000/ov_SC01_000_jr_8013FFD8.c index 5eada007d..51246e0e0 100644 --- a/src/ov_SC01_000/ov_SC01_000_jr_8013FFD8.c +++ b/src/ov_SC01_000/ov_SC01_000_jr_8013FFD8.c @@ -818,7 +818,7 @@ extern u8 D_8017F3A0[]; extern u8 D_8017F49C[]; extern u8 D_8017F51C[]; extern u8 D_8017F3F4[]; -extern void func_8013D53C(void); +extern void func_8013D53C(); extern void func_8013DD68(void); extern void func_8013D8FC(void); extern void func_8013CF68(void); diff --git a/src/ov_SC01_000/ov_SC01_000_jr_80140608.c b/src/ov_SC01_000/ov_SC01_000_jr_80140608.c index 8321983fb..43a289204 100644 --- a/src/ov_SC01_000/ov_SC01_000_jr_80140608.c +++ b/src/ov_SC01_000/ov_SC01_000_jr_80140608.c @@ -818,7 +818,7 @@ extern u8 D_8017F3A0[]; extern u8 D_8017F49C[]; extern u8 D_8017F51C[]; extern u8 D_8017F3F4[]; -extern void func_8013D53C(void); +extern void func_8013D53C(); extern void func_8013DD68(void); extern void func_8013D8FC(void); extern void func_8013CF68(void);