mirror/BFM-decomp
1
0
Fork 0
mirror of https://github.com/Druthulu/BFM-decomp synced 2026-09-26 21:36:06 -04:00
Code Issues Packages Projects Releases Wiki Activity

feat(phase-29): CRACK THE PLUMBING FAILURE — the derived-offset remap bug (cookbook §84); func_8013D53C banked

Browse Source 
Drew: "if it fails, see why and try again with new knowledge." It failed twice, then banked.

ROOT CAUSE, byte-proven: a family_remap member reached match_one MATCH (240 ins) and failed the
whole-binary gate by ONE BYTE. The exemplar carries a deliberate matching idiom — reach a symbol via
a DIFFERENT symbol plus a literal offset, so gcc cannot CSE the two %hi/%lo pairs:
    (*(S9*)&D_801DAA78) = *(S9*)(&D_801DA998 + 0x20);  /* same addr as &D_801DA9B8 */
family_remap substitutes the symbol NAMES correctly and leaves the literal 0x20 — but 0x20 is not a
constant of the algorithm, it is the DISTANCE BETWEEN TWO PER-OVERLAY SYMBOLS:
    exemplar 0x801DA998 + 0x20 = 0x801DA9B8  OK
    member   0x801A5778 + 0x20 = 0x801A5798  WRONG (real symbol 0x801A5790)
    member   0x801A5778 + 0x18 = 0x801A5790  correct
match_one MASKS HI16/LO16 so it is STRUCTURALLY BLIND to this — the §81 blindness in its DATA form.

TWO FIXES WERE EACH INDIVIDUALLY INSUFFICIENT: the byte fix alone re-failed as PLUMBING; the ladder
alone re-failed as DIFF. Together -> BANKED, R22 clean-fleet 140/140.

TWO LADDER CORRECTIONS (my own T0.2 error): bare harvest_verify is the LAST RUNG, not the ladder —
gate_stage runs canon_resident_calls -> cast_call_sites -> reconcile_tu -> ARITY -> sig_unify ->
harvest_verify, so T0.2's "8/8 PLUMBING" measured the UN-RECOVERED rate. And reconcile_decls.py is
RETIRED (R33, superseded by reconcile_tu): asking "what does the FLEET call this symbol?" is wrong by
construction in a loosely-typed engine (548 of its answers conflicted, rewriting 60 of 196 drafts) —
so Drew's suggested tool would have made it worse.

SCOPE, MEASURED (not over-generalised, §80): the idiom appears at only 5 sites corpus-wide — BUT one
gates a 123-member family (133 staged drafts all carry the un-recomputed +0x20 with different
per-overlay bases), so the mechanical fix is worth ~240 ins x 123 ~= 29,520 ins. It does NOT explain
the pool generally: func_80144090 / func_8012CC88 / func_8014D12C have ZERO derived-offset sites and
fail for a different, still-undiagnosed cause.

THE FIX IS MECHANICAL: correct_literal = mapped(aliased_sym) - mapped(base_sym). The remap already
holds both mappings, and the exemplar's own comment names the aliased symbol.

Also observed: the ARITY pre-pass left 40 TUs of caller-decl edits after a 0-bank run (same hygiene
bug as --normalize-self-decls, twice in one session) — reverted, both binaries byte-identical.
This commit is contained in:
Drew T
2026-07-26 13:47:06 -06:00
parent 1d44ce25f3
commit b6bb04ecff
6 changed files with 210 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/matching-cookbook.md
+67
View File
@@ -6447,3 +6447,70 @@ Two case residuals (6 and 11, 8 and 7 diffs) had been written off as "pure alloc
the morph/lerp loop walks **copies** of its two input pointers (`pa = msa; pb = msb;`), not the
originals — and spelling that took both cases to **zero**. **A residual class you assigned on an
earlier base is a hypothesis, not a verdict** (§80). Re-run the negative list after the base moves.
## §84 — The DERIVED-OFFSET remap bug: a hand-computed literal that encodes the DISTANCE between two per-overlay symbols, and why `match_one` is structurally blind to it (Phase 29 SESSION-20, `func_8013D53C`)
A `family_remap` member draft reached **`match_one` MATCH (240 ins)** and **failed the whole-binary
gate**. The whole binary differed by **ONE BYTE**.
### The construct
Cracking the exemplar produced a deliberate matching idiom — reach a symbol *via a different symbol
plus a literal offset*, so gcc cannot CSE the two `%hi/%lo` pairs into one:
```c
/* EXEMPLAR (ov_SC01_077): 0x801DA998 + 0x20 == 0x801DA9B8 ✓ */
(*(S9 *)&D_801DAA78) = *(S9 *)(&D_801DA998 + 0x20); /* same addr as &D_801DA9B8;
distinct sym defeats cse, keeps %hi/%lo */
```
`family_remap` substitutes symbol NAMES correctly — `D_801DA998→D_801A5778`, `D_801DA9B8→D_801A5790`
— **and leaves the literal `0x20` alone.** But `0x20` is not a constant of the algorithm: it is the
**distance between two per-overlay symbols**, and that distance is per-overlay.
```
exemplar: 0x801DA998 + 0x20 = 0x801DA9B8 ✓
member: 0x801A5778 + 0x20 = 0x801A5798 ✗ (the real symbol is 0x801A5790)
member: 0x801A5778 + 0x18 = 0x801A5790 ✓ correct offset is 0x18, not 0x20
```
Changing `0x20`→`0x18` **+ running the full `gate_stage` ladder** banked it byte-identical.
### Why it survived every candidate gate
**`match_one` masks HI16/LO16**, so a wrong `%lo` is invisible to it — it reports a clean MATCH.
Only the whole-binary gate sees the byte. This is the §81 blindness in its DATA form: §81 is
`match_one` blind to a jump table; §84 is `match_one` blind to a mis-derived data address.
**A `match_one` MATCH that fails the whole-binary gate by ONE BYTE is a masked-field bug —
diff the built image against the payload and read the differing word before assuming plumbing.**
The diagnostic that found it in one step:
```
python3 - <<'PY' # built vs extracted payload, byte-diff, map file offset -> vram
a=open('build/<ov>/<ov>','rb').read(); b=open('<target_path>','rb').read()
d=[i for i in range(min(len(a),len(b))) if a[i]!=b[i]]
print(len(d), [hex(BASE+i) for i in d[:8]])
PY
```
### THE FIX IS MECHANICAL — the tool already holds the answer
The remap knows both mappings, and the exemplar's own comment even names the aliased symbol (which
the remap has already substituted: *"same addr as `&D_801A5790`"*). So:
> **`correct_literal = mapped(aliased_sym) − mapped(base_sym)`**
Detect `&SYM + LITERAL` where `SYM_exemplar + LITERAL` equals another mapped symbol's exemplar
address, and recompute the literal from the member's own addresses. **Never carry the exemplar's
literal through a symbol substitution.**
### Scope, measured (do not over-generalise — §80)
The idiom is **rare**: only **5 sites** across the whole matched corpus
(`ov_SC01_077.c` ×1, `ov_SC01_077_jr_8017AE2C.c` ×2, `engine_core.h` ×2). **But one of those five
gates an entire 123-member family** — 133 staged member drafts all carry the un-recomputed `+ 0x20`
with different per-overlay base symbols (`D_801F3058`, `D_801A5778`, `D_8018F9B8`, …), so the single
tool fix is worth ≈ **240 ins × 123 members ≈ 29,520 ins**.
**It does NOT explain the sibling pool generally:** the other three families probed the same day
(`func_80144090`, `func_8012CC88`, `func_8014D12C`) have **zero** derived-offset sites and fail for a
different, still-undiagnosed cause.
### Two ladder lessons banked with it
1. **Bare `harvest_verify` is the LAST rung, not the ladder.** `gate_stage.py` runs
`canon_resident_calls → cast_call_sites → reconcile_tu → ARITY pre-pass → sig_unify →
harvest_verify`. A probe that calls `harvest_verify` directly measures the **un-recovered** rate
and will report PLUMBING for everything the ladder would have cleared. The byte fix AND the ladder
were each individually insufficient here; only together did it bank.
2. **`reconcile_decls.py` is RETIRED** (Phase 26-A, R33) — superseded by `reconcile_tu.py`, because
asking *"what does the FLEET call this symbol?"* is wrong by construction in a loosely-typed
engine (548 of its answers conflicted; it was rewriting 60 of 196 live drafts). Reach for
`reconcile_tu`, never `reconcile_decls`.
docs/progress.fleet.md
+7 -7
View File
@@ -4,23 +4,23 @@
# cross-binary collapsible-byte leverage: docs/duplicates.cross.md.
# THREE progress metrics (all matter — see the labels):
FLEET fn-count byte-ident: 315467 / 353719 = 89.19% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
FLEET instr-weighted : 10589065 / 13141652 = 80.6% (shipped .text across main + resident + 138 overlays; the decomp.dev-DISPLAY number)
FLEET distinct-code(uniq): 3846416 / 5634875 = 68.3% (64911/87459 unique fns; the DISTINCT-RE number)
FLEET fn-count byte-ident: 315468 / 353719 = 89.19% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
FLEET instr-weighted : 10589305 / 13141652 = 80.6% (shipped .text across main + resident + 138 overlays; the decomp.dev-DISPLAY number)
FLEET distinct-code(uniq): 3846656 / 5634875 = 68.3% (64912/87459 unique fns; the DISTINCT-RE number)
MAIN game-code weighted : 436 / 60201 = 0.7% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-06-14; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift)
(fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 10588629 / 13081451 = 80.9%)
(fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 10588869 / 13081451 = 80.9%)
FLEET REAL substantive : 313612 (of which dedup-shared 239530 via 1886 groups / 239604 instances)
FLEET REAL substantive : 313613 (of which dedup-shared 239530 via 1886 groups / 239604 instances)
FLEET LINKED PsyQ objs : 959
FLEET NON_MATCHING : 7 (0 in any default build — G4)
FLEET INCLUDE_ASM stubs : 38245
FLEET INCLUDE_ASM stubs : 38244
FLEET matchable : 353719
| binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % |
|---|---:|---:|---:|---:|---:|---:|
| main | 54 | 2 | 959 | 1055 | 2096 | 50.3% |
| resident | 129 | 0 | 0 | 131 | 145 | 90.3% |
| ov_SC01_000 | 2250 | 1742 | 0 | 2250 | 2403 | 93.6% |
| ov_SC01_000 | 2251 | 1742 | 0 | 2251 | 2403 | 93.7% |
| ov_SC01_001 | 2253 | 1743 | 0 | 2255 | 2466 | 91.4% |
| ov_SC01_004 | 2242 | 1734 | 0 | 2243 | 2414 | 92.9% |
| ov_SC01_005 | 2272 | 1757 | 0 | 2272 | 2503 | 90.8% |
src/ov_SC01_000/ov_SC01_000_jr_801380E0.c
+133 -2
View File
@@ -1891,7 +1891,7 @@ extern u8 D_8017F49C[];
extern u8 D_8017F51C[];
extern u8 D_8017F3F4[];
extern void func_8013D53C(void);
extern void func_8013D53C();
extern void func_8013DD68(void);
extern void func_8013D8FC(void);
extern void func_8013CF68(void);
@@ -1936,7 +1936,138 @@ void func_8013D3D4(int param_1, int param_2)
}
INCLUDE_ASM("asm/ov_SC01_000/nonmatchings/ov_SC01_000_jr_801380E0", func_8013D53C);
/* 9-byte, align-1 -> unaligned block copy */
extern s32 D_801A44F8;
extern s32 D_801A44E0;
extern u8 D_801A5872;
extern u8 D_801A58E6;
extern u8 D_801A579A;
extern u8 D_801A5854;
extern u8 D_801A5778;
extern u8 D_801A5799;
extern s32 D_801A5858;
extern s32 D_801A44F4;
extern s32 D_801A44F0;
extern s32 D_801A44D8;
extern s32 D_801A44E4;
extern s32 D_801A44E8;
extern void *D_801A44C8;
extern s32 D_801A4500;
extern void *D_801A44D0;
void func_8013D53C(void *arg0v) {
extern Rec9 D_8017F538[];
extern Rec12 D_8017F570[];
extern u8 D_80078EAF;
extern u8 D_801A5830;
extern s32 D_801A44EC;
Cmd_8013D53C *arg0 = arg0v;
extern u8 D_8017F574[];
extern u8 D_8017F578[];
extern unsigned char D_801A5790;
extern s16 *D_801A44D4;
extern s32 D_801A4504;
extern s32 D_801A4508;
extern s32 D_801A450C;
s32 s0v;
s32 t9v;
s32 t8v;
u8 b0, b1, b2;
u8 pad[8]; /* dead BLKmode local: frame 0x10 -> 0x18, zero code */
if (!(D_801A44F8 & 1)) {
D_801A44E0 = 1;
} else {
D_801A44E0 = D_80078EAF;
}
b0 = ((u8 *)D_8017F570)[D_801A44E0 * 12];
D_801A5872 = b0;
D_801A58E6 = b0;
b1 = D_8017F574[D_801A44E0 * 12];
D_801A579A = b1;
D_801A5854 = b1;
b2 = D_8017F578[D_801A44E0 * 12];
D_801A5778 = b2;
D_801A5799 = b2;
(*(S9 *)&D_801A5790) = ((S9 *)D_8017F538)[D_801A44E0];
(*(S9 *)&D_801A5830) = *(S9 *)(&D_801A5778 + 0x18); /* same addr as (*(S9 *)&D_801A5790); distinct sym defeats cse, keeps %hi/%lo */
D_801A5858 = 1;
D_801A44F4 = -1;
D_801A44F0 = 0;
D_801A44EC = -1;
D_801A44D8 = 0;
D_801A44E4 = 0;
D_801A44E8 = 0;
if ((D_801A44F8 & 2) && (D_801A44E0 == 4)) {
s0v = (*(s32 * *)&D_801A44C8)[18];
t9v = (*(s32 * *)&D_801A44C8)[19];
t8v = (*(s32 * *)&D_801A44C8)[20];
} else {
s32 *p = (s32 *)(D_801A44E0 * 12 + (s32) (*(s32 * *)&D_801A44C8)); /* block-local: local-alloc ties sum into mul chain */
s0v = p[0];
t9v = p[1];
t8v = p[2];
}
(*(Cmd_8013D53C * *)&D_801A44D4) = arg0;
if (arg0 != 0) {
if (D_801A4500 != 0) {
s32 *p = (s32 *)(D_801A44E0 * 12 + (s32) (*(s32 * *)&D_801A44D0));
D_801A4504 = p[0];
D_801A4508 = p[1];
D_801A450C = p[2];
} else {
while ((arg0->cmd & 0xFFFF) != 0xFF) {
if ((arg0->cmd & 0xFFFF) == 9) {
s32 n;
s32 i;
u16 *src;
u16 *dst;
n = arg0->w * arg0->h;
i = 0;
src = arg0->data;
__asm__("" :: "r"(src)); /* +2 refs on src (depth-2): keeps src above i, below the mfhi temp */
dst = src + n;
if (n > 0) {
do {
u16 px;
s32 r, g, b, out;
__asm__("" :: "r"(i)); /* +3 refs on i (depth-3): lifts i over dst in the $t2 race */
px = *src;
r = ((px & 0x1F) * s0v) / 2560;
g = (((px & 0x3E0) * t9v) / 2560) & 0x3E0;
b = (((px & 0x7C00) * t8v) / 2560) & 0x7C00;
out = r | g | b | (px & 0x8000);
if (out == 0 && px != 0) {
out = 0x8000;
}
*dst = out;
dst++;
i++;
src++;
} while (i < n);
}
}
arg0++;
}
}
}
}
src/ov_SC01_000/ov_SC01_000_jr_8013F350.c
+1 -1
View File
@@ -818,7 +818,7 @@ extern u8 D_8017F3A0[];
extern u8 D_8017F49C[];
extern u8 D_8017F51C[];
extern u8 D_8017F3F4[];
extern void func_8013D53C(void);
extern void func_8013D53C();
extern void func_8013DD68(void);
extern void func_8013D8FC(void);
extern void func_8013CF68(void);
src/ov_SC01_000/ov_SC01_000_jr_8013FFD8.c
+1 -1
View File
@@ -818,7 +818,7 @@ extern u8 D_8017F3A0[];
extern u8 D_8017F49C[];
extern u8 D_8017F51C[];
extern u8 D_8017F3F4[];
extern void func_8013D53C(void);
extern void func_8013D53C();
extern void func_8013DD68(void);
extern void func_8013D8FC(void);
extern void func_8013CF68(void);
src/ov_SC01_000/ov_SC01_000_jr_80140608.c
+1 -1
View File
@@ -818,7 +818,7 @@ extern u8 D_8017F3A0[];
extern u8 D_8017F49C[];
extern u8 D_8017F51C[];
extern u8 D_8017F3F4[];
extern void func_8013D53C(void);
extern void func_8013D53C();
extern void func_8013DD68(void);
extern void func_8013D8FC(void);
extern void func_8013CF68(void);