From c697746462edb960fcbb04d25665d262be14214b Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Thu, 6 Aug 2026 10:52:07 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-30=20S44=20I.0):=20the=20static=20lo?= =?UTF-8?q?ader=20routing=20table=20+=20the=20full=20tool=20audit=20?= =?UTF-8?q?=E2=80=94=20knowledge=20captured?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plan-approved campaign (Fable5Max, ~/.claude/plans/optimized-squishing-engelbart.md). I.0 = capture while hot (R30/R31), before any code: - memory-map.md §"Phase 30 S44": the COMPLETE loader routing table, static-derived (G5) — the EXE's loadDestPtrTable (0x80072C70: resident/overlay/slotA/slotB/type-7), the boot k-set {1,3,8,10,11}, the RESIDENT's index tables D_800D3764 (29x8, MAIN/13-41 -> 0x800CAE08) and D_800D384C (6x8, MAIN/42-47 -> 0x800CCB1C), resident.c:641 (MAIN/12 -> 0x80128158), the SC07 pair's header-derived 0x801A00D8, gbase arithmetic (LIST.CD carries LBA+len ONLY), the slot-adjacency proof, the module-id-word law (word0, dense 0x13..0x73, resident=0x36; MAIN/9-vs-39 duplicate flagged), and "PAC type 1 = uncompressed overlay, type 4 = LZSS". SUPERSEDES P3-T5's "entries [1]+ are runtime-indexed (no static xref)". - disc-completeness.md: the "only knowable by runtime RE" doctrine REFUTED in place (H5, original kept) — 46 of 78 addresses are static; the runtime-only remainder is 28 script modules + 4 stragglers, parked for L3 with evidence. Byte-sum correction (rows 3,406,325 B vs bucket 3,564,021 incl. PAC headers), MAIN/7 raw-path exception, MAIN/0≡1. - tooling-audit.md §S44: EVERY tool classified with file:line — 8 must-change (family_remap VRAM const, Makefile+modules.mk, sig-target generalization, audit_binaries de-ov_, family_hseq/ progress:647/audit_frontier globs, corpus.sig_is_independent), 7 one-line registrations, 5 retirements (disc_code_sweep superseded by disc_audit; reconcile_decls; 3 rollout one-shots; ImportOverlay/VerifyOverlay.java), rest auto-OK/N-A. new_overlay.sh -> new_binary.sh design. - decision-log (R31): the pivot entry — the emulator dependency dissolves; the "modules" mostly dissolve into overlays (~75-77% h_exact-known; 802 novel fns); why the doctrine was missable for 30 phases (a confident negative doctrine is a claim like any other — date it, cite it, re-measure). - cookbook §154 + index regen (454 sections): module-id word / dual base-voting (h_exact ~500:1 + jal-alignment, must AGREE; thin votes => park, P9) / diff a mystery payload's head against classes you already own before inventing a new one. --- docs/cookbook-index.md | 100 +++++++++++++++++++++++++++++------- docs/decision-log.md | 35 +++++++++++++ docs/disc-completeness.md | 14 +++++ docs/matching-cookbook.md | 38 ++++++++++++++ docs/memory-map.md | 56 ++++++++++++++++++++ docs/tooling-audit.md | 98 +++++++++++++++++++++++++++++++++++ phase-ends/CURRENT_PHASE.md | 17 ++++++ 7 files changed, 339 insertions(+), 19 deletions(-) diff --git a/docs/cookbook-index.md b/docs/cookbook-index.md index 38ad29bc2..9cd9df8b3 100644 --- a/docs/cookbook-index.md +++ b/docs/cookbook-index.md @@ -2,7 +2,7 @@ > **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section. > -> `docs/matching-cookbook.md` is ~716 KB / 424 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. +> `docs/matching-cookbook.md` is ~716 KB / 454 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. **How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win. @@ -59,10 +59,10 @@ - **§3-The** — attribution primitive (use this before calling anything a scheduling residual) L6050 - **§3-The** — scheduling rules (refining §135-2 and §135-4) L8902 - **Consequence** — for the family (a real scheduling decision) L10085 -- **§148** — The loop.c hoisting THRESHOLD is arithmetic you can compute, and the `?:` clamp that folds to MIN_EXPR (P30 S42, `func_8017C6F4`, 947 ins) L10098 -- **§3-A.** — `move_movables` hoists iff `threshold × savings × lifetime ≥ insn_count` — and you can read it L10104 +- **§148** — The loop.c hoisting THRESHOLD is arithmetic you can compute, and the `?:` clamp that folds to MIN_EXPR (P30 S42, `func_8017C6F4`, 947 ins) L10135 +- **§3-A.** — `move_movables` hoists iff `threshold × savings × lifetime ≥ insn_count` — and you can read it L10141 -### register allocation & pins (36) +### register allocation & pins (38) - **§10** — Closing the regalloc/scheduling hard tail by hand (LZSS, Phase 7 session F — the full close) L835 - **Residual** — A — commutative `|`/`&`/`+` result lands in the wrong source-operand register L856 @@ -99,12 +99,15 @@ - **§3-The** — same swallow, twice more, in the integration spine L9699 - **§3-B.** — A `?:` on MEMORY operands costs ~16 bytes of invisible frame; on REGISTER operands, zero L10047 - **§3-D.** — A lone `$t8`/`$t9` in the target is RELOAD SCRATCH — reproduce the spill, don't pin the register L10063 -- **§3-C.** — A zero-byte ALLOCNO-PRIORITY slider L10137 +- **§3-C.** — A zero-byte ALLOCNO-PRIORITY slider L10174 +- **§150** — A register ROTATION across symmetric blocks is VARIABLE-IDENTITY evidence, not an allocator tie (P30 S43, `func_8017C6F4`, 947 ins ×4) L10302 +- **§152** — BYTE SIZE is the family key that name- and h_seq-grouping both miss (P30 S43, the 0xECC family: 1 crack → 12 overlays → 11,364 ins) L10408 -### CSE / redundancy / rematerialization (2) +### CSE / redundancy / rematerialization (3) - **§46** — The `func_80178D40` crack (890 ins ×134, the heaviest core in the game): four LOOP-STRUCTURE levers cheap-Opus found by reading loop.c/jump.c/cse.c (Phase 26 session 8, 2026-07-13) L3306 - **§83d** — CSE's quantity budget is WHOLE-FUNCTION, so a local rewrite cannot fix a local symptom L6445 +- **§153** — THE ADDRESS-REMATERIALISATION LAUNDER: a third zero-emission asm lever (P30 S43, `func_8018D98C`, 710 ins) L10456 ### loops & induction variables (9) @@ -116,7 +119,7 @@ - **§66d-1** — What transfers between giants is the LOOP, not the PIN L5273 - **§70** — The giv-init base register: walk the PARAMETER, not a copy of it (Phase 29 SESSION-18, `func_801777BC`) L5612 - **§145** — Three loop/combine levers from the S40 wave-2 drafters (16/16 match_one) L9917 -- **§148** — The loop.c hoisting THRESHOLD is arithmetic you can compute, and the `?:` clamp that folds to MIN_EXPR (P30 S42, `func_8017C6F4`, 947 ins) L10098 +- **§148** — The loop.c hoisting THRESHOLD is arithmetic you can compute, and the `?:` clamp that folds to MIN_EXPR (P30 S42, `func_8017C6F4`, 947 ins) L10135 ### structs, block moves & memcpy (30) @@ -151,7 +154,7 @@ - **§3-The** — DEFINITION-side alias is the only escape when the fleet canon disagrees on a promoting param L9463 - **§3-Two** — errors of mine, both instructive L9999 -### types, signedness & load/store width (31) +### types, signedness & load/store width (33) - **§3-I1** — Unsigned range check: `(x - lo) < (hi-lo)` → `addiu`+`sltiu` L41 - **§3-I2** — Byte mask forces `andi` even after `lbu` L47 @@ -184,6 +187,8 @@ - **§3-The** — type-form rules L8872 - **§143** — `cast_call_sites` read a RETURN STATEMENT as a prototype and deleted it. A 0/39 sweep became 18/39. (P30 S40) L9824 - **Then** — propagation returned 0/137 TWICE — both times a missing TYPE L9990 +- **§154** — Reading a disc payload: the module-id word, static base derivation, and "type 1 = uncompressed overlay" (P30 S44) L10512 +- **§3-C.** — PAC type 1 = the same payload class as type 4, just NOT compressed L10538 ### declarations, prototypes & K&R (55) @@ -287,7 +292,7 @@ - **§127a** — §71 (sibling-first) is the strongest `-O0` lever, and it beats the index L8370 - **§132** — The `JR-PAIR-IN-ONE-O0-OBJECT` "wall" was TWO instrument defects: a merged-double span the carve could not see, and a truncated object no rule deleted (P30 S29, `func_8013B83C` + `func_8013BD74`) L8563 -### family propagation & sweeps (74) +### family propagation & sweeps (78) - **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) L483 - **§11** — Cross-binary dedup & code-sharing (Phase 11 — "one match unlocks many") L908 @@ -363,8 +368,12 @@ - **§3-A.** — The frame has THREE strata, and stratum 3 is unreachable from C L10031 - **§3-E.** — A `qty_compare` TIE is not spelling-reachable — recognise it and stop L10073 - **Consequence** — for the family (a real scheduling decision) L10085 +- **§150** — A register ROTATION across symmetric blocks is VARIABLE-IDENTITY evidence, not an allocator tie (P30 S43, `func_8017C6F4`, 947 ins ×4) L10302 +- **§151** — THE GHOST WEDGE: when a load-before-store transposition is unreachable by ANY statement order (P30 S43, `func_8017EF68`, 969 ins) L10357 +- **When** — to reach for it L10397 +- **§152** — BYTE SIZE is the family key that name- and h_seq-grouping both miss (P30 S43, the 0xECC family: 1 crack → 12 overlays → 11,364 ins) L10408 -### integration / TU plumbing (36) +### integration / TU plumbing (37) - **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) L437 - **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) L483 @@ -402,6 +411,7 @@ - **§3-The** — declaration surface (integration, not codegen) L8931 - **Reconciling** — a gate-refused draft: which way you edit depends on WHERE the TU's decl is L9555 - **§3-The** — same swallow, twice more, in the integration spine L9699 +- **§3-A.** — Payload word0 is a global MODULE ID; code starts after the header L10517 ### build graph, splat & the harness (99) @@ -505,7 +515,7 @@ - **§142** — An open stub whose `h_exact` class is MATCHED elsewhere is FREE. Propagate the body; do not gate a draft. (P30 S39, +7,710 ins in two commands) L9768 - **§3-The** — measurement (do this before any wave; it is ~20 lines and needs no builds) L9780 -### process, measurement & doctrine (59) +### process, measurement & doctrine (63) - **§8e** — The jtbl ALIGNMENT LAW + the pad-spec filter — multi-table .rodata spans (Phase 29, byte-proven; `.run/probe_jtbl/verdict.md`) L530 - **§3-The** — mechanism: game-code dedup is SOURCE-LEVEL, not an object swap (R-D1, the key lesson) L926 @@ -566,8 +576,12 @@ - **§146** — RE-MEASURE A WALL BEFORE YOU RESPECT IT. Both "permanent" giants fell to drafts already on disk. (P30 S6, +50,094 ins) L9967 - **§147** — The three-stratum FRAME LAW, and four "stop searching" verdicts (P30 S42, `func_8017C294`, serial run) L10026 - **§3-C.** — Inner-block declaration does NOT delay slot allocation — BYTE-REFUTED L10058 +- **§3-D.** — "Cheap fuel" that was never probed: 0 of 31 templatable L10283 +- **§3-Two** — corrections to the record L10335 +- **§3-The** — two fallouts, and how to close them (both measured, in order) L10385 +- **What** — does NOT work (14 byte-measured probes) L10476 -### (unbucketed — title matched no symptom vocabulary) (122) +### (unbucketed — title matched no symptom vocabulary) (140) - **§3-How** — to use this L30 - **§1** — Idiom catalog (asm pattern → C that produces it) L39 @@ -689,8 +703,26 @@ - **§144** — THE LITERAL'S SPELLING PICKS THE IMMEDIATE ENCODING (P30 S40 wave 1, `func_801822E0`) L9878 - **§3-Why** — a correct draft can read as an intrinsic wall L9978 - **§3-The** — rule L10013 -- **§3-B.** — `(v < 0x40) ? v : 0x3F` is folded to `MIN_EXPR` and expands to the WRONG SHAPE L10123 -- **§3-D.** — Reproduce the original's BUGS verbatim L10147 +- **§3-B.** — `(v < 0x40) ? v : 0x3F` is folded to `MIN_EXPR` and expands to the WRONG SHAPE L10160 +- **§3-D.** — Reproduce the original's BUGS verbatim L10184 +- **§149** — Four instrument defects in one session, and the two questions they were hiding (P30 S43) L10233 +- **§3-A.** — A prep step that returns its input on failure is indistinguishable from a search that found nothing L10239 +- **§3-B.** — Same address + same name ≠ same body — and the ledger keys on address L10257 +- **§3-C.** — `make: *** [...] Error N` is a summary, never a diagnosis L10273 +- **§3-The** — fix L10309 +- **§3-The** — method that found it (this is the transferable part) L10319 +- **Diagnostic** — order (adopt this) L10346 +- **§3-The** — mechanism (read from cc1's own `-dR` trace, not inferred) L10362 +- **§3-The** — lever — a zero-emission insn that absorbs the blocked tick L10377 +- **§3-The** — finding L10413 +- **§3-The** — key L10422 +- **§3-Two** — cautions that must travel with this technique L10433 +- **§3-The** — companion defect (open) L10444 +- **Symptom** — Symptom L10464 +- **Mechanism** — (gcc source + RTL dumps, not inferred) L10469 +- **§3-The** — cure — a fresh launder per site, each in its own block L10482 +- **Companion** — levers from the same function L10492 +- **§3-B.** — Two static base-derivation methods that must AGREE (use both) L10526 ## All sections, in order @@ -1114,8 +1146,38 @@ - **§3-D.** — A lone `$t8`/`$t9` in the target is RELOAD SCRATCH — reproduce the spill, don't pin the register L10063 - **§3-E.** — A `qty_compare` TIE is not spelling-reachable — recognise it and stop L10073 - **Consequence** — for the family (a real scheduling decision) L10085 -- **§148** — The loop.c hoisting THRESHOLD is arithmetic you can compute, and the `?:` clamp that folds to MIN_EXPR (P30 S42, `func_8017C6F4`, 947 ins) L10098 -- **§3-A.** — `move_movables` hoists iff `threshold × savings × lifetime ≥ insn_count` — and you can read it L10104 -- **§3-B.** — `(v < 0x40) ? v : 0x3F` is folded to `MIN_EXPR` and expands to the WRONG SHAPE L10123 -- **§3-C.** — A zero-byte ALLOCNO-PRIORITY slider L10137 -- **§3-D.** — Reproduce the original's BUGS verbatim L10147 +- **§148** — The loop.c hoisting THRESHOLD is arithmetic you can compute, and the `?:` clamp that folds to MIN_EXPR (P30 S42, `func_8017C6F4`, 947 ins) L10135 +- **§3-A.** — `move_movables` hoists iff `threshold × savings × lifetime ≥ insn_count` — and you can read it L10141 +- **§3-B.** — `(v < 0x40) ? v : 0x3F` is folded to `MIN_EXPR` and expands to the WRONG SHAPE L10160 +- **§3-C.** — A zero-byte ALLOCNO-PRIORITY slider L10174 +- **§3-D.** — Reproduce the original's BUGS verbatim L10184 +- **§149** — Four instrument defects in one session, and the two questions they were hiding (P30 S43) L10233 +- **§3-A.** — A prep step that returns its input on failure is indistinguishable from a search that found nothing L10239 +- **§3-B.** — Same address + same name ≠ same body — and the ledger keys on address L10257 +- **§3-C.** — `make: *** [...] Error N` is a summary, never a diagnosis L10273 +- **§3-D.** — "Cheap fuel" that was never probed: 0 of 31 templatable L10283 +- **§150** — A register ROTATION across symmetric blocks is VARIABLE-IDENTITY evidence, not an allocator tie (P30 S43, `func_8017C6F4`, 947 ins ×4) L10302 +- **§3-The** — fix L10309 +- **§3-The** — method that found it (this is the transferable part) L10319 +- **§3-Two** — corrections to the record L10335 +- **Diagnostic** — order (adopt this) L10346 +- **§151** — THE GHOST WEDGE: when a load-before-store transposition is unreachable by ANY statement order (P30 S43, `func_8017EF68`, 969 ins) L10357 +- **§3-The** — mechanism (read from cc1's own `-dR` trace, not inferred) L10362 +- **§3-The** — lever — a zero-emission insn that absorbs the blocked tick L10377 +- **§3-The** — two fallouts, and how to close them (both measured, in order) L10385 +- **When** — to reach for it L10397 +- **§152** — BYTE SIZE is the family key that name- and h_seq-grouping both miss (P30 S43, the 0xECC family: 1 crack → 12 overlays → 11,364 ins) L10408 +- **§3-The** — finding L10413 +- **§3-The** — key L10422 +- **§3-Two** — cautions that must travel with this technique L10433 +- **§3-The** — companion defect (open) L10444 +- **§153** — THE ADDRESS-REMATERIALISATION LAUNDER: a third zero-emission asm lever (P30 S43, `func_8018D98C`, 710 ins) L10456 +- **Symptom** — Symptom L10464 +- **Mechanism** — (gcc source + RTL dumps, not inferred) L10469 +- **What** — does NOT work (14 byte-measured probes) L10476 +- **§3-The** — cure — a fresh launder per site, each in its own block L10482 +- **Companion** — levers from the same function L10492 +- **§154** — Reading a disc payload: the module-id word, static base derivation, and "type 1 = uncompressed overlay" (P30 S44) L10512 +- **§3-A.** — Payload word0 is a global MODULE ID; code starts after the header L10517 +- **§3-B.** — Two static base-derivation methods that must AGREE (use both) L10526 +- **§3-C.** — PAC type 1 = the same payload class as type 4, just NOT compressed L10538 diff --git a/docs/decision-log.md b/docs/decision-log.md index b50ddab53..2164b3c77 100644 --- a/docs/decision-log.md +++ b/docs/decision-log.md @@ -2204,3 +2204,38 @@ contract, and this makes it enforceable rather than remembered. **Sequencing (Drew's call):** finish the serial crack queue → L1+L2 (cheap, deterministic, and they sharpen L3's target list) → L3 + type-1 onboarding. Fold into **P31**, which already owns bucket T. + +## 2026-08-06 (P30 S44) — the 78-payload campaign: static addresses dissolve the emulator dependency; "modules" mostly dissolve into overlays + +**Context + belief.** `make audit-disc` (S43) enumerated 78 unclaimed code payloads (~3.4 MB). Standing +doctrine (`disc-completeness.md`, from P27): these are "type-1 modules" whose load addresses are "only +knowable by runtime RE" — so onboarding was gated on an emulator session (L3), and the completion +contract carried them as a 39-module backlog. + +**What the measurement said (3 read-only agents, byte-verified).** (1) The load addresses are STATIC +for 46 of 78: the EXE's `loadDestPtrTable` + boot literals + two index tables inside the resident + +`resident.c:641` + the SC07 pair's own headers give every MAIN payload and the SC07 pair a derived +address, corroborated by two independent corpus-side voting methods at ~500:1 margins +(`memory-map.md` §S44). (2) The three biggest "modules" are ORDINARY OVERLAYS stored uncompressed +(type 1 = raw overlay, type 4 = LZSS) for the standard 0x80128158 slot — ~75–77% of their functions +h_exact-identical to the onboarded corpus, 802 genuinely novel across all three. (3) The remainder +tiers honestly: 35 small actor modules at two statically-known ping-pong slots; SC07/3+4 at their own +slot; 28 script modules (7 × 4 per-disc builds) + 4 stragglers genuinely runtime-determined. + +**The pivot.** L3 shrinks from "the onboarding prerequisite" to a small runtime-confirm pass (28+4 +payloads + R34 verification of the static addresses). The campaign inverts: tooling updates → onboard +the big 3 through the EXISTING overlay machinery → dedup-bank the h_exact majority → batch the small +modules — all emulator-free. The "new binary class" tooling burden collapses to: `config/modules.mk`, +a de-ov_'d R36 gate, a vram-derived `family_remap`, glob widenings, and a parameterized +`new_binary.sh`. Full per-tool table: `tooling-audit.md` §S44. + +**Why this was missable for 30 phases.** Each prior tool was correct about its subset and silent about +the rest (the audit's founding observation) — and the doctrine layer had the same shape: the P27 +"only knowable by runtime RE" sentence was true of the tools that existed then, and nobody re-derived +it after the loader cluster was matched (the tables were sitting in matched C + the resident's own +bytes). A confident negative doctrine is a claim like any other — date it, cite its evidence, +re-measure before letting it gate a campaign (the §146/§147 lesson at doctrine scale). + +**Hindsight better path.** When Phase 3 T5 wrote "entries [1]+ are runtime-indexed (no static xref)", +the honest follow-up was a named open question ("WHERE do the indices live?") rather than a doctrine. +The answer was one grep away once the resident was matched in Phase 12. diff --git a/docs/disc-completeness.md b/docs/disc-completeness.md index 3b5e82c43..e9ca1a3e4 100644 --- a/docs/disc-completeness.md +++ b/docs/disc-completeness.md @@ -44,6 +44,20 @@ loads at its own address, the way the resident loads at `0x800CEDF8`. So — unl that address is only knowable by runtime RE (a PCSX-Redux RAM-dump proof, the Phase-3 method). Onboarding them is a Gen2 RE task, deferred with this evidence — NOT a false "complete" while code sits unbuilt. +> **⚠️ 2026-08-06 (S44): the "only knowable by runtime RE" sentence above is REFUTED.** The load +> addresses are **static** for 46 of the 78 unclaimed payloads: the EXE's `loadDestPtrTable` +> (0x80072C70) + the boot loaders' literal `&cdFileLocTable[k]` operands + two index tables INSIDE the +> resident (`D_800D3764` → slot A 0x800CAE08 for MAIN/13…41; `D_800D384C` → slot B 0x800CCB1C for +> MAIN/42…47) + `src/resident/resident.c:641` (MAIN/12 → the standard overlay slot 0x80128158) + the +> SC07 pair's own headers (→ 0x801A00D8). Full routing table with provenance: +> **`docs/memory-map.md` §"Phase 30 S44"**. Independently corroborated by h_exact base voting (~500:1) +> and jal-alignment voting. The genuinely runtime-only remainder is the 28 SC0x script modules + +> MAIN/7, MAIN/9, SC02/9 — parked for L3 with evidence. Additional corrections from the same pass: +> the three biggest "modules" (MAIN/12, SC02/37, SC03/107) are **ordinary overlays stored uncompressed** +> (PAC type 1 = raw overlay, type 4 = LZSS overlay); the 78 ledger rows sum **3,406,325 B** (the +> bucket's 3,564,021 additionally counts PAC headers); `MAIN/7` is a raw file (`FILE_007`, not +> PAC-wrapped); `MAIN/0 ≡ MAIN/1` byte-identical; payload word0 is a global module id (resident=0x36). + ## Consequence for the completion contract (roadmap §1) The contract's binary count is **no longer "136"**. Two corrections: diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 00dc80b4f..7e349aaef 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -10506,3 +10506,41 @@ exist, gate the PLAIN one first. **Symptom lines for the index:** **"an extra `sw $sN` in the prologue"** · **"`la $sN,SYM` + moves where the target rematerialises"** · **"an address argument used twice in one block"** · **"a hoist no respelling reaches"**. + +--- + +## §154 — Reading a disc payload: the module-id word, static base derivation, and "type 1 = uncompressed overlay" (P30 S44) + +Not codegen — payload forensics. Three laws from the 78-unclaimed-payload analysis, each of which +turns a former "needs the emulator" into a static read. + +### A. Payload word0 is a global MODULE ID; code starts after the header +75 of 78 unclaimed payloads begin with a small LE integer forming one dense id space across all discs +(0x13…0x73; the resident is 0x36). Some follow it with a function-pointer table (SC07/3: table to +0xF8; SC07/4: to 0x154) before code. **Consequences:** `sig_image --bootstrap` returns **0 functions** +on any of them if run from offset 0 (its linear partition hits the header, finds no `jr $ra`, stops) — +**always pass `--text-lo` past the header**; and a header's own pointer table dates the base for free +(first table target − first prologue file offset = base). Only payloads that begin directly with code +(a `27bdffe8`-class prologue at offset 0) may be signed bare. + +### B. Two static base-derivation methods that must AGREE (use both) +1. **h_exact voting:** sign the payload at ANY nominal base; for every function h_exact-identical to a + corpus function, `delta = corpus_addr − signed_addr` votes for the true base. On real overlays the + margin is decisive (~500:1 — 218,454 votes vs a 414 runner-up). +2. **jal-alignment voting:** collect distinct internal `jal` targets; the base under which the most + land on actual prologue file offsets wins. Corpus-independent; the control (the resident) reproduces + its known 0x800CEDF8 and ends 4 bytes under the overlay slot. +A payload where the two disagree, or where votes are thin (script modules: 3–14 aligned jals, calls +almost all outward), is **loader-determined** — park it for runtime confirm rather than guessing (P9). +And check the LOADER first: the EXE's `loadDestPtrTable` + the resident's index tables route most +payloads statically (`memory-map.md` §S44) — the vote is then the R34 cross-check, not the source. + +### C. PAC type 1 = the same payload class as type 4, just NOT compressed +The three biggest "mystery modules" were ordinary location overlays for the standard 0x80128158 slot, +stored raw. Their first 192 bytes are byte-identical to built ov_ images; ~75% of their functions are +h_exact-identical to the corpus. **Before inventing a new class for a payload, diff its head against +the classes you already own.** (Corollary of §152: same-bytes is the family key — here at payload +scale.) + +**Symptom lines for the index:** **"sig_image bootstrap finds 0 functions"** · **"a payload with a +small integer first word"** · **"where does this blob load"** · **"a huge type-1 module"**. diff --git a/docs/memory-map.md b/docs/memory-map.md index 9cdd31593..e9e7e20f5 100644 --- a/docs/memory-map.md +++ b/docs/memory-map.md @@ -568,3 +568,59 @@ in retail too (shipped data, not a debug build). The proto's scene-select shares | gamehacking.org #88529 (US) / #93476 (JP) via libretro-database GameShark `.cht` | Player stat block, flags, misc | gamehacking.org Cloudflare-blocks scripts; cht mirror: `raw.githubusercontent.com/libretro/libretro-database/master/cht/Sony%20-%20PlayStation/Brave%20Fencer%20Musashi%20(USA,%20Japan)%20(GameShark).cht` | | jywjyw `bravefencer-hack` `doc/note.md` | JP overlay/memory map, LIST.CD-in-RAM behavior, pointer table | **All addresses JP (SLPS-01490)** — re-derive for US | | Hidden Palace / archive.org | Prototype facts | pages fetchable via `hiddenpalace.org/w/index.php?title=PAGE&action=raw` | + +## Phase 30 S44 — the COMPLETE loader routing table (static-derived; supersedes "runtime-indexed, no static xref") + +> **Provenance (G5):** `static-derived` — read from the EXE bytes (`extracted/retail/SLUS_007.26`, +> vram = fileoff + 0x8000F800), the resident payload bytes (`MAIN.CD.dir/FILE_010.dir/1.1`, fileoff = +> vram − 0x800CEDF8), the matched loader C (`src/800.c`, `src/resident/resident.c`), and the per-overlay +> wrapper asm — by 3 read-only exploration agents, 2026-08-06. Region: **US**. The Phase-3 T5 note +> "entries [1]+ are runtime-indexed (no static xref)" is **superseded**: the indices ARE static, they +> live in the resident and in each overlay, not in the EXE. Independently corroborated by two +> corpus-side methods (h_exact base voting at ~500:1; distinct-jal→prologue alignment voting) — and the +> resident control reproduces its known 0x800CEDF8 and ends at 0x80128154, four bytes under the overlay slot. + +### loadDestPtrTable — 0x80072C70 (EXE fileoff 0x63470), 5 × u32 + +| slot | value | role (byte-proven) | +|---|---|---| +| [0] | **0x800CEDF8** | resident-module slot (boot loaders) | +| [1] | **0x80128158** | location-overlay slot | +| [2] | **0x800CAE08** | module slot A (small actor modules) | +| [3] | **0x800CCB1C** | module slot B (small actor modules) | +| [4] | **0x800C7F08** | PAC-type-7 fixed destination | + +### Who loads what where (all statically enumerated) + +| loader | index source | payloads | dest | +|---|---|---|---| +| 5 boot loaders (literal `&cdFileLocTable[k]` at 0x80010CA4 / 0x80010F1C / 0x800112F0 / 0x80011100 / 0x80011144) | k ∈ {1,3,8,10,11} | MAIN/1,3,8,**10 (=resident)**,11 | `loadDestPtrTable[0]` = 0x800CEDF8 | +| resident `func_800D02D0` | **`D_800D3764`** = 29 × {u32 cdFileLocIdx; u32 param} | MAIN/13…41 (contiguous) | `[2]` = 0x800CAE08 | +| resident `func_800D0488` | **`D_800D384C`** = 6 × {u32,u32} | MAIN/42…47 | `[3]` = 0x800CCB1C | +| resident `func_800CF94C` (`src/resident/resident.c:641`) | `&cdFileLocTable[12]` | MAIN/12 (an UNCOMPRESSED overlay) | `[1]` = 0x80128158 | +| per-overlay wrapper `func_80128CFC` (every overlay) | per-overlay `IDXTAB` (s16, −1-terminated, 37 entries, same list fleet-wide) + `*DESTPTR` (per-overlay initialized word) | resources incl. the SC0x sets | per-overlay dest (e.g. ov_SC01_000: IDXTAB 0x8017EEC8, *0x801A3234 = 0x801A58E8) | +| SC07 endgame pair | header-derived (id word + fn-ptr table; first table target − first prologue fileoff) | SC07/3 (code@0xFC), SC07/4 (code@0x158) | **0x801A00D8** (own slot, overlaps the overlay tail — disc-7 layout) | + +### The arithmetic + +- **Global cdFileLocTable index** = `gbase[cd] + subfile`; gbase = MAIN:0 SC01:49 SC02:135 SC03:178 + SC04:318 SC05:349 SC06:379 SC07:418 (LIST.CD counts 49/86/43/140/31/30/39/29, byte-verified; LIST.CD + carries **LBA + length only**, never load addresses). +- **Slot adjacency proof:** 0x800CAE08 + 7,444 (max slot-A payload, MAIN/34) = 0x800CCB1C; + 0x800CCB1C + 8,920 (max slot-B, MAIN/44) = 0x800CEDF4 → resident at 0x800CEDF8. The three regions are + back-to-back, each sized to its largest member. MAIN/46's self-calls (base+0x724/0x978/0xAB0) confirm slot B. +- **Module-ID law:** payload **word0 is a global module id** (dense 0x13…0x73 across all discs; the + resident is 0x36). 75/78 unclaimed payloads carry it; only the 3 raw uncompressed overlays + (MAIN/12, SC02/37, SC03/107) start directly with code. MAIN/9 and MAIN/39 both carry id 0x2D + (unresolved duplicate). MAIN/0 ≡ MAIN/1 byte-identical (one module stored twice). +- **PAC-type law (extends formats.md):** type **1** = uncompressed code/module payload; type **4** = + the same class LZSS-compressed. The PAC header's bytes 0x10–0x7FF are never read by the loader + (`CdGetSector(lzss_sectorStagingBuf, 4)` reads 4 words) — no address lives in the payload. + +### Statically UNRESOLVED (parked for L3 — runtime confirm, R34) + +The 28 SC0x script modules (SC03/73-79, SC03/132-138, SC04/24-30, SC05/23-29 = 7 modules × 4 per-disc +builds), SC02/9, MAIN/7 (raw file, not PAC), MAIN/9: dest comes through the resourceIdMap / +`StreamLoadStateMachine` descriptor path (`D_80068B60[(loadParam−0x100)*0x10]`) or per-overlay DESTPTR +values — per-disc, not EXE-static. Their jal-vote bases are LOW-CONFIDENCE (3–14 aligned jals, calls +almost entirely outward) and are NOT recorded as addresses here. diff --git a/docs/tooling-audit.md b/docs/tooling-audit.md index 6d404aad2..e1639f5a0 100644 --- a/docs/tooling-audit.md +++ b/docs/tooling-audit.md @@ -1693,3 +1693,101 @@ by a build from stale objects. Cheap, total, and it removes the need to remember **Assertion (R32):** after `build`, assert every `.o` linked into the image is NEWER than every `.s` it includes; fail loud on the first inversion. A build that consumed a stale object must never be allowed to report BYTE-IDENTICAL. + +--- + +# S44 NEWCODE AUDIT (2026-08-06) — every tool vs the 78 unclaimed payloads + +> Drew's directive: *"analyze every single one of our tools and determine how/if it needs to be +> updated to properly account for our new code findings."* Ground truth: 3 read-only exploration +> agents over the full inventory (117 `tools/*.py`, 13 `tools/*.sh`, `tools/bfm_extract/` ×11, +> `tools/ghidra_scripts/` ×11, `tools/workflows/` ×6, `tools/permuter/`, `diff_settings.py`, +> `Makefile`, the config registries). Vendored submodules out of scope. +> +> **The class mostly DISSOLVES:** the 3 big payloads are ordinary overlays (standard slot, existing +> machinery); only the small modules + the SC07 pair need a genuinely new binary class ("md_*", +> `config/modules.mk`). Classification: **(a)** parameterized per-binary · **(b)** derives the binary +> set from configs · **(c)** hardcodes overlay shape · **(d)** binary-agnostic/N-A. + +## Registration surfaces (the choke points) + +| surface | class | point | verdict | +|---|---|---|---| +| `Makefile` | b | `:55` BINARIES; prune `:511`; check-all `:753` | **CODE**: `-include config/modules.mk`, `+ $(MODULE_BINARIES)`; downstream of `$(BINARIES)` auto-OK | +| `config/overlays.mk` | b | generated registry | **NEW SIBLING** `config/modules.mk`, same 18-var block, per-alias VRAM | +| `tools/dup_report.py` BINARIES | b | `:26-180` (sentinel `:167`) | registration line/binary (non-ov_ aliases already take the individual-ingest path `:210`) | +| `tools/progress.py` BINARIES | b | `:23-305` (sentinel `:304`) | registration line/binary | +| `diff_settings.py` BINARIES | b | `:16-437` (sentinel `:437`) | registration line/binary | +| `tools/audit_binaries.py` (R36) | **c** | `onboarded()` `:41-43` globs `splat.ov_*.yaml`; `startswith("ov_")` `:92,:111,:124,:131` | **CODE**: derive from `splat.*.yaml` minus main (R33); keep engine_core-include check ov_-conditional | +| `tools/corpus.py` | b | `:373` from dup_report; `sig_is_independent` `:336` | **CODE** at `:336` (ov_/resident-only ⇒ module sigs untrusted); rest auto-OK | +| `tools/difficulty.py` | b→derived | `cfg_for(alias)` `:22-34` | **auto-OK** (P27 T6 migration) | + +## Must-change (code) + +| tool | defect | fix | +|---|---|---| +| `family_remap.py` | `VRAM = 0x80128158` module const `:30`, used in ALL offset math `:98,:160` (img_path is already yaml-derived) | `vram_base_of(alias)` from `config/splat..yaml` — the pattern `jtbl_carve.overlay_vram_base()` `:65-71` already implements | +| Makefile sig targets | `sig-overlays` hardcodes `OVERLAY_VRAM :=0x80128158` `:292`; `sig-resident` separate | one generalized target over `$(filter-out main,$(BINARIES))` with `$($(a)_VRAM_BASE)` (+ per-alias TEXT_LO); keep old names as aliases | +| `family_hseq.py` | `src/ov_*` `:43` + `sig.ov_*` `:45` globs; self-declared overlays-only `:189,:219` | include resident+modules (glob `sig.*.jsonl` minus main, or read registries) | +| `progress.py --weighted` | `sig.ov_*` glob `:647` + explicit resident `:648` | derive from BINARIES | +| `audit_frontier.py` | `:57-59` same glob shape | same fix | +| `backlog.py` | alias regex `:137` `(ov_…|resident|main)` | add `md_…` | +| `prefetch_fleet.py` | `:67` `("main","resident")` | add modules | +| `dedup_propagate.py` | reads only `overlays.mk` `:44` | also read `modules.mk` | + +## Retire (R33) + +`disc_code_sweep.py` — superseded by `disc_audit.py` (whole-disc partition, both layers, no window, +claims); zero build refs (Makefile mentions it only in a comment); doc refs to update: +`docs/SETUP.md:667`, `docs/disc-completeness.md` · `reconcile_decls.py` — self-declared RETIRED · +`rollout_801457a4_o0.py`, `rollout_whale_o0.py`, `rollout_o0_cluster.py` — one-shot, slot-locked +historical rollouts · `ghidra_scripts/ImportOverlay.java` + `VerifyOverlay.java` — 1-overlay-era +hardcoded tables (`ghidra_import_raw.sh` is the live path). + +## `new_overlay.sh` → `tools/new_binary.sh` + +Overlay-specific: alias pattern `:29`, payload path `${ENTRY}.dec` `:30`, `VRAM=` `:31`, slot literals +re-hardcoded at `:39,:44`, the overlay splat template. **Generic and reusable verbatim:** check.sha + +symbols creation, the 18-var mk block, the sentinel-anchored 3-dict registrar `:104-133` (ast-checked), +extract+build byte-check. ⇒ parameterize {ALIAS, PAYLOAD, VRAM, TEXT_LO, TEMPLATE, REGISTRY}; +`new_overlay.sh` becomes a wrapper with the old defaults. + +## `sig_image.py` — no code change; a USAGE law + +`--bootstrap` linear-partitions from `lo = vram_base` (`:232`, `bootstrap_seeds` `:81-98`) ⇒ assumes +code at file offset 0. 75/78 payloads start with the module-id word (+ sometimes a ptr table) ⇒ 0 +seeds. **Law: pass `--text-lo` past the header** (prologue offsets are in the disc-ledger roster). +`--seeds` accepts a sig jsonl or 0xADDR lines (`:47-59`). + +## Auto-OK once registered — (a) parameterized / (b) derived + +`gate_stage` · `harvest_verify` · `match_one` · `rtu_match` · `masked_diff`/`masked_scorer` · +`family_sweep` (cross-address `--to-addr` EXISTS: `:332-343,:537`) · `family_manifest`(glob fix rides +family_hseq) · `dedup_extend` · `dedup_integrate` · `jtbl_carve` (the model implementation) · +`jtbl_family_bank` · `jr_isolate`/`jr_isolate_all` · `o0_subsplit` · `overlay_src_split` · +`split_src_region` · `blast_radius` (derives from `splat.*.yaml` — best-in-class) · `worklist` · +`exemplar_miner` · `diff_regions` · `lift_types` · `build_engine_types` · `uniquify_type` · +`canon_sig_reconcile` · `recover_giant` · `recover_integration` · `fix_arity_callers` · +`fix_header_decl` · `cast_call_sites` · `sig_unify` · `reconcile_tu` · `canon_draft_decls` · +`canon_resident_calls` · `inject_capped_externs` · `scope_tu_externs` · `scope_data_externs` · +`normalize_self_decls` · `conform_decls` · `blocker_probe` · `demacroize` · `autopsy` · +`residual_class` · `bank_exemplar` · `t7_bank` · `sweep_parallel` · `bulk_harvest` (alias side via +`lora_grind.binaries()` = check-sha glob) · `lora_grind` (`binaries()` auto-OK; reach-glob rides the +hseq fix) · `gen_harvest_targets`(same) · `build_fuel_manifest`(same) · `wave_targets` · +`build_wave_args` · `idiom_loop` · `audit_digest` (via progress.BINARIES) · `lint_symbol_refs` · +`cookbook_index` · `symcheck` · `burndown` · `p16_permute`/`permuter_ils`/`permuter_weights`/ +`p16_improve`/`p16_known_answer` · `grinder` · `auto_driver` · workflows (`worker_wave.js` etc. — +aliases are prompt args) · `glm_reconcile`. + +## N-A (binary-agnostic) + +Extract stack (`bfm_extract/*` — already extracted the payloads) · PsyQ linking (`psyq_*`, +`gen_lib_subsegs`, `ld_interleave`, `make_*_used`, `make_libgs.sh`, `jtbl_rodata_pads`) · LLM tier +(`serve_local`, `api_draft`, `train_lora`, `eval_lora`, `format_finetune`, `export_pairs`, +`idiom_hunt`, `glm_parallel.sh`, `orchestrator`) · permuter internals (`run_masked`, `compile*.sh`) · +Ghidra plumbing (`ghidra_import.sh`, `ghidra_import_raw.sh` — the live module importer, +`ghidra_mcp_*.sh`, the .java scripts except the two retired above) · automation shell +(`auto_status/stop/supervisor.sh`, `treelock.sh`) · `decompile.py` · `match_protos.py` · +`ram_probe.py` · `audit_text_sources.py` · `sweep_citations.py` · `ab_match.js`/`ab_score.py` · +`disc_audit.py` (the new oracle itself; its `claimed-by` derives from `config/check.*.sha`, so newly +onboarded binaries flip to claimed with ZERO wiring). diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 58884c0b2..c855c0832 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -2527,6 +2527,23 @@ and is booked as data. **Not in `tools-health`** — it needs `disks/`, which a **Next (task #11 / L3):** the emulator tour resolves whether these 34 are the 39 type-1 modules, gives their load addresses, and proves completeness against execution. +### ▶ S44-I.0 — the new-code campaign opens: knowledge captured (2026-08-06, Fable5Max plan approved) +Plan: `~/.claude/plans/optimized-squishing-engelbart.md` (Part I this session, Part II fresh). Three +exploration agents broke the 78-payload class open; everything captured while hot (R30/R31): +- **`docs/memory-map.md` §"Phase 30 S44"** — the COMPLETE static loader routing table + (`loadDestPtrTable` slots · boot k-set {1,3,8,10,11} · resident tables `D_800D3764`/`D_800D384C` → + slots A/B 0x800CAE08/0x800CCB1C · MAIN/12 → 0x80128158 · SC07 pair → 0x801A00D8 · gbase arithmetic · + slot-adjacency proof · module-id law · "type 1 = uncompressed overlay"). Supersedes P3-T5's + "runtime-indexed, no static xref" and P27's "only knowable by runtime RE" + (`disc-completeness.md` corrected in place, H5). +- **`docs/tooling-audit.md` §S44** — EVERY tool classified (a/b/c/d) with file:line: 8 code-changes, + 7 registrations, 5 retirements, the rest auto-OK/N-A. The "every single tool" deliverable. +- **`docs/decision-log.md`** — the R31 pivot entry (why L3 shrank; why the doctrine was missable: + a confident negative doctrine is a claim like any other — the §146/§147 lesson at doctrine scale). +- **Cookbook §154** (+ index regen, 454 sections): module-id word / dual base-voting methods / + type-1-is-an-overlay. Key numbers: big-3 base 0x80128158 at ~500:1; ~75-77% h_exact-known; + **802 novel fns**; 46/78 addresses static; 28+4 parked for L3. + ### ▶ S11 — the propagation lag: EXTEND 0/36 -> 31/36, and every blocker was a DECLARATION (2026-08-03/04) Lane 2 of the S10 checkpoint ("26,006 ins, ~0 agent tokens, PARTLY BLOCKED"), taken first on the standing doctrine that the cheap deterministic lever is probed before the expensive agent one.