mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 22:45:39 -04:00
49db81d8ae
- reports (Task 3): tools/{progress,difficulty,dup_report}.py + make report/sig-refresh
(deterministic digests); harvest to 43 REAL matches incl. LzssDecodeSector
- per-module -O0 split (Task 5.5): src/boot.c + Makefile per-file CC1FLAGS (cookbook §6)
- PsyQ SDK linking (sessions C-G): split/build/identify/link/integrate pipeline
(tools/psyq_*.py, make_libgs.sh, ld_interleave.py, split_src_region.py) — libcd (18 obj)
+ FULL libgs (31 obj / 6 blocks) linked BYTE-IDENTICAL from real PsyQ 4.0 libs; byte-
identical with or without the SDK objects (stub fallback); cookbook §7/§8/§9.1-9.5
- LZSS MATCHED byte-for-byte (session F): §5a cross-jump barrier + §10 regalloc/schedule
idioms, ground-truthed against pinned gcc-2.7.2 source (the Gen1 LZSS gate)
- file-loader cluster: 6 matched + 5 NON_MATCHING-drafted (CdReadStateMachine,
CdReadSectorReadyCB, StreamLoadStateMachine new this session, valid under -DNON_MATCHING);
SaveLoadRoutine DEFERRED to Q#5 (save/memcard, documented stub)
- README Gen1-complete; worklog preserved -> phase-ends/logs/Phase7.md
- rules R17 (web-research compiler quirks), R18 (plain-English session summaries),
R19 (preserve phase worklog out of load order)
- MILESTONE: Gen1 EXIT confirmed — byte-identical build, 43 matches incl LZSS, 7 zero-
regression sessions, loader cluster matched/drafted, overlay map emulator-proven
- bumps project version 1.6.0 -> 1.7.0
202 lines
28 KiB
Markdown
202 lines
28 KiB
Markdown
# CURRENT PHASE — Phase 7: Matching loop at scale + Gen1 close-out
|
||
|
||
**Status:** Plan APPROVED 2026-06-14. **REORDERED 2026-06-14** (Drew): bank non-switch wins first; the
|
||
rodata-island foundation + LZSS match are DEFERRED to a focused sub-project after the easier tasks.
|
||
**Full plan:** `~/.claude/plans/confirmed-plan-mode-enabled-jolly-lemon.md`. **Project version:** 1.6.0 → 1.7.0 at PhaseEnd.
|
||
|
||
> In-phase crash/compaction recovery log (P3 step 4). Update after every task.
|
||
|
||
---
|
||
|
||
## Locked decisions (developer-confirmed)
|
||
- **≥25 = real substantive matches** (the 42 splat-auto empties do NOT count). 14 real now → need ≥11 more.
|
||
- **Loader cluster = match-tractable / draft-hard** (NON_MATCHING-draft the hard state machines).
|
||
- **REORDER (2026-06-14):** rodata foundation hit a structural wall (see below); do reports + harvest + non-switch loader FIRST, then a focused LZSS/rodata sub-project. LZSS is still required for Gen1 exit.
|
||
- **⚠️ PHASEEND AFTER LIBGS (Drew 2026-06-14):** do NOT write PhaseEnd_Phase7 / delete this CURRENT_PHASE.md until **libgs is done**. This file holds the libgs placement + working notes (see the session-E "libgs placement" block) and must stay intact so libgs work can resume. Sequence: LZSS → libgs → Task 6 close-out → (only then) Task 7 PhaseEnd. Task graph enforces it (#8 blocked by #9).
|
||
|
||
## Verified baseline (grounded; R14 corrections)
|
||
- Build byte-identical (`143dbb89f34491258bbc27810d0a12ec8b43a8dd`), reproducible. **Only change from committed Phase-6 = one R15 symbol line** (`func_80047CAC = 0x80047CAC; // data`) — fixes a LATENT NON-REPRODUCIBILITY: spimdisasm 1.41.0 auto-detection of that 8-byte inter-fn blob is unstable across clean extracts; declaring it makes `make clean && make extract && make build` deterministic. (Note for PhaseEnd.)
|
||
- Switch cohort ~46 funcs / 53 tables (51 in-image, single-ref/migratable). LzssDecodeSector 0x80018730 = gcc switch via `jtbl_80072A38` (write-free rodata island 0x80072A38–0x80074750 / file 0x63238–0x64F50).
|
||
- All 13 loader-cluster funcs in text region (< 0x800629DC). Authoritative counts: 14 real / 42 empty / 2 NON_MATCHING / 1998 stubs / 2050 matchable.
|
||
|
||
---
|
||
|
||
## Task checklist (REORDERED — execution order top to bottom)
|
||
- [x] **Task 3 — Report machinery** ✓ DONE. `tools/{progress,difficulty,dup_report}.py` + `make report` (deterministic) + `make sig-refresh`. Authoritative baseline **14 real / 42 empty / 2 NM / 1992 stubs / 2050 matchable** (REAL/matchable=0.68%); empties audit 42/42 clean; dup leverage 8 h_exact + 44 h_norm redundant; harvest queue 1119 non-jtbl leaves + 313 trivial. Digests → `docs/{progress,difficulty,duplicates}.md`.
|
||
- [x] **Task 4 — Harvest to ≥25 real** ✓ DONE. **36 real matches** (14 + 22 trivial accessor leaves: getters/setters of globals, one mask, one two-store), build BYTE-IDENTICAL → all 22 byte-perfect. **≥25 bar PASS** (margin 36/25). `.run/harvest.py` = the applier. Dedup-collapse skipped (the sig dup groups are mostly PsyQ library fragments/epilogues, not real funcs). LZSS still required (Task 2′).
|
||
- [x] **Task 5.5 — Per-file -O0 split mechanism** ✓ DONE (session B, Drew-approved). MAJOR FINDING: per-module opt mixing is real (§5.5). The **boot/main/game-mode-dispatch module @ vram 0x80010000–0x800123F0** (~50 funcs incl. start/main/GameModeDispatch/DebugMenuHandler) is **-O0**, not the pinned -O2; everything ≥0x800123F0 is -O2. Mechanism: split text c-subseg → `[0x800,c,boot]→src/boot.c` (-O0 via Makefile target-specific CC1FLAGS) + `[0x2BF0,c,800]→src/800.c` (-O2, matched fns untouched). Regression gate PASS (byte-identical at 100% INCLUDE_ASM). Cookbook §6 added (detect via `21F0A003` frame-ptr sig; `register`-pointer far-member idiom). Reusable for every future flag-divergent module.
|
||
- [x] **Task 5.6 — PsyQ CD struct header** ✓ DONE (session B). `include/psyq/libcd.h` (CdlLOC 4B + CdlFILE 24B, offsets from Ghidra `/LIBCD.H`, G1) + 3 libcd protos; added 4 PsyQ symbols to `config/symbols.us.txt` (CdSearchFile/CdPosToInt/CdIntToPos/VSync — Ghidra-confirmed, R15) and renamed their `func_*` stubs in src/800.c. Build BYTE-IDENTICAL (label-only); header compiles, sizeof asserts hold. Unlocks the file-loader cluster (LoaderInitFileTable, ResourceLoadStateMachine, the jtbl loaders). Cookbook §7 added (PsyQ types/symbols pattern). **NB symbols.us.txt now has 983 lines (4 appended, un-sorted — re-merge note for R15).**
|
||
- [~] **Task 5 — Loader cluster** PARTIAL (42 real total; non-jtbl 5/7 matched). Matched (score 0, byte-identical): **ResourceGetCdLoc** (15), **LoaderResetReadState** (32), **GameModeDispatch** (29, -O0 `register`-ptr far member), **DebugMenuHandler** (30, -O0, reserved-slot local), **CdQueueBusy** (35, -O2 — if/else order + branch polarity), **CdReadRequest** (53, -O2 — early-return fall-through). Triage of the rest:
|
||
- **jtbl → Task 2′ (rodata-gated):** CdReadStateMachine (385), CdReadSectorReadyCB (424, sole LZSS caller), SaveLoadRoutine (1139), StreamLoadStateMachine (459).
|
||
- **non-jtbl COMPLETE (6 matched + 2 drafted):** **LoaderInitFileTable** (133) + **ResourceLoadStateMachine** (211) → **NON_MATCHING-drafted** (logically faithful, libcd.h-typed; per-function residual notes in src/800.c — LoaderInit: regalloc/loop-invariant hoisting + name=base-0x14; ResourceLoad: block placement (state 0→1→2→3 layout vs nested-if) + reserved-local frame 0x38 vs 0x28 + func_8002D4C8 arg & 0xFFFF). **NON_MATCHING count now 4.** Byte-match both in a later structural/permuter pass.
|
||
- **Report tooling fixed for the multi-file split:** `tools/{progress,difficulty}.py` now glob all `src/*.c` + search all `asm/nonmatchings/*` subdirs, and `progress.py` skips `extern …(…);` forward-declarations (they were swallowing the next fn + double-counting). Deterministic; 42 real / 1964 stubs consistent across both. (Note for PhaseEnd.)
|
||
- [x] **Task 2′ — Focused LZSS + rodata-island sub-project** ✓ DONE (session F, 2026-06-15). **LzssDecodeSector MATCHED byte-for-byte** (asm-differ score 0; `make clean && make extract && make build && make check` → `143dbb89… BYTE-IDENTICAL`). The Gen1-exit LZSS gate is satisfied. **43 real matches** (42 + LZSS). The NON_MATCHING guard is DROPPED — the C is the default build. Close required 5 load-bearing constructs (all documented in src/800.c + cookbook §10): (A) decoupled `nh` high-byte var + `(code&0xFF)|(nh<<8)` operand order [local-alloc combine_regs]; (B) `result` set in save predecessors [reorg fill_simple_delay_slots]; (B/state-2) explicit `register $2` local + early read-only input-asm pin [sched list-order]; (B3) NO switch `default`/trailing return → state≥5 reuses the `sltiu` result; (§5a) the zero-byte cross-jump barrier. The surgical rodata carve + ld_interleave place gcc's generated jtbl at 0x80072A38 (links byte-identical). Cookbook §10 added (the reusable regalloc/schedule-tail idioms + the floor-free `.text` metric); §5a cross-linked.
|
||
- [ ] **Task 6 — Gen1-exit close-out** (README, checklist, ≥3-session zero-regression evidence). **xHigh.**
|
||
- [ ] **Task 7 — PhaseEnd_Phase7** (Gen1 synthesis, milestone gate). **Max · Tier 1.**
|
||
|
||
## Current task
|
||
**LZSS DONE (session F, 2026-06-15) — the Gen1-exit LZSS gate is SATISFIED.** `LzssDecodeSector` matches
|
||
byte-for-byte (asm-differ score 0; full clean build `143dbb89… BYTE-IDENTICAL`); the `#ifdef NON_MATCHING`
|
||
guard is dropped — the C is the default build. 43 real matches. Full close detail in the Task 2′ checklist
|
||
entry above + the 5 LOAD-BEARING comments in `src/800.c` + cookbook §10.
|
||
- ✅ **libcd wired into the byte-identical build** (session D) — 58 SDK funcs.
|
||
- ✅ **LZSS matched** (session F) — surgical rodata carve + ld_interleave place gcc's generated switch jtbl at
|
||
0x80072A38 (links byte-identical); the §5a cross-jump barrier + 4 regalloc/schedule idioms (§10) closed it.
|
||
Method: §3a web-research ground-truthed all levers against the pinned gcc-2.7.2 source; a floor-free `.text`
|
||
metric (objcopy `--only-section=.text` + `cmp`, bypasses the rodata/jtbl floor) measured the close; the
|
||
permuter could NOT (score floor + random divergence).
|
||
- ✅ **libgs BLOCK 6 integrated byte-identical** (session F, Drew: "block 6 alone first, then the integration").
|
||
**16 contiguous libgs objects → 26 SDK functions banked** (vram 0x80053AF8–0x80057928), `make clean/extract/
|
||
build/check` → `143dbb89… BYTE-IDENTICAL`, reproducible; fresh-clone stub fallback verified; zero spurious
|
||
warnings. Mechanism (the libcd §9.3 pattern): resegment 800b → [800b pre][libgs6][800b2 post]; regenerate the
|
||
3 split src files; `psyq_integrate libgs6`. **Three new fixes (all in this commit):**
|
||
- **splat instability the resegment exposed:** shrinking 800b shifted spimdisasm's auto-detected boundaries.
|
||
Declared `GsMulCoord2/3` (real MATRIX.o funcs, verified via the PsyQ object — R15, Ghidra-mirror pending);
|
||
and carved the §8 data descriptor table `[0x53198,data,53198]` (boot.c's `D_80062998` was being shadowed by
|
||
a mis-detected `func_80062998`). ld_interleave FRONT_DATA → `53198.data.o`.
|
||
- **psyq_integrate multi-library support:** NOLOAD section names namespaced per-library (`.nl_<tag>_*`) so the
|
||
2nd integration's NOLOAD placement isn't skipped (was discarding OBJT3 .rdata/.bss); trial-link now includes
|
||
already-emitted sibling `*_externals.ld` (kills the "31 UNRESOLVED" false alarm — they're libcd's externals);
|
||
NOLOAD lines globally re-sorted by vram across libraries (kills "dot moved backwards").
|
||
- **reproducibility:** `tools/make_libgs_block6.sh` regenerates the curated 16-object dir (disambiguated:
|
||
GS_131/137 not RVWUNIT/RVWLUNIT, PRESET3/OBJT3 not PRESET2/OBJT2).
|
||
- ✅ **DONE (session G) — full libgs integration.** All **31 libgs objects in 6 contiguous blocks** now linked
|
||
from real PsyQ SDK objects, byte-identical. Blocks 1-5 (15 objects / 29 named funcs) added to block 6's 16
|
||
(20 named funcs) → **≈49 named SDK functions across 31 objects** (count OBJECTS — splat over-segments library
|
||
code into 217 INCLUDE_ASM stubs; session-F's per-block "26" was a looser estimate, superseded). GS_001 stays
|
||
excluded (gsgap3 stub, §9.1). Unified into ONE `psyq_integrate` call (supersedes the block-6-only path).
|
||
Changes: `tools/make_libgs.sh` (31-obj regen dir, **replaces** make_libgs_block6.sh); `psyq_integrate.py`
|
||
gained an optional placement-window arg (GS_106 = 8 ins, only anchors uniquely in 0x80051804-0x80057928);
|
||
`config/splat.us.exe.yaml` 800b resegmented → 800b(pre) + libgs1-5 + gsgap1-5 (+ unchanged libgs6/800b2);
|
||
`Makefile` LIBGS6_* → LIBGS_* (window + 6 stubs); `src/800b.c` trimmed to <0x80051804; **10 new stub src
|
||
files** (libgs1-5.c, gsgap1-5.c). Split DETERMINISTIC across 2 clean extracts; byte-identical WITH or WITHOUT
|
||
the SDK objects (fresh-clone stub fallback). Cookbook §9.5. **Remaining: Task 6 close-out, Task 7 PhaseEnd
|
||
(PhaseEnd gated on libgs per Drew — now UNBLOCKED).**
|
||
NOTE: Gen1 exit needs ≥3 SESSIONS of green `make check` — **satisfied** (A, B, C, D, E, F, +G); Tasks 6/7 pending.
|
||
|
||
### Island ownership map (session E — the decisive finding)
|
||
Rodata island 0x80072A38–0x80074750, 102 jtbls total (~52 in-island). By owner segment:
|
||
- **0x80072A38–0x800734F4 (~35 jtbls): 800-segment GAME code** (targets 0x80018xxx–0x80039xxx; incl.
|
||
`jtbl_80072A38`=LZSS first entry, and S_SCA/SR_SV which resolve to `asm/nonmatchings/800/`). **MIGRATE these.**
|
||
- **0x800737CC–0x800746B0 (~17 jtbls): 800b + LIBRARY** — BIOS(libcd, already integrated), GS_123/PRESET/
|
||
PRESET2/OBJT/OBJT2(libgs), PRNT(libc2), LIBMCRD. **STAY RAW** (untouched flat data → byte-identical).
|
||
- **+24 root cause = 6 library jtbls, ALL in 800b** (PRESET_OBJ_744/8FC, PRESET2_OBJ_4D8/A88, OBJT2_OBJ_614, PRNT_OBJ_24C).
|
||
LZSS+game jtbls are in 800; the only libgs object in 800 (GS_013, 0x8003D40C) has **no jtbl** → 800-only migration has no library TU → no +24.
|
||
|
||
### libgs placement (session E — preserved for the BONUS task #9)
|
||
36/201 located; byte-test disambiguation: **PRESET3** (not PRESET2) @0x80055D40, **OBJT3** (not OBJT2) @0x80057094
|
||
(losers have a spurious .data + .text mismatch); GS_131≡RVWUNIT, GS_137≡RVWLUNIT are .text-identical aliases (keep GS_*).
|
||
GS_106 @0x80053308 fills a gap (narrow-window). GS_013 @0x8003D40C = far outlier in 800 (6 ins). 4 ambiguous
|
||
(GS_101/102/124/125) don't fit gaps → not linked. Main block 0x80051804–0x80057928 (800b), ~4 sub-blocks; gaps 96/48/304 B.
|
||
Patched `tools/psyq_identify.py` to skip data-only objects (no .text, e.g. GLOBAL.o). PRESET3/OBJT3 .rdata land in-island (0x80073c98/0x80073ee8).
|
||
- **libgs BYTE-VERIFIED (session E):** curated `.run/obj40/libgs_used/` (31 objects). `psyq_link_region … --emit .run/libgs_region`
|
||
→ **31/32 byte-identical**, 0 conflicts, 69 externals; `.run/libgs_region.{ld,syms}` emitted. Confirms BFM links real
|
||
PsyQ libgs 4.0 objects byte-for-byte. **GS_001 EXCLUDED** (known-issue: psyq-obj-parser packs scattered PSD* commons
|
||
into `.bss` referenced via `.bss`+offset; 35 words differ at the global-zeroing run — needs per-symbol .bss resolution,
|
||
cookbook §9.1 hard case). Block structure: **6 contiguous blocks + 5 gaps** (80/48/1536[GS_001]/48/304 B, all non-libgs
|
||
→ stay stubs). REMAINING (mechanical): resegment splat 800b into the 6 blocks + 5 gap stubs (~13 subsegs) + split_src +
|
||
wire `psyq_integrate` (6 block stubs) → byte-identical → banks ~63 SDK funcs. Simpler first win = block 6 alone (16 objs
|
||
incl. PRESET3/OBJT3) as [pre][libgs6][post] → ~35 funcs. Deferred at session-E end (long session); clean continuation.
|
||
|
||
## Per-session `make check` green log (≥3 sessions needed for the milestone)
|
||
- 2026-06-14 (session A): `make check` → `143dbb89… BYTE-IDENTICAL` ✓ — baseline restored + reproducibility fix, reports built, **38 real matches** (22 accessor leaves + ResourceGetCdLoc + LoaderResetReadState), build byte-identical throughout. [need ≥2 more sessions]
|
||
- 2026-06-14 (session B): `make check` → `143dbb89… BYTE-IDENTICAL` ✓ — **per-file -O0 split mechanism** (src/boot.c + Makefile per-file flags); **4 real matches** (GameModeDispatch, DebugMenuHandler, CdQueueBusy, CdReadRequest) → **42 real**; **PsyQ libcd.h infra** (CdlLOC/CdlFILE + 4 named symbols, unlocks the loader cluster); **LoaderInitFileTable + ResourceLoadStateMachine NON_MATCHING-drafted** (→ 4 NM) — **Task 5 non-jtbl loaders COMPLETE** (6 matched + 2 drafted); report tooling fixed (multi-file); cookbook §6/§7/T4. Build byte-identical throughout. [need ≥1 more session]
|
||
- 2026-06-15 (session C): `make check` → `143dbb89… BYTE-IDENTICAL` ✓ (full `clean && extract && build`, restored after the Task-2′ experiments). **≥3-session bar MET.** This session: fully diagnosed + built the **rodata-island mechanism** (works); root-caused the +24; **proved the PsyQ-library-linking GO** (see below). No new matches (architectural session). Build green at start and after restore.
|
||
- 2026-06-14 (session D): `make clean && make extract && make build && make check` → `143dbb89… BYTE-IDENTICAL` ✓ — **libcd LINKED INTO THE BUILD** (Drew-approved push-through). The first real PsyQ library is now sourced from real SDK objects in the byte-identical build: **58 libcd SDK functions** linked (not stubs), replacing the libcd-region asm stubs. Idempotent; Makefile-automated; conditional (fresh clone w/o `tools/psyq/` builds via stubs). New committed tooling: `tools/psyq_link.py` (per-object byte-link engine — recovers externals from resolved relocs, weakens psyq-obj-parser's mislabelled `.bss` commons), `psyq_link_lib.py` (whole-lib verify, 18/18 libcd), `psyq_link_region.py` (region link via **NOLOAD** = no data carving), `psyq_integrate.py` (build wiring: splat resegment + .ld swap + external resolution), `split_src_region.py` (H5-safe src split). Cookbook §9.1/9.2/9.3 + R16. **No data carving** (NOLOAD data placement; flat data subseg unchanged). Build green throughout.
|
||
- 2026-06-15 (session E): `make clean && make extract && make build && make check` → `143dbb89… BYTE-IDENTICAL` ✓ — start-of-session baseline + after the **LZSS surgical rodata carve** (jtbl_80072A38 migrated + sandwiched) + LZSS C **NON_MATCHING-guarded** (default build = stub). **≥3-session bar already MET (A/B/C/D); E is margin.** Findings: lean LZSS path proven (no libgs needed for it); LZSS C structurally matches (111/122) but blocked on a gcc cross-jump-merge hard-tail (see LZSS block above); libgs placement+disambiguation done (bonus, task #9, deferred per Drew until before PhaseEnd).
|
||
- 2026-06-15 (session F): `make clean && make extract && make build && make check` → `143dbb89… BYTE-IDENTICAL` ✓ — TWO milestones. **(1) LZSS MATCHED byte-for-byte** (asm-differ score 0; C compiled in, NON_MATCHING guard dropped). Closed the session-E ~4 regalloc/scheduling residuals via §3a web-research (ground-truthed against pinned gcc-2.7.2 `reorg.c`/`jump.c`/`local-alloc.c`) + a floor-free `.text` object metric: decoupled high-byte var + OR operand order (residual A); `result`-in-predecessor + explicit-`$2`-local-with-early-pin (residual B); no-`default` sltiu-reuse (residual B3); §5a barrier retained. **43 real matches; Gen1-exit LZSS gate SATISFIED.** Cookbook §10. **(2) libgs BLOCK 6 integrated** (16 objects / 26 SDK funcs byte-identical; the libcd §9.3 pattern). Fixed a resegment-triggered spimdisasm boundary instability (declared GsMulCoord2/3; carved the §8 data table `[0x53198,data]`) + made psyq_integrate multi-library-safe (per-lib NOLOAD namespacing + global sort; sibling externals in the trial); `tools/make_libgs_block6.sh` for reproducibility; cookbook §9.4. Build byte-identical with OR without the SDK objects; fresh-clone stub fallback verified. Changed: `src/800.c`, `config/{splat.us.exe.yaml,symbols.us.txt}`, `Makefile`, `tools/{psyq_integrate.py,ld_interleave.py,make_libgs_block6.sh}`, `docs/matching-cookbook.md` (zero generated/ROM bulk staged). **Tasks: full libgs → 6 → 7 remain** (PhaseEnd gated on libgs per Drew).
|
||
- 2026-06-15 (session G): `make clean && make extract && make build && make check` → `143dbb89… BYTE-IDENTICAL` ✓ (start-of-session baseline + multiple full clean cycles after the change). **FULL libgs integration DONE** — the remaining 5 blocks unified with block 6 into ONE `psyq_integrate` call: **all 31 libgs objects / 6 contiguous blocks** linked from real SDK objects byte-identical (≈49 named SDK functions). `psyq_integrate.py` gained an optional placement-window arg (GS_106 = 8 ins anchors uniquely only in the libgs window); `tools/make_libgs.sh` replaces `make_libgs_block6.sh` (31-obj dir); config 800b resegmented into 800b(pre)+libgs1-5+gsgap1-5 (+ unchanged libgs6/800b2); `src/800b.c` trimmed (<0x80051804, 0 matched-C lost); 10 new stub src files. Split DETERMINISTIC across 2 clean extracts (no §9.4 boundary instability); byte-identical WITH or WITHOUT the SDK objects (fresh-clone stub fallback verified). Cookbook §9.5. Changed: `config/splat.us.exe.yaml`, `Makefile`, `src/800b.c` + 10 new `src/{libgs1-5,gsgap1-5}.c`, `tools/{psyq_integrate.py,make_libgs.sh}` (−make_libgs_block6.sh), `docs/matching-cookbook.md`, `docs/{progress,difficulty}.md` (digest refresh). **Tasks: 6 (close-out) → 7 (PhaseEnd) remain; libgs gate CLEARED.**
|
||
**(session G cont.)** Then the 4 jtbl loaders: **CdReadStateMachine / CdReadSectorReadyCB / StreamLoadStateMachine NON_MATCHING-drafted** (faithful translations of the Ghidra decompile; all valid C under `-DNON_MATCHING`, verified via a cpp→cc1 syntax check; shared-loader-global types reconciled to the file's s32/void* convention). **SaveLoadRoutine DEFERRED to Q#5** (Drew) — it is a 1139-ins multi-entry save/memcard blob (single `jr $ra`, 3 saveHeaderTemplate entry points, Ghidra returns only an unaff_-register fragment); left a full structured deferral comment in src/800.c. Task 6 close-out: README → Gen1-complete status; Gen1-exit checklist verified (43 REAL incl LZSS, 7 zero-regression sessions, reports auto-generate, loader cluster matched/drafted, overlay map proven). **Gen1 EXIT confirmed by Drew.** 3 rules formalized at PhaseEnd: R17 (web-research compiler quirks), R18 (plain-English session recaps), R19 (preserve worklog → phase-ends/logs/, out of load order). This worklog is itself preserved under R19. **PhaseEnd_Phase7 written; Gen1 COMPLETE.**
|
||
|
||
---
|
||
|
||
## Rodata-island foundation — RESOLVED end-to-end (session C, 2026-06-15)
|
||
The mechanism now WORKS; only the +24 (a known file-split/alignment artifact) blocks full byte-identity, and
|
||
the PsyQ-lib pivot (below) is the chosen fix. Experimental configs saved: `.run/{splat.island.yaml,
|
||
symbols.island.txt,Makefile.island,800.c.fresh-throwaway}`; new committed tool `tools/ld_interleave.py`.
|
||
**Proven mechanism (3 parts, all validated this session):**
|
||
1. **Migration** — dotted `.rodata` sibling named `800` (`[0x63238, .rodata, 800]`) → spimdisasm migrates all
|
||
50 jtbls + single-ref consts into their owning `asm/nonmatchings/800/<fn>.s` (99 .s got `.section .rodata`,
|
||
`.L`-refs resolve intra-object). Multi-ref consts → 70 `INCLUDE_RODATA` lines (needs a FRESH `800.c`; for the
|
||
PERMANENT file, surgically INSERT those 70 lines into the curated 800.c — do NOT regen-fresh, it drops
|
||
comments/H5). All 301 jtbl targets ∈ the `800` text seg (none in `boot`), so the single sibling is correct.
|
||
2. **Placement** — `tools/ld_interleave.py` (wired into `make extract`) rewrites splat's section-major `.main`
|
||
into the real `.data→.rodata→.data` sandwich order (text, front-data@0x800629DC, rodata@0x80072A38,
|
||
tail-data@0x80074750). Front/tail split by object basename. rodata + both data sizes came out **byte-exact**.
|
||
3. **Data-in-text carve** — the 68-B descriptor table at 0x80062998–0x800629DC (ptrs to start/D_80062998/
|
||
D_80074778) must be its own `[0x53198, data, 53198]` subseg or the jumptable analyzer mis-extends the last
|
||
code function across it.
|
||
**The ONLY residual = +24 (ROOT-CAUSED):** 6× `.align 3` jumptable padding nops injected into `.text` (at
|
||
PRESET_OBJ_744, PRESET_OBJ_8FC, PRESET2_OBJ_4D8, PRESET2_OBJ_A88, OBJT2_OBJ_614, PRNT_OBJ_24C). GCC 8-aligns
|
||
each switch jtbl, but the original built these as SEPARATE translation units; our single 800.o concatenation
|
||
adds padding the original lacked. spimdisasm itself printed **8 file-split suggestions** (rodata 0x6324C,
|
||
0x63388, 0x633FC, 0x63920, 0x63C94, 0x64420, 0x64AB4, 0x64CA0). Canonical fix = per-file split — OR the PsyQ-lib
|
||
pivot below (most of these are library code).
|
||
|
||
## PsyQ-library-linking SPIKE — GO PROVEN (session C, the chosen +24 fix + free SDK code)
|
||
**Finding:** BFM's PsyQ library functions are **byte-identical to the real PsyQ SDK objects** → link them
|
||
directly (byte-exact) instead of hand-decompiling, which ALSO gives each library `.o` correct per-object
|
||
alignment (dissolving the library-half of the +24). Validated: `CdPosToInt` (32 instrs) + `CdIntToPos` (65)
|
||
EXACT vs PsyQ **4.7** `libcd.a`; `CdPosToInt` also in **4.0** `LIBCD.LIB`; `PRESET_OBJ_108` (a +24 culprit) is
|
||
in **4.0 `LIBGS.LIB`** → library code, not game. (Raw-byte lib search has false-negatives on relocated funcs,
|
||
e.g. `_spu_FsetPCR`/`OBJT2`/`PRNT` "missed" — needs the real ELF-link test to classify those.)
|
||
**Assets staged (gitignored `tools/psyq/`):** `psyq-obj-parser` (decompme prebuilt, works on `.OBJ`; rejects
|
||
`.LIB` archives — needs splitting); `psyq4.0/` (4.0 tools: CC1PSX/ASPSX/PSYLIB/…); `conv47/` (4.7 pre-converted
|
||
ELF `.a` — quick reference); **`lib40/*.LIB`** = the 20 PsyQ **4.0 USA** libraries (DTL-S2002 R2.0, BFM's exact
|
||
version) extracted from the redump via our `tools/bfm_extract/iso9660.py` walker. Footprint in BFM: ~350 funcs
|
||
(libsnd 131, libapi/gs 69+, libmcrd 63, libsn 37, libcd 28, libspu 21, …) of 2050 matchable.
|
||
**Integration pipeline — BUILT + PROVEN end-to-end (session C):**
|
||
- `tools/psyq_lib_split.py` (committed) — splits a `LIB\x01` archive into its member `.OBJ` (locates each
|
||
member header by the invariant `u32@(header+12) == LNK_offset − header`). LIBCD → 25 objects ✓.
|
||
- `tools/psyq_build_libs.sh` (committed) — `.LIB → .OBJ → psyq-obj-parser → ELF .o → ar` per lib. **Built all
|
||
14 BFM libs → `tools/psyq/lib40_elf/*.a`** (gitignored): LIBCD 25, LIBGS 201, LIBSPU 129, LIBSND 163,
|
||
LIBMCRD 2, LIBSN 51, LIBAPI 90, LIBETC 7, LIBGTE 381, LIBGPU 12, LIBMATH 48, LIBCARD 18, LIBC 56, LIBC2 46.
|
||
- **Byte-match PROVEN at object level:** in libcd `SYS.o`, leaves `CdPosToInt`/`CdIntToPos` are EXACT; relocated
|
||
funcs (`CdComstr` …) differ ONLY at their relocation sites → link byte-exact once relocs resolve to BFM
|
||
symbol addrs. So every step (split, convert, leaf-match, reloc-resolve) is validated.
|
||
|
||
**FULL OBJECT LINK — PROVEN BYTE-IDENTICAL (session C):** libcd `SYS.o` (483 instrs, the full TU: leaves +
|
||
relocated funcs + 21 externals + internal .rdata/.data) links **byte-for-byte identical to BFM**. The pipeline
|
||
+ the 3 last pieces:
|
||
- **Identify placement** (`tools/psyq_identify.py`, committed) — relocation-masked search locates each object's
|
||
`.text` in BFM. libcd: **18/25 objects found, CONTIGUOUS** at 0x80043088–0x80046D1C in object order (the 7
|
||
unused — CDPLAY, C_012–015… — BFM doesn't link). So per-library placement = link the used objects in order at
|
||
the region base; addresses are read off, not guessed.
|
||
- **Recover externals from BFM** — symbols the object references but doesn't define (e.g. libcd's `CD_pos`,
|
||
`CD_com`, `DMACallback`) are NOT in symbols.us.txt, but their addresses are encoded in the EXE's already-
|
||
RESOLVED relocations: parse the object's reloc records, read BFM at each site, reconstruct (R_MIPS_26 →
|
||
target; HI16/LO16 pair → addr). Recovered all 21 for SYS.o. (Feeds symbols.us.txt over time.)
|
||
- **Alignment fix** — psyq-obj-parser sets `.text/.rdata/.data` align=2**3 (8); the original placed them
|
||
4-aligned, so an 8-align bumps them +4. `objcopy --set-section-alignment .rdata=4 .data=4` before linking →
|
||
exact. (The +4 mismatch is the tell.)
|
||
- **Link recipe:** `ld -T <script placing .text@<objaddr> .rdata@<island> .data@<addr>> --defsym <recovered…>
|
||
obj.o` → objcopy .text → byte-compare. Proven on SYS.o.
|
||
|
||
**REMAINING (replication + wiring, NEXT session, Task #5):** generalize the SYS.o recipe to all used objects
|
||
per library (identify region → recover externals → set-align → place sections in order → link), then wire into
|
||
the build: drop the linked functions' INCLUDE_ASM + carve their raw data, add the lib objects to the link via a
|
||
generated `.ld` fragment. ~350 SDK funcs become byte-exact + the library-region jtbl alignment resolves. GAME
|
||
switches + **LZSS** (jtbl_80072A38 = island's first entry, before any misalignment) take the proven
|
||
migration+ld_interleave path above.
|
||
|
||
## Blockers / open items
|
||
- `.LIB`→`.OBJ` splitter (LIB\x01 format) — the gate for the lib-linking integration.
|
||
- Which 4.x version matches each BFM lib object best (4.0 USA primary; BFM mixes 4.0+4.2 stamps, so some objects
|
||
may need 4.2/4.3 libs — determine per-lib during integration via the ELF-link byte test).
|
||
- LZSS via the proven migration+ld_interleave path (independent of the lib pivot).
|
||
|
||
## Notes
|
||
- **Rule candidate (PhaseEnd, Drew-flagged session E):** web-research the compiler internals (real compiler
|
||
source e.g. `pmret/gcc-papermario`) + decomp community for **compiler-quirk residuals** (cross-jump,
|
||
scheduling, regalloc) — a proven escalation tier above the permuter, below decomp.me. Found the LZSS
|
||
cross-jump barrier. Captured in cookbook §3a/§5a + memory `web-research-compiler-quirks`.
|
||
- **Rule candidate (PhaseEnd, Drew-flagged session G):** end every session/phase with a few-sentence
|
||
plain-English recap of what we did and WHY (no jargon) — keeps Drew oriented on real progress without
|
||
tracking the byte-level detail. Captured in memory `session-summary-plain-english`. Apply immediately.
|
||
- **Session-E checkpoint commit:** Drew directed a checkpoint commit (deviates from R8's strict
|
||
one-commit-at-phase-end; consistent with the session A–D checkpoint commits in the git log). Commit local in
|
||
WSL (no push, no Co-Authored-By per R5); Drew pushes via GitHub Desktop (R6).
|
||
- `.run/merge_matches.py` = regenerate-800.c + re-apply-matches helper. **H5 caveat:** regen-fresh drops
|
||
file-level/stub comments; for the permanent 800.c, surgically insert the 70 INCLUDE_RODATA lines instead.
|
||
- `tools/ld_interleave.py` (committed) = the `.data→.rodata→.data` linker-script interleaver.
|