From 1f5cc439812e47f119e80deb44186ef33ab6779f Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Mon, 15 Jun 2026 19:22:36 -0600 Subject: [PATCH] =?UTF-8?q?feat(phase-8):=20close-out=20=E2=80=94=20PhaseE?= =?UTF-8?q?nd=5FPhase8=20+=20worklog=20archived=20(v1.8.0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - PhaseEnd_Phase8.md: 8 PsyQ libraries linked byte-identical, byte-identical coverage 20.31% -> 50.24% (LINKED 340 -> 959), 9 game-code matches (REAL 43 -> 52), milestone confirmed by Drew (gate 2) - rule R22 (verify byte-matches from a clean rebuild, never incremental) - CURRENT_PHASE.md -> phase-ends/logs/Phase8.md (R19) - bumps project version 1.7.0 -> 1.8.0 --- phase-ends/PhaseEnd_Phase8.md | 137 ++++++++++++++++++ .../{CURRENT_PHASE.md => logs/Phase8.md} | 0 2 files changed, 137 insertions(+) create mode 100644 phase-ends/PhaseEnd_Phase8.md rename phase-ends/{CURRENT_PHASE.md => logs/Phase8.md} (100%) diff --git a/phase-ends/PhaseEnd_Phase8.md b/phase-ends/PhaseEnd_Phase8.md new file mode 100644 index 000000000..cd580ef66 --- /dev/null +++ b/phase-ends/PhaseEnd_Phase8.md @@ -0,0 +1,137 @@ +# PhaseEnd — Phase 8: Finish the EXE — remaining PsyQ libraries + final harvest +**Date:** 2026-06-15 · **Project Version:** 1.8.0 · **Phase Status:** Complete · **Generation:** Gen2 (first phase) + +> First Gen2 phase (the global count continues from Gen1's 1–7). The granular per-library trail +> (survey, per-integration addresses, the hard-case diagnostics) is preserved on-demand at +> **`phase-ends/logs/Phase8.md`** (R19 — NOT auto-loaded; consult only when researching a mechanism). +> Owner decisions (2026-06-15): progress metric = a **LINKED** category; exit bar = **full ceiling** +> (link every footprint-confirmed library; defer the rest as documented honest stubs). + +## Build Log + +**Files created/changed and complete — do not recreate:** + +*Tooling (new, committed):* +- `tools/gen_lib_subsegs.py` — generate splat subseg lines + the `psyq_integrate` stub list for a + MULTI-block library; block ends use the last object's `.text` **section size** (8-aligned), not the + instruction count (the boundary gotcha). Used for libgte / the combined sound + apicard regions. +- `tools/make_snd_used.py` — build `.run/obj40/snd_used`: the **combined libspu+libsnd** curated dir + (merge by vram, pick the byte-matching object per aliased address, exclude 4 unreconcilable addresses). +- `tools/make_apicard_used.py` — build `.run/obj40/apicard_used`: the **combined libapi+libcard** 800c2 + curated dir (C112 dedup, 0 exclusions). + +*Tooling (modified):* +- `tools/progress.py` — **LINKED category** (owner decision Q1): INCLUDE_ASM stubs in integrated library + subsegs counted as a distinct byte-identical bucket, separate from REAL and stubs. The linked-subseg set + is parsed from the Makefile's `psyq_integrate` calls (single source of truth), resolving `$(VAR)` stub + lists. Re-baselined libcd+libgs (340) → LINKED. + +*Config / build:* +- `config/splat.us.exe.yaml` — resegmented for 8 libraries: `libetc`; `libgpu`(EXT+PRIM)+`800c`; + `libmcrd1/2`+`800c2/800c3`; `libc2_1/2`; `libgte1..22`+`800b/800b_2..7`; the combined sound region + `snd1..9`+`sgap*`; the combined apicard region `apicard1..4`+`800c2*`. (Game-code subseg names multiplied + as regions were carved; all boundaries section-size-correct.) +- `Makefile` — per-library `LIB*_/SND_/APICARD_` vars + gated `psyq_integrate` calls (conditional on the + SDK ELF dir → fresh-clone stub fallback) + each library's `-T ` on the `ld` line. `LIBGTE_STUBS`/ + `SND_STUBS`/`APICARD_STUBS` vars hold the long stub lists. +- `src/*.c` — **committed stub records** for every new library subseg (`libetc`, `libgpu`, `libmcrd1/2`, + `libc2_1/2`, `libgte1..22`, `snd1..9`, `apicard1..4`) and the new game-code fragments (`800c`, `800c2`, + `800c3`, `800b`, `800b_2..7`, `sgap`, `sgap_2..8`); **9 game-code REAL matches** added to `src/800.c` + (trivial D_* accessors). `src/800.c`/`800b.c`/`800b2.c` trimmed (matched C preserved). + +*Docs / knowledge base:* +- `docs/psyq-worklist.md` — **new**: byte-confirmed footprint map (9 footprint libs / 3 skipped), the + per-library recipe, the boundary gotcha, and the **deferred/excluded-objects table**. +- `docs/matching-cookbook.md` — **§9.6** "Scaling library linking to the whole EXE" (R16 flywheel): survey- + first, `gen_lib_subsegs` + the section-size boundary, combined-region for interleaved libs, scattered-`.bss` + cross-object exclusion-by-address, and the **clean-rebuild-to-verify** gotcha. +- `docs/SETUP.md` — the 3 new tools added to the inventory (R21). `docs/progress.md` — regenerated. +- `phase-ends/CURRENT_PHASE.md` → **`phase-ends/logs/Phase8.md`** (R19 archive). `phase-ends/PhaseEnd_Phase8.md` — this file. + +**Local artifacts (gitignored / regenerable — do NOT recreate; a fresh clone redoes them):** +- `.run/obj40/{libgpu_used,snd_used,apicard_used}` — curated SDK-object dirs (regenerate: `psyq_build_libs.sh` + then the `make_*_used.py` scripts / a 2-file `cp` for libgpu_used). Build is byte-identical without them. +- `build/psyq/*` — prepared objects + `*_externals.ld` (regenerated each `make build`). `.run/survey/*` — the placement survey. + +**Tools/packages installed:** None (used the existing Phase-4/6 toolchain entirely). + +**Verification results (literal):** +- **MILESTONE — full clean cycle, BOTH ways byte-identical:** `make clean && make extract && make build && + make check` → `build/us/SLUS_007.26` SHA1 `143dbb89f34491258bbc27810d0a12ec8b43a8dd` (a) **with** all 8 + newly-linked PsyQ libraries present, and (b) **without** any SDK ELF dir (the fresh-clone all-stubs + fallback) — both `[ OK ] BYTE-IDENTICAL`. +- **`make report` (deterministic):** REAL **52** · LINKED **959** · NON_MATCHING 7 · empties 42 · stubs 1036 + · blobs 2. **byte-identical/matchable 1053/2096 = 50.24%** (was 20.31% at phase start with libcd+libgs only). +- **8 PsyQ libraries linked byte-identical** (object counts): libetc 5, libgpu 2 (EXT+PRIM), libmcrd 2, + libc2 17, libgte 53/58, libspu+libsnd 60 (combined), libapi+libcard 22 (combined 800c2). Per-region + byte-verified by `psyq_link_region` before wiring; each integration **dual byte-gated** (with + without). +- **0 NON_MATCHING in the default build** (the 7 are `#ifdef NON_MATCHING`-guarded) — G4 preserved. +- **3 libraries have zero footprint** (recorded skipped): libmath, libc (BFM links libc2), libsn. +- `git status`: only `config/`, `Makefile`, `tools/`, `src/`, `docs/`, `phase-ends/` tracked — **zero + ROM-derived/generated bulk staged** (curated dirs + build/ + `.run/` gitignored). + +**Milestone achieved:** The EXE rebuilds **byte-for-byte identical with AND without** all 8 newly-linked PsyQ +libraries (fresh-clone-safe), `make report` shows byte-identical coverage risen **20.31% → 50.24%** (LINKED +340 → 959, REAL 43 → 52), 0 NON_MATCHING in the default build, and every footprint library is linked or a +documented honest deferral — confirmed by Drew (gate 2). + +**Next:** **Phase 9 — Binary-agnostic toolchain refactor** (Gen2 roadmap). Start by parameterizing the +single-binary tools (`psyq_*.py`, `progress/difficulty/dup_report.py`, the Makefile) by `--binary`/ +`--text-vram`, proving the refactor a no-op via the EXE rebuilding SHA1 `143dbb89…`. + +## Deviations +| Item | Plan | Actual | Reason | +|---|---|---|---| +| Integration order | ascending difficulty (libgpu→…→libgte) | **value-first** (libetc/gpu/mcrd/c2 → libgte → sound → apicard) | libcard/libapi turned out fiddliest (dense aliases); libgte/sound are highest-value and matched-C-free | +| Interleaved libs | one library per task | **combined-region integration** (libspu+libsnd as one; libapi+libcard as one) | the libs interleave object-by-object; two passes would tangle (each's objects span the other's gaps) | +| Multi-block resegment | (manual per the §9.5 pattern) | built **`gen_lib_subsegs.py`** (libgte=22, sound=9, apicard=4 blocks) | hand-editing ~50 subsegs is impractical/error-prone; the tool bakes in the section-size boundary | +| libgpu / sound completeness | link the whole library | **excluded scattered-`.bss` objects** (libgpu SYS.o; sound S_R/S_GRMDT/VM_F; + S_IH false-positive) | the §9.1 GS_001 class (cross-object form): `.bss`+offset refs the original linker scattered; honest stubs (P9) | +| libgte / libapi completeness | whole library | **5 libgte libgs-gap objects + libapi's 800c3 remnant DEFERRED** | low value vs the gsgap sub-split / extra region resegment; documented, byte-identical stubs | +| Harvest size | a modest bank | **9 trivial game-code accessors** (REAL 43→52) | "bank, don't exhaust"; more trivial leaves remain | +| Boundary discovery | (not anticipated) | **section-size boundary gotcha** found+fixed at libc2 (SETJMP.o `.text` 0x80 vs 0x78 ins) | a too-low boundary shifted the whole image; lesson → cookbook §9.6 + the tool | +| Verification | (incremental implied) | **clean-rebuild required** (an incremental build falsely diffed in libmcrd) | the incremental `psyq_integrate` `.ld` rewrite can go stale → R22 candidate | +| difficulty.md regen | regen in the harvest | **deferred** (sig-refresh needs the Ghidra server stopped) | next-session follow-up; not a milestone item | + +## Commit Message +``` +feat(phase-8): finish the EXE — 8 PsyQ libraries linked byte-identical + harvest (v1.8.0) + +- 8 libraries linked byte-identical (libetc/libgpu/libmcrd/libc2/libgte/libspu+libsnd/ + libapi+libcard); byte-identical coverage 20.31% -> 50.24% (LINKED 340 -> 959) +- new tooling: gen_lib_subsegs.py (multi-block subseg generator, section-size boundaries), + make_snd_used.py + make_apicard_used.py (combined-region curated dirs), progress.py LINKED +- combined-region integration for interleaved libs (libspu+libsnd, libapi+libcard); value- + first order; scattered-.bss objects excluded as honest stubs (libgpu SYS.o, sound, GS_001) +- 9 game-code accessors harvested (REAL 43 -> 52); cookbook §9.6 + SETUP tool inventory +- MILESTONE: make clean && build && check -> 143dbb89 byte-identical WITH and WITHOUT all + SDK objects (fresh-clone-safe); 0 NON_MATCHING in default build; deferrals documented +- worklog archived -> phase-ends/logs/Phase8.md (R19); bumps 1.7.0 -> 1.8.0 +``` + +## Rules Added This Phase +| Rule | Reason | +|---|---| +| **R22 — Verify byte-matches from a CLEAN rebuild, never an incremental build.** A byte check (`make check` / SHA1) is trustworthy only after `make clean && make extract && make build`. The `psyq_integrate` step rewrites the linker script in place, so an *incremental* build after a source change can re-run it on an already-rewritten `.ld` and transiently mis-resolve a sibling library's externals — producing a **false** diff (or, in principle, a false pass). Extends G3/P9. | A Phase-8 harvest incrementally "diffed" in libmcrd; the identical source built byte-identical from a clean tree. Caught a verification-integrity hole that could otherwise mislead a match/no-match conclusion. | + +*(Not elevated to rules — captured as cookbook §9.6 techniques: the section-size boundary rule, the combined-region pattern for interleaved libraries, scattered-`.bss` exclusion-by-address, survey-first. Owner decision recorded: library linking is reported as a distinct **LINKED** bucket, never folded into REAL — a P9 measurement-honesty practice, implemented in progress.py.)* + +## PhaseEnd Changelog +**v1.7.0 → v1.8.0 — Phase 8 complete (first Gen2 phase).** The EXE is driven to its practical matching +ceiling: **8 more PsyQ libraries linked byte-identical** (libetc, libgpu, libmcrd, libc2, libgte, libspu, +libsnd, libapi, libcard), taking byte-identical-from-source coverage from **20.31% to 50.24%** (LINKED 340 → +959) — proven byte-identical **with and without** every SDK object (fresh-clone-safe). The phase's reusable +output: **`gen_lib_subsegs.py`** (mechanical multi-block resegmentation with section-size-correct +boundaries), the **combined-region** technique for interleaved libraries (`make_snd_used.py` / +`make_apicard_used.py`, alias dedup by byte-match + scattered-`.bss` exclusion-by-address), and the +**LINKED** progress metric (Makefile-sourced). A modest game-code harvest banked **9** trivial accessors +(REAL 43 → 52). Three zero-footprint libraries skipped (libmath/libc/libsn); a small set of scattered-`.bss` +objects and two low-value remnants (libgte's libgs-gap objects, libapi's 800c3 cluster) deferred as +documented honest stubs — none block the byte-identical build. One rule (R22 clean-rebuild verification). +Knowledge fed back (cookbook §9.6, R16). The EXE is now the fully-understood, half-from-source reference +that Gen2's binary-agnostic refactor (Phase 9), resident-blob (Phase 10), and overlays build on. + +## 🛑 Stop Here +PhaseEnd written; `CURRENT_PHASE.md` archived → `phase-ends/logs/Phase8.md` (R19). The Phase-8 work is +committed (`commit:0030`→ this commit); **Drew pushes** (R6). Gen2 is underway — do NOT start Phase 9 here. +Start a **fresh session** (effort **Max**, plan mode) for **Phase 9 — Binary-agnostic toolchain refactor** +(a Tier-1 Phase-Start planning task). Keep this file forever. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase8.md similarity index 100% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase8.md