BFM-decomp/phase-ends/PhaseEnd_Phase8.md

PhaseEnd — Phase 8: Finish the EXE — remaining PsyQ libraries + final harvest

Date: 2026-06-15 · Project Version: 1.8.0 · Phase Status: Complete · Generation: Gen2 (first phase)

First Gen2 phase (the global count continues from Gen1's 1–7). The granular per-library trail (survey, per-integration addresses, the hard-case diagnostics) is preserved on-demand at phase-ends/logs/Phase8.md (R19 — NOT auto-loaded; consult only when researching a mechanism). Owner decisions (2026-06-15): progress metric = a LINKED category; exit bar = full ceiling (link every footprint-confirmed library; defer the rest as documented honest stubs).

Build Log

Files created/changed and complete — do not recreate:

Tooling (new, committed):

• tools/gen_lib_subsegs.py — generate splat subseg lines + the psyq_integrate stub list for a MULTI-block library; block ends use the last object's .text section size (8-aligned), not the instruction count (the boundary gotcha). Used for libgte / the combined sound + apicard regions.
• tools/make_snd_used.py — build .run/obj40/snd_used: the combined libspu+libsnd curated dir (merge by vram, pick the byte-matching object per aliased address, exclude 4 unreconcilable addresses).
• tools/make_apicard_used.py — build .run/obj40/apicard_used: the combined libapi+libcard 800c2 curated dir (C112 dedup, 0 exclusions).

Tooling (modified):

• tools/progress.py — LINKED category (owner decision Q1): INCLUDE_ASM stubs in integrated library subsegs counted as a distinct byte-identical bucket, separate from REAL and stubs. The linked-subseg set is parsed from the Makefile's psyq_integrate calls (single source of truth), resolving $(VAR) stub lists. Re-baselined libcd+libgs (340) → LINKED.

Config / build:

• config/splat.us.exe.yaml — resegmented for 8 libraries: libetc; libgpu(EXT+PRIM)+800c; libmcrd1/2+800c2/800c3; libc2_1/2; libgte1..22+800b/800b_2..7; the combined sound region snd1..9+sgap*; the combined apicard region apicard1..4+800c2*. (Game-code subseg names multiplied as regions were carved; all boundaries section-size-correct.)
• Makefile — per-library LIB*_/SND_/APICARD_ vars + gated psyq_integrate calls (conditional on the SDK ELF dir → fresh-clone stub fallback) + each library's -T <syms> on the ld line. LIBGTE_STUBS/ SND_STUBS/APICARD_STUBS vars hold the long stub lists.
• src/*.c — committed stub records for every new library subseg (libetc, libgpu, libmcrd1/2, libc2_1/2, libgte1..22, snd1..9, apicard1..4) and the new game-code fragments (800c, 800c2, 800c3, 800b, 800b_2..7, sgap, sgap_2..8); 9 game-code REAL matches added to src/800.c (trivial D_* accessors). src/800.c/800b.c/800b2.c trimmed (matched C preserved).

Docs / knowledge base:

• docs/psyq-worklist.md — new: byte-confirmed footprint map (9 footprint libs / 3 skipped), the per-library recipe, the boundary gotcha, and the deferred/excluded-objects table.
• docs/matching-cookbook.md — §9.6 "Scaling library linking to the whole EXE" (R16 flywheel): survey- first, gen_lib_subsegs + the section-size boundary, combined-region for interleaved libs, scattered-.bss cross-object exclusion-by-address, and the clean-rebuild-to-verify gotcha.
• docs/SETUP.md — the 3 new tools added to the inventory (R21). docs/progress.md — regenerated.
• phase-ends/CURRENT_PHASE.md → phase-ends/logs/Phase8.md (R19 archive). phase-ends/PhaseEnd_Phase8.md — this file.

Local artifacts (gitignored / regenerable — do NOT recreate; a fresh clone redoes them):

• .run/obj40/{libgpu_used,snd_used,apicard_used} — curated SDK-object dirs (regenerate: psyq_build_libs.sh then the make_*_used.py scripts / a 2-file cp for libgpu_used). Build is byte-identical without them.
• build/psyq/* — prepared objects + *_externals.ld (regenerated each make build). .run/survey/* — the placement survey.

Tools/packages installed: None (used the existing Phase-4/6 toolchain entirely).

Verification results (literal):

• MILESTONE — full clean cycle, BOTH ways byte-identical: make clean && make extract && make build && make check → build/us/SLUS_007.26 SHA1 143dbb89f34491258bbc27810d0a12ec8b43a8dd (a) with all 8 newly-linked PsyQ libraries present, and (b) without any SDK ELF dir (the fresh-clone all-stubs fallback) — both [ OK ] BYTE-IDENTICAL.
• make report (deterministic): REAL 52 · LINKED 959 · NON_MATCHING 7 · empties 42 · stubs 1036 · blobs 2. byte-identical/matchable 1053/2096 = 50.24% (was 20.31% at phase start with libcd+libgs only).
• 8 PsyQ libraries linked byte-identical (object counts): libetc 5, libgpu 2 (EXT+PRIM), libmcrd 2, libc2 17, libgte 53/58, libspu+libsnd 60 (combined), libapi+libcard 22 (combined 800c2). Per-region byte-verified by psyq_link_region before wiring; each integration dual byte-gated (with + without).
• 0 NON_MATCHING in the default build (the 7 are #ifdef NON_MATCHING-guarded) — G4 preserved.
• 3 libraries have zero footprint (recorded skipped): libmath, libc (BFM links libc2), libsn.
• git status: only config/, Makefile, tools/, src/, docs/, phase-ends/ tracked — zero ROM-derived/generated bulk staged (curated dirs + build/ + .run/ gitignored).

Milestone achieved: The EXE rebuilds byte-for-byte identical with AND without all 8 newly-linked PsyQ libraries (fresh-clone-safe), make report shows byte-identical coverage risen 20.31% → 50.24% (LINKED 340 → 959, REAL 43 → 52), 0 NON_MATCHING in the default build, and every footprint library is linked or a documented honest deferral — confirmed by Drew (gate 2).

Next: Phase 9 — Binary-agnostic toolchain refactor (Gen2 roadmap). Start by parameterizing the single-binary tools (psyq_*.py, progress/difficulty/dup_report.py, the Makefile) by --binary/ --text-vram, proving the refactor a no-op via the EXE rebuilding SHA1 143dbb89….

Deviations

Item	Plan	Actual	Reason
Integration order	ascending difficulty (libgpu→…→libgte)	value-first (libetc/gpu/mcrd/c2 → libgte → sound → apicard)	libcard/libapi turned out fiddliest (dense aliases); libgte/sound are highest-value and matched-C-free
Interleaved libs	one library per task	combined-region integration (libspu+libsnd as one; libapi+libcard as one)	the libs interleave object-by-object; two passes would tangle (each's objects span the other's gaps)
Multi-block resegment	(manual per the §9.5 pattern)	built gen_lib_subsegs.py (libgte=22, sound=9, apicard=4 blocks)	hand-editing ~50 subsegs is impractical/error-prone; the tool bakes in the section-size boundary
libgpu / sound completeness	link the whole library	excluded scattered-.bss objects (libgpu SYS.o; sound S_R/S_GRMDT/VM_F; + S_IH false-positive)	the §9.1 GS_001 class (cross-object form): .bss+offset refs the original linker scattered; honest stubs (P9)
libgte / libapi completeness	whole library	5 libgte libgs-gap objects + libapi's 800c3 remnant DEFERRED	low value vs the gsgap sub-split / extra region resegment; documented, byte-identical stubs
Harvest size	a modest bank	9 trivial game-code accessors (REAL 43→52)	"bank, don't exhaust"; more trivial leaves remain
Boundary discovery	(not anticipated)	section-size boundary gotcha found+fixed at libc2 (SETJMP.o .text 0x80 vs 0x78 ins)	a too-low boundary shifted the whole image; lesson → cookbook §9.6 + the tool
Verification	(incremental implied)	clean-rebuild required (an incremental build falsely diffed in libmcrd)	the incremental psyq_integrate .ld rewrite can go stale → R22 candidate
difficulty.md regen	regen in the harvest	deferred (sig-refresh needs the Ghidra server stopped)	next-session follow-up; not a milestone item

Commit Message

feat(phase-8): finish the EXE — 8 PsyQ libraries linked byte-identical + harvest (v1.8.0)

- 8 libraries linked byte-identical (libetc/libgpu/libmcrd/libc2/libgte/libspu+libsnd/
  libapi+libcard); byte-identical coverage 20.31% -> 50.24% (LINKED 340 -> 959)
- new tooling: gen_lib_subsegs.py (multi-block subseg generator, section-size boundaries),
  make_snd_used.py + make_apicard_used.py (combined-region curated dirs), progress.py LINKED
- combined-region integration for interleaved libs (libspu+libsnd, libapi+libcard); value-
  first order; scattered-.bss objects excluded as honest stubs (libgpu SYS.o, sound, GS_001)
- 9 game-code accessors harvested (REAL 43 -> 52); cookbook §9.6 + SETUP tool inventory
- MILESTONE: make clean && build && check -> 143dbb89 byte-identical WITH and WITHOUT all
  SDK objects (fresh-clone-safe); 0 NON_MATCHING in default build; deferrals documented
- worklog archived -> phase-ends/logs/Phase8.md (R19); bumps 1.7.0 -> 1.8.0

Rules Added This Phase

Rule

Reason

R22 — Verify byte-matches from a CLEAN rebuild, never an incremental build. A byte check (make check / SHA1) is trustworthy only after make clean && make extract && make build. The psyq_integrate step rewrites the linker script in place, so an incremental build after a source change can re-run it on an already-rewritten .ld and transiently mis-resolve a sibling library's externals — producing a false diff (or, in principle, a false pass). Extends G3/P9.

A Phase-8 harvest incrementally "diffed" in libmcrd; the identical source built byte-identical from a clean tree. Caught a verification-integrity hole that could otherwise mislead a match/no-match conclusion.

(Not elevated to rules — captured as cookbook §9.6 techniques: the section-size boundary rule, the combined-region pattern for interleaved libraries, scattered-.bss exclusion-by-address, survey-first. Owner decision recorded: library linking is reported as a distinct LINKED bucket, never folded into REAL — a P9 measurement-honesty practice, implemented in progress.py.)

PhaseEnd Changelog

v1.7.0 → v1.8.0 — Phase 8 complete (first Gen2 phase). The EXE is driven to its practical matching ceiling: 8 more PsyQ libraries linked byte-identical (libetc, libgpu, libmcrd, libc2, libgte, libspu, libsnd, libapi, libcard), taking byte-identical-from-source coverage from 20.31% to 50.24% (LINKED 340 → 959) — proven byte-identical with and without every SDK object (fresh-clone-safe). The phase's reusable output: gen_lib_subsegs.py (mechanical multi-block resegmentation with section-size-correct boundaries), the combined-region technique for interleaved libraries (make_snd_used.py / make_apicard_used.py, alias dedup by byte-match + scattered-.bss exclusion-by-address), and the LINKED progress metric (Makefile-sourced). A modest game-code harvest banked 9 trivial accessors (REAL 43 → 52). Three zero-footprint libraries skipped (libmath/libc/libsn); a small set of scattered-.bss objects and two low-value remnants (libgte's libgs-gap objects, libapi's 800c3 cluster) deferred as documented honest stubs — none block the byte-identical build. One rule (R22 clean-rebuild verification). Knowledge fed back (cookbook §9.6, R16). The EXE is now the fully-understood, half-from-source reference that Gen2's binary-agnostic refactor (Phase 9), resident-blob (Phase 10), and overlays build on.

🛑 Stop Here

PhaseEnd written; CURRENT_PHASE.md archived → phase-ends/logs/Phase8.md (R19). The Phase-8 work is committed (c27a4432e→ this commit); Drew pushes (R6). Gen2 is underway — do NOT start Phase 9 here. Start a fresh session (effort Max, plan mode) for Phase 9 — Binary-agnostic toolchain refactor (a Tier-1 Phase-Start planning task). Keep this file forever.