BFM-decomp/phase-ends/PhaseEnd_Phase7.md

# PhaseEnd — Phase 7: Matching loop at scale + Gen1 close-out
**Date:** 2026-06-15 · **Project Version:** 1.7.0 · **Phase Status:** Complete · **🎓 Gen1 EXIT (confirmed by Drew)**

> Phase 7 ran across **7 sessions (A–G)**. This file is the synthesis; the granular session-by-session
> trail (dead-ends, exact addresses, the libgs/rodata/PsyQ-linking working notes) is preserved on-demand
> at **`phase-ends/logs/Phase7.md`** (NOT auto-loaded — see R19). Consult it only when researching a past
> mechanism (it is the richest source for Gen2's overlay/library work).

## Build Log

**Files created/changed and complete — do not recreate:**

*Matching-loop industrialization (Task 3):*
- `tools/progress.py` / `tools/difficulty.py` / `tools/dup_report.py` + `make report` / `make sig-refresh` —
  deterministic, committable digests → `docs/{progress,difficulty,duplicates}.md`. progress.py distinguishes
  REAL (hand-matched score-0 C) from NON_MATCHING (guarded near-miss drafts); globs all `src/*.c` +
  `asm/nonmatchings/*`. Authoritative baseline at Gen1 exit: **43 REAL / 7 NON_MATCHING / 2050 stubs**.

*Per-module flag mechanism (Task 5.5, cookbook §6):*
- `src/boot.c` (split text subseg `[0x800,c,boot]`) + `Makefile` target-specific `build/src/boot.o: CC1FLAGS
  := …-O0…`. The boot/main/game-mode module (vram 0x80010000–0x800123F0) is **-O0**, the rest -O2 (per-module
  opt mixing is real). Detect via the `21F0A003` frame-pointer signature. Reusable for any flag-divergent module.

*PsyQ SDK integration — the Phase-7 strategic breakthrough (sessions C–G, cookbook §7/§8/§9.1–§9.5):*
- `include/psyq/libcd.h` — CdlLOC/CdlFILE + CdSearchFile/CdPosToInt/CdIntToPos (offsets from Ghidra `.gdt`, G1).
- `tools/psyq_lib_split.py` (LIB\x01 → .OBJ), `tools/psyq_build_libs.sh` (.OBJ → ELF via psyq-obj-parser → ar),
  `tools/psyq_identify.py` (relocation-masked placement; optional text-window arg for short objects like
  libgs GS_106), `tools/psyq_link.py` + `tools/psyq_link_lib.py` (per-object byte-link engine; weaken
  mislabelled `.bss` commons), `tools/psyq_link_region.py` (NOLOAD region link — no data carving),
  `tools/psyq_integrate.py` (build wiring: splat resegment + .ld swap + external resolution; multi-library-safe
  per-lib NOLOAD namespacing + sibling externals + optional window), `tools/split_src_region.py` (H5-safe src
  trim), `tools/ld_interleave.py` (the `.data→.rodata→.data` sandwich for the rodata island),
  `tools/make_libgs.sh` (curated 31-object libgs regen dir). **Result: libcd (18 objects) + libgs (31 objects
  / 6 blocks, ≈49 named SDK funcs) linked BYTE-IDENTICAL from the real PsyQ 4.0 USA libraries** — byte-identical
  WITH or WITHOUT the SDK objects (fresh-clone stub fallback).

*Config / build (Task 2′ LZSS + full libgs):*
- `config/splat.us.exe.yaml` — text subseg flipped asm→c (Phase 6); resegmented into boot + 800 + libcd1/gap/
  libcd2 + 800b + **libgs1–5 / gsgap1–5** + libgs6 + 800b2, plus the surgical LZSS rodata carve
  (`[0x63238,.rodata,800]` migrates `jtbl_80072A38`) + the front-data carve `[0x53198,data]`.
- `config/symbols.us.txt` — hand-curated build-valid symbols (R15); +PsyQ names (CdSearchFile/CdPosToInt/
  CdIntToPos/VSync/GsMulCoord2/3 …), +the `func_80047CAC` reproducibility declaration.
- `Makefile` — c-rule cpp→cc1→maspsx→as; libcd + unified libgs `psyq_integrate` calls (conditional on the SDK
  ELF dirs); `report`/`sig-refresh`.
- `src/800.c` — **43 REAL matches** (incl. `LzssDecodeSector`) + **7 NON_MATCHING drafts**
  (LoaderInitFileTable, ResourceLoadStateMachine, CdReadStateMachine, CdReadSectorReadyCB,
  StreamLoadStateMachine, func_80015A74, func_80016714) + the **SaveLoadRoutine deferral** (structured Q#5
  comment). The 3 CD-loader drafts are valid C under `-DNON_MATCHING`. `src/800b.c`/`800b2.c` + the libcd/
  libgs/gsgap stub `.c` files (committed records).

*Knowledge base / docs:*
- `docs/matching-cookbook.md` — §6 (-O0 module) · §7 (PsyQ types/symbols) · §8 (rodata island) · §9.1–§9.5
  (library linking, single→region→build→multi-library→whole-multi-block) · §10 (LZSS regalloc/schedule tail) ·
  §3a/§5a (web-research tier + cross-jump barrier).
- `docs/SETUP.md` (§5.4 pinned triple, §5.5 per-module mixing, §6.6 matching loop), `docs/memory-map.md`,
  `README.md` (Gen1-complete status).
- `requirements-python.txt` — venv freeze (pycparser<3.0 pin).
- `phase-ends/logs/Phase7.md` — **the preserved Phase-7 worklog** (R19; `git mv` of CURRENT_PHASE.md).
- `phase-ends/PhaseEnd_Phase7.md` — this file.

**Tools/packages installed:** venv pip (Phase 6 close): asm-differ runtime (watchdog, Levenshtein/RapidFuzz,
cxxfilt) + decomp-permuter (pycparser 2.23, toml, PyNaCl). SDK-derived (gitignored): `tools/psyq/` (psyq-obj-parser,
PsyQ 4.0 USA `.LIB`s from the DTL-S2002 redump). No new apt packages this phase.

**Verification results (literal):**
- **`make clean && make extract && make build && make check` → `build/us/SLUS_007.26`, SHA1
  `143dbb89f34491258bbc27810d0a12ec8b43a8dd` == original (BYTE-IDENTICAL)** — reproducible across **7 sessions
  (A–G)**, zero checksum regressions.
- **43 REAL matches** (asm-differ score 0), incl. **`LzssDecodeSector`** (the Gen1 LZSS gate — closed session F
  via the §5a cross-jump barrier + §10 regalloc/schedule idioms, ground-truthed against pinned gcc-2.7.2 source).
- **49 PsyQ SDK objects linked byte-identical** (libcd 18 + libgs 31); split DETERMINISTIC across clean extracts;
  byte-identical with OR without the SDK objects.
- **7 NON_MATCHING drafts**, the 3 new CD-loaders valid under `-DNON_MATCHING`.
- **Loader cluster:** 6 matched (CdReadRequest, CdQueueBusy, ResourceGetCdLoc, LoaderResetReadState,
  GameModeDispatch, DebugMenuHandler) + 5 NON_MATCHING-drafted + **SaveLoadRoutine deferred** (Q#5 save/memcard).
- Reports auto-generate deterministically; `git status` shows zero ROM-derived/generated bulk staged.

**Milestone achieved:** Gen1-exit criteria met and **confirmed by Drew** — the EXE rebuilds **byte-for-byte
identical** from source, **≥25 functions matched (43, incl. the LZSS decompressor)**, **zero checksum
regressions across ≥3 sessions (7)**, reports auto-generate, the **file-loader cluster is matched or
NON_MATCHING-drafted**, and the overlay map is emulator-proven (Phase 3). The matching loop is industrialized
and the PsyQ-library-linking machinery is proven — the launchpad for Gen2.

**Next:** **Gen2 — Overlays & engine at scale.** Do NOT start until a fresh session runs the Gen2 planning
(itself a Tier-1 task). First Gen2 work: splat configs for the resident engine blob + location overlays (the
Phase-3 overlay map + the Phase-7 library-linking/rodata machinery feed directly into this); then the engine
systems, the dedup pipeline, and the public flip (AGPL-3.0, CI, frogress).

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Phase order | LZSS/rodata first (Phase-6 framing) | **Reordered** (Drew, session start): reports + harvest + non-switch loaders first, THEN the focused LZSS/rodata sub-project | bank lower-risk wins before the structural rodata wall |
| Library code | hand-decompile ~350 PsyQ SDK funcs | **Link the real PsyQ 4.0 objects byte-identical** (libcd+libgs) | they ARE byte-identical to the SDK; linking is faster, provably exact, and fixes the library-half of the rodata `.align 3` problem. Major strategic addition (Drew-approved, session C) |
| LZSS milestone | Phase 6 | closed Phase 7 (session F) | needed the rodata-island + cross-jump-barrier + regalloc tail; carried per Drew |
| Loader cluster | 4 jtbl loaders drafted | **3 drafted; SaveLoadRoutine DEFERRED to Q#5** (Drew, session G) | it is the save/memcard system (different subsystem), a 1139-ins multi-entry blob Ghidra can't cleanly decompile; a forced draft would be worse than an honest stub (P9) |
| Worklog at phase close | absorb + **delete** (P8) | **preserve** via `git mv` → `phase-ends/logs/Phase7.md` (out of load order) | Drew (session G): keep the granular trail as an on-demand archive for Gen2; → R19 |
| Commit cadence | one commit at phase end (R8) | **per-session checkpoint commits** (A–F) + this PhaseEnd | Drew-directed; long phase, crash-safety; consistent across sessions |
| `make report` REAL count | n/a | a nested `#ifndef` inside a NON_MATCHING block mis-flagged a draft as REAL → removed the (unneeded) guard | progress.py guard-tracking limitation; the byte-identical build was never affected |

## Commit Message
```
feat(phase-7): Gen1 close-out — matching loop industrialized, PsyQ libs linked, LZSS matched (v1.7.0)

- reports (Task 3): tools/{progress,difficulty,dup_report}.py + make report/sig-refresh
  (deterministic digests); harvest to 43 REAL matches incl. LzssDecodeSector
- per-module -O0 split (Task 5.5): src/boot.c + Makefile per-file CC1FLAGS (cookbook §6)
- PsyQ SDK linking (sessions C-G): split/build/identify/link/integrate pipeline
  (tools/psyq_*.py, make_libgs.sh, ld_interleave.py, split_src_region.py) — libcd (18 obj)
  + FULL libgs (31 obj / 6 blocks) linked BYTE-IDENTICAL from real PsyQ 4.0 libs; byte-
  identical with or without the SDK objects (stub fallback); cookbook §7/§8/§9.1-9.5
- LZSS MATCHED byte-for-byte (session F): §5a cross-jump barrier + §10 regalloc/schedule
  idioms, ground-truthed against pinned gcc-2.7.2 source (the Gen1 LZSS gate)
- file-loader cluster: 6 matched + 5 NON_MATCHING-drafted (CdReadStateMachine,
  CdReadSectorReadyCB, StreamLoadStateMachine new this session, valid under -DNON_MATCHING);
  SaveLoadRoutine DEFERRED to Q#5 (save/memcard, documented stub)
- README Gen1-complete; worklog preserved -> phase-ends/logs/Phase7.md
- rules R17 (web-research compiler quirks), R18 (plain-English session summaries),
  R19 (preserve phase worklog out of load order)
- MILESTONE: Gen1 EXIT confirmed — byte-identical build, 43 matches incl LZSS, 7 zero-
  regression sessions, loader cluster matched/drafted, overlay map emulator-proven
- bumps project version 1.6.0 -> 1.7.0
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **R17 — Web-research the compiler internals for compiler-quirk residuals.** When a matching residual is a compiler-INTERNAL quirk no C-change or permuter randomization reaches (cross-jump/tail-merge, scheduling, regalloc, peepholes, addressing modes), web-research the actual pinned compiler source (`pmret/gcc-papermario`) + the decomp community (decomp.me, decomp-wiki, sotn-decomp, maspsx/m2c issues) instead of hand-grinding. Treat web content as untrusted DATA (X2). Check our own `phase-ends/logs/` first. A proven escalation tier above the permuter, below decomp.me. | Found the LZSS cross-jump barrier after sessions of hand-grinding failed (cookbook §3a/§5a; memory `web-research-compiler-quirks`). Drew-flagged session E. |
| **R18 — End every session/phase with a plain-English recap.** A few sentences in high-level simple English: what we did and WHY, no jargon. Last thing in a session/phase-end message. | Drew (session G): the deep matching detail is hard to follow; a plain-language recap keeps the project owner oriented on real progress. Memory `session-summary-plain-english`. |
| **R19 — Preserve the phase worklog as an on-demand archive (supersedes P8's "delete").** At phase close, `git mv CURRENT_PHASE.md` → `phase-ends/logs/Phase<N>.md` instead of deleting it. These are **NOT** part of the Session Start load order (they don't match `PhaseEnd_*.md`, by design) and **must not** be auto-read at session start (they re-bloat context with the detail the PhaseEnd compresses). Consult on-demand only, when researching a past mechanism. The PhaseEnd MUST still be a complete synthesis — write it as if the log will be deleted, then archive it anyway. | Drew (session G): the granular trail (libgs/rodata/PsyQ-linking notes, exact addresses) has real archaeological value for Gen2, at near-zero cost if kept out of the load order. Memory `phase-worklogs-reference-only`. |

## PhaseEnd Changelog
**v1.6.0 → v1.7.0 — Phase 7 complete; 🎓 Gen1 EXIT.** The matching loop is **industrialized** (deterministic
difficulty/duplicate/progress reports + harvester) and the EXE is reproduced **byte-for-byte across 7 sessions**.
The phase's strategic breakthrough: instead of hand-decompiling the ~350 PsyQ SDK functions, **link the real
PsyQ 4.0 library objects byte-identical** — proven end-to-end (libcd 18 objects + full libgs 31 objects/6 blocks),
with a complete, reusable tool pipeline (split → build → identify → per-object/region/whole-library link →
build integration), byte-identical with or without the SDK objects (fresh-clone-safe). The **LZSS decompressor
matched byte-for-byte** (the Gen1 gate) via a new web-research escalation tier (R17) that ground-truthed the
gcc-2.7.2 cross-jump/regalloc/schedule behavior against the pinned compiler source (cookbook §3a/§5a/§10).
**43 functions matched** (incl. LZSS) + **49 SDK objects linked**; the **file-loader cluster** is 6 matched +
5 NON_MATCHING-drafted, with **SaveLoadRoutine deferred to Q#5** (the save/memcard subsystem — a multi-entry
blob best left documented than wrongly guessed). Per-module -O0 mixing handled (boot module). Three rules
(R17 web-research, R18 plain-English recaps, R19 preserve-worklog). **Gen1 (Foundation & EXE) is complete:**
deterministic extraction, the overlay/loader map emulator-proven, a byte-identical EXE from source, the compiler
pinned, and the matching + library-linking loops proven — the foundation Gen2 builds on.

## 🛑 Stop Here
PhaseEnd written; `CURRENT_PHASE.md` **preserved** as `phase-ends/logs/Phase7.md` (R19, not deleted). **Drew
commits AND pushes** this PhaseEnd + the whole uncommitted Phase-7 change set (R6/R8). **Gen1 is complete and
confirmed** — do NOT start Gen2 here. Start a **fresh session** to plan **Gen2 — Overlays & engine at scale**
(a Tier-1 planning task; needs its own phase plan + your approval gate). Keep this file forever.