BFM-decomp/phase-ends/PhaseEnd_Phase2.md

# PhaseEnd — Phase 2: Deterministic disc & .CD extraction pipeline
**Date:** 2026-06-13 · **Project Version:** 1.2.0 · **Phase Status:** Complete

## Build Log

**Files created/changed and complete — do not recreate:**
- `tools/bfm_extract/extract.py` — unified full-disc CLI. Walks the ISO root (over the frozen `iso9660.Iso9660Image`), writes every file, splits `.CD`→PAC→type-4, writes the manifest; `--list` / `--verify`. Composes `cd_archive` + `pac` + `lzss` + `manifest`; reuses `extract_exe` EXE constants. Also contains the multi-track `AudioTrack` reader for the 3 `.DA` files.
- `tools/bfm_extract/cd_archive.py` — `.CD` TOC walker (`parse_toc`, `iter_subfiles`) + `classify_subfile` (pac/sqv/raw). Excludes LIST.CD by caller contract.
- `tools/bfm_extract/pac.py` — PAC chain splitter (`is_pac`, `split_pac`); `{index}.{type}` model with `+0x08` and full-header capture.
- `tools/bfm_extract/lzss.py` — game-semantics LZSS decoder (`pos==0` terminator, −1 bias, length cross-check). Pure function; ring re-zeroed per call.
- `tools/bfm_extract/test_lzss.py` — 5 standalone synthetic unit tests (literals, terminator, −1 bias, overlapping copy, cross-check warnings).
- `tools/bfm_extract/manifest.py` — deterministic JSON-Lines manifest (`build`/`render`/`write`/`verify`); `manifest.jsonl` + `manifest.sha1`.
- `tools/bfm_extract/crosscheck.py` — CUE `brave` oracle harness: builds brave, maps type-only names → `{index}.{type}`, byte-compares, classifies F6/shadowed, writes `_build/divergence.log`.
- `tools/brave-CUE/posix_shim.h` — glibc build shim (`filelength`→`fstat`; 1-arg `mkdir` macro) so the GPL `brave.c`/`common.inc` compile **unmodified**.
- `tools/bfm_extract/extract_exe.py` — added `EXPECTED_EXE_SHA1` constant (round-trip anchor).
- `.gitignore` — `/extracted/*` + `!`-exceptions for `SLUS_007.26`/`manifest.jsonl`/`manifest.sha1`; `/tools/brave-CUE/_build/`.
- `docs/SETUP.md` — §4.4 note: disc staged on ext4 (Phase 2), WSL `--verify-disc` PASS (closes Phase-1 deferral).
- `docs/formats.md` — §2.2 verified US sub-file counts; §3.1 `+0x08` finding; §3.2 type distribution; §4.4 F6 resolution; §5/§7 SQV/.DA; §8 F1/F2/F4/F5/F6/F7 status + Phase-2 summary block.
- `CLAUDE.md` — Reasoning & Model Protocol: corrected the "Ultracode is an effort level" error; added the Effort-map check rule (→ R7); Session Start now states next-task effort.
- `docs/effort-map.md` — **new** evolvable reference: verified effort mechanics, two-axis depth/breadth model, Max-default policy, per-phase Gen1 map.
- `phase-ends/CURRENT_PHASE.md` — **absorbed into this file and deleted** (P8).

**Local artifacts (gitignored / outside the repo — present on this machine, do NOT recreate; a fresh clone must redo them):**
- `disks/Brave Fencer Musashi (USA) (Track 1..4).bin` + `.cue` — all 4 tracks copied once from `/mnt/z` (Track 1 = 364,846,944 B; tracks 2-4 = CD-DA audio). Gitignored.
- `extracted/` bulk — 1801 artifacts (raw `.CD`, `FILE_%03d`, `{index}.{type}`, `.dec`, FMV, `.DA` audio). Regenerable via `extract.py`. **Committed** subset: `manifest.jsonl`, `manifest.sha1`, `SLUS_007.26`.
- `tools/brave-CUE/_build/brave` (+ `divergence.log`) — compiled oracle + scratch (gitignored).

**Tools/packages installed:** gcc 13.3.0 (`Ubuntu 13.3.0-6ubuntu2~24.04.1`) — to build the `brave` cross-validation oracle (user-approved; `build-essential` is a Phase-4 dependency regardless).

**Verification results (literal):**
- LZSS unit tests: **5/5 pass**.
- `extract.py`: 27 files (24 Track-1 verbatim + 3 `.DA` raw CD-DA audio); 447 sub-files, 348 PAC chains, **1189 PAC entries** (138 type-4 decoded, **0 LZSS warnings**), 98 SQV. **EXE round-trip PASS** (SHA1 `143dbb89f34491258bbc27810d0a12ec8b43a8dd`).
- Manifest: **1801 artifacts**; `manifest.sha1 = e5c6019249dec3c71bb2cf40658a78439dddb037`; **identical across re-runs** (deterministic); `--verify` **PASS**.
- `crosscheck.py --sample all` (brave built with gcc): **1484 raw payloads byte-identical** (0 diff); **138/138 type-4 F6** (ours = the game-correct prefix; brave over-decodes exactly 2 B per stream = 276 B total); **0 raw mismatches, 0 decode mismatches — PASS**. 14 "shadowed" entries in MAIN.CD confirm the brave naming collision our `{index}.{type}` fixes.
- `git status`: only tooling/docs/manifest tracked; bulk payloads + `_build/` ignored (**zero ROM-derived files staged**).
- Open-question results: **F5** US counts (MAIN=49, SC01=86, **SC02=43**, SC03=140, SC04=31, SC05=30, SC06=39, SC07=29); **F2** `+0x08`=0 for all type-4 (size-hint refuted; non-zero only type 0=301, type 6=29); **F1** type 7=139 (common), no type 5; **F6** resolved (no within-payload divergence).

**Milestone achieved:** `python3 tools/bfm_extract/extract.py` extracts the full disc in one command, MAIN.CD yields its 49 sub-files, the decompressed type-4 payloads are byte-identical to CUE's `brave` up to the game `pos==0` terminator (138/138), the SHA1 manifest is deterministic, and `git status` shows zero ROM-derived files staged.

**Next:** Phase 3 — File-loader & overlay-map RE (+ PCSX-Redux harness). Start with Ghidra seeded with the provenance-tagged ledger from `docs/memory-map.md`. (Phase 3 is RE work — MCP precondition G2 applies; and per `docs/effort-map.md` the US address derivation + RAM-dump proof are **Max** tasks.)

## Deviations
| Item | Plan | Actual | Reason |
|---|---|---|---|
| Disc staging | Track 1 only | All 4 tracks + `.cue` staged on ext4 | User directed extracting the `.DA` files, whose data lives in tracks 2-4 |
| `.DA` files | Out of scope / skip | Extracted as raw 2352-B/sector CD-DA audio (tracks 2-4) | User "extract every byte"; `.DA` ISO entries point past Track 1 into the audio tracks |
| Cross-val tool | brave.exe (Windows) via wibo/wine | Build `brave.c` from source via `posix_shim.h` (gcc) | Cleaner than Wine; GPL upstream untouched; gcc is a Phase-4 dep anyway (user-approved early install) |
| Commit cadence | One commit per task (P4/R6) | One commit at phase end | User directive 2026-06-13 (→ rule R8) |
| Effort-map work | (not in the phase plan) | Added `docs/effort-map.md` + R7 + CLAUDE.md correction at session start | User requested before planning the phase |
| F4 (LIST.CD regen) | Resolve during the run | Deferred (extractor excludes LIST.CD; regen+diff is a cheap follow-up) | Optional; not a milestone item |
| Output tree | (raw split only) | Keep raw `.CD` + raw sub-files + `{index}.{type}` + `.dec` (all gitignored bulk) | Faithful "every byte" + diffable-against-brave structure |

## Commit Message
```
feat(phase-2): deterministic disc/.CD/PAC/LZSS extraction pipeline

- extract.py: one-command full-disc extraction (27 files: 24 Track-1 verbatim
  + 3 .DA as raw CD-DA audio from tracks 2-4) over the frozen iso9660.py
- cd_archive.py + pac.py: .CD TOC walker + PAC splitter, {index}.{type} naming
  (no collisions; 447 sub-files, 1189 entries), SQV detection (98)
- lzss.py: game-semantics decoder (pos==0 terminator, -1 bias) + 5 unit tests;
  138 type-4 decoded clean (0 warnings)
- manifest.py: deterministic JSON-Lines manifest (1801 artifacts) + --verify
- crosscheck.py + tools/brave-CUE/posix_shim.h: build CUE's brave from source
  (GPL untouched), byte-compare; 1484 raw identical, 138/138 type-4 F6
- docs/formats.md: F5 counts (SC02=43), F2 refuted, F6 resolved, F1 type-7=139
- docs/effort-map.md + CLAUDE.md: Effort-map check rule (R7), Ultracode fix
- .gitignore: /extracted/* + !manifest/EXE exceptions; brave _build/
- install gcc 13.3.0 for the oracle
- milestone: one command extracts the full disc; MAIN.CD=49 sub-files; decoder
  agrees byte-for-byte with CUE up to the game pos==0 terminator; zero ROM
  staged; project version 1.1.0 -> 1.2.0
```

## Rules Added This Phase
| Rule | Reason |
|---|---|
| **R7 — Effort-map check.** Before each Phase Start, state the recommended effort for planning that phase per `docs/effort-map.md` and confirm the developer has it set; annotate every task in the plan with its effort; restate the recommended effort whenever presenting the NEXT task. `docs/effort-map.md` is the evolvable source of truth (it also records the verified harness mechanics: effort ladder `low..xHigh/Max`; Max & Ultracode are session-only; Ultracode = xHigh + multi-agent orchestration, not a deeper level). | User requested a persistent effort guide and noticed Ultracode "reverting to xHigh"; corrects CLAUDE.md's mislabeling of Ultracode as an effort level. |
| **R8 — One commit at phase end (supersedes the per-task-commit part of P4/R6).** Work accumulates UNCOMMITTED during a phase; `CURRENT_PHASE.md` is the per-task crash-recovery log; a single commit lands at phase close. The user commits AND pushes it (R6 phase-boundary exception, extended to the whole phase). | User directive 2026-06-13 — fewer commits to manage with the WSL-commit / GitHub-Desktop-push workflow. |

## PhaseEnd Changelog
**v1.1.0 → v1.2.0** — Phase 2 complete. The deterministic disc/.CD/PAC/LZSS extraction pipeline is live (pure-stdlib Python over the frozen `iso9660.py`): one command extracts the full disc into 1801 manifest-tracked artifacts; the game-semantics LZSS decoder decodes all 138 type-4 overlays cleanly and is byte-validated against CUE's `brave` (built from source). Open questions resolved: F5 (US sub-file counts, SC02=43), F2 (`+0x08` is not a size hint), F6 (no within-payload game/tool divergence); F1 partially (type 7 common). Two rules added (R7 effort-map check, R8 one-commit-at-phase-end). gcc 13.3.0 installed. Deviations: all 4 tracks staged + `.DA` extracted as raw audio (user scope change).

## 🛑 Stop Here
Start a fresh session for the next phase. Keep this file forever.