diff --git a/docs/SETUP.md b/docs/SETUP.md index 3bcd8e29b..66de46282 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -400,7 +400,7 @@ Phase 18 (raise the match-% ceiling by understanding gcc-2.7.2's blocking codege | **vanilla gcc-2.7.2** (the REAL source of our pinned cc1) | authoritative source for the codegen map (cookbook §31/§34 + `docs/gcc-2.7.2-map/`): `sched.c`/`reorg.c` (scheduling), `local-alloc.c`/`global.c`/`reload1.c` (regalloc), `loop.c` (IV/hoist), `cse.c`/`expr.c` (CSE/aliasing). Staged Phase 23; **completed Phase 24 T5** (18 `.c` files — +toplev.c/function.c/flow.c/… from the FSF tarball; sched.c/loop.c/mips.c verified byte-identical to vanilla → §34's `local-alloc.c:1441` 3-qty-sort-bug citation is sound). **P31 T2: +6 files** — `calls.c` + `caller-save.c` (both cited by §172's frame-residue producer model, previously MISSING) + `integrate.c`/`optabs.c`/`varasm.c`/`recog.c` — from GNU ftp `gcc-2.7.2.tar.gz`, sha256 `7cd8bce5c3aeec59a72ecc2d3d5123864a817b14cdbd0680b1a969c3bccc5da5`. | `tools/reference/gcc-2.7.2` | GNU 2.7.2 release | (re-fetch: GNU ftp `gcc-2.7.2.tar.gz`; tarball kept at `.run/t7/fable/gcc-2.7.2.tar.gz`) | | `ladysilverberg/xenogears-decomp` | Square, Oct 1998, **gcc-2.7.2-psx + -cdk** (our exact compiler); mine transferable quirk idioms + `gears.toml` per-module presets | `tools/reference/xenogears-decomp` | `main` | `f27c0768b1ad10812cec776cadadb85ae70aadee` | | **PsyQ Runtime Library 4.2 + the libpad 4.2.1 patch** (P31 S79, 2026-09-04, task #13) | archive.org item `play-station-programmer-tool-runtime-library-version-4.2.7z` (383 KB) → tracked as `tools/psyq/PlayStation_Programmer_Tool_-_Runtime_Library_Version_4.2.7z` (sha256 in `CHECKSUMS.sha256`) → `tools/psyq/lib42/` (the 4.2 LIBs, 1998-01-21) + `tools/psyq/lib421/` (`LIB/42PATCH/J421PD.ZIP` = SCE R&D "libpad.lib 4.2.1 for the DUAL SHOCK", 1998-02-26: LIBPAD.LIB 4.2.1 + LIBAPI.LIB 4.2 + LIBPAD.H/LIBAPI.H/KERNEL.H) → ELF via `psyq_lib_split.py` + `psyq-obj-parser` into `.run/obj42/{libpad421,libapi42}`. **This is the EXE's libpad/libapi**: 7 + 39 objects place and link byte-identical (`psyq_link.py` PASS ×46) — the whole 0x8005CE18–0x8005FC68 band + the apicard-region libapi pieces. The 4.2.1 headers are the band's prototype oracle | `tools/psyq/lib42/`, `lib421/` (gitignored, regenerable) | — | — | -| **PsyQ 4.6 libraries** (P31 S78, 2026-09-04) | `Psy-Q_46.zip` (psx.arthus.net; sha256 in `tools/psyq/CHECKSUMS.sha256`) → `tools/psyq/lib46/*.LIB` (23 libs) → ELF via `psyq_lib_split.py` + `psyq-obj-parser` into `.run/obj46//`; placed against the EXE (`.run/survey46/`). Only `PDMAIINI.o` is byte-identical in the libpad-4.2.1 band | `tools/psyq/lib46/` (gitignored, sha-recorded) | — | — | +| **PsyQ 4.6 libraries** (P31 S78, 2026-09-04) | `Psy-Q_46.zip` (psx.arthus.net; sha256 in `tools/psyq_CHECKSUMS.sha256`) → `tools/psyq/lib46/*.LIB` (23 libs) → ELF via `psyq_lib_split.py` + `psyq-obj-parser` into `.run/obj46//`; placed against the EXE (`.run/survey46/`). Only `PDMAIINI.o` is byte-identical in the libpad-4.2.1 band | `tools/psyq/lib46/` (gitignored, sha-recorded) | — | — | | **PsyQ 4.5 toolkit** (P31 S78) | `PSYQ_SDevTC_v4.5.zip` (psx.arthus.net; 7 split zips, unpacked on demand; sha-recorded) | `tools/psyq/` | — | — | | **PsyQ 4.0 dev-CD SDK tree** (P31 T2, 2026-08-14) | the SDK **sample C source** (400 `.c`: 373 in `PSX/SAMPLE/` — CD/GRAPHICS/SOUND/MODULE/CMPLR/…) = the canonical source shapes era game code was written from → drafting-agent style priors, esp. for main (see `docs/psyq-sample-idioms.md`). Also `GNU/SNGNUVER.TXT` — SN's build history (`2.7.2.SN32.3.7.0002`, 14.5.97) naming SN's exact patches vs vanilla (only `UNROLL.C` codegen-relevant). | `tools/reference/psyq-sdk` | n/a (extracted, not cloned) | regenerate: walk the on-disk Track-1 image (`tools/psyq/…(Track 1).bin`, MODE2/2352) with `tools/bfm_extract/iso9660.py` — 2,374 files / 231.6 MB, 7 out-of-track `.DA` audio skipped (script pattern preserved in `phase-ends/logs/` P31 T2) | @@ -645,7 +645,7 @@ Full how-to in `docs/matching-cookbook.md` §11. Command crib: `h_norm` is self-consistent within the overlay fleet (not Ghidra-byte-exact — D2); overlay boundaries via linear partition + `detect_code_end` (BFS fails — overlays dispatch via function-pointer tables, not `jal`). - **PsyQ provenance (R24)**: the resident is PsyQ **4.7** (`tools/psyq/conv47/`, sha-recorded in - `tools/psyq/CHECKSUMS.sha256`) — Phase 12 links its embedded SDK code from 4.7, not the EXE's 4.0 libs. + `tools/psyq_CHECKSUMS.sha256`) — Phase 12 links its embedded SDK code from 4.7, not the EXE's 4.0 libs. ## §6.9 Running a matching wave — see `docs/wave-playbook.md` @@ -1069,6 +1069,34 @@ fills fast). Nothing is leaking — but the host does not get the memory back on * `gate_main` **REFUSES a draft containing its own `INCLUDE_ASM`** (substituting it restores the stub, so the build passes for free and the function counts as banked), and counts banks from the SOURCE. +### P33 B4 (S86, 2026-09-06) — `tools/fetch_psyq.sh`: the OPTIONAL PsyQ SDK objects, user-supplied and verified +- **What it is for.** Byte-identity never needs Sony's libraries (without them main links its INCLUDE_ASM tiles — the + fresh-clone proof in B3 built all 218 that way). `tools/fetch_psyq.sh [--disc ] [--from ] + [--no-build]` is for a developer who wants `make check BINARY=main` to link Sony's REAL objects (the 1,256 LINKED + functions): it obtains and verifies every piece against `tools/psyq_CHECKSUMS.sha256` (rows relative to `tools/psyq/`, + which is gitignored — nothing Sony-copyrighted is redistributed), then runs the existing builders and `make sdk-dual`. +- **Pieces and provenance:** `psyq-obj-parser` — decomp.me's release tarball + `https://github.com/decompme/compilers/releases/download/compilers/psyq-obj-parser.tar.gz` (tarball sha256 `353495f1…`, + the static x86-64 binary inside `4fba623a…` — identical to the copy this project used since Phase 7); the **Runtime + Library 4.2** 7z — archive.org `https://archive.org/download/play-station-programmer-tool-runtime-library-version-4.2.7z/ + PlayStation_Programmer_Tool_-_Runtime_Library_Version_4.2.7z` (383,431 B, sha256 `e4f5a678…`) → `tools/psyq/lib421/` + (the `LIB/42PATCH/J421PD.ZIP` payload: LIBPAD.LIB 4.2.1 + LIBAPI.LIB 4.2, both sha256-rowed); the **20 PsyQ 4.0 libraries + — NO public download**: they are `PSX/LIB/*.LIB` on the redump disc *PlayStation - Programmer Tools - Run-time Library + 4.0 (USA) (PC-CD ROM Release 2.0)* (DTL-S2002, ISO volume `PROGTOOL`); `tools/psyq_libs_from_disc.py + ` extracts them with the project's own ISO9660 walker (`tools/bfm_extract/iso9660.py`, the same reader + `make disc-extract` uses) — every LIB is checked against its recorded sha256 (LIBCD.LIB from the disc == the copy used + since Phase 7). Without `--disc`/`--from` and without `tools/psyq/lib40/` the script REFUSES and names the disc (R43). +- **Builders it runs (unchanged, all regenerable):** `tools/psyq_build_libs.sh LIBCD LIBETC LIBGPU LIBMCRD LIBC2 LIBGTE + LIBGS LIBSPU LIBSND LIBCARD` (→ `.run/obj40/` + `tools/psyq/lib40_elf/*.a`), `tools/make_libgs.sh` (→ + `.run/obj40/libgs_used`), `tools/make_snd_used.py` (→ `.run/obj40/snd_used`), the lib421 ELF step (→ + `.run/obj42/{libpad421,libapi42}`), `tools/make_apicard_used.py` (→ `.run/obj42/apicard_used`); then `make -j$(nproc) + sdk-dual`. Which region needs which: `.run/obj40/{libcd,libetc,libgpu,libmcrd,libc2,libgte}` + `libgs_used` + `snd_used` + from the 4.0 LIBs; `.run/obj42/{apicard_used,libapi42,libpad421}` from LIBCARD 4.0 + the 4.2 archive. `Psy-Q_46.zip`, + `PSYQ_SDevTC_v4.5.zip`, `conv47/` and the 4.0 Win32 tools feed NO linked region (research artifacts; archive repo only). +- **Controls (S86):** no source → refuse (rc 1, the message); `--from` a directory with one corrupted LIB → `sha256 MISMATCH + for tools/psyq/lib40/LIBTAP.LIB`, rc 1, nothing built; the disc path → see `phase-ends/CURRENT_PHASE.md` B4 for the run. +- `tools/psyq/CHECKSUMS.sha256` → **`tools/psyq_CHECKSUMS.sha256`** (git mv; the directory it lived in leaves git at C3). + ### P33 B3 (S86, 2026-09-06) — `tools/bootstrap.sh` / `make bootstrap`: the fresh-clone setup - **`tools/bootstrap.sh`** (idempotent, never sudo): apt packages of §4.5 checked with `dpkg -s` and the missing ones printed as ONE `sudo apt-get install` line (never run); `.venv` created from `requirements-python.txt` (pinned; a no-op when diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 4026bcba2..6a976b990 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -53,7 +53,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo re-tightened; the 4 splat preset headers tracked; `clean` fixed) — Max — see Log 2026-09-06 B1 - [x] **B2** 34 absolute includes → `../shared/` + portable-include audit + 15-binary re-gate — xHigh — see Log 2026-09-06 B2 - [x] **B3** `tools/bootstrap.sh` / `make bootstrap` + check-env extensions + fresh-clone proof 218/218 — xHigh — see Log 2026-09-06 B3 -- [ ] **B4** `tools/fetch_psyq.sh` + CHECKSUMS rows (20 lib40 LIBs, psyq-obj-parser) BEFORE `tools/psyq/` leaves git — xHigh +- [x] **B4** `tools/fetch_psyq.sh` + CHECKSUMS rows (20 lib40 LIBs, psyq-obj-parser) BEFORE `tools/psyq/` leaves git — xHigh — see Log 2026-09-06 B4 - [ ] **B5** Ghidra regenerability (ExportAnnotations/ImportAnnotations/ghidra_rebuild.sh `--proof` on SLUS_007.26 + resident; roster; ExportSymbols R15 fix; path hardcodes; hooks) — Max, MCP stopped - [ ] **B6** `dumps/CHECKSUMS.sha1` + INDEX.md rewrite + memory-map Source-index row — Low/xHigh @@ -158,16 +158,30 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact `tools/bootstrap.sh` created its own `.venv`, extracted both cc1s, fetched the submodules, `check-env: OK` → `disks` symlinked → `disc-extract: OK` → `extract-all: 217 extracted, 0 failed of 217` → **`check-all: 218 passed, 0 failed of 218`**, EXIT=0, **4 m 18 s wall** (user 45 m 48 s) — with NO SDK objects in the clone (`.run/obj40` absent), i.e. every binary built the way a - public user builds it. Clone deleted afterwards (3.1 GB). SETUP: P33 B3 section (R21). + public user builds it. Clone deleted afterwards (3.1 GB). SETUP: P33 B3 section (R21). Commit `commit:4018`. +- **2026-09-06 (S86) — B4 (the optional SDK, user-supplied and verified).** `git mv tools/psyq/CHECKSUMS.sha256 + tools/psyq_CHECKSUMS.sha256` (done here rather than in C3 so the script's path is final) + rows for the parser tarball + (`353495f1…`, decomp.me's release), the parser binary (`4fba623a…`, identical to our Phase-7 copy) and the 20 PsyQ 4.0 + LIBs; `docs/SETUP.md` + `.gitignore` references repointed. New `tools/psyq_libs_from_disc.py` (the 20 `PSX/LIB/*.LIB` off + the DTL-S2002 redump disc via our own ISO walker; refuses a disc without `PSX/LIB`; asserts 20) and `tools/fetch_psyq.sh + [--disc|--from] [--no-build]` (parser → RTL 4.2 7z → lib421 → the 4.0 LIBs, each sha256-verified before use; then + `psyq_build_libs.sh` ×10, `make_libgs.sh`, `make_snd_used.py`, the lib421 ELF step, `make_apicard_used.py`, `make sdk-dual`). + **Controls:** no 4.0 source → refuses naming the redump title (rc 1) after verifying the parser + 4.2 pieces; `--from` a dir + with one corrupted LIB → `sha256 MISMATCH for tools/psyq/lib40/LIBTAP.LIB`, rc 1, nothing built. **The run:** with + `tools/psyq/lib40/` moved aside, `fetch_psyq.sh --disc ""` extracted the 20 LIBs from the disc + (`diff -rq` against the tracked copies: identical), verified them, rebuilt `.run/obj40/*`, `.run/obj42/*`, + `tools/psyq/lib40_elf/`, and `sdk-dual: OK` — **36.9 s wall**, EXIT=0 (`.run/P33/b4_fetch.log`). SETUP: P33 B4 (R21). -## 🛑 SESSION CHECKPOINT — A1–A4 ✓, B1–B3 ✓; NEXT = B4 (`tools/fetch_psyq.sh` + the SDK checksums, xHigh) (2026-09-06, written by session bd19e14a "S86"; SUPERSEDES the earlier blocks) +## 🛑 SESSION CHECKPOINT — A1–A4 ✓, B1–B4 ✓ (8 tasks; P6 rules check done after B4); NEXT = B5 Ghidra regenerability (Max, MCP stopped) (2026-09-06, written by session bd19e14a "S86"; SUPERSEDES the earlier blocks) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest -(R1–R73), then WAIT for Drew. **NEXT = B4** (xHigh: `tools/fetch_psyq.sh` — see §3 step 1 for the measured facts). If -`git log -1 --format=%s` does not start with `feat(phase-33): B3`, B3's commit did not land: `tools/bootstrap.sh` must print -`check-env: OK` (1–2 s on this tree) before committing; the proof log is `.run/P33/b3_fresh.log`. +(R1–R73), then WAIT for Drew. **NEXT = B5** (Max; the Ghidra text export + rebuild proof — the plan's Block B5 and §3 step 1 +below; the headless MCP server holds the project lock: `tools/ghidra_mcp_stop.sh` FIRST, then every headless step; the +SessionStart hook restarts the server next session and R29 applies). If `git log -1 --format=%s` does not start with +`feat(phase-33): B4`, B4's commit did not land: `tools/fetch_psyq.sh --no-build` must print `verified …` for every piece and +exit 0 before committing (`.run/P33/b4_fetch.log` is the full run). ### 1. Where we are **Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). R65–R73 ratified at @@ -236,27 +250,23 @@ items can be cut). Harness tasks: #1 A1 done, #2 A2 done, #3 A3 next … #41 G2. ### 3. NEXT — in order 0. **Preflight:** `git status --short | grep -v ghidra/` (empty) · `git log -1 --format='%h %s'` · `df -h ~` (≈13 GB free) · `.venv/bin/python -c 'import splat'` · `ls build/us/SLUS_007.26.map` (main is built; `make check BINARY=main` if not). -1. **B4** (xHigh) — measured facts (S86): the 20 PsyQ **4.0** LIBs come from the redump disc "PlayStation - Programmer Tools - - Run-time Library 4.0 (USA) (PC-CD ROM Release 2.0)" (DTL-S2002; our copy: `tools/psyq/PlayStation - Programmer Tools - - Run-time Library 4.0 (USA) (PC-CD ROM Release 2.0) (Track 1).bin`, 270 MB, gitignored; volume `PROGTOOL`) — its ISO - directory `PSX/LIB/` holds exactly the 20 `*.LIB` (names `LIBCD.LIB;1` …), and `tools/bfm_extract/iso9660.py` - (`img.find('PSX')` → `img.find('LIB', psx)` → `img.iter_directory(lib)` / `img.extract_file(name, lib)`) extracts them - byte-identical to `tools/psyq/lib40/` (LIBCD.LIB sha256 `db4e1a71…` both ways). The RTL 4.2 7z's archive.org URL is - `https://archive.org/download/play-station-programmer-tool-runtime-library-version-4.2.7z/PlayStation_Programmer_Tool_-_Runtime_Library_Version_4.2.7z` - (383,431 B; sha256 already recorded `e4f5a678…`). `psyq-obj-parser`: `https://github.com/decompme/compilers/releases/ - download/compilers/psyq-obj-parser.tar.gz` — tarball sha256 `353495f13f6756773cd905ba3a0743843232f95719e306fc6ee56cbf7d731a3e` - (3,507,359 B), the binary inside is BYTE-IDENTICAL to our tracked `tools/psyq/psyq-obj-parser` (sha256 - `4fba623a31bb5830ceea95b5081bc13504629686b0b4904c533b7be2a5f2377a`). Plan: `git mv tools/psyq/CHECKSUMS.sha256 - tools/psyq_CHECKSUMS.sha256` NOW (the plan put it in C3; doing it in B4 keeps the script's path final) and add rows for the - 20 lib40 LIBs + the parser tarball + the parser binary; `tools/fetch_psyq.sh [--disc ] [--from DIR]`: - (1) parser: download + verify or reuse; (2) RTL 4.2 7z: download + verify → `lib42/` + `lib421/` (the J421PD.ZIP payload) - → `.run/obj42/{libpad421,libapi42}` via `psyq_lib_split.py` + the parser; (3) the 4.0 LIBs: from `--disc` via the ISO - walker (new `tools/psyq_libs_from_disc.py`) or `--from DIR`, verified against the checksum rows → `tools/psyq/lib40/`; - REFUSE with the redump title when neither is given and `lib40/` is absent; (4) the builders: `tools/psyq_build_libs.sh` - (LIBCD LIBETC LIBGPU LIBMCRD LIBC2 LIBGTE LIBGS LIBSPU LIBSND LIBCARD), `tools/make_libgs.sh`, `tools/make_snd_used.py`, - `tools/make_apicard_used.py`; (5) `make -j$(nproc) sdk-dual`. Verify on this machine by rebuilding into the real dirs - (they are regenerable) and the dual passing; the checksum rows are verified by `sha256sum --check`. -2. Then B5 (MCP stopped) → B6 → B7 → B8 → A5 → B9/C3 … per the task list. After every task: tick the box, +1. **B5** (Max) — facts gathered (S86): the existing scripts and their conventions are in `tools/ghidra_scripts/` + (`ExportSymbols.java` symbols-only, hardcoded output + blind overwrite — the R15 violation to fix; `ApplySymbols.java` the + proven text→DB mirror with `BFMAPPLY` counts; `DumpFunctionSignatures.java` has the `jsonEsc` helper + JSONL style to reuse; + `DefineFunctions.java` reads `~/bfm-decomp/.run/_funcs.txt` (hardcoded `user.home`); `ImportPsyqGdt.java` hardcodes + `/home/musashi/ghidra_12.1_PUBLIC/.../psyq400.gdt`); wrappers `tools/ghidra_import.sh ` (PSX loader auto-detect + + analysis + gdt + info; `-overwrite`), `tools/ghidra_import_raw.sh ` (BinaryLoader; stages the blob to + `.run/`), `tools/ghidra_apply_symbols.sh [PROG] [files…]`, `tools/ghidra_mcp_{start,stop,verify}.sh` — ALL hardcode + `PROJ_DIR="$HOME/bfm-decomp/ghidra"` and `SCRIPTS="$HOME/bfm-decomp/tools/ghidra_scripts"`; `tools/prefetch_fleet.py` + (`vram_of(ov)` from the yaml, `import_overlay`, `run_define`, `mcp_stop_if_running`). The project has **129 programs** + (`ghidra/bfm.rep/idata/*/*.prp`, `.prp` XML with `STATE NAME=… VALUE=…`), `.run/_funcs.txt` exists for 125. The MCP + server is SERVING on :8080 (two analyzeHeadless pids) — stop it first. Headless `-process` without a name processes every + program in the project folder (one `-readOnly` run can export all 129). Design per the plan: `ExportAnnotations.java + [symbols…]`, `ImportAnnotations.java `, `tools/ghidra_rebuild.sh [--proof|--into-live]` + (scratch project under `.run/ghidra_rebuild/`), baseline-before-import to filter analysis noise, negative control, roster + `config/ghidra/ROSTER.md`, proof REQUIRED on `SLUS_007.26` + `resident`; the R15 fix + the path hardcodes + the + `.claude/settings.json` hooks. +2. Then B6 → B7 → B8 → A5 → B9/C3 … per the task list. Next P6 rules check after B8 (12 tasks done). After every task: tick the box, add a Log line, refresh this checkpoint block (the 🛑 block is the ONLY in-phase context the next session inherits), commit. Next P6 rules check after B4 (8 tasks done). `extracted/proto/` (sep8 + aug31 EXEs) is regenerated from the prototype discs by `tools/bfm_extract/extract_proto_exe.py` (docstring examples) — B5 needs both files. diff --git a/tools/fetch_psyq.sh b/tools/fetch_psyq.sh new file mode 100644 index 000000000..f2fa15ecb --- /dev/null +++ b/tools/fetch_psyq.sh @@ -0,0 +1,132 @@ +#!/usr/bin/env bash +# tools/fetch_psyq.sh — OPTIONAL: obtain Sony's PsyQ SDK pieces and build the ELF object dirs that main's link +# swaps in for its INCLUDE_ASM library tiles (P33 B4). The build is byte-identical WITHOUT any of this +# (`make sdk-dual` proves both legs); this only makes `make check BINARY=main` link Sony's REAL objects. +# The repository redistributes none of it (tools/psyq/ is gitignored); every artifact is verified against +# tools/psyq_CHECKSUMS.sha256 before use (rows are relative to tools/psyq/). +# +# tools/fetch_psyq.sh [--disc ] [--from ] [--no-build] +# +# What is needed, and where it comes from: +# psyq-obj-parser decomp.me's release tarball (URL below); tarball + binary sha256 recorded. +# RTL 4.2 + libpad 4.2.1 archive.org (URL below) -> tools/psyq/lib421/ (the J421PD.ZIP payload: LIBPAD.LIB 4.2.1 +# + LIBAPI.LIB 4.2) -> .run/obj42/{libpad421,libapi42}. +# PsyQ 4.0 libraries (20) NO public URL. They are PSX/LIB/*.LIB on the redump disc "PlayStation - Programmer +# Tools - Run-time Library 4.0 (USA) (PC-CD ROM Release 2.0)" (DTL-S2002, volume +# PROGTOOL): pass its Track-1 .bin with --disc (tools/psyq_libs_from_disc.py extracts +# them with the project's own ISO walker) or a directory holding the 20 files with +# --from. Without either, and without tools/psyq/lib40/ already present, this REFUSES. +# Then the existing builders run — psyq_build_libs.sh (10 libs -> .run/obj40/), make_libgs.sh +# (.run/obj40/libgs_used), make_snd_used.py (.run/obj40/snd_used), the lib421 ELF step, make_apicard_used.py +# (.run/obj42/apicard_used) — and `make -j$(nproc) sdk-dual` proves main byte-identical WITH and WITHOUT them. +set -euo pipefail +REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +cd "$REPO" +say() { printf 'fetch_psyq: %s\n' "$*"; } +die() { printf 'fetch_psyq: %s\n' "$*" >&2; exit 1; } + +PARSER_URL='https://github.com/decompme/compilers/releases/download/compilers/psyq-obj-parser.tar.gz' +RTL42_URL='https://archive.org/download/play-station-programmer-tool-runtime-library-version-4.2.7z/PlayStation_Programmer_Tool_-_Runtime_Library_Version_4.2.7z' +RTL42='PlayStation_Programmer_Tool_-_Runtime_Library_Version_4.2.7z' +CK="$REPO/tools/psyq_CHECKSUMS.sha256" +PSYQ="$REPO/tools/psyq" +SCRATCH="$REPO/.run/psyq_fetch" +DISC=""; FROM=""; BUILD=1 +while [ $# -gt 0 ]; do + case "$1" in + --disc) DISC="$2"; shift 2 ;; + --from) FROM="$2"; shift 2 ;; + --no-build) BUILD=0; shift ;; + -h|--help) sed -n '2,24p' "$0"; exit 0 ;; + *) die "unknown argument: $1 (see --help)" ;; + esac +done +mkdir -p "$PSYQ" "$SCRATCH" + +# verify ONE row of the checksum file for a path relative to tools/psyq/ (exit 1 on mismatch or no row) +ck1() { + local rel="$1" + local want; want=$(grep -E " $rel\$" "$CK" | awk '{print $1}' | head -1) + [ -n "$want" ] || die "no checksum row for $rel in $CK" + local got; got=$(sha256sum "$PSYQ/$rel" | awk '{print $1}') + [ "$got" = "$want" ] || die "sha256 MISMATCH for tools/psyq/$rel: got $got, recorded $want — refusing to use it" + say "verified tools/psyq/$rel" +} +fetch() { # url -> tools/psyq/, unless a verified copy is already there + local url="$1" name="$2" + if [ -f "$PSYQ/$name" ] && ck1 "$name" 2>/dev/null; then return 0; fi + say "downloading $name" + curl -fL --retry 3 --max-time 600 -o "$PSYQ/$name.part" "$url" + mv "$PSYQ/$name.part" "$PSYQ/$name" + ck1 "$name" +} + +# 1) psyq-obj-parser (the .OBJ -> ELF converter every builder needs) +if [ -x "$PSYQ/psyq-obj-parser" ] && ck1 psyq-obj-parser 2>/dev/null; then + : +else + fetch "$PARSER_URL" psyq-obj-parser.tar.gz + tar xzf "$PSYQ/psyq-obj-parser.tar.gz" -C "$SCRATCH" + p=$(find "$SCRATCH" -type f -name psyq-obj-parser | head -1) + [ -n "$p" ] || die "psyq-obj-parser not found inside the tarball" + install -m 0755 "$p" "$PSYQ/psyq-obj-parser" + ck1 psyq-obj-parser +fi + +# 2) the Runtime Library 4.2 archive -> lib421/ (LIBPAD.LIB 4.2.1 + LIBAPI.LIB 4.2) +fetch "$RTL42_URL" "$RTL42" +if [ ! -f "$PSYQ/lib421/LIBPAD.LIB" ] || [ ! -f "$PSYQ/lib421/LIBAPI.LIB" ]; then + command -v 7z >/dev/null || die "7z not found (apt: p7zip-full)" + rm -rf "$SCRATCH/rtl42"; 7z x -y -bd -o"$SCRATCH/rtl42" "$PSYQ/$RTL42" >/dev/null + z=$(find "$SCRATCH/rtl42" -iname 'J421PD.ZIP' | head -1) + [ -n "$z" ] || die "LIB/42PATCH/J421PD.ZIP not found inside $RTL42" + mkdir -p "$PSYQ/lib421"; unzip -o -q "$z" -d "$PSYQ/lib421" +fi +ck1 lib421/LIBPAD.LIB; ck1 lib421/LIBAPI.LIB + +# 3) the 20 PsyQ 4.0 libraries -> lib40/ +need40=0 +for L in LIBAPI LIBC LIBC2 LIBCARD LIBCD LIBCOMB LIBDS LIBETC LIBGPU LIBGS LIBGTE LIBGUN LIBMATH LIBMCRD LIBPRESS LIBSIO LIBSN LIBSND LIBSPU LIBTAP; do + [ -f "$PSYQ/lib40/$L.LIB" ] || need40=1 +done +if [ "$need40" = 1 ]; then + if [ -n "$DISC" ]; then + say "extracting PSX/LIB/*.LIB from the Run-time Library 4.0 disc" + python3 tools/psyq_libs_from_disc.py "$DISC" "$PSYQ/lib40" >/dev/null + elif [ -n "$FROM" ]; then + mkdir -p "$PSYQ/lib40"; cp "$FROM"/*.LIB "$PSYQ/lib40/" + else + die "the PsyQ 4.0 libraries are not present and have no public download. They are PSX/LIB/*.LIB on the redump disc + \"PlayStation - Programmer Tools - Run-time Library 4.0 (USA) (PC-CD ROM Release 2.0)\" (DTL-S2002, volume PROGTOOL): + re-run with --disc \"\" or --from . + (Not needed for byte-identity: without them main links its INCLUDE_ASM tiles and still matches.)" + fi +fi +for L in LIBAPI LIBC LIBC2 LIBCARD LIBCD LIBCOMB LIBDS LIBETC LIBGPU LIBGS LIBGTE LIBGUN LIBMATH LIBMCRD LIBPRESS LIBSIO LIBSN LIBSND LIBSPU LIBTAP; do + ck1 "lib40/$L.LIB" >/dev/null +done +say "verified the 20 PsyQ 4.0 libraries in tools/psyq/lib40/" + +[ "$BUILD" = 1 ] || { say "--no-build: SDK pieces in place, nothing built"; exit 0; } + +# 4) the builders (all regenerable; outputs under .run/obj40, .run/obj42, tools/psyq/lib40_elf) +say "psyq_build_libs.sh (10 libraries -> .run/obj40/)" +tools/psyq_build_libs.sh LIBCD LIBETC LIBGPU LIBMCRD LIBC2 LIBGTE LIBGS LIBSPU LIBSND LIBCARD +say "make_libgs.sh (-> .run/obj40/libgs_used)" +tools/make_libgs.sh >/dev/null +say "make_snd_used.py (-> .run/obj40/snd_used)" +python3 tools/make_snd_used.py >/dev/null +say "lib421 -> .run/obj42/{libpad421,libapi42}" +for pair in LIBPAD:libpad421 LIBAPI:libapi42; do + L=${pair%%:*}; d=".run/obj42/${pair##*:}" + rm -rf "$d"; mkdir -p "$d" + python3 tools/psyq_lib_split.py "$PSYQ/lib421/$L.LIB" "$d" >/dev/null + for o in "$d"/*.obj; do "$PSYQ/psyq-obj-parser" "$o" -o "${o%.obj}.o" >/dev/null 2>&1 || true; done +done +say "make_apicard_used.py (-> .run/obj42/apicard_used)" +python3 tools/make_apicard_used.py >/dev/null + +# 5) prove both legs +say "make -j$(nproc) sdk-dual" +make --no-print-directory -j"$(nproc)" sdk-dual +say "OK — the SDK object dirs are in place and main is byte-identical WITH and WITHOUT them" diff --git a/tools/psyq_CHECKSUMS.sha256 b/tools/psyq_CHECKSUMS.sha256 new file mode 100644 index 000000000..3f58dff52 --- /dev/null +++ b/tools/psyq_CHECKSUMS.sha256 @@ -0,0 +1,68 @@ +# tools/psyq/CHECKSUMS.sha256 — integrity record for the hard-to-source PsyQ archive (R20). +# Verify: cd tools/psyq && sha256sum --check CHECKSUMS.sha256 +# +# psyq-4.7-converted.zip — the community-converted PsyQ 4.7 SDK (ELF .o objects + headers, +# extracted under conv47/psyq-4_7-converted/{lib,include}; 55 objects). PROVENANCE/USE: the +# resident engine blob is PsyQ 4.7 (DetectPsyQ 4.7.0 + the DsMix/libsnd signature hit — R24), +# so Phase 12 links the resident's embedded PsyQ code (libsnd / libgte / libspu) from conv47/, +# NOT the EXE's 4.0 libs (which won't byte-match the 4.7 objects). Git already tracks these +# files (R20 backup); this file records integrity + the Phase-12 provenance. (The 4.0 .LIBs in +# lib40/ are the EXE's, used Phases 7-8.) +10ffdcd912feffddebe6955ea346bbd5de2f912ce952bfc703994cde49180c2a psyq-4.7-converted.zip +# +# P31 S78 (2026-09-04) — two more hard-to-source SDK archives, fetched from psx.arthus.net/sdk/Psy-Q/ +# (X2: treated as data). PROVENANCE/USE: the EXE's 0x8005CF68-0x8005FC68 band is PsyQ LIBPAD 4.2.1 + +# LIBAPI 4.2 (the loader's per-version signature sets place PADENTRY/PADCMD/PADPORTD/PADSEQD/COUNTER/ +# FIRST/PAD/PATCH/CHCLRPAD byte-exact; docs/psyq-worklist.md "S78"). 4.6's libs were converted to ELF +# (tools/psyq/lib46/ -> .run/obj46/) and placed: only PDMAIINI.o (0x8005D8B4, 68 ins) is byte-identical +# in that band; 4.6/4.7 libpad otherwise differ from 4.2.1. Kept as the nearest surviving neighbours +# until a 4.2.1/4.3 LIBPAD.LIB surfaces (task #13). The 4.5 toolkit zip (7 split archives) is unpacked +# only on demand. +ab473f979652c9254c18f041b8325522774d9f25bb157faa19f475e12383cbd4 Psy-Q_46.zip +e7e4269efcf5482d9c4c5b64610dd4e4daf4bfc00ebe0600597100975748e52b PSYQ_SDevTC_v4.5.zip +# +# P31 S79 (2026-09-04, task #13) — THE archive that links main's 0x8005CE18-0x8005FC68 band: archive.org item +# `play-station-programmer-tool-runtime-library-version-4.2.7z` ("Play Station Programmer Tool Runtime Library +# Version 4.2", X2: data). Contents: PsyQ Runtime Library 4.2 (LIB/*.LIB + INCLUDE, 1998-01-21) AND the +# LIB/42PATCH/J421PD.ZIP patch = "Libpad.lib version 4.2.1 for the Analog Controller (DUAL SHOCK)", SCE R&D, +# 1998-02-26, shipped with libapi.lib 4.2 + LIBPAD.H/LIBAPI.H/KERNEL.H. Extracted: tools/psyq/lib42/ (the 4.2 +# LIBs + the two patch dirs) and tools/psyq/lib421/ (the J421PD payload), both gitignored/regenerable from the +# 7z; ELF via psyq_lib_split.py + psyq-obj-parser into .run/obj42/{libpad421,libapi42}. RESULT: libpad 4.2.1 +# places 7/11 objects (PADENTRY PADMAIN PADCMD PADIF PADPORTD PADSEQD WAITRC2 = the whole band's pad code, +# PADMAIN 760 ins exact) and libapi 4.2 places 39/88 (the 21 band trampolines, COUNTER, L02/L03, and the +# apicard-region C112/A50/A51/A54/A65/A67/A69/FIRST/A66/PAD/A18-21/PATCH/CHCLRPAD); ALL 46 link byte-identical +# (psyq_link.py PASS). docs/psyq-worklist.md "S79 task #13". Plain libpad 4.2 (LIB/LIBPAD.LIB) places only 4. +e4f5a6785e981051d910c3534a0f05ea1d4f6ef4fbcfb1d09db93eca6223e5fe PlayStation_Programmer_Tool_-_Runtime_Library_Version_4.2.7z +# inner payloads (lib421/, from J421PD.ZIP): +eb41454b3678fdeee1065ff2c892882cdeadd1d04c9d8ad0c7a7c8ba41012fc4 lib421/LIBPAD.LIB +88294cd8eb6f0b623e8847628e375250411da5b36d67bca178b0d274d76928c5 lib421/LIBAPI.LIB +# +# P33 B4 (2026-09-06) — the rows tools/fetch_psyq.sh verifies before building the SDK object dirs. This file now lives at +# tools/psyq_CHECKSUMS.sha256 (tools/psyq/ itself is gitignored: Sony-copyrighted, user-supplied, never distributed). +# Verify: cd tools/psyq && sha256sum --check ../psyq_CHECKSUMS.sha256 +# psyq-obj-parser: https://github.com/decompme/compilers/releases/download/compilers/psyq-obj-parser.tar.gz (the +# tarball AND the static x86-64 binary inside it; identical to the copy this project used since Phase 7). +353495f13f6756773cd905ba3a0743843232f95719e306fc6ee56cbf7d731a3e psyq-obj-parser.tar.gz +4fba623a31bb5830ceea95b5081bc13504629686b0b4904c533b7be2a5f2377a psyq-obj-parser +# PsyQ 4.0 libraries — NO public URL: PSX/LIB/*.LIB on the redump disc "PlayStation - Programmer Tools - Run-time +# Library 4.0 (USA) (PC-CD ROM Release 2.0)" (DTL-S2002, volume PROGTOOL); tools/psyq_libs_from_disc.py extracts them. +3c082a4fd4b00c5b191b717a96035e5453b7db2f5b906b2bd68e5fca6350031d lib40/LIBAPI.LIB +14847db1c85806630867c1be95e7440474fef47997f398863380360695adee6e lib40/LIBC.LIB +3d6c0464728e014cb341a8eae337303c7c03e0dcd8c897a2549ef16e1e231c46 lib40/LIBC2.LIB +d398c47818824ce96c280ba9ae7a6cdf7bab5386c82292566d1caf723a540764 lib40/LIBCARD.LIB +db4e1a716b3b49a0776fb85c7802d839e5cae4acadaeaf3b16621aa5c61ad79d lib40/LIBCD.LIB +d6c8b3312e9420dcd539c3f8cdbd20fd1a2fa6d2c1e1d2b3196e23af08c14120 lib40/LIBCOMB.LIB +0b152beb505441d5b7b7c8198ec2dcba597d935f8d8867b748e51622b160a01b lib40/LIBDS.LIB +71a57b08c6a406148061cb5bb66aeadcbfac736b4d45d0ccc7f26635aaadae54 lib40/LIBETC.LIB +d296e33ff3599a9962d28d59fd1883e20160f252ca85b28b8a44333199d331bf lib40/LIBGPU.LIB +d3341cd5c813f53078a72376c80d15f6f502ecc44487e2bb21ed95d585d83046 lib40/LIBGS.LIB +bec029730c9a8bc529b1980519c5cacbbbb63132c28cb524fa72ea6e4b0debea lib40/LIBGTE.LIB +12ff5443876496bf35023a04bb7c052076b82e00a1027202bb3624483bb55ba4 lib40/LIBGUN.LIB +f4cf1719fb07b6daa02dba5144d8d5b09c504ad745af061ffbadc70d6f1b5bbc lib40/LIBMATH.LIB +8cc75e2b840944906d9ce8bf33e91e603cfd56bdcbb5fe9044ffeb7a2bc3807d lib40/LIBMCRD.LIB +f78f42abbd8a21ef56f0f6bdfcc4f1b9f1b4b903dfd8c3ae3add6a7f355d8a6c lib40/LIBPRESS.LIB +9ff3e830d32692ac72a5d675188f770707fbb31b77c7538b496616372acc39c4 lib40/LIBSIO.LIB +1ef99108b065166613a4bf6c5b46a74bd0f584e8a570926e8e0c66497e64f547 lib40/LIBSN.LIB +f70695798e42b4c94d907bf1ef8af6f90618ca42a1b302c122efc8de51fda53a lib40/LIBSND.LIB +ee099de953b355e580e34a83e37b44e4276cb14202afbfd2ea4ba2d4926e6d13 lib40/LIBSPU.LIB +cd4c5d17246fdf301126f4f37388516bcc0e73782f11ed3f521314d23b898654 lib40/LIBTAP.LIB diff --git a/tools/psyq_libs_from_disc.py b/tools/psyq_libs_from_disc.py new file mode 100644 index 000000000..720257e86 --- /dev/null +++ b/tools/psyq_libs_from_disc.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python3 +"""psyq_libs_from_disc.py — the 20 PsyQ 4.0 `PSX/LIB/*.LIB` files, extracted from the runtime-library disc. + +The EXE links PsyQ 4.0 objects (main's LINKED regions). Sony's libraries are not redistributable and have no +public download; they are on the redump disc "PlayStation - Programmer Tools - Run-time Library 4.0 (USA) +(PC-CD ROM Release 2.0)" (DTL-S2002; ISO volume `PROGTOOL`), directory `PSX/LIB/`. This walks that disc's +Track-1 image with the project's own MODE2/2352 ISO9660 reader (tools/bfm_extract/iso9660.py — the same one +`make disc-extract` uses on the game disc) and writes the `.LIB` files out, printing each file's sha256 so +`tools/psyq_CHECKSUMS.sha256` can be checked against them. Called by tools/fetch_psyq.sh --disc. (P33 B4) + + python3 tools/psyq_libs_from_disc.py "<... Run-time Library 4.0 ... (Track 1).bin>" tools/psyq/lib40 + +Refuses (R43) a disc without `PSX/LIB` and a directory that does not hold exactly 20 `.LIB` files (R32). +""" +import hashlib +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent / "bfm_extract")) +from iso9660 import Iso9660Image, Iso9660Error # noqa: E402 + +EXPECTED_LIBS = 20 + + +def main(argv): + if len(argv) != 3: + sys.exit(__doc__) + binp, out = pathlib.Path(argv[1]), pathlib.Path(argv[2]) + if not binp.is_file(): + sys.exit(f"psyq_libs_from_disc: no such disc image: {binp}") + out.mkdir(parents=True, exist_ok=True) + try: + with Iso9660Image(binp) as img: + psx = img.find("PSX") + lib = img.find("LIB", psx) if psx is not None else None + if lib is None or not lib.is_dir: + sys.exit(f"psyq_libs_from_disc: no PSX/LIB directory on {binp.name} (volume {img.volume_id!r}) — " + "expected the Run-time Library 4.0 disc (volume PROGTOOL)") + n = 0 + for rec in sorted(img.iter_directory(lib), key=lambda r: r.raw_name): + if rec.is_dir: + continue + name = rec.raw_name.split(";")[0] + if not name.upper().endswith(".LIB"): + continue + data = img.extract_file(rec.raw_name, lib) + (out / name).write_bytes(data) + print(f"{hashlib.sha256(data).hexdigest()} {out / name} ({len(data)} B)") + n += 1 + vol = img.volume_id + except Iso9660Error as e: + sys.exit(f"psyq_libs_from_disc: {e}") + print(f"psyq_libs_from_disc: {n} .LIB files -> {out}/ (volume {vol!r})", file=sys.stderr) + if n != EXPECTED_LIBS: + sys.exit(f"psyq_libs_from_disc: expected {EXPECTED_LIBS} .LIB files in PSX/LIB, found {n} (R32)") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv))