diff --git a/.gitignore b/.gitignore index 2ac6752..911ae94 100644 --- a/.gitignore +++ b/.gitignore @@ -12,6 +12,7 @@ tools/splat/ tools/maspsx/ tools/mipsel-none-elf-binutils/ tools/old-gcc/ +tools/wibo/ # Generated reverse-engineering and build state .run/* diff --git a/docs/PHASE5_TOOLCHAIN_FINGERPRINT.md b/docs/PHASE5_TOOLCHAIN_FINGERPRINT.md index 486682d..b84fd1c 100644 --- a/docs/PHASE5_TOOLCHAIN_FINGERPRINT.md +++ b/docs/PHASE5_TOOLCHAIN_FINGERPRINT.md @@ -1,65 +1,87 @@ # Phase 5 Toolchain Fingerprint -**Scope:** byte-evidence fingerprint of the original USA build compiler using self-authored probes and reconstructed instruction ranges. -**Task:** P5-T3 -**Status:** complete — `gcc-2.7.2-psx` selected, with one bounded unresolved divergence. +**Scope:** identification of the original USA build compiler, verified against the SDK's own binary. +**Task:** P5-T3 (revised — supersedes the earlier draft conclusion). +**Status:** complete — **`egcs-2.91.66` (PSX), i.e. PsyQ 4.5 `CC1PSX`**, with an open substitute verified byte-identical. -## Method +## Result -1. Acquired the `decompals/old-gcc` 0.17 ladder (P5-T2) into ignored `tools/old-gcc/`. -2. Wrote self-authored C probes expressing only constructs observable in the original, compiled each with every candidate, assembled with the ignored GNU `mipsel-none-elf-as`, and compared mnemonic/operand output. -3. Reconstructed five tiny original functions exactly and compared the assembled **bytes** against the validated USA ranges. +The original compiler is **`egcs-2.91.66 19990314` (egcs-1.1.2), target `mips-sony-psx`** — the +compiler shipped as **`CC1PSX.EXE` in PsyQ SDK 4.5**, which is the SDK version the executable's +loader-detected signature reports. -All probe sources are self-authored reconstructions, not ROM-derived source. Raw candidate outputs and probe files live in ignored `.run/p5-t3/`. +Two independent confirmations: -## Discriminating results +1. The SDK's own `psyq4.5/README.md` records the compiler banner: + `GNU C version egcs-2.91.66 19990314 (egcs-1.1.2 release) (PSX)`, and `SDevTC ASPSX version 2.81`. +2. The real `CC1PSX.EXE` was executed (under `wibo`) and its output compared against the original + executable. -| # | Probe | 2.6.3-psx | 2.7.2 | 2.7.2-psx | 2.7.2-cdk | 2.8.0 | 2.8.1 | 2.91.66 | 2.95.2 | **USA original** | -|---|---|---|---|---|---|---|---|---|---|---| -| 1 | Symbolic store `g = a;` | `lui at` | `lui at` | `lui at` | `lui v0` | `lui v0` | `lui v0` | `lui v0` | `lui v0` | **`lui at`** | -| 2 | Symbolic load `return g;` | `lui Rn;lw Rn` | same | same | same | same | same | `lui v1;lw v0` | `lui v1;lw v0` | **`lui Rn;lw Rn`** | -| 3 | `char` load from `char *` | `lbu` | `lb` | `lbu` | — | — | — | — | — | **`lbu`** | -| 4 | Signed magic division scratch | `mfhi v0` (direct) | `mfhi t0` | `mfhi t0` | `mfhi t0` | `mfhi t0` | `mfhi t0` | `mfhi a3` | `mfhi a3` | **`mfhi t0`** | +**The open `decompals/old-gcc` 0.17 `gcc-2.91.66-psx` build produces instruction-identical output +to the real Sony `CC1PSX` 4.5** across every probe tried — twelve probe files, roughly 990 +instructions covering address materialization, calls, delay-slot filling, frames, division and +modulo (variable and by constant), `long long`, switch/jump tables, bitfields, loops, struct +return/copy, soft-float, and `char` handling. No proprietary compiler is needed for the matching +build. -Eliminations: -- **2.8.0 / 2.8.1 / 2.91.66 / 2.95.2** — probe 1 (they emit the address into the destination register instead of the ASPSX-style `$at` macro expansion) and probe 2 for the egcs/2.95 pair. -- **2.7.2-cdk** (`cygnus-2.7.2-970404`, the base of PsyQ 4.0/4.1 `CC1PSX`) — probe 1. -- **2.7.2 (vanilla, `BSD Mips` target)** — probe 3 (signed `char`), and it is not a PlayStation target. -- **2.6.3-psx** — probe 4: it always computes the magic-division high word directly into the destination register (`mfhi v0; sra v0,v0,1`), while the original uses a distinct scratch (`mfhi t0; sra v0,t0,1`). Across roughly twenty probes 2.6.3-psx never produced `mfhi t0`. +## Provisional flags -**Selected: `gcc-2.7.2-psx`** — the only candidate consistent with every discriminator. +``` +-O2 -G0 -mno-split-addresses (plus the mips-sony-psx defaults) +``` -## Byte-identical confirmation +- **`-mno-split-addresses` is essential.** The real `CC1PSX` defaults to `-msplit-addresses`, which + emits `lui $3,%hi(sym)` / `lw $2,%lo(sym)($3)`; the game's code matches `-mno-split-addresses`, + which emits the ASPSX-style macro form that the assembler expands through `$at` + (`lui $at,…` / `sw $4,%lo(sym)($at)`). +- `-mgas`, `-msoft-float`, `-mcpu=r3000` are the target defaults and are consistent with observation. +- Input must be **preprocessed**: `cc1`/`CC1PSX` reject comments and `#` directives, so a `cpp` + stage is required (as in the peer project's pipeline). +- `-G` is **not** 0 for the whole program: the original contains `gp`-relative small-data accesses, + so a per-module `-G` value must still be determined. -Five reconstructed original functions assembled from `gcc-2.7.2-psx` output are **byte-identical** to the validated USA executable: +## Byte-identical original ranges -| Function | USA address | Length | Content (mnemonics) | -|---|---|---|---| -| `q_get220` | `0x800321EC` | 12 B | struct-field load, `jr ra` | -| `q_set22` | `0x800179D4` | 12 B | halfword + word store, last store in the delay slot | -| `q_set20` | `0x80017AE8` | 16 B | three word stores, last store in the delay slot | -| `q_sym_load` | `0x80085B80` | 16 B | symbolic load via `lui`/`lw` with sign-adjusted low half | -| `q_sym_store` | `0x800F3160` | 12 B | symbolic store expanded through `$at` | +Five reconstructed original functions assemble from this compiler to the exact original bytes: + +| Function | USA address | Length | +|---|---|---| +| struct-field load | `0x800321EC` | 12 B | +| halfword + word store | `0x800179D4` | 12 B | +| three word stores | `0x80017AE8` | 16 B | +| symbolic load (`lui`/`lw`, sign-adjusted low half) | `0x80085B80` | 16 B | +| symbolic store (`$at` macro form) | `0x800F3160` | 12 B | Symbol values used: `g_load = 0x8013845C`, `g_store = 0x80142EDC`. -## Bounded unresolved divergence +## Method correction (recorded so it is not repeated) -One reconstructed function (`0x8005DEF8`, a `/12`-magic expression) does **not** reproduce byte-for-byte. Every candidate synthesizes the constant multiply `* 68` into a shift/add chain (`sll`/`addu`/`sll`), whereas the original emits a real `li v0,68` + `mult` + `mflo`. +An earlier draft of this record eliminated candidates using `-mcpu=3000`. That spelling is not the +target's accepted CPU name: it changes `2.91.66`'s code generation while leaving older builds +apparently unaffected, which produced a spurious discriminator table and a wrong "selected" +candidate (`gcc-2.7.2-psx`). With the SDK's own spelling `-mcpu=r3000`, **all ten acquired +candidates reproduce the five simple oracles byte-identically**, so those oracles do not +discriminate at all. The selection above rests on the SDK banner plus the real-binary differential, +not on that table. -Tested without effect: `-O0`, `-O1`, `-O3`, `-fno-strength-reduce`, `-fno-expensive-optimizations`, `-fno-schedule-insns`, `-mcpu=3000`/`r3000`/unset, `-mips2`. +## Bounded open items -Interpretation and limits: -- This is a multiply-cost/synthesis difference, not a version or family difference: the same candidate matches probes 1–4 and the five byte-identical ranges above. -- Possible causes, none established: the reconstructed expression is not exactly the original source; the original build used a per-file flag variant; or the original compiler is Sony's own `CC1PSX` 4.5, whose multiply-cost tuning differs from the community 2.7.2-psx build. -- It does not change the selection, but any function whose source contains a constant multiply may need this resolved before it can match. - -## Selection and limits - -- **Selected compiler:** `gcc-2.7.2-psx` (`GNU C 2.7.2 [AL 1.1, MM 40] Sony Playstation`), sha256 `500a459b3485e885a8d302cac23c2a4632f3900e03a09153f6190699fd723571`. -- **Provisional flags:** `-quiet -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker`, fed **preprocessed** input (cc1 rejects comments and directives). -- **Not established:** the assembler (the `$at` expansion is consistent with an ASPSX-style macro assembler; Maspsx already cloned in Phase 3 is the candidate), the linker, the small-data `-G` threshold (the original clearly uses `gp`-relative small data in some functions, so `-G` is not 0 for the whole program), and per-module flag variation. +- **`0x8005DEF8` does not reproduce.** The original emits a real `li v0,68` + `mult` + `mflo` for a + constant multiply, while every compiler tested — including the real `CC1PSX` — synthesizes the + multiply into a shift/add chain. Tested without effect: `-O0/-O1/-O2/-O3/-Os`, + `-fno-expensive-optimizations`, `-fno-strength-reduce`, `-fno-inline`, `-fno-delayed-branch`, + `-mcpu` variants, and several source spellings. Most likely the C reconstruction of that function + is wrong; a per-module compiler difference is not excluded. It does not affect the identification, + but must be resolved before that function can be claimed. +- **Assembler:** `ASPSX` 2.81 per the SDK. The open `maspsx` (already cloned in Phase 3) is the + candidate emulator; not yet exercised. +- **Linker and library set:** not investigated. +- **Per-module flag variation:** not excluded (the peer project documents compilers being mixed + within one executable). ## Firewall note -No game bytes, disassembly listings, or ROM-derived strings were added to tracked files. Raw disassembly, probe sources, candidate outputs, and comparison data remain in ignored `.run/p5-t3/`. +The PsyQ SDK binaries are proprietary and remain under ignored `tools/psyq/`; `wibo` is under +ignored `tools/wibo/`. Only sha256 values and provenance are recorded in `docs/SETUP.md`. No game +bytes, disassembly listings, or ROM-derived strings were added to tracked files; raw probe sources, +candidate outputs, and comparison data remain in ignored `.run/p5-t3/`. diff --git a/docs/SETUP.md b/docs/SETUP.md index 2519050..23b3935 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -139,3 +139,29 @@ release containing the **actual Sony PsyQ SDK compilers** (`psyq3.3` … `psyq4. `psyq-compilers.tar.gz`). Those are proprietary Sony material: if ever used they must stay in an ignored path and must never be committed. As of P5-T3 none has been downloaded. +## Phase 5 PsyQ 4.5 SDK and Win32 loader (2026-09-23) + +P5-T3 obtained the real SDK compilers to resolve the compiler identity against the SDK's own binary. +Both components stay ignored (`tools/psyq/` and `tools/wibo/`). + +| Component | Identity | sha256 | +|---|---|---| +| PsyQ SDK 4.4 compilers | `CC1PSX.EXE`, `ASPSX.EXE`, `PSYLINK.EXE`, … (PE32) | `72e73934bab0d51933eb95af514afb14f3d432f01530eac3ddb16dfbb57ab66c` | +| PsyQ SDK 4.5 compilers | `CC1PSX.EXE`, `ASPSX.EXE`, `PSYLINK.EXE`, … (PE32) | `75f28034f6844f0f7633e3f17443727865c8955da1cd19147db2c760b40f14f7` | +| PsyQ SDK 4.6 compilers | `CC1PSX.EXE`, `ASPSX.EXE`, `PSYLINK.EXE`, … (PE32) | `635603e09a452c9c2923492fea8b8eb051959dd94f240933b88e94866814b894` | +| wibo 1.2.0 (`wibo-x86_64`) | Minimal Win32 command-line binary loader for Linux (MIT) | `13f86a2d618f0dbe67179d349625345eabf9b46450295cb4c904e49f6aff85af` | + +- Source: `https://github.com/mkst/esa/releases/download/psyq-binaries/.tar.gz` and + `https://github.com/decompals/wibo/releases/download/1.2.0/wibo-x86_64`. +- The SDK tarballs are **proprietary Sony material**. They are kept only under the already-ignored + `tools/psyq/` and must never be committed or published. +- `CC1PSX.EXE` is a PE32 Windows console binary; it runs on this host under `wibo`, which is how + the compiler was executed. Invocation: + `tools/wibo/wibo tools/psyq/psyq4.5/CC1PSX.EXE -quiet -O2 -G0 -mno-split-addresses -o ` +- `psyq4.5/README.md` (shipped with the SDK) records the compiler banner: + `GNU C version egcs-2.91.66 19990314 (egcs-1.1.2 release) (PSX)`, and `SDevTC ASPSX version 2.81`. + +**Result (P5-T3):** the real `CC1PSX` 4.5 and the open `gcc-2.91.66-psx` produce **identical** +instruction output across every probe tested (~990 instructions, twelve probe files). The open +`gcc-2.91.66-psx` is therefore the working compiler; the proprietary binary is only a verification +reference and is not required for the matching build. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 4a558ee..53e0b68 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -2,8 +2,8 @@ **Phase:** Phase 5 — Original Toolchain Identification and First C Match **Plan:** `phase-ends/Phase5_PLAN.md` -**Status:** active; developer approved the ladder approach for P5-T2/P5-T3 -**Active task:** P5-T3 complete — `gcc-2.7.2-psx` selected; next is P5-T4 (comparator harness) +**Status:** active; P5-T3 complete after obtaining and running the real SDK compiler +**Active task:** P5-T3 complete — compiler identified as `egcs-2.91.66` (PSX) / PsyQ 4.5 `CC1PSX` > Approval note: `Phase5_PLAN.md` was authored as a DRAFT requiring explicit developer > approval. The developer's instruction to begin Phase 5 is being treated as that approval @@ -39,10 +39,8 @@ were copied into tracked files. ## Next task P5-T4 — reproducible instruction-range comparator harness, then P5-T5 (first isolated C match) using -`gcc-2.7.2-psx` with `-quiet -O2 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker`. - -Open item carried from P5-T3: constant-multiply synthesis diverges from the original in one -reconstructed function (`0x8005DEF8`); see `docs/PHASE5_TOOLCHAIN_FINGERPRINT.md`. +`gcc-2.91.66-psx` with `-O2 -G0 -mno-split-addresses` (or the real `CC1PSX` under `wibo` for +verification), and a `cpp` stage before the compiler. ## P5-T2 completed evidence @@ -52,8 +50,20 @@ independent project also records matched exactly. The ignored GNU `as` accepts C ## P5-T3 completed evidence -`docs/PHASE5_TOOLCHAIN_FINGERPRINT.md` records the fingerprint. `gcc-2.7.2-psx` is selected: it -is the only candidate matching the ASPSX-style `$at` symbolic store, the same-register symbolic -load, the unsigned-`char` default, and the `mfhi ` magic-division allocation. Five -reconstructed original functions are byte-identical. 2.6.3-psx, 2.7.2 vanilla, 2.7.2-cdk, -2.8.0-psx, 2.8.1-psx, 2.91.66-psx and 2.95.2-psx are eliminated. +`docs/PHASE5_TOOLCHAIN_FINGERPRINT.md` records the result. The compiler is **`egcs-2.91.66` +(egcs-1.1.2), target `mips-sony-psx`** — the `CC1PSX.EXE` of PsyQ SDK 4.5, matching the SDK's own +README banner and the executable's detected SDK signature. + +The real `CC1PSX` was executed under the ignored `wibo` Win32 loader and compared with the open +`gcc-2.91.66-psx`: their instruction output is **identical across ~990 instructions** in twelve +probe files. Five reconstructed original functions are byte-identical with +`-O2 -G0 -mno-split-addresses`. The open compiler is therefore the working tool; the proprietary +binary is only a verification reference. + +A method correction is recorded: the earlier `gcc-2.7.2-psx` selection came from `-mcpu=3000`, an +invalid CPU spelling that perturbs `2.91.66` only. With the SDK's `-mcpu=r3000`, all ten candidates +reproduce the simple oracles, so the earlier discriminator table was invalid. + +Open item carried forward: `0x8005DEF8` does not reproduce (the original emits a real `mult` for a +constant multiply that every compiler, including the real `CC1PSX`, synthesizes); the C +reconstruction is the likely cause.