diff --git a/docs/COMPILING_OVERLAYS.md b/docs/COMPILING_OVERLAYS.md new file mode 100644 index 00000000..8261847d --- /dev/null +++ b/docs/COMPILING_OVERLAYS.md @@ -0,0 +1,167 @@ +# Compiling overlays from `overlay_captures.json` + +**Audience:** developers preparing a release who want to ship as much +native-compiled overlay coverage as possible, plus modders and players who want +to pre-build coverage for their own machine. + +> **What `overlay_captures.json` is.** PS1 games stream *overlays* — chunks of +> code loaded from disc into RAM at runtime and later overwritten. The static +> recompiler never saw them at build time, so on first visit they run on the +> dirty-RAM interpreter (correct, but slower). With `[runtime] overlay_cache = +> true`, the runtime records each overlay it sees (bytes + executed PCs) into +> `overlay_captures.json` next to the exe. Compiling that file turns those +> overlays into native code. +> +> ⚠️ **Privacy:** `overlay_captures.json` contains snapshots of the game's own +> code read from *your* disc. Keep it private — do not post it publicly. + +§0 frames the production goal; §1 builds the gcc cache you ship; §2 bakes +overlays straight into the executable. + +--- + +## 0. The production goal: ship a full gcc cache + +**A good release ships as many developer-built gcc shards as possible.** gcc +produces the best-optimized native code, and a shard the developer already built +runs at full speed on the player's *first* visit to that area — no interpreter +hitch, no wait for a background compile. So the intended workflow is: + +1. Play through the game (or the areas you care about) on a dev box with + **gcc + Python**, with `overlay_cache = true`, to accumulate + `overlay_captures.json` coverage. +2. Pre-build the gcc cache from it (§1) and ship that `cache/` folder in the + release. + +**The runtime auto-compile is a fallback that fills the gaps, not the plan.** +With `overlay_cache = true`, if a player reaches an area your shipped cache +doesn't cover, the runtime compiles it in the background so the *next* visit is +native. On a dev box that background compile uses gcc; on a **player box with no +toolchain**, it falls back to the fully self-contained bundle beside the exe +(`overlay_toolchain/`: embedded Python + **TinyCC** + the recompiler + this +script + headers). tcc keeps such players from ever being stuck on the +interpreter, but it optimizes worse than gcc — which is exactly why you want the +developer's gcc shards to cover as much as possible up front. + +The rest of this doc is how you build that cache (§1) or bake overlays straight +into the binary (§2). + +--- + +## 1. Build the gcc cache (the shard you ship) + +This produces the `cache/…/*.dll` shards you ship in the release, ready from the +first launch. It's the same cache the runtime would eventually build in the +background — but done up front, with gcc, by you. + +```sh +python psxrecomp/tools/compile_overlays.py \ + --captures /overlay_captures.json \ + --game-toml game.toml \ + --recompiler psxrecomp/recompiler/build/psxrecomp-game.exe \ + --runtime-include psxrecomp/runtime/include \ + --out-dir /cache \ + --gcc C:/msys64/mingw64/bin/gcc.exe \ + --cps +``` + +Then just launch the game — the loader rescans the cache dir and picks the +shards up automatically. + +**The flags that matter:** + +| Flag | Why | +|------|-----| +| `--captures` | The `overlay_captures.json` to read. If the runtime set `PSX_OVERLAY_CAPTURES`, that wins; for a manual run pass this explicitly. | +| `--game-toml` | Reads the game id (used in the cache path) and the `[widescreen]` site lists. | +| `--recompiler` | `psxrecomp-game.exe`. **Must be built from the same source tree** — a stale binary is rejected up front (see "Staleness guard"). | +| `--runtime-include` | The runtime `include/` dir. Supplies the codegen version + hash that namespace the cache. | +| `--out-dir` | Cache root. Point it at the `cache` folder next to the exe. | +| `--gcc` | Absolute path to your mingw gcc, e.g. `C:/msys64/mingw64/bin/gcc.exe`. gcc gives the best-optimized shards. | +| `--cps` | **Required** so the generated code matches a CPS runtime build. If the runtime is CPS and you omit this, the DLLs misbehave. Match the runtime. | +| `--compiler tcc` | Use the bundled TinyCC instead of gcc (toolchain-free). tcc shards land in their own `tcc/` cache namespace. | +| `--jobs N` | Parallel workers (default: CPU cores − 2). Regions compile independently, so this is a large speedup on a full capture set. `--jobs 1` forces the sequential path. | +| `--force` | Rebuild shards even if they already exist. Normally the compile is **incremental** — it skips any DLL already present and any function already covered by an existing shard. | + +**Where the shards land.** The cache is namespaced so different games, compilers, +CPU architectures, and codegen versions never mix: + +``` +///-/cg_/_.dll + _.ranges +``` + +e.g. `cache/SLUS-01395/gcc/win-x64/cg7_1a2b3c4d/000E7000_B476006F.dll`. The +loader computes the exact same path, so if you build into the right game's cache +folder it just works. (If you build with a recompiler/headers from a *different* +codegen version, the shards land in a different `cg…` folder and the current +runtime ignores them — that's the version guard working, not a bug.) + +--- + +## 2. Bake overlays into the executable (`--static`) + +This is the literal "compile them **along with** the executable": instead of +per-area DLLs loaded at runtime, every captured overlay is emitted into a single +C file that is compiled and linked **into the runtime binary**. + +```sh +python psxrecomp/tools/compile_overlays.py \ + --static \ + --captures /overlay_captures.json \ + --game-toml game.toml \ + --recompiler psxrecomp/recompiler/build/psxrecomp-game.exe \ + --runtime-include psxrecomp/runtime/include \ + --out-dir generated/ \ + --gcc C:/msys64/mingw64/bin/gcc.exe \ + --cps +``` + +Output: `generated/overlays_static.c`, containing every in-overlay +`func_XXXXXXXX` plus an auto-generated `psx_overlay_dispatch()` switch. Add that +file to the runtime build (it's a normal translation unit) and rebuild the exe. +No `cache/` folder and no DLL loading are involved at runtime. + +**Trade-off vs. the DLL cache:** `--static` is a fixed snapshot chosen at build +time — it will *not* grow as the player explores new areas. The DLL cache (§0/§1) +is the mechanism that keeps improving with play. Use `--static` when you want a +single self-contained binary with a known coverage set baked in; use the DLL +cache for everything else. + +--- + +## Staleness guard (read this if it refuses to run) + +The recompiler *binary* and the codegen *source* are tied together by a hash. +Before compiling anything, the script runs `psxrecomp-game.exe --codegen-hash` +and compares it to the hash baked into `--runtime-include`. If they differ — or +the binary is too old to support the flag — it **aborts** rather than silently +emitting shards with stale codegen (a real bug class: read-tag matches, +semantics are old). The fix is always the same: + +```sh +cmake --build psxrecomp/recompiler/build --target psxrecomp-game +``` + +Rebuild the recompiler from the current tree, then re-run. + +--- + +## Quick reference + +- **I'm shipping a release:** play to accumulate coverage, then pre-build the gcc + cache into `/cache` and ship it — the more you cover, the less the player + ever falls back to tcc or the interpreter (§0/§1). +- **I want the overlays inside the .exe itself:** `--static` → `overlays_static.c` + linked into the runtime (§2). +- **What about areas I didn't cover?** `overlay_cache = true` lets the runtime + fill the gaps on the player's box (gcc if present, else bundled tcc) — a + fallback, not a substitute for shipping gcc shards (§0). +- **It won't compile / rejects the recompiler:** rebuild `psxrecomp-game` + (staleness guard). +- **The shards build but the runtime ignores them:** check the `cg_` and + `-` folder match the runtime you're launching, and that `--cps` + matched the runtime build. + +Design/rationale for the whole system lives in `overlay-recompilation-design.md`, +`overlay-plan.md`, and `FEATURES.md` in this folder.