diff --git a/ISSUES.md b/ISSUES.md index d494c6ee..b0122059 100644 --- a/ISSUES.md +++ b/ISSUES.md @@ -515,8 +515,16 @@ discovery seeds). Either way, then close Issue #3 alongside. ## Issue #7 — sljit live execution is unvalidated (pure-live save-load wedge) -**Status:** open, root-caused — fix in progress (branch `feat/sljit-backend`) +**Status:** CLOSED — superseded, not fixed. The sljit Tier-2 backend this issue +describes was removed wholesale in `5b7e69b4` (2026-07-15), which deleted +`lib/sljit/`, `overlay_sljit.{c,h}` and the sljit paths in `code_provider.c` / +`overlay_loader.c`. The toolchain-free role it filled is now served by the +bundled TinyCC tier (introduced `c2a3f6ad`, 2026-06-25), which compiles to a DLL +through a subprocess rather than JITting in-process, so it does not share this +defect's live-execution/save-load shape. Nothing here is actionable; retained as +history. **Date opened:** 2026-06-15 +**Date closed:** 2026-07-15 (by removal of the subsystem) **Area:** overlay Tier-2 sljit backend (`runtime/src/overlay_loader.c`, `overlay_sljit.c`, `code_provider.c`, `overlay_sljit.c` resolution) diff --git a/THIRD_PARTY_ATTRIBUTION.md b/THIRD_PARTY_ATTRIBUTION.md index b57d5f90..0b1b85d2 100644 --- a/THIRD_PARTY_ATTRIBUTION.md +++ b/THIRD_PARTY_ATTRIBUTION.md @@ -13,14 +13,24 @@ Normal runtime builds stage it automatically so players supply only a disc; `bios/OpenBIOS.LICENSE` always rides alongside the shipped image. -## sljit — stack-less JIT compiler (Tier-2 in-process overlay backend) +## TinyCC (TCC) — toolchain-free overlay compiler shipped to players -[sljit](https://github.com/zherczeg/sljit) by Zoltan Herczeg, licensed -**BSD-2-Clause**. Vendored at `lib/sljit/` (source `lib/sljit/sljit_src/`, -license `lib/sljit/LICENSE`) and compiled into the runtime as the self-contained -Tier-2 overlay JIT backend (`runtime/src/overlay_sljit.c`). No external toolchain -dependency; sljit auto-detects the host architecture. See `SLJIT.md` (repo root / -workspace) for the backend design. +[TinyCC](https://bellard.org/tcc/) by Fabrice Bellard and contributors, licensed +**LGPL-2.1**. Not vendored in this repository — no TinyCC source or binary is +tracked here. It is invoked as a **separate subprocess** by +`tools/compile_overlays.py` (`--compiler tcc`, `--tcc <binary>`) to build overlay +shards into a DLL, so players need no compiler of their own. Nothing in the +runtime links against libtcc, so this is aggregation with a separate program +rather than LGPL linkage. + +The runtime expects an end-user bundle at +`<exe_dir>/overlay_toolchain/{python/, tcc/tcc.exe, compile_overlays.py, …}` +(`runtime/src/main.cpp`). **No script in this repository populates that +directory**, so if release packaging supplies it, that step lives outside this +repo and the TinyCC license notice must be shipped alongside it there. + +Developers with `gcc` on `PATH` use the gcc tier instead; the bundled tcc matters +only for end-user release packages (`docs/BUILDING.md`). ## JRickey / gba-recomp — verified-enhancement shadow + screen color science diff --git a/docs/ASYNC_OVERLAY_COMPILE.md b/docs/ASYNC_OVERLAY_COMPILE.md index 28eeba45..8da4eaff 100644 --- a/docs/ASYNC_OVERLAY_COMPILE.md +++ b/docs/ASYNC_OVERLAY_COMPILE.md @@ -1,5 +1,29 @@ # Async background overlay compilation — implementation plan +> **SUPERSEDED IN PART — the sljit tier described below no longer exists.** +> +> This document was written while sljit was the toolchain-free fallback. That +> tier was replaced by tcc in `c2a3f6ad` (2026-06-25) and removed outright in +> `5b7e69b4` (2026-07-15), which deleted `lib/sljit/`, `overlay_sljit.{c,h}` and +> `overlay_compile_worker.{c,h}`. +> +> **The backend tiers today are `static → gcc → tcc`** (`docs/ARCHITECTURE.md`): +> gcc is the development default when it is on `PATH`, and TinyCC is bundled +> beside the shipped executable in `overlay_toolchain/` so players need no +> compiler. Both are external compilers invoked as a subprocess that produce a +> **DLL**; neither is an in-process JIT. +> +> That difference matters when reading this file: every passage below about +> serializing a `.sljit` blob, `overlay_sljit_deserialize`, `try_sljit_region`, +> `register_sljit_candidate`, or a per-fragment in-memory JIT registry describes +> a mechanism that was never how tcc works and no longer exists at all. Those +> passages are **not** transferable to the tcc tier by substituting the name. +> Read them as history. The async worker/file-handoff architecture, the +> gcc-versus-fallback mutual exclusivity, and the CPS work are still accurate. +> +> Current behaviour lives in `runtime/src/overlay_backend.c`, +> `runtime/src/code_provider.c` and `tools/compile_overlays.py`. + Status: **LARGELY ALREADY BUILT + VALIDATED under CPS (2026-06-18).** The "planned" feature below turned out to mostly exist: `overlay_capture.c` (`overlay_autocapture_tick` — autocapture on interp pressure, ON whenever the cache is on) + `autocompile.c` (background compile: probes for a diff --git a/docs/OVERLAY_CACHE_V2.md b/docs/OVERLAY_CACHE_V2.md index 6e89b230..7e8a0153 100644 --- a/docs/OVERLAY_CACHE_V2.md +++ b/docs/OVERLAY_CACHE_V2.md @@ -184,8 +184,11 @@ KEYING for interior shards: key/validate by the COVERED block ranges + a hash of ONLY the covered instruction bytes (NOT a guessed enclosing host range — that's what pulled in volatile/data bytes). The loader validates only those covered ranges. Dispatch priority for a PC: full host > recovered host > interior -fragment > sljit > interp; an interior shard covers only its exact legal entries, -never claims the whole host. +fragment > interp; an interior shard covers only its exact legal entries, +never claims the whole host. (This chain listed an `sljit` tier between the +interior fragment and the interpreter when written; the sljit tier was removed +in `5b7e69b4`, 2026-07-15. Shards are produced by gcc or the bundled tcc today, +and both land in the host/fragment classes above.) PHASES: P1 separate the build classes (stop poisoning — REGION_TRUSTED only emits the all-roots DLL). P2 interior-entry fragment compiler (do this FIRST for the FMV diff --git a/docs/RECURSION_BUG.md b/docs/RECURSION_BUG.md index 4bc4e387..589324ac 100644 --- a/docs/RECURSION_BUG.md +++ b/docs/RECURSION_BUG.md @@ -1269,6 +1269,10 @@ it decelerates and is unrelated to the (now-fixed) host-stack leak. Run gameplay ### 25.5 Stage 3 — overlay (sljit + gcc-DLL) CPS parity (2026-06-18) +> Historical session log. The sljit tier was removed in `5b7e69b4` (2026-07-15); +> the gcc-DLL findings below remain current, and the toolchain-free role sljit +> filled is now the bundled tcc tier. A gcc miss falls to tcc, then interp. + **sljit JIT (overlay_sljit.c), scoped:** `jr $ra` publishes `cpu->pc`; call-containing fragments **decline to the interp** (CPS-safe, depth-1); `jr rX` already tail-transferred. So only leaf shards JIT — `jr` covered, calls interp. Validated: boots, leak flat, shards diff --git a/docs/ecosystem-watch.md b/docs/ecosystem-watch.md index 212446f1..9ff2e7af 100644 --- a/docs/ecosystem-watch.md +++ b/docs/ecosystem-watch.md @@ -5,6 +5,11 @@ exact external state that was inspected, separates reusable framework work from title-specific work, and prevents an old fork from silently replacing newer framework fixes. +> Note: entries evaluated on or before 2026-07-13 predate the removal of the +> sljit Tier-2 backend (`5b7e69b4`, 2026-07-15). Where an entry describes an +> external branch touching "SLJIT", that surface no longer exists here; the +> equivalent path today is the bundled tcc tier plus the dirty interpreter. + ## Audit baseline and rules - Evaluated through `2026-07-13T21:25:28-07:00` (America/Los_Angeles), against `mstan/psxrecomp` diff --git a/docs/internal/BACKDROP_PRELOAD.md b/docs/internal/BACKDROP_PRELOAD.md index c1724870..4c722b23 100644 --- a/docs/internal/BACKDROP_PRELOAD.md +++ b/docs/internal/BACKDROP_PRELOAD.md @@ -98,7 +98,10 @@ unlikely; assert/log the per-function window count at build time. already forwards `--ws-config` (game.toml), so `auto_backdrop` reaches it. Bump `PSX_OVERLAY_CODEGEN_VER` and rebuild the overlay cache (gcc//cgN). - Backends: a backdrop generator may run as a gcc-cache DLL (pass handles it), - a sljit shard, or interpreted. For full coverage like `auto_screen_x`, the + a sljit shard, or interpreted. [sljit was removed 2026-07-15 (`5b7e69b4`); the + toolchain-free tier is now bundled tcc, which also produces a DLL — so only the + DLL and interpreted cases exist today. See `runtime/src/overlay_backend.c`.] + For full coverage like `auto_screen_x`, the sljit emitter + dirty_ram_interp need the same force-start-0 / widen-end at the detected sites. Options: (a) export the detected site map to the runtime so sljit/interp can apply it, or (b) re-derive the signature in a sljit/interp diff --git a/docs/internal/COSIM_ORACLE.md b/docs/internal/COSIM_ORACLE.md index f9c267f1..048ae334 100644 --- a/docs/internal/COSIM_ORACLE.md +++ b/docs/internal/COSIM_ORACLE.md @@ -116,7 +116,7 @@ transfer transient state that a cycle-exact lockstep needs. precise-slice / dirty-interp resume state if it affects delivery. **Exclude from the hash (host-only, would cause false divergences):** CPUState function -pointers + `sljit_helpers`, jmpbufs, fiber handles, malloc addresses, file handles, +pointers, jmpbufs, fiber handles, malloc addresses, file handles, struct padding, uninitialized bytes. Serialize little-endian explicitly; serialize arrays in fixed order; serialize any event/deadline queue **sorted** by a deterministic key (guest_cycle, priority, type, seq). diff --git a/docs/internal/README.md b/docs/internal/README.md index 5ab8d0de..702e49c2 100644 --- a/docs/internal/README.md +++ b/docs/internal/README.md @@ -38,5 +38,8 @@ behind a specific subsystem. - [`BACKDROP_PRELOAD.md`](BACKDROP_PRELOAD.md) — widescreen backdrop preload. ## Overlay cache internals -- [`SLJIT_PERSIST_CACHE.md`](SLJIT_PERSIST_CACHE.md) — persisted overlay-shard - cache design. +- `SLJIT_PERSIST_CACHE.md` — **removed.** It described the persisted shard cache + of the sljit Tier-2 backend, which was deleted in `5b7e69b4` (2026-07-15). + For the current model see `runtime/src/overlay_loader.c` (per-entry candidate + validity), `tools/compile_overlays.py` (shard formation and the `.ranges` + manifest), and [`../COMPILING_OVERLAYS.md`](../COMPILING_OVERLAYS.md). diff --git a/docs/internal/SLJIT_PERSIST_CACHE.md b/docs/internal/SLJIT_PERSIST_CACHE.md deleted file mode 100644 index 0b539c85..00000000 --- a/docs/internal/SLJIT_PERSIST_CACHE.md +++ /dev/null @@ -1,163 +0,0 @@ -# Persisted sljit shard cache — design + work to be done - -> **STATUS (implemented):** Stage 1 (position-independent shards via the -> cpu-relative `CPUState.sljit_helpers` table) and Stage 2 (serialize on JIT → -> `//sljit//cg/_.sljit`, reload + regenerate -> at init, re-validate through the existing dispatch crc + diff gate) are both -> done and validated: a clean two-launch test reloaded 12 shards from disk and ran -> them at a fresh ASLR base with no stale-pointer crash, gameplay rendering -> correctly. KEY GOTCHA found in testing: `sljit_serialize_compiler` must run -> BEFORE `sljit_generate_code` (generate finalizes the compiler → `compiler->error -> = SLJIT_ERR_COMPILED` → serialize then returns NULL), and serialize with -> `options = 0` (NOT `IGNORE_DEBUG`) so an argument-checks build can deserialize. -> Persist timing is write-on-generate (negligible cost, crash-safe). The notes -> below are the original design. - -Branch: `feat/sljit-persist-cache` (off `master` @ 0c1f005). Deferred from the -2026-06-15 sljit-default-on session because it has an **ASLR prerequisite** that -makes it a focused feature, not a quick add. In-session sljit re-JIT already -works (sljit is default-on, validated, gcc > sljit > interp); this feature only -adds **cross-session persistence** so subsequent launches reuse shards instead of -re-JITing the tail. - -## Goal - -> "sljit is supposed to have its own cache too — separate from gcc, lower -> priority. Load into memory in subsequent sessions any sljit shard that doesn't -> have a higher-priority gcc DLL." — user, 2026-06-15 - -Persist validated sljit shards to disk; reload them at startup; use a reloaded -shard only where no higher-priority gcc DLL covers the same function. Priority -stays **gcc > sljit > interp**. - -## The blocker (why v1 punted, why Stage 1 exists) - -`sljit_serialize_compiler()` serializes the LIR **verbatim, including immediate -operands**. Per sljitLir.h it restores jump *targets*, but NOT arbitrary -`MOV reg, IMM` operands. Our shards bake **absolute host function pointers** as -immediates, so a shard serialized in one process and reloaded in another calls -**stale addresses** (the runtime loads at a different base each launch — ASLR) -→ crash. So persistence is impossible until shards are position-independent -w.r.t. host pointers. - -### The exact hazard sites (runtime/src/overlay_sljit.c, as of 0c1f005) - -Three sites bake **four** distinct helper pointers via `SLJIT_IMM (uintptr_t)fn`: - -| Site (fn) | line | helper(s) baked | -|---|---|---| -| `emit_helper2` | ~349 | `psx_sljit_memx`, `psx_sljit_cop2` (LWL/LWR/SWL/SWR, COP2/GTE, LWC2/SWC2) | -| `emit_ws_cull` | ~365 | `psx_ws_cull_sltiu` (widescreen render-funnel cull widen) | -| `PEND_CALL` | ~921 | `psx_sljit_call` (jal/jalr call contract) | - -The load/store icalls (~320/337) are **already reload-safe** — they load the fn -from `SLJIT_MEM1(R_CPU), offsetof(CPUState, read_word/write_word/...)`, i.e. -cpu-relative. That is the exact pattern Stage 1 copies for the helpers. - -## Stage 1 — position-independent shards (PREREQUISITE, correctness-critical) - -Route every host-helper call through a **cpu-relative helper-pointer table** -instead of a baked immediate, so the serialized LIR contains zero absolute host -addresses. - -1. Add a fixed-order helper table reachable from `CPUState` (runtime/include/ - cpu_state.h). Either a `const void *const *sljit_helpers;` pointer field - (append at END so existing gcc-DLL field offsets are unchanged — gcc codegen - is unaffected; gcc DLLs resolve `psx_ws_cull_sltiu` via the loader's import - fixups, not baked pointers, so ONLY sljit needs this) or an inline array. -2. Define a stable enum order, e.g. - `enum { SLJIT_HLP_MEMX, SLJIT_HLP_COP2, SLJIT_HLP_WS_CULL, SLJIT_HLP_CALL, SLJIT_HLP_COUNT };` - The order is part of the on-disk format — never reorder; only append. Fold - the order/contents into a serialization version (see Stage 2 / cg). -3. Populate the table once at startup (where CPUState is initialized) with the - real addresses. -4. Change the 3 emit sites: instead of `MOV R2, IMM (uintptr_t)fn`, emit - `MOV Rtab, MEM1(R_CPU)[offsetof(CPUState, sljit_helpers)]` then - `MOV Rfn, MEM1(Rtab)[k*sizeof(void*)]` then icall `Rfn`. (`emit_helper2` - takes a helper-enum index instead of a raw `fn`.) Mind scratch-register - usage — the table load needs a scratch that doesn't clobber the call args - (R0..R3 are args; use R4/a free scratch for the table+fn, as PEND_CALL - already uses R4 for the fn ptr). -5. **This is a strict win on its own** and is independently testable: with - Stage 1 alone (no disk cache yet), in-session JIT must still pass the existing - live playtest (no regression: same shard behaviour, just one extra indirection - per helper call). Verify before building Stage 2. - -Risk: a missed/mis-ordered helper pointer = every shard that uses it crashes. -Test by exercising each helper class (a GTE-heavy scene for COP2, an unaligned -load for memx, a jal-heavy overlay for call, a 16:9 overlay-funnel scene for -ws_cull). - -## Stage 2 — the disk cache (serialize / reload / re-validate) - -Mirror the gcc cache layout + versioning exactly. - -**Path:** `//sljit//cg/_.sljit` -(the `sljit//` namespace is already RESERVED in overlay_loader.c's -scan; `cg` = `PSX_OVERLAY_CODEGEN_VER` from overlay_api.h, same as gcc — so a -codegen/emitter change auto-invalidates stale sljit blobs too. Also bump a -dedicated serialization tag if the helper-table order changes.) - -**Serialize on compile** (overlay_sljit.c, after `sljit_generate_code` succeeds): -- `buf = sljit_serialize_compiler(C, SLJIT_SERIALIZE_IGNORE_DEBUG, &sz)` BEFORE - `sljit_free_compiler(C)` (serialize doesn't modify the compiler, so order vs - generate_code is flexible; keep both before free). -- Compute `crc_code` over live RAM `[entry_phys, entry_phys+frag_words*4)` with - the same `crc32_compute` the loader uses (so the filename CRC matches what - dispatch re-hashes). -- Write a small header + the serialized buffer: - `{ magic, format_ver, helper_order_ver, entry_phys, code_lo, code_len, - insns, crc_code, blob_size }` then the `sljit_uw` blob. -- Gate writing on a config/env (don't write during shadow/diff runs; only persist - shards that have been VALIDATED — or persist eagerly and rely on reload-time - re-validation, see below. Eager is simpler and safe given re-validation.). - -**Reload at init** (overlay_loader.c, extend `scan_cache_dir`): -- Scan `sljit//cg/*.sljit`. For each: read header, check - magic/format/helper_order, `C = sljit_deserialize_compiler(blob, sz, 0, NULL)`, - `code = sljit_generate_code(C, 0, NULL)`, `sljit_free_compiler(C)`. -- Register as a sljit candidate (`dll = -1`) over `[code_lo, code_len)` with - `crc_code` — exactly like `register_sljit_candidate`, just sourced from disk. - -**Re-validation (the safety model — no new trust surface):** a reloaded shard -is NOT trusted blindly. The existing dispatch path already (a) re-hashes live RAM -vs `crc_code` every dispatch (a shard for a different scene/variant simply never -matches → never runs), and (b) routes `dll<0` shards through the same-state diff -gate (`want_diff = s_sljit_live && c->dll < 0`) until the verify budget, with -device-touch/diverging shards pinned to interp. So a persisted shard re-earns -trust each session exactly like a fresh JIT. This is what makes Stage 2 safe. - -**Priority gcc > sljit > interp:** unchanged — gcc DLLs register as `dll>=0` and -the existing obsolete logic (`sljit shard obsoleted by gcc DLL`) retires a sljit -shard when a gcc DLL covers the same live function. Ensure load order or the -dispatch chain keeps gcc ahead (gcc DLLs loaded lazily on dispatch miss via -`try_load_region`; persisted sljit shards loaded at init — confirm a gcc DLL that -arrives later still supersedes the pre-registered sljit shard). - -## Verification plan - -- Stage 1: in-session playtest unchanged (no crash, same diff stats) — gates - Stage 2. -- Stage 2: launch once (populate the sljit cache), relaunch, confirm shards load - from disk (`sljit_status` shows reloaded count), the game runs (no stale-pointer - crash — proves Stage 1), gcc-covered functions still use gcc, and the - native:interp ratio matches the in-session-JIT baseline. Force a helper-class - per shard (GTE/memx/call/ws_cull) on reload to prove the indirection. -- 4:3 / widescreen-off unaffected throughout. - -## Files - -- `runtime/include/cpu_state.h` — helper table field (Stage 1). -- `runtime/src/overlay_sljit.c` — helper-table emit (Stage 1) + serialize (Stage 2). -- `runtime/src/overlay_loader.c` — reload + register at init (Stage 2); the - `sljit//` scan is already reserved here. -- `runtime/include/overlay_api.h` — `PSX_OVERLAY_CODEGEN_VER` (+ a helper-order - tag) gates the sljit blob namespace. -- startup (CPUState init site) — populate the helper table (Stage 1). - -## Context (what already shipped on master, 0c1f005) - -sljit is merged + default-on + validated; one shared `psx_ws_cull_sltiu` helper -is routed by gcc/sljit/interp (overlay 3D widens in 16:9 on every path); the gcc -cache is versioned `gcc//cg/`. See `SLJIT.md`, `WIDESCREEN.md`, and -memory `sljit_backend_state` / `native_wide_fov_autocull`.