From 55cec09acd7e6dfc40f38bd88d2b2efd24b391b4 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Sun, 16 Aug 2026 18:16:34 -0700 Subject: [PATCH] docs: retire stale sljit references, attribute TinyCC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sljit Tier-2 in-process JIT was replaced by the tcc tier in c2a3f6ad (2026-06-25) and removed outright in 5b7e69b4 (2026-07-15). That removal reworded the C-level comments but touched no markdown, so the prose still described sljit as a live, default-on backend. Backends today are static -> gcc -> tcc: gcc is the development default when it is on PATH, tcc is the toolchain-free path for players. Both are external compilers invoked as a subprocess that produce a DLL; neither is an in-process JIT. docs/ARCHITECTURE.md, docs/BUILDING.md, docs/EXECUTION_MODEL.md, docs/COMPILING_OVERLAYS.md and README.md already said this correctly and are unchanged here. - THIRD_PARTY_ATTRIBUTION.md: the sljit entry claimed code vendored at lib/sljit/ and compiled into the runtime. That directory does not exist. Replaced with a TinyCC entry, which was missing entirely despite tcc being the compiler intended to reach end users. - docs/internal/SLJIT_PERSIST_CACHE.md: deleted. It documented the persisted shard cache of a subsystem that no longer exists, as "implemented" and "default-on". docs/internal/README.md updated to point at the current model instead of a dead link. - docs/ASYNC_OVERLAY_COMPILE.md: superseded-in-part header. Deliberately NOT rewritten sljit->tcc: the passages about serializing .sljit blobs, overlay_sljit_deserialize and an in-memory JIT registry describe a mechanism tcc never used, so renaming it would trade one false claim for another. The async worker/file-handoff design and the CPS work remain accurate. - docs/OVERLAY_CACHE_V2.md: sljit dropped from the dispatch priority chain. - ISSUES.md: Issue #7 closed as superseded — the backend it reported against was deleted rather than fixed. - docs/internal/COSIM_ORACLE.md: sljit_helpers left CPUState with the tier. - Dated session logs (RECURSION_BUG.md 25.5, BACKDROP_PRELOAD.md, ecosystem-watch.md) keep their narrative and get a pointer to the removal. One thing this does not fix: docs assert the tcc toolchain is "downloaded at release-packaging time and bundled beside the game exe". runtime/src/main.cpp does look for /overlay_toolchain/, but no script in this repository creates it. The attribution entry now states only what the code does and notes that if packaging supplies the bundle, it happens outside this repo and must carry the TinyCC notice. Flagged rather than papered over. Co-Authored-By: Claude Opus 5 (1M context) --- ISSUES.md | 10 +- THIRD_PARTY_ATTRIBUTION.md | 24 ++-- docs/ASYNC_OVERLAY_COMPILE.md | 24 ++++ docs/OVERLAY_CACHE_V2.md | 7 +- docs/RECURSION_BUG.md | 4 + docs/ecosystem-watch.md | 5 + docs/internal/BACKDROP_PRELOAD.md | 5 +- docs/internal/COSIM_ORACLE.md | 2 +- docs/internal/README.md | 7 +- docs/internal/SLJIT_PERSIST_CACHE.md | 163 --------------------------- 10 files changed, 74 insertions(+), 177 deletions(-) delete mode 100644 docs/internal/SLJIT_PERSIST_CACHE.md diff --git a/ISSUES.md b/ISSUES.md index d494c6ee..b0122059 100644 --- a/ISSUES.md +++ b/ISSUES.md @@ -515,8 +515,16 @@ discovery seeds). Either way, then close Issue #3 alongside. ## Issue #7 — sljit live execution is unvalidated (pure-live save-load wedge) -**Status:** open, root-caused — fix in progress (branch `feat/sljit-backend`) +**Status:** CLOSED — superseded, not fixed. The sljit Tier-2 backend this issue +describes was removed wholesale in `5b7e69b4` (2026-07-15), which deleted +`lib/sljit/`, `overlay_sljit.{c,h}` and the sljit paths in `code_provider.c` / +`overlay_loader.c`. The toolchain-free role it filled is now served by the +bundled TinyCC tier (introduced `c2a3f6ad`, 2026-06-25), which compiles to a DLL +through a subprocess rather than JITting in-process, so it does not share this +defect's live-execution/save-load shape. Nothing here is actionable; retained as +history. **Date opened:** 2026-06-15 +**Date closed:** 2026-07-15 (by removal of the subsystem) **Area:** overlay Tier-2 sljit backend (`runtime/src/overlay_loader.c`, `overlay_sljit.c`, `code_provider.c`, `overlay_sljit.c` resolution) diff --git a/THIRD_PARTY_ATTRIBUTION.md b/THIRD_PARTY_ATTRIBUTION.md index b57d5f90..0b1b85d2 100644 --- a/THIRD_PARTY_ATTRIBUTION.md +++ b/THIRD_PARTY_ATTRIBUTION.md @@ -13,14 +13,24 @@ Normal runtime builds stage it automatically so players supply only a disc; `bios/OpenBIOS.LICENSE` always rides alongside the shipped image. -## sljit — stack-less JIT compiler (Tier-2 in-process overlay backend) +## TinyCC (TCC) — toolchain-free overlay compiler shipped to players -[sljit](https://github.com/zherczeg/sljit) by Zoltan Herczeg, licensed -**BSD-2-Clause**. Vendored at `lib/sljit/` (source `lib/sljit/sljit_src/`, -license `lib/sljit/LICENSE`) and compiled into the runtime as the self-contained -Tier-2 overlay JIT backend (`runtime/src/overlay_sljit.c`). No external toolchain -dependency; sljit auto-detects the host architecture. See `SLJIT.md` (repo root / -workspace) for the backend design. +[TinyCC](https://bellard.org/tcc/) by Fabrice Bellard and contributors, licensed +**LGPL-2.1**. Not vendored in this repository — no TinyCC source or binary is +tracked here. It is invoked as a **separate subprocess** by +`tools/compile_overlays.py` (`--compiler tcc`, `--tcc <binary>`) to build overlay +shards into a DLL, so players need no compiler of their own. Nothing in the +runtime links against libtcc, so this is aggregation with a separate program +rather than LGPL linkage. + +The runtime expects an end-user bundle at +`<exe_dir>/overlay_toolchain/{python/, tcc/tcc.exe, compile_overlays.py, …}` +(`runtime/src/main.cpp`). **No script in this repository populates that +directory**, so if release packaging supplies it, that step lives outside this +repo and the TinyCC license notice must be shipped alongside it there. + +Developers with `gcc` on `PATH` use the gcc tier instead; the bundled tcc matters +only for end-user release packages (`docs/BUILDING.md`). ## JRickey / gba-recomp — verified-enhancement shadow + screen color science diff --git a/docs/ASYNC_OVERLAY_COMPILE.md b/docs/ASYNC_OVERLAY_COMPILE.md index 28eeba45..8da4eaff 100644 --- a/docs/ASYNC_OVERLAY_COMPILE.md +++ b/docs/ASYNC_OVERLAY_COMPILE.md @@ -1,5 +1,29 @@ # Async background overlay compilation — implementation plan +> **SUPERSEDED IN PART — the sljit tier described below no longer exists.** +> +> This document was written while sljit was the toolchain-free fallback. That +> tier was replaced by tcc in `c2a3f6ad` (2026-06-25) and removed outright in +> `5b7e69b4` (2026-07-15), which deleted `lib/sljit/`, `overlay_sljit.{c,h}` and +> `overlay_compile_worker.{c,h}`. +> +> **The backend tiers today are `static → gcc → tcc`** (`docs/ARCHITECTURE.md`): +> gcc is the development default when it is on `PATH`, and TinyCC is bundled +> beside the shipped executable in `overlay_toolchain/` so players need no +> compiler. Both are external compilers invoked as a subprocess that produce a +> **DLL**; neither is an in-process JIT. +> +> That difference matters when reading this file: every passage below about +> serializing a `.sljit` blob, `overlay_sljit_deserialize`, `try_sljit_region`, +> `register_sljit_candidate`, or a per-fragment in-memory JIT registry describes +> a mechanism that was never how tcc works and no longer exists at all. Those +> passages are **not** transferable to the tcc tier by substituting the name. +> Read them as history. The async worker/file-handoff architecture, the +> gcc-versus-fallback mutual exclusivity, and the CPS work are still accurate. +> +> Current behaviour lives in `runtime/src/overlay_backend.c`, +> `runtime/src/code_provider.c` and `tools/compile_overlays.py`. + Status: **LARGELY ALREADY BUILT + VALIDATED under CPS (2026-06-18).** The "planned" feature below turned out to mostly exist: `overlay_capture.c` (`overlay_autocapture_tick` — autocapture on interp pressure, ON whenever the cache is on) + `autocompile.c` (background compile: probes for a diff --git a/docs/OVERLAY_CACHE_V2.md b/docs/OVERLAY_CACHE_V2.md index 6e89b230..7e8a0153 100644 --- a/docs/OVERLAY_CACHE_V2.md +++ b/docs/OVERLAY_CACHE_V2.md @@ -184,8 +184,11 @@ KEYING for interior shards: key/validate by the COVERED block ranges + a hash of ONLY the covered instruction bytes (NOT a guessed enclosing host range — that's what pulled in volatile/data bytes). The loader validates only those covered ranges. Dispatch priority for a PC: full host > recovered host > interior -fragment > sljit > interp; an interior shard covers only its exact legal entries, -never claims the whole host. +fragment > interp; an interior shard covers only its exact legal entries, +never claims the whole host. (This chain listed an `sljit` tier between the +interior fragment and the interpreter when written; the sljit tier was removed +in `5b7e69b4`, 2026-07-15. Shards are produced by gcc or the bundled tcc today, +and both land in the host/fragment classes above.) PHASES: P1 separate the build classes (stop poisoning — REGION_TRUSTED only emits the all-roots DLL). P2 interior-entry fragment compiler (do this FIRST for the FMV diff --git a/docs/RECURSION_BUG.md b/docs/RECURSION_BUG.md index 4bc4e387..589324ac 100644 --- a/docs/RECURSION_BUG.md +++ b/docs/RECURSION_BUG.md @@ -1269,6 +1269,10 @@ it decelerates and is unrelated to the (now-fixed) host-stack leak. Run gameplay ### 25.5 Stage 3 — overlay (sljit + gcc-DLL) CPS parity (2026-06-18) +> Historical session log. The sljit tier was removed in `5b7e69b4` (2026-07-15); +> the gcc-DLL findings below remain current, and the toolchain-free role sljit +> filled is now the bundled tcc tier. A gcc miss falls to tcc, then interp. + **sljit JIT (overlay_sljit.c), scoped:** `jr $ra` publishes `cpu->pc`; call-containing fragments **decline to the interp** (CPS-safe, depth-1); `jr rX` already tail-transferred. So only leaf shards JIT — `jr` covered, calls interp. Validated: boots, leak flat, shards diff --git a/docs/ecosystem-watch.md b/docs/ecosystem-watch.md index 212446f1..9ff2e7af 100644 --- a/docs/ecosystem-watch.md +++ b/docs/ecosystem-watch.md @@ -5,6 +5,11 @@ exact external state that was inspected, separates reusable framework work from title-specific work, and prevents an old fork from silently replacing newer framework fixes. +> Note: entries evaluated on or before 2026-07-13 predate the removal of the +> sljit Tier-2 backend (`5b7e69b4`, 2026-07-15). Where an entry describes an +> external branch touching "SLJIT", that surface no longer exists here; the +> equivalent path today is the bundled tcc tier plus the dirty interpreter. + ## Audit baseline and rules - Evaluated through `2026-07-13T21:25:28-07:00` (America/Los_Angeles), against `mstan/psxrecomp` diff --git a/docs/internal/BACKDROP_PRELOAD.md b/docs/internal/BACKDROP_PRELOAD.md index c1724870..4c722b23 100644 --- a/docs/internal/BACKDROP_PRELOAD.md +++ b/docs/internal/BACKDROP_PRELOAD.md @@ -98,7 +98,10 @@ unlikely; assert/log the per-function window count at build time. already forwards `--ws-config` (game.toml), so `auto_backdrop` reaches it. Bump `PSX_OVERLAY_CODEGEN_VER` and rebuild the overlay cache (gcc//cgN). - Backends: a backdrop generator may run as a gcc-cache DLL (pass handles it), - a sljit shard, or interpreted. For full coverage like `auto_screen_x`, the + a sljit shard, or interpreted. [sljit was removed 2026-07-15 (`5b7e69b4`); the + toolchain-free tier is now bundled tcc, which also produces a DLL — so only the + DLL and interpreted cases exist today. See `runtime/src/overlay_backend.c`.] + For full coverage like `auto_screen_x`, the sljit emitter + dirty_ram_interp need the same force-start-0 / widen-end at the detected sites. Options: (a) export the detected site map to the runtime so sljit/interp can apply it, or (b) re-derive the signature in a sljit/interp diff --git a/docs/internal/COSIM_ORACLE.md b/docs/internal/COSIM_ORACLE.md index f9c267f1..048ae334 100644 --- a/docs/internal/COSIM_ORACLE.md +++ b/docs/internal/COSIM_ORACLE.md @@ -116,7 +116,7 @@ transfer transient state that a cycle-exact lockstep needs. precise-slice / dirty-interp resume state if it affects delivery. **Exclude from the hash (host-only, would cause false divergences):** CPUState function -pointers + `sljit_helpers`, jmpbufs, fiber handles, malloc addresses, file handles, +pointers, jmpbufs, fiber handles, malloc addresses, file handles, struct padding, uninitialized bytes. Serialize little-endian explicitly; serialize arrays in fixed order; serialize any event/deadline queue **sorted** by a deterministic key (guest_cycle, priority, type, seq). diff --git a/docs/internal/README.md b/docs/internal/README.md index 5ab8d0de..702e49c2 100644 --- a/docs/internal/README.md +++ b/docs/internal/README.md @@ -38,5 +38,8 @@ behind a specific subsystem. - [`BACKDROP_PRELOAD.md`](BACKDROP_PRELOAD.md) — widescreen backdrop preload. ## Overlay cache internals -- [`SLJIT_PERSIST_CACHE.md`](SLJIT_PERSIST_CACHE.md) — persisted overlay-shard - cache design. +- `SLJIT_PERSIST_CACHE.md` — **removed.** It described the persisted shard cache + of the sljit Tier-2 backend, which was deleted in `5b7e69b4` (2026-07-15). + For the current model see `runtime/src/overlay_loader.c` (per-entry candidate + validity), `tools/compile_overlays.py` (shard formation and the `.ranges` + manifest), and [`../COMPILING_OVERLAYS.md`](../COMPILING_OVERLAYS.md). diff --git a/docs/internal/SLJIT_PERSIST_CACHE.md b/docs/internal/SLJIT_PERSIST_CACHE.md deleted file mode 100644 index 0b539c85..00000000 --- a/docs/internal/SLJIT_PERSIST_CACHE.md +++ /dev/null @@ -1,163 +0,0 @@ -# Persisted sljit shard cache — design + work to be done - -> **STATUS (implemented):** Stage 1 (position-independent shards via the -> cpu-relative `CPUState.sljit_helpers` table) and Stage 2 (serialize on JIT → -> `//sljit//cg/_.sljit`, reload + regenerate -> at init, re-validate through the existing dispatch crc + diff gate) are both -> done and validated: a clean two-launch test reloaded 12 shards from disk and ran -> them at a fresh ASLR base with no stale-pointer crash, gameplay rendering -> correctly. KEY GOTCHA found in testing: `sljit_serialize_compiler` must run -> BEFORE `sljit_generate_code` (generate finalizes the compiler → `compiler->error -> = SLJIT_ERR_COMPILED` → serialize then returns NULL), and serialize with -> `options = 0` (NOT `IGNORE_DEBUG`) so an argument-checks build can deserialize. -> Persist timing is write-on-generate (negligible cost, crash-safe). The notes -> below are the original design. - -Branch: `feat/sljit-persist-cache` (off `master` @ 0c1f005). Deferred from the -2026-06-15 sljit-default-on session because it has an **ASLR prerequisite** that -makes it a focused feature, not a quick add. In-session sljit re-JIT already -works (sljit is default-on, validated, gcc > sljit > interp); this feature only -adds **cross-session persistence** so subsequent launches reuse shards instead of -re-JITing the tail. - -## Goal - -> "sljit is supposed to have its own cache too — separate from gcc, lower -> priority. Load into memory in subsequent sessions any sljit shard that doesn't -> have a higher-priority gcc DLL." — user, 2026-06-15 - -Persist validated sljit shards to disk; reload them at startup; use a reloaded -shard only where no higher-priority gcc DLL covers the same function. Priority -stays **gcc > sljit > interp**. - -## The blocker (why v1 punted, why Stage 1 exists) - -`sljit_serialize_compiler()` serializes the LIR **verbatim, including immediate -operands**. Per sljitLir.h it restores jump *targets*, but NOT arbitrary -`MOV reg, IMM` operands. Our shards bake **absolute host function pointers** as -immediates, so a shard serialized in one process and reloaded in another calls -**stale addresses** (the runtime loads at a different base each launch — ASLR) -→ crash. So persistence is impossible until shards are position-independent -w.r.t. host pointers. - -### The exact hazard sites (runtime/src/overlay_sljit.c, as of 0c1f005) - -Three sites bake **four** distinct helper pointers via `SLJIT_IMM (uintptr_t)fn`: - -| Site (fn) | line | helper(s) baked | -|---|---|---| -| `emit_helper2` | ~349 | `psx_sljit_memx`, `psx_sljit_cop2` (LWL/LWR/SWL/SWR, COP2/GTE, LWC2/SWC2) | -| `emit_ws_cull` | ~365 | `psx_ws_cull_sltiu` (widescreen render-funnel cull widen) | -| `PEND_CALL` | ~921 | `psx_sljit_call` (jal/jalr call contract) | - -The load/store icalls (~320/337) are **already reload-safe** — they load the fn -from `SLJIT_MEM1(R_CPU), offsetof(CPUState, read_word/write_word/...)`, i.e. -cpu-relative. That is the exact pattern Stage 1 copies for the helpers. - -## Stage 1 — position-independent shards (PREREQUISITE, correctness-critical) - -Route every host-helper call through a **cpu-relative helper-pointer table** -instead of a baked immediate, so the serialized LIR contains zero absolute host -addresses. - -1. Add a fixed-order helper table reachable from `CPUState` (runtime/include/ - cpu_state.h). Either a `const void *const *sljit_helpers;` pointer field - (append at END so existing gcc-DLL field offsets are unchanged — gcc codegen - is unaffected; gcc DLLs resolve `psx_ws_cull_sltiu` via the loader's import - fixups, not baked pointers, so ONLY sljit needs this) or an inline array. -2. Define a stable enum order, e.g. - `enum { SLJIT_HLP_MEMX, SLJIT_HLP_COP2, SLJIT_HLP_WS_CULL, SLJIT_HLP_CALL, SLJIT_HLP_COUNT };` - The order is part of the on-disk format — never reorder; only append. Fold - the order/contents into a serialization version (see Stage 2 / cg). -3. Populate the table once at startup (where CPUState is initialized) with the - real addresses. -4. Change the 3 emit sites: instead of `MOV R2, IMM (uintptr_t)fn`, emit - `MOV Rtab, MEM1(R_CPU)[offsetof(CPUState, sljit_helpers)]` then - `MOV Rfn, MEM1(Rtab)[k*sizeof(void*)]` then icall `Rfn`. (`emit_helper2` - takes a helper-enum index instead of a raw `fn`.) Mind scratch-register - usage — the table load needs a scratch that doesn't clobber the call args - (R0..R3 are args; use R4/a free scratch for the table+fn, as PEND_CALL - already uses R4 for the fn ptr). -5. **This is a strict win on its own** and is independently testable: with - Stage 1 alone (no disk cache yet), in-session JIT must still pass the existing - live playtest (no regression: same shard behaviour, just one extra indirection - per helper call). Verify before building Stage 2. - -Risk: a missed/mis-ordered helper pointer = every shard that uses it crashes. -Test by exercising each helper class (a GTE-heavy scene for COP2, an unaligned -load for memx, a jal-heavy overlay for call, a 16:9 overlay-funnel scene for -ws_cull). - -## Stage 2 — the disk cache (serialize / reload / re-validate) - -Mirror the gcc cache layout + versioning exactly. - -**Path:** `//sljit//cg/_.sljit` -(the `sljit//` namespace is already RESERVED in overlay_loader.c's -scan; `cg` = `PSX_OVERLAY_CODEGEN_VER` from overlay_api.h, same as gcc — so a -codegen/emitter change auto-invalidates stale sljit blobs too. Also bump a -dedicated serialization tag if the helper-table order changes.) - -**Serialize on compile** (overlay_sljit.c, after `sljit_generate_code` succeeds): -- `buf = sljit_serialize_compiler(C, SLJIT_SERIALIZE_IGNORE_DEBUG, &sz)` BEFORE - `sljit_free_compiler(C)` (serialize doesn't modify the compiler, so order vs - generate_code is flexible; keep both before free). -- Compute `crc_code` over live RAM `[entry_phys, entry_phys+frag_words*4)` with - the same `crc32_compute` the loader uses (so the filename CRC matches what - dispatch re-hashes). -- Write a small header + the serialized buffer: - `{ magic, format_ver, helper_order_ver, entry_phys, code_lo, code_len, - insns, crc_code, blob_size }` then the `sljit_uw` blob. -- Gate writing on a config/env (don't write during shadow/diff runs; only persist - shards that have been VALIDATED — or persist eagerly and rely on reload-time - re-validation, see below. Eager is simpler and safe given re-validation.). - -**Reload at init** (overlay_loader.c, extend `scan_cache_dir`): -- Scan `sljit//cg/*.sljit`. For each: read header, check - magic/format/helper_order, `C = sljit_deserialize_compiler(blob, sz, 0, NULL)`, - `code = sljit_generate_code(C, 0, NULL)`, `sljit_free_compiler(C)`. -- Register as a sljit candidate (`dll = -1`) over `[code_lo, code_len)` with - `crc_code` — exactly like `register_sljit_candidate`, just sourced from disk. - -**Re-validation (the safety model — no new trust surface):** a reloaded shard -is NOT trusted blindly. The existing dispatch path already (a) re-hashes live RAM -vs `crc_code` every dispatch (a shard for a different scene/variant simply never -matches → never runs), and (b) routes `dll<0` shards through the same-state diff -gate (`want_diff = s_sljit_live && c->dll < 0`) until the verify budget, with -device-touch/diverging shards pinned to interp. So a persisted shard re-earns -trust each session exactly like a fresh JIT. This is what makes Stage 2 safe. - -**Priority gcc > sljit > interp:** unchanged — gcc DLLs register as `dll>=0` and -the existing obsolete logic (`sljit shard obsoleted by gcc DLL`) retires a sljit -shard when a gcc DLL covers the same live function. Ensure load order or the -dispatch chain keeps gcc ahead (gcc DLLs loaded lazily on dispatch miss via -`try_load_region`; persisted sljit shards loaded at init — confirm a gcc DLL that -arrives later still supersedes the pre-registered sljit shard). - -## Verification plan - -- Stage 1: in-session playtest unchanged (no crash, same diff stats) — gates - Stage 2. -- Stage 2: launch once (populate the sljit cache), relaunch, confirm shards load - from disk (`sljit_status` shows reloaded count), the game runs (no stale-pointer - crash — proves Stage 1), gcc-covered functions still use gcc, and the - native:interp ratio matches the in-session-JIT baseline. Force a helper-class - per shard (GTE/memx/call/ws_cull) on reload to prove the indirection. -- 4:3 / widescreen-off unaffected throughout. - -## Files - -- `runtime/include/cpu_state.h` — helper table field (Stage 1). -- `runtime/src/overlay_sljit.c` — helper-table emit (Stage 1) + serialize (Stage 2). -- `runtime/src/overlay_loader.c` — reload + register at init (Stage 2); the - `sljit//` scan is already reserved here. -- `runtime/include/overlay_api.h` — `PSX_OVERLAY_CODEGEN_VER` (+ a helper-order - tag) gates the sljit blob namespace. -- startup (CPUState init site) — populate the helper table (Stage 1). - -## Context (what already shipped on master, 0c1f005) - -sljit is merged + default-on + validated; one shared `psx_ws_cull_sltiu` helper -is routed by gcc/sljit/interp (overlay 3D widens in 16:9 on every path); the gcc -cache is versioned `gcc//cg/`. See `SLJIT.md`, `WIDESCREEN.md`, and -memory `sljit_backend_state` / `native_wide_fov_autocull`.