docs: retire stale sljit references, attribute TinyCC

The sljit Tier-2 in-process JIT was replaced by the tcc tier in c2a3f6ad
(2026-06-25) and removed outright in 5b7e69b4 (2026-07-15). That removal
reworded the C-level comments but touched no markdown, so the prose still
described sljit as a live, default-on backend.

Backends today are static -> gcc -> tcc: gcc is the development default when
it is on PATH, tcc is the toolchain-free path for players. Both are external
compilers invoked as a subprocess that produce a DLL; neither is an in-process
JIT. docs/ARCHITECTURE.md, docs/BUILDING.md, docs/EXECUTION_MODEL.md,
docs/COMPILING_OVERLAYS.md and README.md already said this correctly and are
unchanged here.

- THIRD_PARTY_ATTRIBUTION.md: the sljit entry claimed code vendored at
  lib/sljit/ and compiled into the runtime. That directory does not exist.
  Replaced with a TinyCC entry, which was missing entirely despite tcc being
  the compiler intended to reach end users.
- docs/internal/SLJIT_PERSIST_CACHE.md: deleted. It documented the persisted
  shard cache of a subsystem that no longer exists, as "implemented" and
  "default-on". docs/internal/README.md updated to point at the current model
  instead of a dead link.
- docs/ASYNC_OVERLAY_COMPILE.md: superseded-in-part header. Deliberately NOT
  rewritten sljit->tcc: the passages about serializing .sljit blobs,
  overlay_sljit_deserialize and an in-memory JIT registry describe a mechanism
  tcc never used, so renaming it would trade one false claim for another. The
  async worker/file-handoff design and the CPS work remain accurate.
- docs/OVERLAY_CACHE_V2.md: sljit dropped from the dispatch priority chain.
- ISSUES.md: Issue #7 closed as superseded — the backend it reported against
  was deleted rather than fixed.
- docs/internal/COSIM_ORACLE.md: sljit_helpers left CPUState with the tier.
- Dated session logs (RECURSION_BUG.md 25.5, BACKDROP_PRELOAD.md,
  ecosystem-watch.md) keep their narrative and get a pointer to the removal.

One thing this does not fix: docs assert the tcc toolchain is "downloaded at
release-packaging time and bundled beside the game exe". runtime/src/main.cpp
does look for <exe_dir>/overlay_toolchain/, but no script in this repository
creates it. The attribution entry now states only what the code does and notes
that if packaging supplies the bundle, it happens outside this repo and must
carry the TinyCC notice. Flagged rather than papered over.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Matthew Stanley
2026-08-16 18:16:34 -07:00
parent 1181c9c9e9
commit 55cec09acd
10 changed files with 74 additions and 177 deletions
+9 -1
View File
@@ -515,8 +515,16 @@ discovery seeds). Either way, then close Issue #3 alongside.
## Issue #7 — sljit live execution is unvalidated (pure-live save-load wedge)
**Status:** open, root-caused — fix in progress (branch `feat/sljit-backend`)
**Status:** CLOSED — superseded, not fixed. The sljit Tier-2 backend this issue
describes was removed wholesale in `5b7e69b4` (2026-07-15), which deleted
`lib/sljit/`, `overlay_sljit.{c,h}` and the sljit paths in `code_provider.c` /
`overlay_loader.c`. The toolchain-free role it filled is now served by the
bundled TinyCC tier (introduced `c2a3f6ad`, 2026-06-25), which compiles to a DLL
through a subprocess rather than JITting in-process, so it does not share this
defect's live-execution/save-load shape. Nothing here is actionable; retained as
history.
**Date opened:** 2026-06-15
**Date closed:** 2026-07-15 (by removal of the subsystem)
**Area:** overlay Tier-2 sljit backend (`runtime/src/overlay_loader.c`,
`overlay_sljit.c`, `code_provider.c`, `overlay_sljit.c` resolution)
+17 -7
View File
@@ -13,14 +13,24 @@ Normal runtime builds stage it automatically so players supply only a disc;
`bios/OpenBIOS.LICENSE` always rides alongside the shipped image.
## sljit — stack-less JIT compiler (Tier-2 in-process overlay backend)
## TinyCC (TCC) — toolchain-free overlay compiler shipped to players
[sljit](https://github.com/zherczeg/sljit) by Zoltan Herczeg, licensed
**BSD-2-Clause**. Vendored at `lib/sljit/` (source `lib/sljit/sljit_src/`,
license `lib/sljit/LICENSE`) and compiled into the runtime as the self-contained
Tier-2 overlay JIT backend (`runtime/src/overlay_sljit.c`). No external toolchain
dependency; sljit auto-detects the host architecture. See `SLJIT.md` (repo root /
workspace) for the backend design.
[TinyCC](https://bellard.org/tcc/) by Fabrice Bellard and contributors, licensed
**LGPL-2.1**. Not vendored in this repository — no TinyCC source or binary is
tracked here. It is invoked as a **separate subprocess** by
`tools/compile_overlays.py` (`--compiler tcc`, `--tcc &lt;binary&gt;`) to build overlay
shards into a DLL, so players need no compiler of their own. Nothing in the
runtime links against libtcc, so this is aggregation with a separate program
rather than LGPL linkage.
The runtime expects an end-user bundle at
`&lt;exe_dir&gt;/overlay_toolchain/{python/, tcc/tcc.exe, compile_overlays.py, …}`
(`runtime/src/main.cpp`). **No script in this repository populates that
directory**, so if release packaging supplies it, that step lives outside this
repo and the TinyCC license notice must be shipped alongside it there.
Developers with `gcc` on `PATH` use the gcc tier instead; the bundled tcc matters
only for end-user release packages (`docs/BUILDING.md`).
## JRickey / gba-recomp — verified-enhancement shadow + screen color science
+24
View File
@@ -1,5 +1,29 @@
# Async background overlay compilation — implementation plan
> **SUPERSEDED IN PART — the sljit tier described below no longer exists.**
>
> This document was written while sljit was the toolchain-free fallback. That
> tier was replaced by tcc in `c2a3f6ad` (2026-06-25) and removed outright in
> `5b7e69b4` (2026-07-15), which deleted `lib/sljit/`, `overlay_sljit.{c,h}` and
> `overlay_compile_worker.{c,h}`.
>
> **The backend tiers today are `static → gcc → tcc`** (`docs/ARCHITECTURE.md`):
> gcc is the development default when it is on `PATH`, and TinyCC is bundled
> beside the shipped executable in `overlay_toolchain/` so players need no
> compiler. Both are external compilers invoked as a subprocess that produce a
> **DLL**; neither is an in-process JIT.
>
> That difference matters when reading this file: every passage below about
> serializing a `.sljit` blob, `overlay_sljit_deserialize`, `try_sljit_region`,
> `register_sljit_candidate`, or a per-fragment in-memory JIT registry describes
> a mechanism that was never how tcc works and no longer exists at all. Those
> passages are **not** transferable to the tcc tier by substituting the name.
> Read them as history. The async worker/file-handoff architecture, the
> gcc-versus-fallback mutual exclusivity, and the CPS work are still accurate.
>
> Current behaviour lives in `runtime/src/overlay_backend.c`,
> `runtime/src/code_provider.c` and `tools/compile_overlays.py`.
Status: **LARGELY ALREADY BUILT + VALIDATED under CPS (2026-06-18).** The "planned" feature below
turned out to mostly exist: `overlay_capture.c` (`overlay_autocapture_tick` — autocapture on
interp pressure, ON whenever the cache is on) + `autocompile.c` (background compile: probes for a
+5 -2
View File
@@ -184,8 +184,11 @@ KEYING for interior shards: key/validate by the COVERED block ranges + a hash of
ONLY the covered instruction bytes (NOT a guessed enclosing host range — that's
what pulled in volatile/data bytes). The loader validates only those covered
ranges. Dispatch priority for a PC: full host > recovered host > interior
fragment > sljit > interp; an interior shard covers only its exact legal entries,
never claims the whole host.
fragment > interp; an interior shard covers only its exact legal entries,
never claims the whole host. (This chain listed an `sljit` tier between the
interior fragment and the interpreter when written; the sljit tier was removed
in `5b7e69b4`, 2026-07-15. Shards are produced by gcc or the bundled tcc today,
and both land in the host/fragment classes above.)
PHASES: P1 separate the build classes (stop poisoning — REGION_TRUSTED only emits
the all-roots DLL). P2 interior-entry fragment compiler (do this FIRST for the FMV
+4
View File
@@ -1269,6 +1269,10 @@ it decelerates and is unrelated to the (now-fixed) host-stack leak. Run gameplay
### 25.5 Stage 3 — overlay (sljit + gcc-DLL) CPS parity (2026-06-18)
> Historical session log. The sljit tier was removed in `5b7e69b4` (2026-07-15);
> the gcc-DLL findings below remain current, and the toolchain-free role sljit
> filled is now the bundled tcc tier. A gcc miss falls to tcc, then interp.
**sljit JIT (overlay_sljit.c), scoped:** `jr $ra` publishes `cpu->pc`; call-containing
fragments **decline to the interp** (CPS-safe, depth-1); `jr rX` already tail-transferred.
So only leaf shards JIT — `jr` covered, calls interp. Validated: boots, leak flat, shards
+5
View File
@@ -5,6 +5,11 @@ exact external state that was inspected, separates reusable framework work from
title-specific work, and prevents an old fork from silently replacing newer
framework fixes.
> Note: entries evaluated on or before 2026-07-13 predate the removal of the
> sljit Tier-2 backend (`5b7e69b4`, 2026-07-15). Where an entry describes an
> external branch touching "SLJIT", that surface no longer exists here; the
> equivalent path today is the bundled tcc tier plus the dirty interpreter.
## Audit baseline and rules
- Evaluated through `2026-07-13T21:25:28-07:00` (America/Los_Angeles), against `mstan/psxrecomp`
+4 -1
View File
@@ -98,7 +98,10 @@ unlikely; assert/log the per-function window count at build time.
already forwards `--ws-config` (game.toml), so `auto_backdrop` reaches it. Bump
`PSX_OVERLAY_CODEGEN_VER` and rebuild the overlay cache (gcc/<arch-abi>/cgN).
- Backends: a backdrop generator may run as a gcc-cache DLL (pass handles it),
a sljit shard, or interpreted. For full coverage like `auto_screen_x`, the
a sljit shard, or interpreted. [sljit was removed 2026-07-15 (`5b7e69b4`); the
toolchain-free tier is now bundled tcc, which also produces a DLL — so only the
DLL and interpreted cases exist today. See `runtime/src/overlay_backend.c`.]
For full coverage like `auto_screen_x`, the
sljit emitter + dirty_ram_interp need the same force-start-0 / widen-end at the
detected sites. Options: (a) export the detected site map to the runtime so
sljit/interp can apply it, or (b) re-derive the signature in a sljit/interp
+1 -1
View File
@@ -116,7 +116,7 @@ transfer transient state that a cycle-exact lockstep needs.
precise-slice / dirty-interp resume state if it affects delivery.
**Exclude from the hash (host-only, would cause false divergences):** CPUState function
pointers + `sljit_helpers`, jmpbufs, fiber handles, malloc addresses, file handles,
pointers, jmpbufs, fiber handles, malloc addresses, file handles,
struct padding, uninitialized bytes. Serialize little-endian explicitly; serialize
arrays in fixed order; serialize any event/deadline queue **sorted** by a deterministic
key (guest_cycle, priority, type, seq).
+5 -2
View File
@@ -38,5 +38,8 @@ behind a specific subsystem.
- [`BACKDROP_PRELOAD.md`](BACKDROP_PRELOAD.md) — widescreen backdrop preload.
## Overlay cache internals
- [`SLJIT_PERSIST_CACHE.md`](SLJIT_PERSIST_CACHE.md) — persisted overlay-shard
cache design.
- `SLJIT_PERSIST_CACHE.md` — **removed.** It described the persisted shard cache
of the sljit Tier-2 backend, which was deleted in `5b7e69b4` (2026-07-15).
For the current model see `runtime/src/overlay_loader.c` (per-entry candidate
validity), `tools/compile_overlays.py` (shard formation and the `.ranges`
manifest), and [`../COMPILING_OVERLAYS.md`](../COMPILING_OVERLAYS.md).
-163
View File
@@ -1,163 +0,0 @@
# Persisted sljit shard cache — design + work to be done
> **STATUS (implemented):** Stage 1 (position-independent shards via the
> cpu-relative `CPUState.sljit_helpers` table) and Stage 2 (serialize on JIT →
> `<cache>/<game>/sljit/<arch-abi>/cg<N>/<entry8>_<crc8>.sljit`, reload + regenerate
> at init, re-validate through the existing dispatch crc + diff gate) are both
> done and validated: a clean two-launch test reloaded 12 shards from disk and ran
> them at a fresh ASLR base with no stale-pointer crash, gameplay rendering
> correctly. KEY GOTCHA found in testing: `sljit_serialize_compiler` must run
> BEFORE `sljit_generate_code` (generate finalizes the compiler → `compiler->error
> = SLJIT_ERR_COMPILED` → serialize then returns NULL), and serialize with
> `options = 0` (NOT `IGNORE_DEBUG`) so an argument-checks build can deserialize.
> Persist timing is write-on-generate (negligible cost, crash-safe). The notes
> below are the original design.
Branch: `feat/sljit-persist-cache` (off `master` @ 0c1f005). Deferred from the
2026-06-15 sljit-default-on session because it has an **ASLR prerequisite** that
makes it a focused feature, not a quick add. In-session sljit re-JIT already
works (sljit is default-on, validated, gcc > sljit > interp); this feature only
adds **cross-session persistence** so subsequent launches reuse shards instead of
re-JITing the tail.
## Goal
> "sljit is supposed to have its own cache too — separate from gcc, lower
> priority. Load into memory in subsequent sessions any sljit shard that doesn't
> have a higher-priority gcc DLL." — user, 2026-06-15
Persist validated sljit shards to disk; reload them at startup; use a reloaded
shard only where no higher-priority gcc DLL covers the same function. Priority
stays **gcc > sljit > interp**.
## The blocker (why v1 punted, why Stage 1 exists)
`sljit_serialize_compiler()` serializes the LIR **verbatim, including immediate
operands**. Per sljitLir.h it restores jump *targets*, but NOT arbitrary
`MOV reg, IMM` operands. Our shards bake **absolute host function pointers** as
immediates, so a shard serialized in one process and reloaded in another calls
**stale addresses** (the runtime loads at a different base each launch — ASLR)
→ crash. So persistence is impossible until shards are position-independent
w.r.t. host pointers.
### The exact hazard sites (runtime/src/overlay_sljit.c, as of 0c1f005)
Three sites bake **four** distinct helper pointers via `SLJIT_IMM (uintptr_t)fn`:
| Site (fn) | line | helper(s) baked |
|---|---|---|
| `emit_helper2` | ~349 | `psx_sljit_memx`, `psx_sljit_cop2` (LWL/LWR/SWL/SWR, COP2/GTE, LWC2/SWC2) |
| `emit_ws_cull` | ~365 | `psx_ws_cull_sltiu` (widescreen render-funnel cull widen) |
| `PEND_CALL` | ~921 | `psx_sljit_call` (jal/jalr call contract) |
The load/store icalls (~320/337) are **already reload-safe** — they load the fn
from `SLJIT_MEM1(R_CPU), offsetof(CPUState, read_word/write_word/...)`, i.e.
cpu-relative. That is the exact pattern Stage 1 copies for the helpers.
## Stage 1 — position-independent shards (PREREQUISITE, correctness-critical)
Route every host-helper call through a **cpu-relative helper-pointer table**
instead of a baked immediate, so the serialized LIR contains zero absolute host
addresses.
1. Add a fixed-order helper table reachable from `CPUState` (runtime/include/
cpu_state.h). Either a `const void *const *sljit_helpers;` pointer field
(append at END so existing gcc-DLL field offsets are unchanged — gcc codegen
is unaffected; gcc DLLs resolve `psx_ws_cull_sltiu` via the loader's import
fixups, not baked pointers, so ONLY sljit needs this) or an inline array.
2. Define a stable enum order, e.g.
`enum { SLJIT_HLP_MEMX, SLJIT_HLP_COP2, SLJIT_HLP_WS_CULL, SLJIT_HLP_CALL, SLJIT_HLP_COUNT };`
The order is part of the on-disk format — never reorder; only append. Fold
the order/contents into a serialization version (see Stage 2 / cg<N>).
3. Populate the table once at startup (where CPUState is initialized) with the
real addresses.
4. Change the 3 emit sites: instead of `MOV R2, IMM (uintptr_t)fn`, emit
`MOV Rtab, MEM1(R_CPU)[offsetof(CPUState, sljit_helpers)]` then
`MOV Rfn, MEM1(Rtab)[k*sizeof(void*)]` then icall `Rfn`. (`emit_helper2`
takes a helper-enum index instead of a raw `fn`.) Mind scratch-register
usage — the table load needs a scratch that doesn't clobber the call args
(R0..R3 are args; use R4/a free scratch for the table+fn, as PEND_CALL
already uses R4 for the fn ptr).
5. **This is a strict win on its own** and is independently testable: with
Stage 1 alone (no disk cache yet), in-session JIT must still pass the existing
live playtest (no regression: same shard behaviour, just one extra indirection
per helper call). Verify before building Stage 2.
Risk: a missed/mis-ordered helper pointer = every shard that uses it crashes.
Test by exercising each helper class (a GTE-heavy scene for COP2, an unaligned
load for memx, a jal-heavy overlay for call, a 16:9 overlay-funnel scene for
ws_cull).
## Stage 2 — the disk cache (serialize / reload / re-validate)
Mirror the gcc cache layout + versioning exactly.
**Path:** `<cache_dir>/<game_id>/sljit/<arch-abi>/cg<N>/<phys8>_<crc8>.sljit`
(the `sljit/<arch-abi>/` namespace is already RESERVED in overlay_loader.c's
scan; `cg<N>` = `PSX_OVERLAY_CODEGEN_VER` from overlay_api.h, same as gcc — so a
codegen/emitter change auto-invalidates stale sljit blobs too. Also bump a
dedicated serialization tag if the helper-table order changes.)
**Serialize on compile** (overlay_sljit.c, after `sljit_generate_code` succeeds):
- `buf = sljit_serialize_compiler(C, SLJIT_SERIALIZE_IGNORE_DEBUG, &sz)` BEFORE
`sljit_free_compiler(C)` (serialize doesn't modify the compiler, so order vs
generate_code is flexible; keep both before free).
- Compute `crc_code` over live RAM `[entry_phys, entry_phys+frag_words*4)` with
the same `crc32_compute` the loader uses (so the filename CRC matches what
dispatch re-hashes).
- Write a small header + the serialized buffer:
`{ magic, format_ver, helper_order_ver, entry_phys, code_lo, code_len,
insns, crc_code, blob_size }` then the `sljit_uw` blob.
- Gate writing on a config/env (don't write during shadow/diff runs; only persist
shards that have been VALIDATED — or persist eagerly and rely on reload-time
re-validation, see below. Eager is simpler and safe given re-validation.).
**Reload at init** (overlay_loader.c, extend `scan_cache_dir`):
- Scan `sljit/<arch-abi>/cg<N>/*.sljit`. For each: read header, check
magic/format/helper_order, `C = sljit_deserialize_compiler(blob, sz, 0, NULL)`,
`code = sljit_generate_code(C, 0, NULL)`, `sljit_free_compiler(C)`.
- Register as a sljit candidate (`dll = -1`) over `[code_lo, code_len)` with
`crc_code` — exactly like `register_sljit_candidate`, just sourced from disk.
**Re-validation (the safety model — no new trust surface):** a reloaded shard
is NOT trusted blindly. The existing dispatch path already (a) re-hashes live RAM
vs `crc_code` every dispatch (a shard for a different scene/variant simply never
matches → never runs), and (b) routes `dll<0` shards through the same-state diff
gate (`want_diff = s_sljit_live && c->dll < 0`) until the verify budget, with
device-touch/diverging shards pinned to interp. So a persisted shard re-earns
trust each session exactly like a fresh JIT. This is what makes Stage 2 safe.
**Priority gcc > sljit > interp:** unchanged — gcc DLLs register as `dll>=0` and
the existing obsolete logic (`sljit shard obsoleted by gcc DLL`) retires a sljit
shard when a gcc DLL covers the same live function. Ensure load order or the
dispatch chain keeps gcc ahead (gcc DLLs loaded lazily on dispatch miss via
`try_load_region`; persisted sljit shards loaded at init — confirm a gcc DLL that
arrives later still supersedes the pre-registered sljit shard).
## Verification plan
- Stage 1: in-session playtest unchanged (no crash, same diff stats) — gates
Stage 2.
- Stage 2: launch once (populate the sljit cache), relaunch, confirm shards load
from disk (`sljit_status` shows reloaded count), the game runs (no stale-pointer
crash — proves Stage 1), gcc-covered functions still use gcc, and the
native:interp ratio matches the in-session-JIT baseline. Force a helper-class
per shard (GTE/memx/call/ws_cull) on reload to prove the indirection.
- 4:3 / widescreen-off unaffected throughout.
## Files
- `runtime/include/cpu_state.h` — helper table field (Stage 1).
- `runtime/src/overlay_sljit.c` — helper-table emit (Stage 1) + serialize (Stage 2).
- `runtime/src/overlay_loader.c` — reload + register at init (Stage 2); the
`sljit/<arch-abi>/` scan is already reserved here.
- `runtime/include/overlay_api.h` — `PSX_OVERLAY_CODEGEN_VER` (+ a helper-order
tag) gates the sljit blob namespace.
- startup (CPUState init site) — populate the helper table (Stage 1).
## Context (what already shipped on master, 0c1f005)
sljit is merged + default-on + validated; one shared `psx_ws_cull_sltiu` helper
is routed by gcc/sljit/interp (overlay 3D widens in 16:9 on every path); the gcc
cache is versioned `gcc/<arch-abi>/cg<N>/`. See `SLJIT.md`, `WIDESCREEN.md`, and
memory `sljit_backend_state` / `native_wide_fov_autocull`.