mirror/psxrecomp
1
0
Fork 0
mirror of https://github.com/RetroPortingToolKit/psxrecomp synced 2026-10-26 02:11:36 -04:00
Code Issues Packages Projects Releases Wiki Activity

overlay: off-thread sljit compile worker + observability; disable sljit tier by default (emitter bug)

Browse Source 
Off-thread sljit compile worker (overlay_compile_worker.c/.h): on a dispatch
gap-fill miss the dispatch thread snapshots the region bytes and ENQUEUES; a
background worker JITs from the snapshot (never live RAM) and publishes the shard
to the on-disk cache via an atomic temp+rename; dispatch picks it up on a later
miss via an idempotent rescan, so the candidate table stays single-threaded. This
removes the synchronous JIT frame hitch. Validated on Tomba + MMX6: compiles
off-thread (one MMX6 compile was 26ms, paid entirely off the dispatch thread with
no frame-pacing spike), shards persist + reload, queue never backs up.

Observability (ring-buffer doctrine): always-on recent-compile ring + per-compile
timing in the worker, the resolved on-disk cache dir, and a tier-enabled flag, all
surfaced via a new sljit_async TCP debug command (documented in TCP_COMMANDS.md).

DISABLE the sljit tier by default (g_sljit_tier_enabled = 0): the sljit EMITTER
mistranslates some MMX6 overlay instruction(s) -- sljit-only save-load softlocks/
loopbacks while PURE INTERP works (discriminator), and it reproduces with the
worker OFF in the synchronous path (PSX_SLJIT_SYNC=1), so it is the emitter, not
the async worker. Until the emitter bug is found + fixed: no shard READ
(scan_sljit_cache_dir skipped), no GENERATE (worker not started + sljit_live
forced 0 + dispatch gap-fill gated), every overlay miss falls to the interpreter.
gcc (DLL cache + autocompile) and overlay capture are untouched, so priority is
gcc > interp and dispatch misses are still recorded for future gcc coverage. gcc
shards are bundled in release zips, so shipped builds are unaffected.
PSX_SLJIT_ENABLE=1 re-enables the tier for emitter-fix testing.

Runtime-only, no regen.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Matthew Stanley
2026-06-25 10:13:53 -07:00
parent c35c04f310
commit 71d6550eef
7 changed files with 468 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TCP_COMMANDS.md
+1
View File
@@ -47,6 +47,7 @@ Columns: **N** = native, **D** = DuckStation oracle.
| `dma_state` | ✓ | ✓ | — | DPCR, DICR, all 7 channel states (madr/bcr/chcr) |
| `event_state` | | ✓ | — | EvCB table summary (stub on DS — events are BIOS-level) |
| `overlay_state` | | ✓ | — | Current overlay info |
| `sljit_async` | ✓ | | — | Off-thread sljit compile-worker telemetry: `worker_running`, `async_on`, always-on counters (`enqueued`/`compiled`/`failed`/`queued_now`), `last_compile_ms`/`max_compile_ms` (off-thread compile time), resolved `cache_dir`, and a `recent[]` ring of the last 64 compiles (per-shard `phys`/`crc`/`code_len`/`ms`/`ok`). Query it to prove the worker compiles off the dispatch thread without a frame hitch |
| `cdrom_sector_dump` | ✓ | | `offset`, `len` | Dump bytes from the last CD-ROM sector observed by the controller, including LBA/mode metadata |
| `cdrom_sector_history` | ✓ | | `count`, optional `lba` | Dump newest CD-ROM sector history entries, including raw XA subheader fields, CPU/audio delivery flags, and the first 128 bytes |
| `cdrom_sector_history_clear` | ✓ | | — | Reset the CD-ROM sector history ring |
runtime/include/overlay_compile_worker.h
+75
View File
@@ -0,0 +1,75 @@
#ifndef PSX_OVERLAY_COMPILE_WORKER_H
#define PSX_OVERLAY_COMPILE_WORKER_H
#include <stdint.h>
#ifdef __cplusplus
extern "C" {
#endif
/*
* Off-main-thread sljit compile worker.
*
* The dispatch (guest-fiber) thread, on an sljit gap-fill miss, snapshots the
* region bytes and ENQUEUES here instead of JIT'ing inline (which used to spike
* the frame). A single background worker pops the queue, JITs the fragment FROM
* THE SNAPSHOT (never live guest RAM), and writes the serialized shard into the
* on-disk cache via overlay_loader_async_publish(). The dispatch thread picks
* the shard up on a later miss (cache-dirty -> idempotent rescan -> register),
* so the candidate table stays single-threaded: the worker NEVER touches it.
*
* Safety invariants (see docs/ASYNC_OVERLAY_COMPILE.md §3/§5):
* - candidate tables / sljit registry: dispatch thread only.
* - guest RAM: worker reads only the enqueue-time snapshot copy.
* - the sljit emitter's file-static scratch is safe because the worker is the
* SOLE runtime caller of the compiler (dispatch enqueues; selftest is init-time).
*/
/* Start/stop the worker. Idempotent. stop() signals + joins. */
void overlay_compile_worker_start(void);
void overlay_compile_worker_stop(void);
/* Enqueue a region for background JIT. `vaddr` is the VIRTUAL entry (carries
* KSEG bits the guest uses for return_pc/jal targets). `src` points at the
* live region bytes [phys .. phys+snap_len); enqueue COPIES them, so the caller
* keeps ownership of `src`. `crc` is a content hash of the snapshot window,
* used only for (phys,crc) dedup. Deduped: a QUEUED/COMPILING/DONE (phys,crc)
* is ignored. Cheap no-op if the worker isn't running. */
void overlay_compile_worker_enqueue(uint32_t vaddr, uint32_t phys,
const uint8_t *src, uint32_t snap_len,
uint32_t crc);
/* Telemetry for the debug server (any out-ptr may be NULL). */
void overlay_compile_worker_stats(uint64_t *enqueued, uint64_t *compiled,
uint64_t *failed, uint32_t *queued_now);
/* 1 if the worker thread is currently running. */
int overlay_compile_worker_running(void);
/* Last / worst single-compile wall time (ms, SDL_GetTicks resolution). The whole
* point of the worker is that this time is paid OFF the dispatch thread, so a
* nonzero max here alongside a flat dispatch-thread ce_profile is the proof. */
void overlay_compile_worker_timing(uint32_t *last_ms, uint32_t *max_ms);
/* Always-on ring of the most-recent compile results (one entry PER shard the
* worker has produced or rejected). Query this for the window of interest —
* never arm-then-capture. */
typedef struct {
uint32_t vaddr; /* virtual entry enqueued */
uint32_t phys; /* physical entry (dedup key) */
uint32_t crc; /* content crc of the enqueue-time snapshot */
uint32_t code_lo; /* compiled range start (phys) */
uint32_t code_len; /* compiled range length in bytes (0 if fail) */
uint32_t compile_ms; /* wall time for this compile (off-thread) */
uint8_t ok; /* 1 = published a shard, 0 = emitter declined */
} OverlayCompileEvent;
/* Copy up to `cap` most-recent events into `out` in chronological order (oldest
* first), returning the count copied. Thread-safe. */
int overlay_compile_worker_recent(OverlayCompileEvent *out, int cap);
#ifdef __cplusplus
}
#endif
#endif /* PSX_OVERLAY_COMPILE_WORKER_H */
runtime/runtime.cmake
+1
View File
@@ -153,6 +153,7 @@ set(PSXRECOMP_RUNTIME_SOURCES
${PSXRECOMP_ROOT}/runtime/src/card_data_writes.c
${PSXRECOMP_ROOT}/runtime/src/overlay_capture.c
${PSXRECOMP_ROOT}/runtime/src/overlay_loader.c
${PSXRECOMP_ROOT}/runtime/src/overlay_compile_worker.c
${PSXRECOMP_ROOT}/runtime/src/overlay_sljit.c
${PSXRECOMP_ROOT}/runtime/src/autocompile.c
${PSXRECOMP_ROOT}/runtime/src/code_provider.c
runtime/src/debug_server.c
+52
View File
@@ -21,6 +21,7 @@
#include "overlay_capture.h"
#include "code_provider.h"
#include "overlay_sljit.h"
#include "overlay_compile_worker.h"
#include "cpu_state.h"
#include "dma.h"
#include "gpu.h"
@@ -8539,6 +8540,56 @@ static void handle_sljit_status(int id, const char *json)
overlay_loader_sljit_persist_dbg());
}
/* sljit_async: off-thread compile-worker telemetry. Reports the always-on
* counters (enqueued/compiled/failed/queued_now), last/worst off-thread compile
* time, the resolved on-disk shard dir, and the recent-compile ring. This is the
* ring-buffer proof that the worker actually does work: a rising `compiled` with
* a populated `recent[]` (each with a phys/crc/code_len/ms) while the dispatch
* thread's ce_profile stays flat == JIT moved off the hot path. */
static void handle_sljit_async(int id, const char *json)
{
extern void overlay_loader_sljit_cache_dir(char *buf, int n);
extern int overlay_loader_sljit_async_on(void);
extern int overlay_loader_sljit_tier_enabled(void);
(void)json;
uint64_t enq = 0, done = 0, failed = 0;
uint32_t queued = 0, last_ms = 0, max_ms = 0;
overlay_compile_worker_stats(&enq, &done, &failed, &queued);
overlay_compile_worker_timing(&last_ms, &max_ms);
char dir[768] = {0};
overlay_loader_sljit_cache_dir(dir, (int)sizeof(dir));
/* escape backslashes in the Windows path for valid JSON */
char dir_esc[1536]; int di = 0;
for (int i = 0; dir[i] && di < (int)sizeof(dir_esc) - 2; i++) {
if (dir[i] == '\\') dir_esc[di++] = '\\';
dir_esc[di++] = dir[i];
}
dir_esc[di] = '\0';
OverlayCompileEvent evs[64];
int ne = overlay_compile_worker_recent(evs, 64);
char out[16384];
int n = snprintf(out, sizeof(out),
"{\"id\":%d,\"ok\":true,\"tier_enabled\":%d,\"worker_running\":%d,\"async_on\":%d,"
"\"enqueued\":%llu,\"compiled\":%llu,\"failed\":%llu,\"queued_now\":%u,"
"\"last_compile_ms\":%u,\"max_compile_ms\":%u,\"cache_dir\":\"%s\","
"\"recent\":[",
id, overlay_loader_sljit_tier_enabled(),
overlay_compile_worker_running(), overlay_loader_sljit_async_on(),
(unsigned long long)enq, (unsigned long long)done,
(unsigned long long)failed, queued, last_ms, max_ms, dir_esc);
for (int i = 0; i < ne && n < (int)sizeof(out) - 160; i++) {
n += snprintf(out + n, sizeof(out) - n,
"%s{\"vaddr\":\"0x%08X\",\"phys\":\"0x%08X\",\"crc\":\"0x%08X\","
"\"code_lo\":\"0x%08X\",\"code_len\":%u,\"ms\":%u,\"ok\":%d}",
i ? "," : "", evs[i].vaddr, evs[i].phys, evs[i].crc,
evs[i].code_lo, evs[i].code_len, evs[i].compile_ms, evs[i].ok);
}
n += snprintf(out + n, sizeof(out) - n, "]}\n");
send_fmt("%s", out);
}
/* sljit_try <hex_addr>: force a one-shot JIT of the leaf function at a phys
* address from live RAM (bypasses the diff gate — a probe) and report whether
* the emitter accepted it. Lets the leaf emitter be exercised against a known
@@ -9578,6 +9629,7 @@ static const CmdEntry s_commands[] = {
{ "turbo_loads", handle_turbo_loads },
{ "autocompile_status", handle_autocompile_status },
{ "sljit_status", handle_sljit_status },
{ "sljit_async", handle_sljit_async },
{ "sljit_try", handle_sljit_try },
{ "autocompile_run", handle_autocompile_run },
{ "overlay_rescan", handle_overlay_rescan },
runtime/src/main.cpp
+2
View File
@@ -580,7 +580,9 @@ static std::filesystem::path default_memcard_dir(const char* argv0) {
static void close_controller(void);
extern "C" void overlay_compile_worker_stop(void);
static void shutdown_runtime(void) {
overlay_compile_worker_stop(); /* signal + join the off-thread sljit worker */
memcard_flush_all();
overlay_capture_write_json();
if (sdl_audio_device) {
runtime/src/overlay_compile_worker.c
+204
View File
@@ -0,0 +1,204 @@
#include "overlay_compile_worker.h"
#include "code_provider.h"
#include "overlay_sljit.h"
#include "crc32.h"
#include <SDL.h>
#include <stdlib.h>
#include <string.h>
/* Dispatch-side publish (overlay_loader.c): atomically write the serialized
* shard into the on-disk cache + flag the loader to rescan-on-miss. It touches
* only the filesystem and read-only-after-init cache-path globals, so it is safe
* to call from this worker thread (it never touches the candidate table). */
extern void overlay_loader_async_publish(uint32_t entry_phys, uint32_t lo,
uint32_t len, uint32_t crc,
const void *blob, unsigned long blob_size);
#define WQ_CAP 256 /* bounded work queue */
#define STATE_CAP 4096 /* per-(phys,crc) dedup map */
#define SNAP_MAX (8u * 1024u) /* max window we JIT (SLJIT_MAX_FRAG_INSNS*4) */
typedef struct {
uint32_t vaddr, phys, crc, snap_len;
uint8_t *snap; /* worker-owned copy of the region bytes */
} WorkItem;
enum { ST_QUEUED = 1, ST_COMPILING = 2, ST_DONE = 3, ST_FAILED = 4 };
typedef struct { uint32_t phys, crc; uint8_t state; } StateEnt;
static SDL_Thread *s_thread;
static SDL_mutex *s_mtx;
static SDL_cond *s_cv;
static int s_running;
static WorkItem s_q[WQ_CAP];
static int s_q_head, s_q_tail, s_q_n;
static StateEnt s_state[STATE_CAP];
static int s_state_n;
static uint64_t s_enq, s_done, s_failed;
/* Always-on recent-compile ring (CLAUDE.md ring-buffer doctrine): every compile,
* success or decline, lands here so a probe can QUERY what the worker did in the
* window of interest rather than time-and-capture. Guarded by s_mtx. */
#define EVT_CAP 64
static OverlayCompileEvent s_evt[EVT_CAP];
static uint64_t s_evt_seq; /* total events recorded ever */
static uint32_t s_last_ms, s_max_ms;
/* dedup map — all accesses hold s_mtx. */
static StateEnt *state_find(uint32_t phys, uint32_t crc) {
for (int i = 0; i < s_state_n; i++)
if (s_state[i].phys == phys && s_state[i].crc == crc) return &s_state[i];
return NULL;
}
static StateEnt *state_add(uint32_t phys, uint32_t crc, uint8_t st) {
if (s_state_n >= STATE_CAP) {
/* Full: evict the first finished entry (the on-disk cache is the source
* of truth, so a later re-miss can re-enqueue if ever needed). COMPILING/
* QUEUED entries are never evicted (in flight). */
for (int i = 0; i < s_state_n; i++)
if (s_state[i].state == ST_DONE || s_state[i].state == ST_FAILED) {
s_state[i] = s_state[--s_state_n];
break;
}
if (s_state_n >= STATE_CAP) return NULL;
}
StateEnt *e = &s_state[s_state_n++];
e->phys = phys; e->crc = crc; e->state = st;
return e;
}
static int worker_main(void *unused) {
(void)unused;
const CodeProvider *cp = code_provider_sljit();
SDL_LockMutex(s_mtx);
for (;;) {
while (s_running && s_q_n == 0) SDL_CondWait(s_cv, s_mtx);
if (!s_running && s_q_n == 0) break;
WorkItem it = s_q[s_q_head];
s_q_head = (s_q_head + 1) % WQ_CAP;
s_q_n--;
{ StateEnt *e = state_find(it.phys, it.crc); if (e) e->state = ST_COMPILING; }
SDL_UnlockMutex(s_mtx);
/* Compile off-thread FROM THE SNAPSHOT (base = phys, so the fragment's
* byte offset into the buffer is entry_phys - phys). Never live RAM. */
int ok = 0;
uint32_t ev_lo = 0, ev_len = 0;
Uint32 t0 = SDL_GetTicks();
if (cp && cp->compile_fragment && it.snap) {
CompiledFragment frag;
memset(&frag, 0, sizeof frag);
cp->compile_fragment(it.vaddr, it.snap, it.snap_len, it.phys, &frag);
if (frag.fn && frag.serialized && frag.serialized_size && frag.code_len &&
frag.code_lo >= it.phys &&
(frag.code_lo - it.phys) + frag.code_len <= it.snap_len) {
/* crc over the EXACT compiled range within the snapshot — this is
* what dispatch re-hashes live RAM against when it loads the shard
* (matches persist_sljit_shard's crc-over-code-range). */
uint32_t off = frag.code_lo - it.phys;
uint32_t crc = crc32_update(0xFFFFFFFFu, it.snap + off, frag.code_len)
^ 0xFFFFFFFFu;
overlay_loader_async_publish(it.phys, frag.code_lo, frag.code_len,
crc, frag.serialized, frag.serialized_size);
ev_lo = frag.code_lo; ev_len = frag.code_len;
ok = 1;
}
if (frag.serialized) overlay_sljit_free_serialized(frag.serialized);
}
uint32_t dur = (uint32_t)(SDL_GetTicks() - t0);
free(it.snap);
SDL_LockMutex(s_mtx);
{ StateEnt *e = state_find(it.phys, it.crc); if (e) e->state = ok ? ST_DONE : ST_FAILED; }
if (ok) s_done++; else s_failed++;
/* record into the recent-compile ring */
OverlayCompileEvent *ev = &s_evt[s_evt_seq % EVT_CAP];
ev->vaddr = it.vaddr; ev->phys = it.phys; ev->crc = it.crc;
ev->code_lo = ev_lo; ev->code_len = ev_len;
ev->compile_ms = dur; ev->ok = (uint8_t)ok;
s_evt_seq++;
s_last_ms = dur; if (dur > s_max_ms) s_max_ms = dur;
}
SDL_UnlockMutex(s_mtx);
return 0;
}
void overlay_compile_worker_start(void) {
if (s_thread) return;
if (!s_mtx) s_mtx = SDL_CreateMutex();
if (!s_cv) s_cv = SDL_CreateCond();
if (!s_mtx || !s_cv) return;
s_running = 1;
s_thread = SDL_CreateThread(worker_main, "psx-sljit-compile", NULL);
if (!s_thread) s_running = 0;
}
void overlay_compile_worker_stop(void) {
if (!s_thread) return;
SDL_LockMutex(s_mtx);
s_running = 0;
SDL_CondSignal(s_cv);
SDL_UnlockMutex(s_mtx);
SDL_WaitThread(s_thread, NULL);
s_thread = NULL;
SDL_LockMutex(s_mtx);
while (s_q_n) { free(s_q[s_q_head].snap); s_q_head = (s_q_head + 1) % WQ_CAP; s_q_n--; }
SDL_UnlockMutex(s_mtx);
}
void overlay_compile_worker_enqueue(uint32_t vaddr, uint32_t phys,
const uint8_t *src, uint32_t snap_len,
uint32_t crc) {
if (!s_thread || !src || snap_len == 0) return;
if (snap_len > SNAP_MAX) snap_len = SNAP_MAX;
SDL_LockMutex(s_mtx);
if (state_find(phys, crc) || s_q_n >= WQ_CAP) { SDL_UnlockMutex(s_mtx); return; }
uint8_t *copy = (uint8_t *)malloc(snap_len);
if (!copy) { SDL_UnlockMutex(s_mtx); return; }
memcpy(copy, src, snap_len);
if (!state_add(phys, crc, ST_QUEUED)) { free(copy); SDL_UnlockMutex(s_mtx); return; }
WorkItem *it = &s_q[s_q_tail];
it->vaddr = vaddr; it->phys = phys; it->crc = crc; it->snap_len = snap_len; it->snap = copy;
s_q_tail = (s_q_tail + 1) % WQ_CAP;
s_q_n++;
s_enq++;
SDL_CondSignal(s_cv);
SDL_UnlockMutex(s_mtx);
}
void overlay_compile_worker_stats(uint64_t *enqueued, uint64_t *compiled,
uint64_t *failed, uint32_t *queued_now) {
if (s_mtx) SDL_LockMutex(s_mtx);
if (enqueued) *enqueued = s_enq;
if (compiled) *compiled = s_done;
if (failed) *failed = s_failed;
if (queued_now) *queued_now = (uint32_t)s_q_n;
if (s_mtx) SDL_UnlockMutex(s_mtx);
}
int overlay_compile_worker_running(void) { return s_thread != NULL; }
void overlay_compile_worker_timing(uint32_t *last_ms, uint32_t *max_ms) {
if (s_mtx) SDL_LockMutex(s_mtx);
if (last_ms) *last_ms = s_last_ms;
if (max_ms) *max_ms = s_max_ms;
if (s_mtx) SDL_UnlockMutex(s_mtx);
}
int overlay_compile_worker_recent(OverlayCompileEvent *out, int cap) {
if (!out || cap <= 0) return 0;
if (s_mtx) SDL_LockMutex(s_mtx);
uint64_t total = s_evt_seq;
int avail = (total < (uint64_t)EVT_CAP) ? (int)total : EVT_CAP;
if (avail > cap) avail = cap;
/* chronological (oldest first): the oldest of the kept window is at
* (seq - avail). */
uint64_t start = total - (uint64_t)avail;
for (int i = 0; i < avail; i++)
out[i] = s_evt[(start + (uint64_t)i) % EVT_CAP];
if (s_mtx) SDL_UnlockMutex(s_mtx);
return avail;
}
runtime/src/overlay_loader.c
+133 -10
View File
@@ -389,6 +389,27 @@ static void sljit_mark_tried(uint32_t phys) {
static void persist_sljit_shard(uint32_t entry_phys, uint32_t lo, uint32_t len,
const void *blob, unsigned long blob_size);
#include "overlay_compile_worker.h"
/* Off-main-thread sljit compile (docs/ASYNC_OVERLAY_COMPILE.md). The background
* worker JITs from a snapshot and PUBLISHES the shard to the on-disk cache +
* sets s_async_cache_dirty; the DISPATCH thread rescans-and-registers on its
* next miss (candidate table stays single-threaded). PSX_SLJIT_SYNC=1 forces
* the legacy synchronous JIT-on-miss path. */
static volatile LONG s_async_cache_dirty = 0;
static int g_sljit_async = 1;
/* MASTER SAFETY GATE (2026-06-25): the sljit EMITTER mistranslates some overlay
* instruction(s). Proven on MMX6: sljit-only save-load softlocks/loopbacks, PURE
* INTERP works (discriminator), and it reproduces in the SYNCHRONOUS path — so it
* is the emitter, not the async worker. Until the emitter bug is found + fixed the
* whole sljit tier is DISABLED by default: no shard READ (scan_sljit_cache_dir),
* no shard GENERATE (worker / sync try / live gap-fill), every overlay miss falls
* to the interpreter. gcc (DLL cache + autocompile) and overlay capture are
* untouched, so priority is gcc > interp and dispatch misses are still recorded
* for future gcc coverage. PSX_SLJIT_ENABLE=1 re-enables it for emitter-fix
* testing. See memory mmx6_sljit_shard_malformed. */
static int g_sljit_tier_enabled = 0;
/* Attempt an in-process JIT of the leaf function at `phys` from live RAM, and
* register the shard as a candidate on success. Gated by the caller to the
* validation configuration only (backend==sljit + diff_mode); see the dispatch
@@ -570,6 +591,14 @@ static void sljit_cache_dir(char *buf, size_t n) {
(unsigned)PSX_OVERLAY_CODEGEN_HASH);
}
/* Public: the resolved on-disk sljit shard dir, so the sljit_async debug command
* can report exactly where the worker writes (no guessing about cache location). */
void overlay_loader_sljit_cache_dir(char *buf, int n) {
if (buf && n > 0) sljit_cache_dir(buf, (size_t)n);
}
int overlay_loader_sljit_async_on(void) { return g_sljit_async; }
int overlay_loader_sljit_tier_enabled(void) { return g_sljit_tier_enabled; }
#ifdef _WIN32
static void sljit_mkdir_p(const char *path) {
char tmp[768];
@@ -617,6 +646,60 @@ static void persist_sljit_shard(uint32_t entry_phys, uint32_t lo, uint32_t len,
#endif
}
/* Worker-thread publish (overlay_compile_worker.c): write a freshly-JIT'd shard
* into the on-disk cache ATOMICALLY (temp + rename, so dispatch's scan never
* reads a partial file) with an EXPLICIT crc (computed by the worker over the
* snapshot's compiled range, NOT live RAM), then flag dispatch to rescan. Only
* the filesystem + read-only-after-init cache-path globals are touched — never
* the candidate table — so it is safe off the dispatch thread. */
void overlay_loader_async_publish(uint32_t entry_phys, uint32_t lo, uint32_t len,
uint32_t crc, const void *blob,
unsigned long blob_size) {
#ifdef _WIN32
if (!s_sljit_persist || !blob || blob_size == 0 || len == 0) return;
char dir[768]; sljit_cache_dir(dir, sizeof(dir));
sljit_mkdir_p(dir);
char path[860], tmp[920];
snprintf(path, sizeof(path), "%s/%08X_%08X.sljit", dir, entry_phys & 0x1FFFFFFFu, crc);
snprintf(tmp, sizeof(tmp), "%s.tmp%lu", path, (unsigned long)GetCurrentThreadId());
FILE *f = fopen(tmp, "wb");
if (!f) return;
SljitBlobHeader hh;
hh.magic = SLJIT_BLOB_MAGIC; hh.format_ver = SLJIT_BLOB_FORMAT_VER;
hh.helper_order_ver = SLJIT_HELPER_ORDER_VER;
hh.entry_phys = entry_phys & 0x1FFFFFFFu; hh.code_lo = lo & 0x1FFFFFFFu;
hh.code_len = len; hh.crc_code = crc; hh.blob_size = (uint32_t)blob_size;
int wok = (fwrite(&hh, sizeof hh, 1, f) == 1) &&
(fwrite(blob, 1, blob_size, f) == blob_size);
fclose(f);
if (!wok) { remove(tmp); return; }
if (!MoveFileExA(tmp, path, MOVEFILE_REPLACE_EXISTING)) { remove(tmp); return; }
s_sljit_persist_writes++; /* telemetry-only; benign cross-thread incr */
InterlockedExchange(&s_async_cache_dirty, 1);
#else
(void)entry_phys; (void)lo; (void)len; (void)crc; (void)blob; (void)blob_size;
#endif
}
/* Idempotency for the on-miss rescan: a dispatch-thread-only set of .sljit
* filenames already deserialized+registered, so re-running scan_sljit_cache_dir
* after a worker publish never double-registers. */
#define MAX_LOADED_BLOBS 8192
static char s_loaded_blobs[MAX_LOADED_BLOBS][24]; /* "XXXXXXXX_XXXXXXXX.sljit" */
static int s_loaded_blob_n;
static int blob_already_loaded(const char *name) {
for (int i = 0; i < s_loaded_blob_n; i++)
if (!strcmp(s_loaded_blobs[i], name)) return 1;
return 0;
}
static void blob_mark_loaded(const char *name) {
if (s_loaded_blob_n < MAX_LOADED_BLOBS) {
strncpy(s_loaded_blobs[s_loaded_blob_n], name, 23);
s_loaded_blobs[s_loaded_blob_n][23] = '\0';
s_loaded_blob_n++;
}
}
/* Reload persisted shards at init: deserialize + regenerate + register. The crc
* comes from the blob header (overlay RAM may not be loaded yet; dispatch
* re-hashes live RAM against it before ever running the shard). */
@@ -630,6 +713,7 @@ static void scan_sljit_cache_dir(void) {
if (fh == INVALID_HANDLE_VALUE) return;
do {
s_reload_seen++;
if (blob_already_loaded(fd.cFileName)) continue; /* idempotent rescan */
char full[900];
snprintf(full, sizeof(full), "%s/%s", dir, fd.cFileName);
FILE *f = fopen(full, "rb");
@@ -650,6 +734,7 @@ static void scan_sljit_cache_dir(void) {
register_sljit_candidate(hd.entry_phys, (OverlayFn)fn,
hd.code_lo, hd.code_len, hd.crc_code);
s_sljit_reloaded++;
blob_mark_loaded(fd.cFileName);
} while (FindNextFileA(fh, &fd));
FindClose(fh);
if (s_sljit_reloaded) loader_log("sljit cache: reloaded %u shard(s)", s_sljit_reloaded);
@@ -861,8 +946,18 @@ void overlay_loader_init(const char *cache_dir, const char *game_id) {
* runs when [runtime] overlay_cache is on), so persist JIT'd shards and reload
* any from a prior session now. Reloaded shards register as dll<0 candidates
* and re-earn trust via the normal per-dispatch crc + diff gate. */
s_sljit_persist = 1;
scan_sljit_cache_dir();
{ const char *e = getenv("PSX_SLJIT_ENABLE"); if (e && *e && *e != '0') g_sljit_tier_enabled = 1; }
if (g_sljit_tier_enabled) {
s_sljit_persist = 1;
scan_sljit_cache_dir();
{ const char *e = getenv("PSX_SLJIT_SYNC"); if (e && *e && *e != '0') g_sljit_async = 0; }
if (g_sljit_async) overlay_compile_worker_start(); /* off-thread JIT */
} else {
/* Tier disabled (default): no read, no generate. persist off so the worker
* publish + persist_sljit_shard are inert even if reached. gcc > interp. */
s_sljit_persist = 0;
loader_log("sljit tier DISABLED (emitter bug; gcc>interp). PSX_SLJIT_ENABLE=1 to re-enable");
}
/* Live-mode policy is applied later via overlay_loader_apply_live_policy(),
* AFTER code_provider_init resolves the backend (the default depends on it). */
s_active = 1;
@@ -893,6 +988,9 @@ void overlay_loader_apply_live_policy(void) {
if (overlay_backend_active() == OVERLAY_BACKEND_GCC) live = 0;
const char *e = getenv("PSX_OVERLAY_SLJIT_LIVE");
if (e && *e) live = (*e != '0') ? 1 : 0;
/* MASTER GATE: the sljit tier is disabled by default (emitter bug) — never go
* live, regardless of backend/env, until PSX_SLJIT_ENABLE=1. */
if (!g_sljit_tier_enabled) live = 0;
s_sljit_live = live;
loader_log("sljit live policy: active_backend=%s sljit_avail=%d env=%s -> live=%d",
overlay_backend_name(overlay_backend_active()),
@@ -1028,22 +1126,47 @@ int overlay_loader_dispatch(CPUState *cpu, uint32_t addr) {
uint32_t phys = addr & 0x1FFFFFFFu;
int head = idx_head(phys);
if (head < 0 && s_active && overlay_cache_window_contains(phys)) {
try_load_region(phys);
head = idx_head(phys);
/* Pick up shards the background compile worker just published (idempotent
* rescan; runs only when a publish has flipped the dirty flag). */
if (g_sljit_tier_enabled && g_sljit_async && InterlockedExchange(&s_async_cache_dirty, 0)) {
scan_sljit_cache_dir();
head = idx_head(phys);
}
if (head < 0) {
try_load_region(phys);
head = idx_head(phys);
}
/* sljit JIT-on-miss (SLJIT.md §7 step 5). Fires when the active backend is
* sljit and EITHER the same-state differential is armed (dev validation
* path — shards created then diffed vs interp) OR live mode is on (the
* production model — shards created and run live, no per-shard diff).
* Never inside a shadow run. */
if (head < 0 && (s_diff_mode || s_sljit_live) && !s_in_shadow &&
if (head < 0 && g_sljit_tier_enabled && (s_diff_mode || s_sljit_live) && !s_in_shadow &&
code_provider_sljit() != NULL) {
/* head < 0 == no gcc DLL (or any candidate) registered for this region
* after the cache-load attempt above => gcc-absent => sljit fills the
* gap (priority gcc > sljit > interp). Fires whenever sljit is
* AVAILABLE, not only when it is the exclusive backend, so sljit and
* gcc run complementarily. */
try_sljit_region(addr); /* virtual entry (carries KSEG bits) */
head = idx_head(phys);
* gap (priority gcc > sljit > interp). */
if (g_sljit_async) {
/* OFF-THREAD: snapshot the region bytes + enqueue for the worker;
* interpret THIS pass. The shard registers on a later miss via the
* rescan above — no synchronous JIT spike on the dispatch thread. */
if (!sljit_already_tried(phys)) {
extern int dirty_ram_is_dirty(uint32_t phys);
uint8_t *ram = memory_get_ram_ptr();
if (ram && dirty_ram_is_dirty(phys)) { /* only JIT runtime code */
sljit_mark_tried(phys);
uint32_t snap = 8u * 1024u; /* SLJIT_MAX_FRAG_INSNS*4 */
if (phys + snap > 2u * 1024u * 1024u)
snap = 2u * 1024u * 1024u - phys;
uint32_t crc = crc32_update(0xFFFFFFFFu, ram + phys, snap)
^ 0xFFFFFFFFu;
overlay_compile_worker_enqueue(addr, phys, ram + phys, snap, crc);
}
}
} else {
try_sljit_region(addr); /* legacy synchronous path (PSX_SLJIT_SYNC=1) */
head = idx_head(phys);
}
}
}