Register configured range_sites with the dirty-RAM interpreter so overlay-resident classifiers use the same two-margin widening already emitted by native codegen.
Co-authored-by: Opok <novelasevilliousesp@gmail.com>
Add opt-in slti/sltiu far-bound scaling with matched native and interpreter semantics, correct immediate sign extension, and an overlay ABI forwarder for live aspect state.
Co-authored-by: Opok <novelasevilliousesp@gmail.com>
Landed while root-causing the Tomba 2 warm-cache boot death; each of these
was a hole that blocked attribution on the production binary:
- psx_fatal_halt: halt-and-serve is now keyed on the TCP listener being
live (debug_server_get_status) instead of the build flavor. A production
run with PSX_DEBUG_SERVER=1 + PSX_EXIT_HALT=1 now stays inspectable at
the fatal instead of exit(1)ing the evidence away.
- debug_server_get_status: was declared in debug_server.h but never
implemented; implemented (listener state, port, saved socket error).
- event_ring: EVENT_RING_ENABLED=1 unconditionally. The ring records
edges only; its production disable predated the production listener and
contradicted its own ALWAYS-ON design note.
- sp_ring (new): stack-domain (64 KB) transition ring at fntrace_record --
one entry per dispatch whose guest SP crossed a domain, i.e. the
provenance record for 'who installed this stack pointer'.
- disp_ring (new): 128-entry unconditional dispatch tail (target/ra/sp/
cycle) so the final control-flow sequence before a death is always
reconstructable post-mortem.
- overlay_loader: the native-call ring (overlay_native_ring recent[]) is
now recorded in production too -- each entry names the requested PC and
the claiming candidate CRC, which is what finally attributed the
wrong-variant native run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add an opt-in horizontal low-edge transform for subu rd,zero,rt classifiers, with matching native and dirty-RAM interpreter behavior and guarded overlay variants.
Co-authored-by: Opok <novelasevilliousesp@gmail.com>
Audit fixes for the async shard-publication pipeline (45a2dda):
- autocompile_shutdown(): stops the pipeline, kills the compiler tree via a
kill-on-close job object (assigned while the child is SUSPENDED so cmd.exe
cannot spawn grandchildren outside it), and JOINS the watcher + preparer —
never abandoning a thread inside the Windows loader lock at ExitProcess.
Wired first into shutdown_runtime(). Crash exits reap the tree via the job.
- Failed off-thread prepares retry (3 attempts, 100 ms) on the worker before
the artifact silently reverts to a synchronous emu-thread first-map; retry
and giveup counters exposed in autocompile_status_json.
- Capacity caps fail closed and LOUD: cand_register shouts once at CAND_CAP;
load_one_dll_prepared refuses at MAX_LOADED_DLLS before registering, so
dll_already_loaded can never lose track and double-register on rescan.
- Publication confinement is filesystem-resolved: published_path_valid
canonicalizes the cache root and the pipe path (GetFinalPathNameByHandle /
realpath), prefix-checks canonical forms, and the worker maps the canonical
string while holding a no-delete-share guard handle (TOCTOU). `..`/junction
spellings that passed the lexical check are now rejected — proven by the
new overlay_path_canon_test boundary test.
- test_autocompile_publication gains a shutdown-mid-run scenario asserting
prepared == committed + discarded with nothing queued; perf guards updated
to require validate -> guard-handle -> canonical first-map ordering.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two provably-correct framework fixes (independent of the AOT work) plus the
play-free overlay-extraction spike tooling and design doc.
Framework fixes (accurate, safe, complete):
- ABI v14: add gte_precision_store_word to OverlayCallbacks + preamble shim +
runtime population. The emitter emits a direct call for every swc2 (GTE
store-word); it is stateful runtime code (gte.cpp) so overlay DLLs must
forward or fail to LINK (undefined reference). Play-captured shards rarely
compiled a swc2; fuller static coverage surfaced the gap. Same pattern as
the v13 ws_player_x_bound / v10 gte_read_ctrl shims.
- Gate splitgen OFF in overlay mode (main_psx.cpp): the current recompiler's
unconditional split-TU output deletes <stem>_full.c and emits only shards,
which broke ALL overlay compilation (compile_overlays.py reads _full.c).
Overlays are single small TUs with no parallel-compile benefit; emit the
monolith. Restores overlay compile on the current tree.
AOT spike (docs/AOT_OVERLAY_PLAN.md + tools/aot_overlay_spike/, WIP):
- Play-free static overlay extraction from the disc image. Validated:
Tomba 1 reproduces 90% of played functions + ~1964 unplayed, live-verified
in-game (Dwarf Village runs native, zero runtime compile). Tomba 2 yields
661 functions byte-identical to the shipped/played vault (provably real),
all from the self-describing MAIN.EXE producer; the header-table BIN
producer needs loader-table RE (base recovery unreliable). Generality is
producer-dependent. Spike scripts are Tomba-specific proofs of concept
with hardcoded paths, not a finished tool.
- Correctness is guaranteed by the existing per-function code_crc dispatch
guard: a mis-positioned shard fails audit at compile or never fires at
runtime -> coverage loss, never wrong execution. Autocompile self-heals gaps.
CROSS-CUTTING: the ABI v14 change is part of the overlay codegen-hash source
set, so it ROLLS the cg tag (1f2ad026 -> e8484213). On master + a fw bump,
every title's overlay cache invalidates and needs regen+reshard. Branch only;
NOT pushed; other games' caches untouched. Needs user validation before merge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tier mis-compiled and has been hard-gated off since 2026-06-25; overlay
gaps fall to gcc/tcc shards and the dirty-RAM interpreter, unchanged.
Re-do on master of the removal validated on the stale branch (6da8c02),
reconciled with everything master grew since the fork:
- Delete overlay_sljit.{c,h}, overlay_compile_worker.{c,h}, lib/sljit,
runtime/tests/sljit_*.c, and the runtime.cmake sources/include-dirs.
- Strip the sljit provider/backend from code_provider.{c,h},
overlay_backend.{c,h} ("sljit" in overlay_backend/PSX_OVERLAY_BACKEND now
degrades to auto so stale configs still boot), main.cpp, debug_server.c
(sljit_status/sljit_async/sljit_try TCP commands), overlay_loader.{c,h}
(live policy, JIT-on-miss, persisted-shard cache, async worker glue,
psx_sljit_call/cop2/memx host helpers — reachable only via the removed
cpu->sljit_helpers table), and cpu_state.h (SLJIT_HLP_* enum + the
sljit_helpers[] member; CPUState layout change reshards via the cg tag).
- Master-only entanglement resolved before deletion: 28165d6's Einhander WS
player-X bound emit existed in three paths (gcc code_generator, sljit,
dirty_ram_interp). The gcc shard link contract now forwards
psx_ws_player_x_bound (ABI v13, previous commit), so deleting the sljit
emit path loses no widescreen behavior; ffc6688's game-dispatch block
inside the dead psx_sljit_call helper goes with it.
- Reword comments that described sljit as a live backend (gpu.{c,h},
ws_*_detect.h, cosim_state.c, dirty_ram_interp.c, autocompile.h,
boot_state.h, config_loader.h, compile_overlays.py, and the CPS marker
comments the recompiler emits).
recompiler psxrecomp-game builds clean; game runtimes rebuilt + boot-verified
separately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
28165d6 (Einhander native-wide player bounds) rewrites configured
[widescreen] signed_x_bound LUI sites into psx_ws_player_x_bound(vanilla)
calls in code_generator.cpp — and that site loop runs in overlay_mode too, so
the call lands in overlay-resident code. But the shard link contract never
forwarded the symbol: any shard containing a bound site fails to LINK
(undefined psx_ws_player_x_bound) and that variant runs interpreted forever —
the same silent class as v10 gte_read_ctrl and v12 psx_rfe_mark_escape (now
loud under the shard observability suite, but still a coverage hole).
The clamp computes against the host's live widescreen state (gpu.c), so a
local copy can't work: add the ws_player_x_bound callback (appended last),
the preamble forwarder (identity fallback => correct 4:3 on a host that
predates it), and the loader init wiring. Bump the ABI to 13 so stale DLL
mixes are rejected and regenerate.
This also unblocks removing the sljit tier: sljit's emit_ws_player_x_bound
was the only other native path carrying the clamp; with the gcc shard path
forwarding it (and dirty_ram_interp already handling the site), the feature
no longer depends on sljit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-land of the shard-observability suite from 6da8c02 (stale branch), merged
against current master (keeps master's overlay_ext() platform suffixes,
recompiler_patch hash sources, and the unconditional v12 rfe forwarder).
- compile_overlays.py: ShardStats tallies every shard outcome — per-shard
"SHARD FAIL [class]" lines, a SHARD BUILD SUMMARY, a machine-readable
"PSX_SHARD_RESULT ok=N failed=M skipped=K" line, and a non-zero exit when
any shard that should have built failed (previously a run that built zero
shards exited 0, indistinguishable from a clean run).
- New --check preflight: build every shard into a throwaway dir and report
pass/fail — answers "would this game's shards compile against the current
recompiler + headers?" without a runtime.
- Retain audit-failed interior-fragment C (<phys>_<interior>_fragment_FAILED.c)
for triage, plus a persistent doomed-interior memo (interior_fail_memo.txt)
keyed (phys,interior,region_crc) so autocompile stops re-walking
data-as-code fragments every cycle (~90x off the V8 fragment pass).
- Extract DISPATCH_PREAMBLE to runtime/include/overlay_dispatch_preamble.c.inc
(single source of truth for the shard link contract, loaded by
load_dispatch_preamble(); missing file is a hard error).
- codegen_hash_sources.cmake: fold the shard COMPILE surface (overlay_api.h,
cpu_state.h, the .inc) into the cg tag so header/ABI/link-contract drift
reshards same-tree instead of silently breaking every new shard compile.
- autocompile.c: parse PSX_SHARD_RESULT from the child-output tail, expose
shard_ok/shard_fail/shard_skipped/shard_fail_total over autocompile_status;
ALWAYS rescan on completion so shards that did build load even when the
run also reported failures.
Live proof on the stale branch: Tomba's first autocompile surfaced 9 shard
link failures (undefined psx_rfe_mark_escape — since fixed by v12) and V8
surfaced 91 benign data-as-code fragment failures, all previously silent.
Codegen hash changes => every title reshards once on next build (expected;
loader rejects/regenerates stale shards).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port the reusable Gaussian interpolation slice from PR #16 while keeping its reverb engine on a separate branch.
Co-authored-by: Martin Penkava <mpenkava1337@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Apply the PS1 1023x511 size rule to every polygon triangle, line, and polyline continuation before widescreen transforms while preserving textured-command state latching.
Co-authored-by: Martin Penkava <mpenkava1337@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The emitter emits psx_rfe_mark_escape() at every rfe — including in
overlay-resident exception-context code — but the dispatch preamble
never provided it, so any rfe-containing overlay failed to LINK
(undefined reference) and ran interpreted forever: the gte_read_ctrl
silent-coverage-blocker class again (11 of Tomba's vault shards hit
it). It mutates shared RFE-pending escape state (interrupts.c), so the
shim forwards unconditionally through OverlayCallbacks; ABI bump to 12
rejects pre-shim caches.
Found live during LOAD_TIME_ZERO E0 bring-up: the user's first pig
load triggered autocapture and the compile died on this link error.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Port Douglas Vanderveer's manifest-driven macOS/Linux overlay discovery onto the current cache index and exact-dispatch loader. Validate both filename key fields, preserve ABI cleanup and rescan idempotence, and cover the POSIX primitives with a real shared-library fixture.
Co-authored-by: douglasjv <vanderveerdouglas@gmail.com>
- psx_cyc.h / psx_cycles.c: _BitScanForward/_BitScanReverse shims for
__builtin_ctz/__builtin_clz under _MSC_VER
- full_function_emitter.cpp / main_psx.cpp: emit a portable .CRT$XCU
init-pointer instead of GCC-only __attribute__((constructor)) in
generated dispatch/game code
- main.cpp: declare cross-language globals extern "C" so MSVC links
them against their C definitions (no-op for GCC/Clang)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cue parser kept a single bin_filename that every FILE line
overwrote, so a redump-style dump (one .bin per track) mounted only
the LAST track file - the CD-DA audio bin - as the entire disc: no
ISO9660 filesystem, no boot EXE, black screen at startup (reported
against Tomba! 2, whose USA disc is data + one CD-DA track).
Replace the single ifstream with a BinSegment table: one entry per
FILE line, opened for the reader's lifetime and laid out at cumulative
disc-relative sector offsets in cue order. ReadSector/ReadRawSector
resolve a disc LBA to its owning segment; INDEX times, which a cue
states relative to the owning file, are converted to disc-relative
track starts once the segment layout is known. Single-file images are
a one-segment table with unchanged addressing.
Also fixed in passing: Open() never cleared tracks_ from a previous
open (reopening a reader accumulated stale TOC entries), per-read
sector-geometry probing is now computed once per segment, a cue whose
FILE is missing or non-BINARY (WAVE/MP3) fails Open() instead of
mis-mapping the TOC, and GetSectorCount() reports the whole-disc
total.
Verified: Tomba! 2 single-bin dump split at the track-2 boundary
(concatenation SHA1-identical to the original); a host-side test
opens both cues and confirms all 186430 sectors byte-identical and
the TOC identical (track 2 at LBA 174674, audio); the game boots to
intro FMV, title, and attract gameplay from the multi-file dump with
CD audio flowing, and the single-file dump boots identically on the
same binary.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Closes the latent Rule-18 soundness hole: kernel-window static dispatch
hits (RAM [0x500,0x8500), the BIOS's boot-time ROM copy) now require the
live RAM bytes of the function's full reachable extent to byte-match the
ROM source (lazy verify, cached, precisely invalidated on writes); patched
bodies divert to the faithful interpreter. Emitter ships
psx_bios_kernel_bodies[] (499 entries); runtime guard in memory.c; TCP
kernel_bless observability; PSX_KERNEL_BLESS=0 A/B switch.
USER-VALIDATED on Tomba (SCUS-94236): multiple play runs, LLE + HLE boots,
pads/menus/events, title 1.00x / village 0.92x parity; 208-214/219 kernel
functions verified clean, the 5 mismatches (runtime-patched pad-driver
family) faithfully interpreted as before.
NOTE: rolls the codegen hash — regen BIOS+game generated C and recompile
overlay shards with a same-tree recompiler on each title's next fw bump
(the overlay loader fail-closes loudly on tag mismatch).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generated dispatcher ran kernel-window table hits (RAM [0x500,0x8500),
the BIOS's boot-time copy of ROM 0x1FC10000+) natively with NO check that
the live RAM bytes still match the ROM image the functions were compiled
from — unsound for runtime-patched kernel code (the BIOS installs pad/SIO
handlers INSIDE compiled bodies; live diff: 15 clusters, 348 bytes). It
never misfired only because entry points happened to avoid patched keys.
A static kernel hit now requires psx_kernel_bless_dispatchable(): the
emitter emits psx_bios_kernel_bodies[] (key -> full RAM extent of the code
reachable from it; continuations carry their parent's extent, 499 entries),
and the runtime lazily byte-compares each entry's body against the ROM
source, caches the verdict, and invalidates precisely on writes into the
body (store chokepoint, DMA/EXE range marks, savestate restore). Clean ->
guarded native; patched/unverifiable -> the faithful dirty-RAM interpreter,
exactly as Rule 18 prescribes. This implements the runtime half of the
relocation-manifest contract (docs/RELOCATION_MANIFEST_FORMAT.md).
Measured on Tomba (SCUS-94236): 208-214 of 219 verified functions clean
(the 5 mismatches are the genuinely patched pad-driver family, faithfully
interpreted); millions of guarded native hits per boot; title 1.00x,
village 0.92x (parity — kernel keys were already native, now they are
PROVEN native); LLE and HLE boots + gameplay input verified. Kernel-window
interp measures 1.3% of interp burden (dirty_ram_stats), so this is a
soundness fix, not a speedup; the dominant interp cost is overlay-region
coverage (cg-tag hygiene), tracked separately.
Observability: TCP kernel_bless (entries/clean/mismatch/native_hits/
verifies/invalidations), always-on. PSX_KERNEL_BLESS=0 reverts to
unguarded-table behavior for A/B.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Tomba 16:9: rooms, sky falls, and task-card sequences presented stretched
because native-wide's 2D-backdrop stretch fires wholesale in scenes whose
background phase never ends. Such scenes now present native 4:3 from the
faithful canonical buffer, keyed on DEEP UNTAGGED POLYGON OVERHANG (>=24px
past the canonical window, >=4 polys, sustained 2 frames) — the only world
signal that survived adversarial scenes (shaded prims, GTE volume, any-poly
and shallow overhang all falsified by census evidence; lineage in gpu.c).
Scoped to sprite-tag titles; full_2d (MMX6) and gte_game_mode (Ape) paths
unchanged. Plus: native-wide never stretches a canonical-width game frame
(fallback pillarboxes); always-on present-classification ring (TCP
present_ring) across all backends; debug input injection drives the full
pad model (hybrid d-pad drop + stick axes); savestate TCP honesty; census
tagged column + ws_hud_mode A/B (HUD-corners deferred: ENHANCEMENTS.md W2).
USER-VALIDATED on Tomba (SCUS-94236).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Tomba nw_hud_corners experiment surfaced two blockers, both documented
in ENHANCEMENTS.md W2 with census evidence: composite dialog boxes spanning
zone boundaries tear (end caps pulled to opposite edges), and the AP counter
renders through the TAGGED sprite funnel so the untagged-only gate skips it.
Groundwork kept for the next attempt:
- WsCensusEntry gains a `tagged` column (ws_census CSV) so funnel membership
is measurable per prim.
- TCP ws_hud_mode {"tag_rects":0|1} live-A/Bs the tagged-rect gate.
Tomba's game.toml reverts nw_hud_corners to false (its own repo).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Matthew Stanley