The continuation-passing leak fix (RECURSION_BUG.md §25) is validated (Tomba boots+plays,
ce_profile flat through a 56.9k-frame soak, full overlay CPS). Flip the gate in all three
emitter sites so a normal regen emits CPS; PSX_CPS=0 is the legacy escape hatch. Verified:
default regen -> CPS markers present; PSX_CPS=0 -> none; default-CPS Tomba boots clean, leak flat.
Doc §25.6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The user-requested async overlay tiering (interp-now / compile-async / native-next, gcc-if-present
else sljit, always async, file-based handoff) is LARGELY ALREADY IMPLEMENTED — overlay_capture.c
(autocapture on interp pressure) + autocompile.c (background compile via a configured command,
toolchain-probed, watcher thread, file-based) + the code_provider abstraction. It just needed
CPS wiring.
This commit:
- overlay_loader_apply_live_policy: when the active backend is gcc, force sljit-live OFF so a gcc
box is gcc>interp (NOT the previous complementary gcc+sljit). Toolchain-less boxes stay
sljit>interp. PSX_OVERLAY_SLJIT_LIVE still overrides. ("mutually exclusive" per user.)
- ASYNC_OVERLAY_COMPILE.md: re-headed to reflect the feature is built+validated under CPS, with
the deltas this session added (compile_overlays --cps from §25.5, the gcc-only policy, the
dev-config overlay_autocompile_cmd --cps).
VALIDATED under CPS: autocapture fires -> background compile_overlays --cps produces CPS overlay
DLLs -> loader loads them -> dispatch_native rises; gcc box shows 0 sljit shards; ce_profile flat.
The native overlay cache auto-builds as you play (each scene compiles on its first sustained
visit). PRODUCTION game.toml overlay_autocompile_cmd still needs --cps appended when CPS ships.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build plan for the next session: interp-now / compile-async / native-next overlay tiering
(gcc-if-toolchain-present else sljit, mutually exclusive, always async). FILE-BASED handoff —
the worker writes .dll/.sljit artifacts; the dispatch fiber loads them via the existing
try_load_region / deserialize paths, so the candidate table stays single-threaded (no race).
Motivated by the CPS title screen being interp-bound (12.2M interp vs 14.8K native) because its
overlays aren't gcc-compiled + the sync sljit JIT-on-miss hitches the dispatch thread.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RECURSION_BUG.md §25.5. Brings the overlay native path under continuation-passing so the
idle-freeze leak fix holds with overlay_cache=true (the production gcc>sljit>interp path).
Runtime CPS-mode flag g_psx_cps_mode (overlay_loader.c) set by a __attribute__((constructor))
emitted in the BIOS + game CPS dispatch (full_function_emitter.cpp / main_psx.cpp). Gates all
overlay/sljit CPS behavior; legacy unaffected.
sljit JIT (overlay_sljit.c), scoped:
- jr $ra publishes cpu->pc (was a bare return). jr rX already tail-transferred.
- call-containing fragments DECLINE to the interpreter (CPS-safe, depth-1) — leaf shards still
JIT. (Full sljit call coverage / per-fragment entry-switch is future work.)
gcc-DLL path — FULL native overlay coverage (validated):
- compile_overlays.py --cps sets PSX_CPS for the recompiler -> CPS overlay C (tail-transfer +
per-function entry-switch; --overlay mode only affects widescreen, not CPS). DISPATCH_PREAMBLE
psx_syscall decl fixed void->int.
- PSX_OVERLAY_CODEGEN_VER 3->4: fresh cg4/ cache namespace; legacy unit-model DLLs (cg3) ignored.
- overlay_loader_dispatch: under CPS, when a dispatched addr is not an overlay ENTRY but lies
inside a registered candidate's code range (a tail-transferred call's continuation),
overlay_find_by_range() re-enters that function with cpu->pc=addr so its entry-switch routes
to the block. gcc DLLs are the trusted tier (diff-validated at entry) -> run native directly.
This is the CPS equivalent of the old unit-model that fixed the dwarf->overworld blue screen.
Validated: CPS overlay DLLs compile + load + dispatch_native>0; with continuation routing
dispatch_interp_fallback -> 0 (full native), ce_profile FLAT (6 KB), boots clean. Addresses are
KSEG0-consistent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The complete §25 fix: make BOTH recompiler emitters emit guest calls as
TAIL-TRANSFERS driven by the unified flat trampoline + the guest's own $ra/$sp,
instead of nesting one host C frame per guest call. The host stack can no longer
mirror unbounded guest re-entrancy — the architectural root of the ~14-min idle
freeze. Gated gen-time by PSX_CPS; legacy codegen byte-identical when unset.
VALIDATED (user): full-CPS Tomba boots BIOS -> FMV -> New Game -> village idle and
plays; ce_profile cur_max_kb FLAT at 5 KB from frame ~9.9k through 56,890 (turbo
soak) — blew past the ~50k-frame freeze threshold with the host stack bounded
(legacy climbed +1.17 KB/frame -> 48 MB -> freeze).
KEY CORRECTION to the prior session's §25.1: there are TWO emitters. The BIOS
(full_function_emitter.cpp) was done last session but the §8 leaking chain
(func_8001A954 ...) is GAME code, emitted by code_generator.cpp (via main_psx.cpp)
— never converted. Also, §25.1's "continuations not in the dispatch table"
hypothesis was wrong (they were always added); the real boot regressions were
found by dumping the always-on fntrace ring on the abnormal psx_dispatch return.
Emitter changes (gated PSX_CPS):
- full_function_emitter.cpp (BIOS): BIOS A0/B0/C0 vector runtime-fallback
tail-transfers instead of nested psx_dispatch_call (was the frame-0 exit).
- code_generator.cpp/.h (game): jal/jalr/jr-$ra/jr-table/branch-split/j-split ->
tail-transfer (jalr captures rs before the link, alias-safe); each call's
return point registered as a dispatchable continuation; entry-switch (keyed on
cpu->pc) at the top of every function AND every overlapping-alias body. 0
psx_call_contract / call_by_address emitted under CPS.
- main_psx.cpp: psx_dispatch_game_compiled gains continuation cases
(case cont: cpu->pc=cont; func_owner(cpu)); entries clear cpu->pc.
- strict_translator.cpp: syscall emits `if (psx_syscall(...)) return;` so a void
syscall falls through to the inline post-syscall jr $ra (real-HW EPC+4).
Runtime changes (the legacy "return via nested dispatch / cpu->pc=0" idioms that
exit the flat CPS trampoline):
- traps.c: psx_syscall returns int (0=void/fall-through, 1=transfer);
Enter/ExitCriticalSection return 0; case-3 ChangeThread-success returns 0 — the
fiber scheduler leaves cpu->pc=0 on resume, which was the frame-712 exit.
- cpu_state.h / overlay_api.h: psx_syscall signature void->int; overlay callback
ABI bumped 4->5 (stale overlay caches rejected).
- dirty_ram_interp.c: gate interp_enter_compiled behind PSX_HAS_GAME_DISPATCH
(un-breaks the BIOS-only build).
- main.cpp: dump the always-on fntrace ring tail + stack to psx_cps_exit_trace.json
on an abnormal psx_dispatch return (diagnostic; CLAUDE.md ring-buffer model).
Docs: RECURSION_BUG.md §25.2-§25.4. Remaining: dwarf->overworld playtest;
Stage 3 (sljit/overlay CPS parity); then consider making CPS default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The universal fix for the idle-freeze host-stack leak (RECURSION_BUG.md §25): make
the recompiler emit guest calls as TAIL-TRANSFERS driven by the trampoline + the
guest's own $ra/$sp, instead of nesting one psx_dispatch_impl per guest call frame.
This makes the host stack incapable of mirroring unbounded guest re-entrancy — the
architectural root of the leak — for ANY game. Perf cost accepted ("right before
fast").
Stage 1 (full_function_emitter.cpp), GATED behind gen-time env PSX_CPS:
- jal/jalr (orphaned-delay + pending-delay paths): emit `cpu->pc=target; return;`
(tail-transfer) + register the return address as a dispatchable continuation,
instead of nested psx_dispatch + psx_call_contract. JALR captures the target reg
BEFORE the link write (rd==rs alias-safe).
- jr $ra (all 3 sites): publish `cpu->pc = cpu->gpr[31]; return;` under CPS so the
trampoline dispatches the caller's continuation (legacy `return;` relied on the
nested-dispatch C-return). Required — without it the BIOS exits at frame 0.
GATING PROVEN SAFE: regen+build WITHOUT PSX_CPS boots Tomba normally — default
codegen byte-unaffected; CPS fully opt-in.
OPEN (CPS path only): with PSX_CPS=1 the BIOS boots main() then exits at frame 0
(execution completed, PC=0). Root cause (high confidence): the trampoline must
dispatch mid-function continuation addresses, but the dispatch table holds only
function entries → a return to a continuation misses → psx_unknown_dispatch →
pc=0 → exit. NEXT: add registered continuations to the dispatch table (map
continuation_addr → containing func), then re-regen PSX_CPS=1 and run the §25 gate
(ce_profile flat + oracle-identical + 15-min soak + dwarf->overworld). Stages 2
(interp interp_enter_compiled surface) + 3 (sljit/overlay parity) follow.
Safe revert point for the whole feature: commit ae47e69.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Oracle disambiguation (RECURSION_BUG.md §23–§24). Traced one func_8001A954
invocation on real HW (psx-beetle) vs the recomp in the IDENTICAL Tomba idle
state (New Game village dialogue; object 0x801FD950, state/scene/jumptable=0).
VERDICT: real HW's per-frame chain is BALANCED — func_8001A954 is entered once
per frame from main_loop (JAL 0x8001A630, sp=0x801FE3E0) and RETURNS once per
frame (jr ra @0x8001A9E8 -> 0x8001A638, SP fully restored); no re-entry, the
overlay 0x80063864 never calls back (refutes §21/§22's "call-cycle"). The recomp
runs the same guest flow and the guest SP unwinds identically, but the HOST stack
leaks +1.17 KB/frame (ce_profile, live-reconfirmed: 20099->22215 KB / 1807 frames)
because the per-frame compiled chain (func_8001A954->AC00->B2B4->B5A8->func_80046264,
emitted as DIRECT func_X(cpu) host calls) re-enters itself via func_80046264's
CD-DMA-dirty hop and never unwinds.
native_stack early-trip (PSX_STACK_GUARD_KB=6144, frame 7592, 6 MB) pins the
leaking host cycle. Cross-checked: xprobe mixed_depth bounded at 6, dirty-insn/
dirty-block show NO literal func_8001A954 transfer, and a954=1/frame is a SURFACED
guest tail-transfer (j/jr) to func_8001A954 that the oracle proves real HW never
makes — i.e. there is both (A) a correctness divergence (spurious per-frame
re-entry) and (B) the architectural host-mirroring limitation that amplifies it.
Fix scope CORRECTED from §23's "quick contained runtime patch": the UNIVERSAL fix
is §11 — route in-range guest calls through the depth-managed psx_dispatch_impl
trampoline instead of mirroring them as direct host calls, so the host stack stays
bounded regardless of guest re-entrancy (real or fabricated). §12 (CD-DMA dirty
precision) and fixing the specific divergence are surgical mitigations, not
universal. Performance loss accepted for now (static recomp: right before fast).
Instruments added (runtime-only, no regen):
- SP capture in the fntrace control-flow ring on BOTH backends (GPR[29] read live):
psx-beetle (beetle_libretro.cpp + beetle_debug_server.c) and the recomp
(fntrace.h/.c + debug_server.c) — enables the func_8001A954 SP/RA-lifecycle oracle.
- psx-beetle default debug port 4380 -> 4382 (+ --port N override): 4380 collided
with psxref and cdirecomp/CdiRuntime.exe; two LISTENers on one port silently
misroute connections (the phantom "oracle keeps dying" early in the session).
Oracle tooling: the control-flow oracle is psx-beetle (Beetle interprets -> records
every J/JAL/JR/JALR), NOT psxref (no PC trace; the assumed emu_step/emu_trace_addr
never existed). Loads Tomba via: psx-beetle.exe --disc tomba.cue <bios> --port 4382.
Next: implement the §11 universal fix (codegen + full regen).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pinned the exact leaking crossing (xprobe detail): entry to func_8001A954 is a single
JAL, src 0x8001A630 (the main loop's only call site), once/frame. Implemented (A):
surface interp JAL/JALR to game-text targets to the owner trampoline (gated by
psx_game_address_in_text so overlay-native keeps the nesting contract). §19 result: leak
STILL CLIMBS (~0.875 KB/frame, down from 1.17) -> reduced, not eliminated -> REVERTED.
It is a guest CALL-CYCLE (main_loop 0x8001A51C <-> func_8001A954): surfacing the interp
side just relocates the nest to the compiled-side jal back to the main loop. Can't fix a
call-cycle by surfacing one crossing. Leak is ~1 host frame PER FRAME (not per cycle
iteration) -> the loop mostly returns; exactly one call/frame fails to unwind. Real HW
idles forever (balanced), so either we mishandle one specific return/frame (Hyp C,
contained) or it needs the §11 continuation-passing redesign. DECISIVE NEXT = oracle
(psxref): does real HW return from func_8001A954 to 0x8001A634 each frame, or never?
Stop surfacing single crossings (block-loop §20 + JAL/JALR §22 both dead ends).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ~14-min Tomba idle freeze is a gradual per-frame host-stack LEAK (not a
within-one-frame spiral, not the gradual-marking theory). Confirmed via a new
compiled-entry depth profile (§19): per-frame MAX guest-stack-used climbs dead-linear
+1.17 KB/frame from gameplay start to 48 MB on the 64 MB main-thread stack -> native
stack guard -> freeze. Once per frame an interpreted overlay (reached via
func_80046264's dirty hop jal 0x80063864) does jalr func_8001A954, nesting a host call
that never unwinds (the per-frame loop never returns). Full writeup RECURSION_BUG.md
§16-§21.
Instruments (always-on rings, runtime-only, no regen):
- §17 stack_profile: per-vblank host-stack sample (main.cpp; stack_profile TCP cmd).
- §18 xprobe / xprobe_arm: interp<->compiled boundary crossing flight recorder
(dirty_ram_interp.c; per-frame summary + per-crossing detail).
- §19 ce_profile: per-compiled-entry depth profile + raw TEB at the trip
(debug_server.c; ce_profile TCP cmd) — the trustworthy instrument; the regression
check for the fix (max_kb must stay FLAT).
- PSX_STACK_GUARD_KB early-trip + PSX_FIBER_STACK_KB (debug_server.c / traps.c).
Fix progress:
- Block-loop tail-transfer surface (dirty_ram_interp.c ~1514): surface compiled
tail-transfers to the trampoline instead of nesting. Correct for tail-transfers but
RULED OUT as the leak (§19 climb unchanged) — the leak is the CALL path (jalr/jal).
- Real fix (next): surface compiled-game CALLS to the owner trampoline, gated to
compiled-game targets only (a naive all-targets surface reintroduces the
dwarf->overworld lost-continuation bug). Needs a non-nesting membership test
(generated psx_dispatch_game_has) or §12 CD-DMA dirty-marking precision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run #1: interp->0x8001A954 edge is ORDINARY (1/frame). Run #2: dispatch_8001A954=0
(dirty_ram_dispatch never called with 0x8001A954) AND recursion_func VARIES
(0x8004DEE0/0x8004DFA0/0x8005FF60) -> address-specific counters are the wrong
instrument. dirty_block_tail shows the real cycle: dirty_ram_dispatch(overlay
0x8011-0x8013) called from main-EXE 0x8002-0x8005 = the per-frame MESSAGE/RENDER
overlay system recursing. SITE_CAP 256->32 fixes the report-JSON overflow. Pivot:
use data-driven rings (native_stack/dirty_block_tail/dispatch_tail) from a clean
report, not guessed addresses; then last-writer slice on the loop termination var.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Per-frame count of the re-entry edge interp->func_8001A954 (the per-frame loop
head) + cycle counter, in a 256-frame ring the crash report dumps
(dirty_ram_re954_json). Answers ChatGPT's step-1 question: is the re-entry
ORDINARY bounded per-frame behavior that stops terminating at ~50k, or a new edge
— and whether cycles advance (it's within ONE frame; 23k laps ~= 23k cycles <<
565k/frame, so no VBlank before overflow => control-flow cycle, not starvation).
docs/RECURSION_BUG.md §15 captures the full strategy reset (stop RAM-diffing;
triggered control-flow recorder + backward last-writer slice; narrow oracle use
only on the resulting small address set). The separately-navigated oracle RAM-diff
(§14) was a dead end (alignment noise).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Capture everything for recall in a future session:
- Resolved the overworld-load regression (stale build artifact, not tooling/master).
- 3 organic captures (deterministic: v0=0x8004DEE0, ra=v0+8, pc=0; both New-Game
and Dwarf-Forest reproduce).
- The contract mechanism (S7): psx_dispatch_impl tail-call trampoline + psx_call_contract
bail keep COMPILED tail-loops flat; the dirty interpreter bypasses them
(exec_one JAL/JALR + dirty_ram_dispatch_inner:~1179 nest psx_dispatch_game_compiled).
Proof it's host-only: guest sp flat (~7KB) across ~23k laps.
- Ghidra map of the guest state machine (S8): func_8001A954 state[0x4a]==1 ->
func_8001AC00 chain -> func_80046264 jumptable (selector @0x8009BCDD, table
@0x80013C60) -> func_8004DFA0 (a render handler that advances no selector).
- External-review verdict (S9): Hypothesis A overwhelmingly likely; B has no
positive evidence; C (interp control-transfer bug fabricating the re-entry) must
be excluded by an oracle PC-trace.
- Oracle disambiguation plan (S10), the single mixed-dispatch-owner fix design
(S11, runtime-only / no regen), CD-DMA dirty-marking critique (S12), and the next
disassembly target around 0x80063864 (S13).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Snapshot of the uncommitted native_stack capture tool (crash_trace.c,
append_native_stack) that recovers the true compiled<->dirty-interp
recursion cycle the recent_fn ring can't show, plus docs/RECURSION_BUG.md
documenting the freeze root cause, the 4-instance soak reproduction
harness, and the overworld-load regression now blocking repro.
This is the build that currently insta-crashes soaks on overworld load;
preserved as-is before hardening so the regression stays reproducible.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SPU Catmull-Rom float shadow (spu_shadow), 15-bit CRT color LUT (raw bit-exact), verifier. Opt-in (env-gated), default OFF: raw output byte-identical and remains the verify oracle. Reverts loud (DEGRADED) on divergence. ShadowVerifier ported from JRickey/gba-recomp (MIT OR Apache-2.0). Compiles clean; full game-build A/B is the remaining integration gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of the dwarf-village->overworld native-overlay blue screen
(deterministic, every run, at native call #70):
The dirty-RAM interpreter's jal/jalr handlers preserved the call's
return-address obligation for statically-compiled callees
(psx_dispatch_game_compiled -> *next_pc_out = return_pc) and for non-local
targets (dispatch_nonlocal_call passes return_pc as stop_addr), but a LOCAL
DIRTY target fell through to a bare `cpu->pc = target; return 1`. When that
target was a NATIVE overlay candidate, the dispatch loop ran it as a C
function; its pc==0 return made psx_dispatch_impl unwind to the C caller,
silently dropping the suspended interpreted caller's continuation.
Concrete case: main-EXE field dispatcher F=0x800338A8 (undiscovered entry —
discovery walked it from 0x338B0; reached via data-table jalr, so it runs
under the interpreter) does `jal 0x8011B0D4` per entity. Under native exec
F's epilogue never ran: guest sp leaked 0x18 per entity until the world
corrupted. Interp-only mode is immune (the callee's own jr ra is interpreted).
Evidence chain (full-register fingerprint diff ON vs OFF localized first
divergence to sp at exec #70; frozen dirty-block ring shows F re-entered at
sp-0x18 with no epilogue between) is in docs §9 and the session memory.
Fix: overlay_loader_call_native() — run a validated native candidate as a
unit and resume at return_pc — called from both jal and jalr handlers,
mirroring the psx_dispatch_game_compiled contract exactly. Side effect:
native overlay coverage rose from 86 calls per village->overworld run to
~112,000 (interpreted calls now route to native candidates).
Verified: dwarf->overworld renders and plays under native overlays.
Same-scene A/B: host CPU 7.31 (native) vs 7.83 (interp) CPU-s per 15s wall.
Timing hypotheses refuted by experiment before fixing: IRQ-check cadence
(suppress armed, identical failure) and device-event granularity
(conservative stepping, identical failure).
Also in this commit (the capture tooling that found it):
- fingerprint ring now stores full entry/exit register files + guest cycle
(FP_CAP 2^20 crc-only -> 2^16 full-reg); overlay_fp_dump takes a path.
- insn_freeze <addr> <nth>: one-shot latch freezing insn/block/flow/fntrace
rings immediately BEFORE the Nth dispatch of a watched candidate, so the
pre-divergence window survives until dumped (ring-first, no arm-then-hope).
- dirty_insn_gate <lo> <hi>: runtime-settable extra PC range for the per-insn
interp log; native entry/exit markers (transferred=200/201) in that ring.
- dirty_insn_dump_file: file dump for the insn ring — the 16MB inline TCP
dump wedged the single-threaded debug server.
- freeze gating in the flow-log path gates ONLY the ring write, never flow.
- debug_client plumbing for the new commands + fntrace_arm/fntrace_dump.
- docs/overlay-recompilation-design.md §9: the dispatch call-contract
invariant, the Bug A case study, verification numbers, and known residue
(data-jump-table discovery-miss class; one unreproduced black-screen
one-off; §8.3 tail-position caveat).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OV-1 resolved by mechanism: invalidations=1 on the BIOS overlay-load write into
0xE7000, 88 funcs unregistered, stale_blocked into the tens of millions (every
one a prevented stale native call). Correctness only so far — native~122 vs
interp~31M — so Inc2 reload-on-return is flagged as the next/biggest benefit
lever.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents what works (capture, evidence-scoped --overlay, boundary gate, clean
88-func DLL, in-village native dispatch) and the open OV-1 issue: blue screen on
village->overworld transition.
OV-1 root-caused via live RAM-vs-captured comparison at the blue screen: the
front of the 0xE7000 region was overwritten (compiled-func addresses now hold
pointer tables: 0xE73C0/0xE7000/0xE8120 DIFFER, 0x100000/0x13B000 MATCH) while
the loader kept all 88 village functions registered (regions_checked=1, never
re-evaluated). The game dispatches into changed content, hits stale village
function pointers, runs wrong code -> blue (frame advancing, no crash, no
exception, 0 misses). The registration outlived its source bytes. Primary fix
is Inc1-D write-invalidation; A/B confirm = rerun with overlay_cache=false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restores entry-scoped overlay analysis (analyze_exact_entries +
split_mid_function_targets=false) as the MANDATORY mode for overlay inputs,
renamed --overlay-exact -> --overlay: the ingester's contract, set
unconditionally by compile_overlays.py for every overlay, never a human toggle.
Evidence-scoped discovery compiles only proven seeds + reachable code and keeps
branch/jump-table targets as in-parent labels — no whole-byte sweep, so it no
longer decodes embedded data tables as code.
Effect on the captured Tomba overlay: the data-as-code explosion collapses from
thousands of UNSUPPORTED to 122 clean functions with 42 residual unsupported
opcodes + 8 bad targets. Those residuals are now a per-function-audit concern
(Inc1-B), not a wholesale region failure. Main-EXE codegen is unchanged (Tomba
still regenerates 2854 functions; split pre-pass still runs in non-overlay mode).
Also adds docs/overlay-recompilation-design.md — the converged design of record
(precision-over-recall; compiled!=registered!=callable; write-invalidation over
CRC; internal-labels-in-parent; the RAM-write-path audit result; the three
implementation increments).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reinstates the runtime-loaded overlay DLL cache (Layer A) and the offline
overlay compilation pipeline that were built on the pruned codex/cache-discovery
branch (tip 2bbafc2) and never merged. Ported onto current master; rebuilt
rather than history-restored.
Runtime (off by default, opt in via [runtime] overlay_cache = true):
- overlay_loader.c/.h, overlay_api.h: CRC32-keyed DLL cache at
cache/<game_id>/<region>_<crc>.dll. On a dirty-RAM dispatch miss the loader
LoadLibrary's the matching DLL, wires runtime callbacks via overlay_init(),
enumerates func_XXXXXXXX exports, and dispatches them BEFORE the interpreter.
- overlay_capture.c/.h: write-once capture set; records overlay DMA bytes and
writes overlay_captures.json (execution-verified seeds) at clean exit.
- dma.c: capture + cd_dma_log hook at CD DMA completion (async model).
- cdrom.c: cdrom_get_setloc_lba() for disc-position correlation.
- dirty_ram_interp.c: g_dirty_ram_exec_pc_table (exec-verified PCs), a
decodability guard on JAL-into-dirty targets, and the overlay dispatch hooks.
- debug_server.c: cd_read_log + overlay_loader_status TCP commands.
- config_loader: [runtime] overlay_cache flag (default false).
Recompiler (compile_overlays.py / extract_overlays.py drive these):
- main_psx.cpp --overlay-exact: analyze only explicit entries + direct JAL
targets and set split_mid_function_targets=false, so overlay basic-block
targets are NOT split into bogus standalone functions (the mid-function-seed
failure mode that produced black-on-load / blue-on-new-game).
- function_analysis.cpp: analyze_exact_entries(); code_generator.cpp: gate the
mid-function split pre-pass behind split_mid_function_targets (default true,
so normal main-EXE codegen is unchanged - Tomba still regens to 2854 funcs).
Deltas from the original: save states dropped (never worked); overlay enable is
runtime-config-gated (off by default) instead of a build-time define; loader
stderr logging converted to the overlay_loader_status TCP path (CLAUDE.md Rule 3).
Recovered from _backups/2026-06-06-consolidation/psxrecomp-all-PRE.bundle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs/disc-speed.md: design doc for per-game disc speed configuration.
docs/overlay-discovery.md: design doc for B+A overlay pre-compilation
and runtime JIT caching (game-side, future work).
tools/dump_tomba_misses.py: one-shot dump of dirty_ram per_pc entries
in the game text range, formatted as seed lines.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Tight-crop (1031x315) so the header isn't dominated by empty grey
above/below the logo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Config loader (recompiler/src/config_loader.{h,cpp}): single source of
truth for both BIOS and game runtime settings (debug_port, window_title,
memcard_dir, disc path). psxrecomp-bios and psxrecomp-game both accept
--config <toml>; runtime takes --game <toml> (no more CLI overrides).
- runtime/src/main.cpp: replaced --disc/--memcard-dir/--game-root CLI
overrides with --game <toml>; game.toml is the single source of truth.
Discless launch (no args) goes to BIOS shell.
- runtime/src/dma.c: fixed update_master_irq() re-asserting I_STAT bit 3
on every DICR read. BIOS shell DMA handler at 0x80058628 only clears
DICR bit 27 (ch3 flag), so level-sensitive re-assert pinned the kernel
in the exception loop. Bit 3 is now edge-triggered (one assert per DMA
completion). Discless boot now reaches MAIN MENU / MEMORY CARD / CD
PLAYER. With-disc Tomba boots through to Whoopee Camp splash.
- Audit tooling: codegen_audit.py (BIOS), codegen_audit_game.py (game),
audit_config.py (config loader for tools), verify_audit_schema.py.
Replaces older tools/_codegen_audit.py probe.
- Docs: CURRENT_STATE.md (project state), UNIFICATION_PLAN.md (Phase C
translator merge plan), docs/config_schema.md (TOML schema reference),
docs/audit_inventory.md (audit tier inventory).
- .gitignore: tools/_*.py convention for ad-hoc scratch probes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: only sector 0 read succeeded; sector ≥1 reads aborted at 2 bytes
because mc_flag was loaded from mc_slots[].flag on every 0x81 select but
never saved back. The clear in MC_CMD lived only in the working static, so
every reselect re-loaded the stale 0x08 ("new card / changed battery") init
value, and the BIOS bailed every multi-sector op as "card was just inserted,
re-init."
After: BIOS shell memcard menu reads the full 15-sector directory area for
both slots. Real save filenames decode (e.g. "BISLPS-01234SAVE00") and per
card icons render in the bottom panes — the Phase 4 success criterion.
Also bundled this session:
- debug_server: B9D0..B9F0 added to always-on wtrace ranges; fn_dump_parse
default window scoped to 1M most-recent entries (filter scans against the
full 128M ring used to freeze the debug server thread for many seconds).
- ISSUES.md + diagnostic tools/*.py written during the audit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the Beetle-discovered chain writer/init PCs as dispatch_miss seeds
so the recompiler emits standalone functions for them at their RAM
destination addresses (via the existing Phase 1e ROM↔RAM alias rule
RAM = ROM - 0xBFC10000 + 0x500):
0xBFC14A54 → 0x00004F54 (chain BUSY-writer)
0xBFC158D0 → 0x00005DD0 (chain init slot-0 BUSY)
0xBFC158D8 → 0x00005DD8 (chain init slot-1 BUSY)
0xBFC159F4 → 0x00005EF4 (chain step 0x02-OPEN writer — KEY)
0xBFC15AA8 → 0x00005FA8 (chain step 0x04-OPEN writer — KEY)
Post-regen dispatch_table contains all 5 as standalone entries. e.g.
func_00005EF4 is a clean 3-instruction leaf (sb $t3, 0($a3); v0=1; jr ra).
Validation (full output in docs/CASE_A_AOT_GAP.md "Vertical slice result"):
- ✓ dispatch_table contains all 5 entries
- ✗ none of the 5 are reached at runtime — chain still doesn't dispatch
- ✗ [0x80007510..0x7570] all-zero — the chain init function FUN_bfc158a8
(= func_00005DA8) never runs in our recomp during this session, gates
haven't even reached the initial 0x01 BUSY state
- ✓ shell IS on memcard screen ([0x80066948] = 0x32)
The seed addition is correct and necessary work — it removes one of two
layers of "writer is unreachable". The remaining layer is upstream:
nothing in our recomp causes FUN_bfc14b00 / FUN_bfc158a8 to run during
the memcard flow. That's a different category from "chain runs but
fails on byte N" and is not chased here per Case A directive.
Beetle wtrace also revealed the writers at 0x5EF4 / 0x5FA8 are tiny
leaf utilities called from BIOS code in BFC08{B98,C94,D08,F28} — NOT
chain-step jump-table targets as initially assumed. The chain step
tables at RAM 0x6c70/0x6c98/0x6ccc point at 0x52xx / 0x56xx-0x5Axx /
0x5Bxx-0x5Dxx ranges. Memo updated accordingly.
Includes 4 new validation scripts under tools/ used during this audit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a RAM-write trace hook to mednafen-PSX (SB/SH/SW handlers fire
g_psxrecomp_wtrace_cb after each write), a ring buffer + TCP commands
in psx-beetleoracle, and verification scripts that drive CROSS in
both Beetle and recomp and diff the gate-byte writers.
Conclusion (docs/CASE_A_AOT_GAP.md): the [0x7568+slot] LSB-clear
writers are at RAM PCs 0x5EF4 / 0x5FA8 (and 0x4F54 sets BUSY).
Recomp never dispatches any of those PCs, never executes the
0x4F00..0x4FFF or 0x5800..0x5FFF RAM windows, and never writes the
gate. Selective coverage pattern (interp reaches 0xCF0/0x52xx/0x6xxx
but not 0x4Fxx/0x5xxx) is install/dispatch path gap, not interpreter
bug. PLAN.md:941 already names this work as a hard gate.
Beetle-side modifications live inside the vendored beetle-psx/ git
clone; docs/beetle_wtrace_hook.patch captures the diff so they
survive a re-clone.
Memcard symptom debugging stops here. Next: trace-generated AOT
RAM-code manifest + recompiler ingestion (Phases B–F).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scaffold for guest-cycle-driven peripheral timing per
docs/CYCLE_TIMING_ARCH.md. The counter is defined and exposed via
freeze_check, but psx_advance_cycles has no callers yet — counter
stays 0. Validation: zero behavior change.
Files:
- runtime/include/psx_cycles.h: extern uint64_t psx_cycle_count;
void psx_advance_cycles(uint32_t cycles); uint64_t psx_get_cycle_count(void);
- runtime/src/psx_cycles.c: scaffold body; peripheral hooks empty
(1.0e-b will add sio_advance, 1.0e-c will add gpu_advance, etc.)
- runtime/CMakeLists.txt: psx_cycles.c added to PSX_SHARED_SOURCES
- runtime/src/debug_server.c: freeze_check exposes psx_cycle_count
- docs/CYCLE_TIMING_ARCH.md: design doc — single cycle clock,
generic per-block emit (no card-specific paths), MMIO-split codegen
per explicit user direction, IRQ-pending-vs-CPU-entry separation,
five-step migration plan (1.0e-a..e)
Validation against 0d15e2b baseline:
- t=20s/30s tx_writes match (1160/5480)
- 4x CROSS succeeds; card txn distribution unchanged (10 reach READ_DATA)
- "MC" bytes still flow at txn 0 bytes 10-11
- psx_cycle_count = 0 throughout (no callers in this slice)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real card protocol restarts on every fresh 0x81 — SELECT deassert is
a bus-wide reset signal. Earlier model preserved mc_state across
deassert via mc_save_slot, which caused every subsequent 0x81 to land
on a stale non-IDLE saved state and trigger abort_reselect bookkeeping.
Audit confirmed the BIOS-visible RX was still correct (the abort path
itself reset mc_state), but the retained sector/cmd/counter context
across logically-independent transactions was wrong per no$psx.
Clear volatile protocol state on SELECT deassert (state, cmd, sector,
sector_msb, sector_lsb, data_idx, checksum on both slots). Preserve
mc_slots[].flag (persistent "new card" metadata) and mc_slots[].data
(sector buffer, harmless to retain).
Result on a 4-press boot run: 6 transactions reach READ_DATA (ts=10)
and successfully read the "MC" header magic (RX bytes 0x4D 0x43);
remaining txns continue as 4-byte presence pings. Pre-fix: 0 reads
of card data ever; post-fix: card protocol can complete the setup
phase and enter data phase.
The byte-15 abort that follows is a separate blocker — kernel chain
dispatcher interleaves pad polling into the card data read. Tracked
for next session via the chain dispatcher disassembly.
Also includes:
- Slot-state capture in SioTraceEntry + sio_trace JSON emission
- dirty_ram_interp: replace mid-block fprintf+fflush with a counter
(CLAUDE.md §3 — synchronous Windows console I/O at high rate
starves the dispatch loop)
- Passive Python TCP probes (raw_tcp, trace_audit, trace_range,
txn_summary, txn_dist, txn_bytes, drs_filter, readram_test,
step_audit{,2}, verify_card_buffer, wtrace_filter)
- docs/MEMCARD_AUDIT_2026_04_28.md (prior session's 7-layer audit)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matthew Stanley