Root cause of the dwarf-village->overworld native-overlay blue screen
(deterministic, every run, at native call #70):
The dirty-RAM interpreter's jal/jalr handlers preserved the call's
return-address obligation for statically-compiled callees
(psx_dispatch_game_compiled -> *next_pc_out = return_pc) and for non-local
targets (dispatch_nonlocal_call passes return_pc as stop_addr), but a LOCAL
DIRTY target fell through to a bare `cpu->pc = target; return 1`. When that
target was a NATIVE overlay candidate, the dispatch loop ran it as a C
function; its pc==0 return made psx_dispatch_impl unwind to the C caller,
silently dropping the suspended interpreted caller's continuation.
Concrete case: main-EXE field dispatcher F=0x800338A8 (undiscovered entry —
discovery walked it from 0x338B0; reached via data-table jalr, so it runs
under the interpreter) does `jal 0x8011B0D4` per entity. Under native exec
F's epilogue never ran: guest sp leaked 0x18 per entity until the world
corrupted. Interp-only mode is immune (the callee's own jr ra is interpreted).
Evidence chain (full-register fingerprint diff ON vs OFF localized first
divergence to sp at exec #70; frozen dirty-block ring shows F re-entered at
sp-0x18 with no epilogue between) is in docs §9 and the session memory.
Fix: overlay_loader_call_native() — run a validated native candidate as a
unit and resume at return_pc — called from both jal and jalr handlers,
mirroring the psx_dispatch_game_compiled contract exactly. Side effect:
native overlay coverage rose from 86 calls per village->overworld run to
~112,000 (interpreted calls now route to native candidates).
Verified: dwarf->overworld renders and plays under native overlays.
Same-scene A/B: host CPU 7.31 (native) vs 7.83 (interp) CPU-s per 15s wall.
Timing hypotheses refuted by experiment before fixing: IRQ-check cadence
(suppress armed, identical failure) and device-event granularity
(conservative stepping, identical failure).
Also in this commit (the capture tooling that found it):
- fingerprint ring now stores full entry/exit register files + guest cycle
(FP_CAP 2^20 crc-only -> 2^16 full-reg); overlay_fp_dump takes a path.
- insn_freeze <addr> <nth>: one-shot latch freezing insn/block/flow/fntrace
rings immediately BEFORE the Nth dispatch of a watched candidate, so the
pre-divergence window survives until dumped (ring-first, no arm-then-hope).
- dirty_insn_gate <lo> <hi>: runtime-settable extra PC range for the per-insn
interp log; native entry/exit markers (transferred=200/201) in that ring.
- dirty_insn_dump_file: file dump for the insn ring — the 16MB inline TCP
dump wedged the single-threaded debug server.
- freeze gating in the flow-log path gates ONLY the ring write, never flow.
- debug_client plumbing for the new commands + fntrace_arm/fntrace_dump.
- docs/overlay-recompilation-design.md §9: the dispatch call-contract
invariant, the Bug A case study, verification numbers, and known residue
(data-jump-table discovery-miss class; one unreproduced black-screen
one-off; §8.3 tail-position caveat).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OV-1 resolved by mechanism: invalidations=1 on the BIOS overlay-load write into
0xE7000, 88 funcs unregistered, stale_blocked into the tens of millions (every
one a prevented stale native call). Correctness only so far — native~122 vs
interp~31M — so Inc2 reload-on-return is flagged as the next/biggest benefit
lever.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Documents what works (capture, evidence-scoped --overlay, boundary gate, clean
88-func DLL, in-village native dispatch) and the open OV-1 issue: blue screen on
village->overworld transition.
OV-1 root-caused via live RAM-vs-captured comparison at the blue screen: the
front of the 0xE7000 region was overwritten (compiled-func addresses now hold
pointer tables: 0xE73C0/0xE7000/0xE8120 DIFFER, 0x100000/0x13B000 MATCH) while
the loader kept all 88 village functions registered (regions_checked=1, never
re-evaluated). The game dispatches into changed content, hits stale village
function pointers, runs wrong code -> blue (frame advancing, no crash, no
exception, 0 misses). The registration outlived its source bytes. Primary fix
is Inc1-D write-invalidation; A/B confirm = rerun with overlay_cache=false.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restores entry-scoped overlay analysis (analyze_exact_entries +
split_mid_function_targets=false) as the MANDATORY mode for overlay inputs,
renamed --overlay-exact -> --overlay: the ingester's contract, set
unconditionally by compile_overlays.py for every overlay, never a human toggle.
Evidence-scoped discovery compiles only proven seeds + reachable code and keeps
branch/jump-table targets as in-parent labels — no whole-byte sweep, so it no
longer decodes embedded data tables as code.
Effect on the captured Tomba overlay: the data-as-code explosion collapses from
thousands of UNSUPPORTED to 122 clean functions with 42 residual unsupported
opcodes + 8 bad targets. Those residuals are now a per-function-audit concern
(Inc1-B), not a wholesale region failure. Main-EXE codegen is unchanged (Tomba
still regenerates 2854 functions; split pre-pass still runs in non-overlay mode).
Also adds docs/overlay-recompilation-design.md — the converged design of record
(precision-over-recall; compiled!=registered!=callable; write-invalidation over
CRC; internal-labels-in-parent; the RAM-write-path audit result; the three
implementation increments).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reinstates the runtime-loaded overlay DLL cache (Layer A) and the offline
overlay compilation pipeline that were built on the pruned codex/cache-discovery
branch (tip 2bbafc2) and never merged. Ported onto current master; rebuilt
rather than history-restored.
Runtime (off by default, opt in via [runtime] overlay_cache = true):
- overlay_loader.c/.h, overlay_api.h: CRC32-keyed DLL cache at
cache/<game_id>/<region>_<crc>.dll. On a dirty-RAM dispatch miss the loader
LoadLibrary's the matching DLL, wires runtime callbacks via overlay_init(),
enumerates func_XXXXXXXX exports, and dispatches them BEFORE the interpreter.
- overlay_capture.c/.h: write-once capture set; records overlay DMA bytes and
writes overlay_captures.json (execution-verified seeds) at clean exit.
- dma.c: capture + cd_dma_log hook at CD DMA completion (async model).
- cdrom.c: cdrom_get_setloc_lba() for disc-position correlation.
- dirty_ram_interp.c: g_dirty_ram_exec_pc_table (exec-verified PCs), a
decodability guard on JAL-into-dirty targets, and the overlay dispatch hooks.
- debug_server.c: cd_read_log + overlay_loader_status TCP commands.
- config_loader: [runtime] overlay_cache flag (default false).
Recompiler (compile_overlays.py / extract_overlays.py drive these):
- main_psx.cpp --overlay-exact: analyze only explicit entries + direct JAL
targets and set split_mid_function_targets=false, so overlay basic-block
targets are NOT split into bogus standalone functions (the mid-function-seed
failure mode that produced black-on-load / blue-on-new-game).
- function_analysis.cpp: analyze_exact_entries(); code_generator.cpp: gate the
mid-function split pre-pass behind split_mid_function_targets (default true,
so normal main-EXE codegen is unchanged - Tomba still regens to 2854 funcs).
Deltas from the original: save states dropped (never worked); overlay enable is
runtime-config-gated (off by default) instead of a build-time define; loader
stderr logging converted to the overlay_loader_status TCP path (CLAUDE.md Rule 3).
Recovered from _backups/2026-06-06-consolidation/psxrecomp-all-PRE.bundle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs/disc-speed.md: design doc for per-game disc speed configuration.
docs/overlay-discovery.md: design doc for B+A overlay pre-compilation
and runtime JIT caching (game-side, future work).
tools/dump_tomba_misses.py: one-shot dump of dirty_ram per_pc entries
in the game text range, formatted as seed lines.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Tight-crop (1031x315) so the header isn't dominated by empty grey
above/below the logo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Config loader (recompiler/src/config_loader.{h,cpp}): single source of
truth for both BIOS and game runtime settings (debug_port, window_title,
memcard_dir, disc path). psxrecomp-bios and psxrecomp-game both accept
--config <toml>; runtime takes --game <toml> (no more CLI overrides).
- runtime/src/main.cpp: replaced --disc/--memcard-dir/--game-root CLI
overrides with --game <toml>; game.toml is the single source of truth.
Discless launch (no args) goes to BIOS shell.
- runtime/src/dma.c: fixed update_master_irq() re-asserting I_STAT bit 3
on every DICR read. BIOS shell DMA handler at 0x80058628 only clears
DICR bit 27 (ch3 flag), so level-sensitive re-assert pinned the kernel
in the exception loop. Bit 3 is now edge-triggered (one assert per DMA
completion). Discless boot now reaches MAIN MENU / MEMORY CARD / CD
PLAYER. With-disc Tomba boots through to Whoopee Camp splash.
- Audit tooling: codegen_audit.py (BIOS), codegen_audit_game.py (game),
audit_config.py (config loader for tools), verify_audit_schema.py.
Replaces older tools/_codegen_audit.py probe.
- Docs: CURRENT_STATE.md (project state), UNIFICATION_PLAN.md (Phase C
translator merge plan), docs/config_schema.md (TOML schema reference),
docs/audit_inventory.md (audit tier inventory).
- .gitignore: tools/_*.py convention for ad-hoc scratch probes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Before: only sector 0 read succeeded; sector ≥1 reads aborted at 2 bytes
because mc_flag was loaded from mc_slots[].flag on every 0x81 select but
never saved back. The clear in MC_CMD lived only in the working static, so
every reselect re-loaded the stale 0x08 ("new card / changed battery") init
value, and the BIOS bailed every multi-sector op as "card was just inserted,
re-init."
After: BIOS shell memcard menu reads the full 15-sector directory area for
both slots. Real save filenames decode (e.g. "BISLPS-01234SAVE00") and per
card icons render in the bottom panes — the Phase 4 success criterion.
Also bundled this session:
- debug_server: B9D0..B9F0 added to always-on wtrace ranges; fn_dump_parse
default window scoped to 1M most-recent entries (filter scans against the
full 128M ring used to freeze the debug server thread for many seconds).
- ISSUES.md + diagnostic tools/*.py written during the audit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the Beetle-discovered chain writer/init PCs as dispatch_miss seeds
so the recompiler emits standalone functions for them at their RAM
destination addresses (via the existing Phase 1e ROM↔RAM alias rule
RAM = ROM - 0xBFC10000 + 0x500):
0xBFC14A54 → 0x00004F54 (chain BUSY-writer)
0xBFC158D0 → 0x00005DD0 (chain init slot-0 BUSY)
0xBFC158D8 → 0x00005DD8 (chain init slot-1 BUSY)
0xBFC159F4 → 0x00005EF4 (chain step 0x02-OPEN writer — KEY)
0xBFC15AA8 → 0x00005FA8 (chain step 0x04-OPEN writer — KEY)
Post-regen dispatch_table contains all 5 as standalone entries. e.g.
func_00005EF4 is a clean 3-instruction leaf (sb $t3, 0($a3); v0=1; jr ra).
Validation (full output in docs/CASE_A_AOT_GAP.md "Vertical slice result"):
- ✓ dispatch_table contains all 5 entries
- ✗ none of the 5 are reached at runtime — chain still doesn't dispatch
- ✗ [0x80007510..0x7570] all-zero — the chain init function FUN_bfc158a8
(= func_00005DA8) never runs in our recomp during this session, gates
haven't even reached the initial 0x01 BUSY state
- ✓ shell IS on memcard screen ([0x80066948] = 0x32)
The seed addition is correct and necessary work — it removes one of two
layers of "writer is unreachable". The remaining layer is upstream:
nothing in our recomp causes FUN_bfc14b00 / FUN_bfc158a8 to run during
the memcard flow. That's a different category from "chain runs but
fails on byte N" and is not chased here per Case A directive.
Beetle wtrace also revealed the writers at 0x5EF4 / 0x5FA8 are tiny
leaf utilities called from BIOS code in BFC08{B98,C94,D08,F28} — NOT
chain-step jump-table targets as initially assumed. The chain step
tables at RAM 0x6c70/0x6c98/0x6ccc point at 0x52xx / 0x56xx-0x5Axx /
0x5Bxx-0x5Dxx ranges. Memo updated accordingly.
Includes 4 new validation scripts under tools/ used during this audit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a RAM-write trace hook to mednafen-PSX (SB/SH/SW handlers fire
g_psxrecomp_wtrace_cb after each write), a ring buffer + TCP commands
in psx-beetleoracle, and verification scripts that drive CROSS in
both Beetle and recomp and diff the gate-byte writers.
Conclusion (docs/CASE_A_AOT_GAP.md): the [0x7568+slot] LSB-clear
writers are at RAM PCs 0x5EF4 / 0x5FA8 (and 0x4F54 sets BUSY).
Recomp never dispatches any of those PCs, never executes the
0x4F00..0x4FFF or 0x5800..0x5FFF RAM windows, and never writes the
gate. Selective coverage pattern (interp reaches 0xCF0/0x52xx/0x6xxx
but not 0x4Fxx/0x5xxx) is install/dispatch path gap, not interpreter
bug. PLAN.md:941 already names this work as a hard gate.
Beetle-side modifications live inside the vendored beetle-psx/ git
clone; docs/beetle_wtrace_hook.patch captures the diff so they
survive a re-clone.
Memcard symptom debugging stops here. Next: trace-generated AOT
RAM-code manifest + recompiler ingestion (Phases B–F).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Scaffold for guest-cycle-driven peripheral timing per
docs/CYCLE_TIMING_ARCH.md. The counter is defined and exposed via
freeze_check, but psx_advance_cycles has no callers yet — counter
stays 0. Validation: zero behavior change.
Files:
- runtime/include/psx_cycles.h: extern uint64_t psx_cycle_count;
void psx_advance_cycles(uint32_t cycles); uint64_t psx_get_cycle_count(void);
- runtime/src/psx_cycles.c: scaffold body; peripheral hooks empty
(1.0e-b will add sio_advance, 1.0e-c will add gpu_advance, etc.)
- runtime/CMakeLists.txt: psx_cycles.c added to PSX_SHARED_SOURCES
- runtime/src/debug_server.c: freeze_check exposes psx_cycle_count
- docs/CYCLE_TIMING_ARCH.md: design doc — single cycle clock,
generic per-block emit (no card-specific paths), MMIO-split codegen
per explicit user direction, IRQ-pending-vs-CPU-entry separation,
five-step migration plan (1.0e-a..e)
Validation against 0d15e2b baseline:
- t=20s/30s tx_writes match (1160/5480)
- 4x CROSS succeeds; card txn distribution unchanged (10 reach READ_DATA)
- "MC" bytes still flow at txn 0 bytes 10-11
- psx_cycle_count = 0 throughout (no callers in this slice)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Real card protocol restarts on every fresh 0x81 — SELECT deassert is
a bus-wide reset signal. Earlier model preserved mc_state across
deassert via mc_save_slot, which caused every subsequent 0x81 to land
on a stale non-IDLE saved state and trigger abort_reselect bookkeeping.
Audit confirmed the BIOS-visible RX was still correct (the abort path
itself reset mc_state), but the retained sector/cmd/counter context
across logically-independent transactions was wrong per no$psx.
Clear volatile protocol state on SELECT deassert (state, cmd, sector,
sector_msb, sector_lsb, data_idx, checksum on both slots). Preserve
mc_slots[].flag (persistent "new card" metadata) and mc_slots[].data
(sector buffer, harmless to retain).
Result on a 4-press boot run: 6 transactions reach READ_DATA (ts=10)
and successfully read the "MC" header magic (RX bytes 0x4D 0x43);
remaining txns continue as 4-byte presence pings. Pre-fix: 0 reads
of card data ever; post-fix: card protocol can complete the setup
phase and enter data phase.
The byte-15 abort that follows is a separate blocker — kernel chain
dispatcher interleaves pad polling into the card data read. Tracked
for next session via the chain dispatcher disassembly.
Also includes:
- Slot-state capture in SioTraceEntry + sio_trace JSON emission
- dirty_ram_interp: replace mid-block fprintf+fflush with a counter
(CLAUDE.md §3 — synchronous Windows console I/O at high rate
starves the dispatch loop)
- Passive Python TCP probes (raw_tcp, trace_audit, trace_range,
txn_summary, txn_dist, txn_bytes, drs_filter, readram_test,
step_audit{,2}, verify_card_buffer, wtrace_filter)
- docs/MEMCARD_AUDIT_2026_04_28.md (prior session's 7-layer audit)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matthew Stanley