Commit Graph

47 Commits

Author SHA1 Message Date
Claude a6ee29a4ea readme: Kula World bring-up on Linux — title screen + the 5 fixes
Adds this fork's Kula World (SCES-01000) end-to-end target to the README with
a title-screen screenshot and a summary of the five runtime bugs fixed during
bring-up (four of them game-agnostic framework improvements): the dirty-RAM
interpreter interrupt-latency fix, the main-RAM mirror, GetID region
derivation, CD status bits, and the SCPH-1001 CD controller version.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-03 06:07:31 +00:00
Claude 590c236933 dirty_ram_interp: poll interrupts at interpreter entry — fixes Kula World wedge (title screen renders)
ROOT CAUSE of the Kula World black-screen wedge: a guest wait loop (libcd's
CdSync spin) that lives entirely in dirty RAM re-enters the dirty-RAM
interpreter once per short (~7-insn) block. The interpreter's only interrupt
poll was gated by the PER-INVOCATION (insns_executed & 0xFFF) counter, which
short blocks never reach — and when the loop exits to dispatch each iteration
that path isn't reached at all. So the interpreter ran millions of
instructions with psx_check_interrupts never called, while SR (IEc+IM2) and
Cause (IP2) fully satisfied the MIPS interrupt-take condition and i_stat had
CDROM+VBLANK pending. The CD IRQ that sets libcd's wait-flag was never
serviced; the loop spun ~14s per cycle until a rare timeout re-dispatched.

Fix: poll interrupts at the interpreter's per-invocation entry, gated by a
global invocation counter (every 64 entries), cpu->pc set for a coherent
resume — the interpreter analogue of a static block-leader poll.

RESULT: Kula World now boots to its title screen and menu. Diagnosed with the
self-stop trap + gdb sampling of SR/Cause/i_stat + the interrupt gate globals
(in_exception, total_checks) + g_dirty_ram_insns_run, which isolated 'CPU
spinning in the interpreter without polling interrupts'. General fix: any
dirty-RAM spin-loop now yields to interrupts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-03 06:02:35 +00:00
Claude 7c8298b12e docs: round 7 — remaining Kula wedge is unserviced CD IRQ (libcd CdSync never completes)
With the boot fixed, live state during the game's retry loop shows the CDROM
interrupt is raised (i_stat bit2) AND enabled (i_mask bit2) and the CPU takes
exceptions (VBLANK cleared each frame), yet the CDROM bit is never cleared and
the response FIFO never drained (irq_flag=0x03, response_read=0). The game's
libcd CD interrupt handler in the BIOS IntRP chain is not servicing the CD
IRQ, so CdSync never completes. Distinct from the CD-response-value bugs fixed
today; a BIOS-IRQ-chain / libcd-handler matter. Documents the next step and
the full session tally in docs/kula-world-findings.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 23:13:55 +00:00
Claude a3f57ec570 cdrom: Test 0x20 reports SCPH-1001 CD version (0x94..), removing spurious boot ReadTOC
The shell CD-init (ROM 0x1DF50) sets kernel flag [0xA000DFFC]=1 when the CD
controller version high byte (Test sub-cmd 0x20) is >= 0x95; the boot CD-open
(0xBFC0D570) then issues a ReadTOC gated on that flag. We returned the
PSone-era 0x97 0x01 0x10 0xC2 (copied from beetle, which hardcodes it) — wrong
for the SCPH-1001 BIOS, whose 1994 sub-CPU reports 0x94 0x09 0x19 0xC0. With
the correct value the flag stays clear, the spurious frame-890 ReadTOC is
gone, and the boot command stream matches the oracle exactly. Found via
self-stop trap + fntrace-all + gdb watchpoint on [0xDFFC] + capstone tracing
the flag setter and its version check.

Also adds a symmetric self-stop trap to psx-beetle's CD-command hook so the
oracle can be frozen at the same boot point for state diffs.

Honest status: this closes the boot-flow divergence but the game still wedges
downstream in its OWN CD-init retry (GetStat/Init loop from game pc 0x63A14);
that is a separate issue, documented in docs/kula-world-findings.md round 6.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 23:10:31 +00:00
Claude 536ad0f886 debug: PSX_FNTRACE_ALL boot trace; root mechanism of Kula bug 3 found
Adds PSX_FNTRACE_ALL=1 (records every dispatch from power-on) and documents
the full mechanistic chain of the wedge (docs/kula-world-findings.md round 5):

The frame-890 ReadTOC that breaks the game's CD reads is gated by kernel flag
[0xA000DFFC]: the CD-open function 0xBFC0D570 does 'beqz [0xDFFC], skip;
jal ReadTOC'. On ours [0xDFFC]=1 -> ReadTOC -> wedge; on the oracle it stays
0 -> reads continue. The flag is set to 1 by func_1FC1DF50 (a CD control
routine) called from shell code at ra=0x80036024. The shells run the same
verbatim bytes but diverge early (ours 0x35000-0x39000 issuing Test/ReadTOC,
oracle 0x30D00-0x33200 issuing Init/GetTN), so a branch input differs. Traced
via self-stop trap + fntrace-all + gdb watchpoint on [0xDFFC] + capstone.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 20:48:04 +00:00
Claude 698e100816 oracle: add guest PC to CD-command trace hook
Extends psxrecomp_cdcmd_cb_t (and the beetle_cdcmd_trace ring + cdrom_cmd_dump
JSON) with the guest PC at dispatch, read via PS_CPU::GetRegister(GSREG_PC).
Lets the boot-command-stream oracle-diff localize the first divergent
command's call site (kernel vs shell vs game) instead of only the command
byte — the next step in isolating the early-boot branch divergence behind
Kula World bug 3. Builds clean; behavior-inert when no listener is attached.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 20:24:50 +00:00
Claude c0ea49832a cdrom: fix 0x1F801800 status bits per oracle (spurious ADPBUSY, missing BUSYSTS)
Bit 2 is ADPBUSY (XA-ADPCM playback in progress) and must idle at 0; we
raised it unconditionally under a mistaken 'ADPCM empty' reading. Bit 7
(BUSYSTS: command written, first response not yet ready) was never raised;
now mapped to the queued-command window, mirroring beetle's
PendingCommandCounter/phase busy window. Verified against beetle cdc.cpp
Read(A==0), which sets bit 2 never and bit 7 during command latency.

Boot-diff verification shows this does NOT change our boot command stream
(the early-boot branch divergence vs the oracle persists — documented in
docs/kula-world-findings.md round 4, with the next sharp step: add guest
PC to the oracle's cdcmd hook and diff the first divergent call site).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 20:18:24 +00:00
Claude 9fee0b5dc0 cdrom: PSX_CD_TRAP_CMD self-stop trap; boot flows diverge from power-on
New debug tool: PSX_CD_TRAP_CMD=<byte> [PSX_CD_TRAP_NTH=<n>] SIGSTOPs the
process the instant the Nth matching CD command dispatches, so a debugger
attaches to the exact moment with zero run-speed cost (a gdb conditional
breakpoint on exec_command slows the emu ~100x and never reaches the wedge).

Used it to pin Kula bug 3 one level deeper: the pre-wedge ReadTOC at frame
~890 is issued by the BIOS BOOT FLOW itself (guest stack shows only
Main/StartKernel-chain ROM addresses, zero game frames) — the boot's
post-EXE-load disc re-check. And the full command-history dump shows our
boot diverges from the oracle's from the VERY FIRST command: the oracle's
kernel opens with Init/Init/GetTN/GetID and never issues Test or ReadTOC;
ours opens with the shell's Test[20]/GetID/ReadTOC check and re-checks
after the EXE load. Same SCPH1001 bytes, different path — an early-boot
branch divergence (recompiled-BIOS or hardware-sim input), documented in
docs/kula-world-findings.md for the first-divergence session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 20:05:05 +00:00
Claude 8861d3e963 docs: sharpen Kula bug 3 — spurious BIOS disc re-identify at frame 890
Full oracle command trace confirms it NEVER issues ReadTOC; ours does, from
the BIOS (Pause->ReadTOC->GetID) at frame 890, with clean pre-read stat
(0x22, no error/shell bits). Same BIOS bytes on both, so our runtime reaching
the disc-identify path is a state-driven BRANCH divergence, not a CD-sim
error: likely a kernel 'disc already identified' flag left unset, or a
recompiler-level game-logic divergence. Next: instruction-level
first-divergence hunt around frame 890.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 19:11:35 +00:00
Claude 99a4002ffb oracle: CD-command trace hook + live-diff localizes Kula World bug 3
Adds a CD-command trace to psx-beetle (beetle_cdcmd_trace_hook.patch:
callback in cdc.cpp's command dispatch, wired through beetle_libretro.cpp
into cdrom_cmd_dump / cdrom_cmd_reset on the oracle debug server). This is
the two-process oracle-diff the setup is built for, now working on Linux.

Live-diff result (docs/kula-world-findings.md, bug 3): the oracle's game
reads level data with a clean Setloc->SeekL->Setmode->ReadN->Pause per
sector (SeekL x18, ReadN x22, GetlocL x89). Our runtime's game, in the
wedge, issues NEITHER SeekL NOR ReadN — only GetStat/Init/Setloc/Pause,
an error/retry branch the oracle never enters.

Crucial: our runtime DID deliver 307 sectors during early boot (ReadN +
data delivery work). Reads only stop at frame ~890 — the BIOS mid-game
region revalidation (ReadTOC+GetID). The region fix made that PASS, but
the post-ReadTOC CD state still stops the game resuming reads. Prime
suspect: our ReadTOC (case 0x1E) is a bare stat+COMPLETE stub that does
not model drive repositioning / the track table the game reads to
recompute its seek. Documented for the next session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 19:05:47 +00:00
Claude 74b3c36779 docs: refine Kula World bug 3 — CD responses are correct, retry loop points at timing/event
The live trace shows the game's CD-init state machine (GetStat/Init then
Setloc/Pause, ~2.5s cadence) receives CORRECT responses at every step —
stat=0x02 (healthy idle drive), INT3 ACK + INT2 COMPLETE both fire and the
game's ISR processes them — yet it retries forever. So bug 3 is NOT a
wrong-response-value bug like bug 2; the game is waiting on a timing or
event-completion condition its init check never sees satisfied. Documents
two next-session paths: (1) rebuild psx-beetle with a CD-command hook and
live-diff the boot->demo sequence, (2) find the game's init-complete branch
condition and check our INT2-COMPLETE delivery timing against it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 18:50:53 +00:00
Claude 3c54c646f6 docs: Kula World bring-up findings (mirror + region fixed, demo-load wedge open)
Records the two fixed bugs (RAM mirror, GetID region) and characterizes
the open third wedge: the game's own CD driver loops Setloc(PVD)/Pause
without ever reading the directory, so the demo level never loads and
the screen stays black. The oracle loads it, so the divergence is
concrete and localized for the next oracle-diff session.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 18:28:27 +00:00
Claude 21e57a82cc docs+patch: psx-beetle oracle on Linux (pinned beetle-psx base + SIO trace hook)
beetle_libretro.cpp targets the C++ beetle-psx tree; upstream master has
been converted to plain C and no longer compiles against it. Documented
the working recipe: pin beetle-psx at 5759277b (the wtrace patch's base,
located via the patch's own blob hash), apply beetle_wtrace_hook.patch,
and apply the new beetle_sio_trace_hook.patch — FrontIO::SetSIOTraceCallback
(fires once per completed SIO byte exchange), a hook beetle_libretro.cpp
required that was missing from the committed patch set.

Verified on Linux/Xvfb: psx-beetle boots Kula World to demo gameplay and
serves ping/screenshot over the shared TCP debug protocol, alongside
kula-runtime — the two-process oracle setup from CLAUDE.md §16 now works
in this environment. Runtime-created per-game .mcr files are gitignored.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BdjxoRB1f9VQsNzZayCgGS
2026-07-02 17:08:19 +00:00
Matthew Stanley 01b4d91e8a WIP: forward-port Ape Escape framework (dev/ape-escape) onto master
Comprehensive 3-way merge of the dev/ape-escape branch into master for
the isolated wt/ape-escape worktree. Brought in every complementary ape
change automatically (dma.c, memory.c interrupt byte/half lanes,
function_analysis discovery, main_bios vector seeds, iso_reader
iso_sector_count, config_loader, crash-trace fields, headers). Resolved
the 13 conflicts by rule:

  - CPS dispatch / recompiler (code_generator, full_function_emitter,
    dirty_ram_interp, traps, main_psx) -> KEEP MASTER (evolved system).
  - CD model (cdrom.c) -> take ape (the model that booted Ape).
  - interrupts.c/.h -> reverted to MASTER (ape's interrupt-delivery is
    dispatch-coupled; the merged Frankenstein stormed).
  - sio/debug_server/freeze_heartbeat/main.cpp -> keep master (evolved).
  - crash_trace.c -> master (ape freeze-dump fns omitted to avoid the
    debug-ring cascade).

STATUS: does NOT boot yet. ape-cdrom + master-interrupts derails to
PC=0 at frame ~2092 (insta-close); ape-cdrom + ape-interrupts storms
(297M exception re-entries) at frame 2077. Root finding: ape's CD
anti-storm is coupled to ape's interrupt-delivery + call-depth + EPC
model, which master replaced with CPS continuation-passing, so the CD
subsystem does not forward-port as pure hardware simulation. Preserved
as the reference state for a design consult on bridging the two models.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 15:55:23 -07:00
Matthew Stanley 7281319005 recursion: make CPS the DEFAULT (opt-out via PSX_CPS=0)
The continuation-passing leak fix (RECURSION_BUG.md §25) is validated (Tomba boots+plays,
ce_profile flat through a 56.9k-frame soak, full overlay CPS). Flip the gate in all three
emitter sites so a normal regen emits CPS; PSX_CPS=0 is the legacy escape hatch. Verified:
default regen -> CPS markers present; PSX_CPS=0 -> none; default-CPS Tomba boots clean, leak flat.
Doc §25.6.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 14:10:44 -07:00
Matthew Stanley 5c893d472a overlay(async): gcc-only on gcc boxes (mutually exclusive with sljit) + async-compile is built
The user-requested async overlay tiering (interp-now / compile-async / native-next, gcc-if-present
else sljit, always async, file-based handoff) is LARGELY ALREADY IMPLEMENTED — overlay_capture.c
(autocapture on interp pressure) + autocompile.c (background compile via a configured command,
toolchain-probed, watcher thread, file-based) + the code_provider abstraction. It just needed
CPS wiring.

This commit:
- overlay_loader_apply_live_policy: when the active backend is gcc, force sljit-live OFF so a gcc
  box is gcc>interp (NOT the previous complementary gcc+sljit). Toolchain-less boxes stay
  sljit>interp. PSX_OVERLAY_SLJIT_LIVE still overrides. ("mutually exclusive" per user.)
- ASYNC_OVERLAY_COMPILE.md: re-headed to reflect the feature is built+validated under CPS, with
  the deltas this session added (compile_overlays --cps from §25.5, the gcc-only policy, the
  dev-config overlay_autocompile_cmd --cps).

VALIDATED under CPS: autocapture fires -> background compile_overlays --cps produces CPS overlay
DLLs -> loader loads them -> dispatch_native rises; gcc box shows 0 sljit shards; ce_profile flat.
The native overlay cache auto-builds as you play (each scene compiles on its first sustained
visit). PRODUCTION game.toml overlay_autocompile_cmd still needs --cps appended when CPS ships.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:54:57 -07:00
Matthew Stanley d99323c72b docs: async background overlay compilation — implementation plan
Build plan for the next session: interp-now / compile-async / native-next overlay tiering
(gcc-if-toolchain-present else sljit, mutually exclusive, always async). FILE-BASED handoff —
the worker writes .dll/.sljit artifacts; the dispatch fiber loads them via the existing
try_load_region / deserialize paths, so the candidate table stays single-threaded (no race).
Motivated by the CPS title screen being interp-bound (12.2M interp vs 14.8K native) because its
overlays aren't gcc-compiled + the sync sljit JIT-on-miss hitches the dispatch thread.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:40:45 -07:00
Matthew Stanley 2d468ee5b8 recursion(stage3): overlay (sljit + gcc-DLL) CPS parity — full native overlay coverage under CPS
RECURSION_BUG.md §25.5. Brings the overlay native path under continuation-passing so the
idle-freeze leak fix holds with overlay_cache=true (the production gcc>sljit>interp path).

Runtime CPS-mode flag g_psx_cps_mode (overlay_loader.c) set by a __attribute__((constructor))
emitted in the BIOS + game CPS dispatch (full_function_emitter.cpp / main_psx.cpp). Gates all
overlay/sljit CPS behavior; legacy unaffected.

sljit JIT (overlay_sljit.c), scoped:
- jr $ra publishes cpu->pc (was a bare return). jr rX already tail-transferred.
- call-containing fragments DECLINE to the interpreter (CPS-safe, depth-1) — leaf shards still
  JIT. (Full sljit call coverage / per-fragment entry-switch is future work.)

gcc-DLL path — FULL native overlay coverage (validated):
- compile_overlays.py --cps sets PSX_CPS for the recompiler -> CPS overlay C (tail-transfer +
  per-function entry-switch; --overlay mode only affects widescreen, not CPS). DISPATCH_PREAMBLE
  psx_syscall decl fixed void->int.
- PSX_OVERLAY_CODEGEN_VER 3->4: fresh cg4/ cache namespace; legacy unit-model DLLs (cg3) ignored.
- overlay_loader_dispatch: under CPS, when a dispatched addr is not an overlay ENTRY but lies
  inside a registered candidate's code range (a tail-transferred call's continuation),
  overlay_find_by_range() re-enters that function with cpu->pc=addr so its entry-switch routes
  to the block. gcc DLLs are the trusted tier (diff-validated at entry) -> run native directly.
  This is the CPS equivalent of the old unit-model that fixed the dwarf->overworld blue screen.

Validated: CPS overlay DLLs compile + load + dispatch_native>0; with continuation routing
dispatch_interp_fallback -> 0 (full native), ce_profile FLAT (6 KB), boots clean. Addresses are
KSEG0-consistent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 13:35:31 -07:00
Matthew Stanley 94a1966444 recursion(fix): universal continuation-passing — FIXES the idle-freeze host-stack leak (BIOS+game)
The complete §25 fix: make BOTH recompiler emitters emit guest calls as
TAIL-TRANSFERS driven by the unified flat trampoline + the guest's own $ra/$sp,
instead of nesting one host C frame per guest call. The host stack can no longer
mirror unbounded guest re-entrancy — the architectural root of the ~14-min idle
freeze. Gated gen-time by PSX_CPS; legacy codegen byte-identical when unset.

VALIDATED (user): full-CPS Tomba boots BIOS -> FMV -> New Game -> village idle and
plays; ce_profile cur_max_kb FLAT at 5 KB from frame ~9.9k through 56,890 (turbo
soak) — blew past the ~50k-frame freeze threshold with the host stack bounded
(legacy climbed +1.17 KB/frame -> 48 MB -> freeze).

KEY CORRECTION to the prior session's §25.1: there are TWO emitters. The BIOS
(full_function_emitter.cpp) was done last session but the §8 leaking chain
(func_8001A954 ...) is GAME code, emitted by code_generator.cpp (via main_psx.cpp)
— never converted. Also, §25.1's "continuations not in the dispatch table"
hypothesis was wrong (they were always added); the real boot regressions were
found by dumping the always-on fntrace ring on the abnormal psx_dispatch return.

Emitter changes (gated PSX_CPS):
- full_function_emitter.cpp (BIOS): BIOS A0/B0/C0 vector runtime-fallback
  tail-transfers instead of nested psx_dispatch_call (was the frame-0 exit).
- code_generator.cpp/.h (game): jal/jalr/jr-$ra/jr-table/branch-split/j-split ->
  tail-transfer (jalr captures rs before the link, alias-safe); each call's
  return point registered as a dispatchable continuation; entry-switch (keyed on
  cpu->pc) at the top of every function AND every overlapping-alias body. 0
  psx_call_contract / call_by_address emitted under CPS.
- main_psx.cpp: psx_dispatch_game_compiled gains continuation cases
  (case cont: cpu->pc=cont; func_owner(cpu)); entries clear cpu->pc.
- strict_translator.cpp: syscall emits `if (psx_syscall(...)) return;` so a void
  syscall falls through to the inline post-syscall jr $ra (real-HW EPC+4).

Runtime changes (the legacy "return via nested dispatch / cpu->pc=0" idioms that
exit the flat CPS trampoline):
- traps.c: psx_syscall returns int (0=void/fall-through, 1=transfer);
  Enter/ExitCriticalSection return 0; case-3 ChangeThread-success returns 0 — the
  fiber scheduler leaves cpu->pc=0 on resume, which was the frame-712 exit.
- cpu_state.h / overlay_api.h: psx_syscall signature void->int; overlay callback
  ABI bumped 4->5 (stale overlay caches rejected).
- dirty_ram_interp.c: gate interp_enter_compiled behind PSX_HAS_GAME_DISPATCH
  (un-breaks the BIOS-only build).
- main.cpp: dump the always-on fntrace ring tail + stack to psx_cps_exit_trace.json
  on an abnormal psx_dispatch return (diagnostic; CLAUDE.md ring-buffer model).

Docs: RECURSION_BUG.md §25.2-§25.4. Remaining: dwarf->overworld playtest;
Stage 3 (sljit/overlay CPS parity); then consider making CPS default.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 12:45:02 -07:00
Matthew Stanley e28c2a0ff9 recursion(fix): WIP universal continuation-passing call/return (gated PSX_CPS) — Stage 1 emitter
The universal fix for the idle-freeze host-stack leak (RECURSION_BUG.md §25): make
the recompiler emit guest calls as TAIL-TRANSFERS driven by the trampoline + the
guest's own $ra/$sp, instead of nesting one psx_dispatch_impl per guest call frame.
This makes the host stack incapable of mirroring unbounded guest re-entrancy — the
architectural root of the leak — for ANY game. Perf cost accepted ("right before
fast").

Stage 1 (full_function_emitter.cpp), GATED behind gen-time env PSX_CPS:
- jal/jalr (orphaned-delay + pending-delay paths): emit `cpu->pc=target; return;`
  (tail-transfer) + register the return address as a dispatchable continuation,
  instead of nested psx_dispatch + psx_call_contract. JALR captures the target reg
  BEFORE the link write (rd==rs alias-safe).
- jr $ra (all 3 sites): publish `cpu->pc = cpu->gpr[31]; return;` under CPS so the
  trampoline dispatches the caller's continuation (legacy `return;` relied on the
  nested-dispatch C-return). Required — without it the BIOS exits at frame 0.

GATING PROVEN SAFE: regen+build WITHOUT PSX_CPS boots Tomba normally — default
codegen byte-unaffected; CPS fully opt-in.

OPEN (CPS path only): with PSX_CPS=1 the BIOS boots main() then exits at frame 0
(execution completed, PC=0). Root cause (high confidence): the trampoline must
dispatch mid-function continuation addresses, but the dispatch table holds only
function entries → a return to a continuation misses → psx_unknown_dispatch →
pc=0 → exit. NEXT: add registered continuations to the dispatch table (map
continuation_addr → containing func), then re-regen PSX_CPS=1 and run the §25 gate
(ce_profile flat + oracle-identical + 15-min soak + dwarf->overworld). Stages 2
(interp interp_enter_compiled surface) + 3 (sljit/overlay parity) follow.

Safe revert point for the whole feature: commit ae47e69.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 08:25:39 -07:00
Matthew Stanley ae47e69902 recursion(oracle): oracle disambiguation DONE + SP-fntrace instruments; verdict = §11 universal fix (docs §23–§24)
Oracle disambiguation (RECURSION_BUG.md §23–§24). Traced one func_8001A954
invocation on real HW (psx-beetle) vs the recomp in the IDENTICAL Tomba idle
state (New Game village dialogue; object 0x801FD950, state/scene/jumptable=0).

VERDICT: real HW's per-frame chain is BALANCED — func_8001A954 is entered once
per frame from main_loop (JAL 0x8001A630, sp=0x801FE3E0) and RETURNS once per
frame (jr ra @0x8001A9E8 -> 0x8001A638, SP fully restored); no re-entry, the
overlay 0x80063864 never calls back (refutes §21/§22's "call-cycle"). The recomp
runs the same guest flow and the guest SP unwinds identically, but the HOST stack
leaks +1.17 KB/frame (ce_profile, live-reconfirmed: 20099->22215 KB / 1807 frames)
because the per-frame compiled chain (func_8001A954->AC00->B2B4->B5A8->func_80046264,
emitted as DIRECT func_X(cpu) host calls) re-enters itself via func_80046264's
CD-DMA-dirty hop and never unwinds.

native_stack early-trip (PSX_STACK_GUARD_KB=6144, frame 7592, 6 MB) pins the
leaking host cycle. Cross-checked: xprobe mixed_depth bounded at 6, dirty-insn/
dirty-block show NO literal func_8001A954 transfer, and a954=1/frame is a SURFACED
guest tail-transfer (j/jr) to func_8001A954 that the oracle proves real HW never
makes — i.e. there is both (A) a correctness divergence (spurious per-frame
re-entry) and (B) the architectural host-mirroring limitation that amplifies it.

Fix scope CORRECTED from §23's "quick contained runtime patch": the UNIVERSAL fix
is §11 — route in-range guest calls through the depth-managed psx_dispatch_impl
trampoline instead of mirroring them as direct host calls, so the host stack stays
bounded regardless of guest re-entrancy (real or fabricated). §12 (CD-DMA dirty
precision) and fixing the specific divergence are surgical mitigations, not
universal. Performance loss accepted for now (static recomp: right before fast).

Instruments added (runtime-only, no regen):
- SP capture in the fntrace control-flow ring on BOTH backends (GPR[29] read live):
  psx-beetle (beetle_libretro.cpp + beetle_debug_server.c) and the recomp
  (fntrace.h/.c + debug_server.c) — enables the func_8001A954 SP/RA-lifecycle oracle.
- psx-beetle default debug port 4380 -> 4382 (+ --port N override): 4380 collided
  with psxref and cdirecomp/CdiRuntime.exe; two LISTENers on one port silently
  misroute connections (the phantom "oracle keeps dying" early in the session).

Oracle tooling: the control-flow oracle is psx-beetle (Beetle interprets -> records
every J/JAL/JR/JALR), NOT psxref (no PC trace; the assumed emu_step/emu_trace_addr
never existed). Loads Tomba via: psx-beetle.exe --disc tomba.cue <bios> --port 4382.

Next: implement the §11 universal fix (codegen + full regen).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 08:03:04 -07:00
Matthew Stanley fd2570ff13 recursion(doc): §22 — fix (A) interp-call surface validated INSUFFICIENT, reverted
Pinned the exact leaking crossing (xprobe detail): entry to func_8001A954 is a single
JAL, src 0x8001A630 (the main loop's only call site), once/frame. Implemented (A):
surface interp JAL/JALR to game-text targets to the owner trampoline (gated by
psx_game_address_in_text so overlay-native keeps the nesting contract). §19 result: leak
STILL CLIMBS (~0.875 KB/frame, down from 1.17) -> reduced, not eliminated -> REVERTED.

It is a guest CALL-CYCLE (main_loop 0x8001A51C <-> func_8001A954): surfacing the interp
side just relocates the nest to the compiled-side jal back to the main loop. Can't fix a
call-cycle by surfacing one crossing. Leak is ~1 host frame PER FRAME (not per cycle
iteration) -> the loop mostly returns; exactly one call/frame fails to unwind. Real HW
idles forever (balanced), so either we mishandle one specific return/frame (Hyp C,
contained) or it needs the §11 continuation-passing redesign. DECISIVE NEXT = oracle
(psxref): does real HW return from func_8001A954 to 0x8001A634 each frame, or never?
Stop surfacing single crossings (block-loop §20 + JAL/JALR §22 both dead ends).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 00:57:46 -07:00
Matthew Stanley add57f5560 recursion: SOLVE root (gradual per-frame host-stack leak) + instruments + pin exact crossing
The ~14-min Tomba idle freeze is a gradual per-frame host-stack LEAK (not a
within-one-frame spiral, not the gradual-marking theory). Confirmed via a new
compiled-entry depth profile (§19): per-frame MAX guest-stack-used climbs dead-linear
+1.17 KB/frame from gameplay start to 48 MB on the 64 MB main-thread stack -> native
stack guard -> freeze. Once per frame an interpreted overlay (reached via
func_80046264's dirty hop jal 0x80063864) does jalr func_8001A954, nesting a host call
that never unwinds (the per-frame loop never returns). Full writeup RECURSION_BUG.md
§16-§21.

Instruments (always-on rings, runtime-only, no regen):
- §17 stack_profile: per-vblank host-stack sample (main.cpp; stack_profile TCP cmd).
- §18 xprobe / xprobe_arm: interp<->compiled boundary crossing flight recorder
  (dirty_ram_interp.c; per-frame summary + per-crossing detail).
- §19 ce_profile: per-compiled-entry depth profile + raw TEB at the trip
  (debug_server.c; ce_profile TCP cmd) — the trustworthy instrument; the regression
  check for the fix (max_kb must stay FLAT).
- PSX_STACK_GUARD_KB early-trip + PSX_FIBER_STACK_KB (debug_server.c / traps.c).

Fix progress:
- Block-loop tail-transfer surface (dirty_ram_interp.c ~1514): surface compiled
  tail-transfers to the trampoline instead of nesting. Correct for tail-transfers but
  RULED OUT as the leak (§19 climb unchanged) — the leak is the CALL path (jalr/jal).
- Real fix (next): surface compiled-game CALLS to the owner trampoline, gated to
  compiled-game targets only (a naive all-targets surface reintroduces the
  dwarf->overworld lost-continuation bug). Needs a non-nesting membership test
  (generated psx_dispatch_game_has) or §12 CD-DMA dirty-marking precision.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-18 00:15:15 -07:00
Matthew Stanley 08484afc68 recursion: flight-recorder runs #1/#2 findings + fix buffer overflow (doc §16)
Run #1: interp->0x8001A954 edge is ORDINARY (1/frame). Run #2: dispatch_8001A954=0
(dirty_ram_dispatch never called with 0x8001A954) AND recursion_func VARIES
(0x8004DEE0/0x8004DFA0/0x8005FF60) -> address-specific counters are the wrong
instrument. dirty_block_tail shows the real cycle: dirty_ram_dispatch(overlay
0x8011-0x8013) called from main-EXE 0x8002-0x8005 = the per-frame MESSAGE/RENDER
overlay system recursing. SITE_CAP 256->32 fixes the report-JSON overflow. Pivot:
use data-driven rings (native_stack/dirty_block_tail/dispatch_tail) from a clean
report, not guessed addresses; then last-writer slice on the loop termination var.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 21:12:34 -07:00
Matthew Stanley 2475ee0550 recursion: re-entry flight recorder + strategy reset (doc §15)
Per-frame count of the re-entry edge interp->func_8001A954 (the per-frame loop
head) + cycle counter, in a 256-frame ring the crash report dumps
(dirty_ram_re954_json). Answers ChatGPT's step-1 question: is the re-entry
ORDINARY bounded per-frame behavior that stops terminating at ~50k, or a new edge
— and whether cycles advance (it's within ONE frame; 23k laps ~= 23k cycles <<
565k/frame, so no VBlank before overflow => control-flow cycle, not starvation).

docs/RECURSION_BUG.md §15 captures the full strategy reset (stop RAM-diffing;
triggered control-flow recorder + backward last-writer slice; narrow oracle use
only on the resulting small address set). The separately-navigated oracle RAM-diff
(§14) was a dead end (alignment noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 20:42:06 -07:00
Matthew Stanley afd843cdc9 docs(recursion): CORRECT the dialogue-trigger error — it is the ~14-min IDLE freeze (frame ~50k, location-independent); the boundary fix is symptom-only; NEXT = oracle idle-diff (turbo both, find first divergence vs real HW). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> 2026-06-17 20:01:22 -07:00
Matthew Stanley 0d3a1623e6 docs(recursion): full root-cause + fix design + oracle plan
Capture everything for recall in a future session:
- Resolved the overworld-load regression (stale build artifact, not tooling/master).
- 3 organic captures (deterministic: v0=0x8004DEE0, ra=v0+8, pc=0; both New-Game
  and Dwarf-Forest reproduce).
- The contract mechanism (S7): psx_dispatch_impl tail-call trampoline + psx_call_contract
  bail keep COMPILED tail-loops flat; the dirty interpreter bypasses them
  (exec_one JAL/JALR + dirty_ram_dispatch_inner:~1179 nest psx_dispatch_game_compiled).
  Proof it's host-only: guest sp flat (~7KB) across ~23k laps.
- Ghidra map of the guest state machine (S8): func_8001A954 state[0x4a]==1 ->
  func_8001AC00 chain -> func_80046264 jumptable (selector @0x8009BCDD, table
  @0x80013C60) -> func_8004DFA0 (a render handler that advances no selector).
- External-review verdict (S9): Hypothesis A overwhelmingly likely; B has no
  positive evidence; C (interp control-transfer bug fabricating the re-entry) must
  be excluded by an oracle PC-trace.
- Oracle disambiguation plan (S10), the single mixed-dispatch-owner fix design
  (S11, runtime-only / no regen), CD-DMA dirty-marking critique (S12), and the next
  disassembly target around 0x80063864 (S13).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 18:12:17 -07:00
Matthew Stanley d9b6ae5e16 recursion: capture native_stack tooling + working doc
Snapshot of the uncommitted native_stack capture tool (crash_trace.c,
append_native_stack) that recovers the true compiled<->dirty-interp
recursion cycle the recent_fn ring can't show, plus docs/RECURSION_BUG.md
documenting the freeze root cause, the 4-instance soak reproduction
harness, and the overworld-load regression now blocking repro.

This is the build that currently insta-crashes soaks on overworld load;
preserved as-is before hardening so the regression stays reproducible.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 16:02:14 -07:00
Matthew Stanley cb00b9c0af feat(psx): verified-enhancement shadow — SPU audio + 15-bit CRT LUT
SPU Catmull-Rom float shadow (spu_shadow), 15-bit CRT color LUT (raw bit-exact), verifier. Opt-in (env-gated), default OFF: raw output byte-identical and remains the verify oracle. Reverts loud (DEGRADED) on divergence. ShadowVerifier ported from JRickey/gba-recomp (MIT OR Apache-2.0). Compiles clean; full game-build A/B is the remaining integration gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 04:25:56 -07:00
Matthew Stanley 70c1eb8a51 Merge master (Bug B pacing fix + overlay cache campaign) into dev/ape-escape
Brings the ape-escape framework (real-EPC interrupt delivery,
g_psx_call_depth safe-point gating, continuation-entry codegen, DualShock
controller config, exception/entryint/irq-check rings, interp
dispatch_until + exit logging + static-text fallback) up to master
035a9fa (frame-pacing TOCTOU fix, vsync self-heal, watchdog classifier,
overlay JIT cache + autocapture/autocompile, Bug A interp call contract,
prod build hygiene).

Key reconciliations (ambiguity resolved toward master per plan):
- interrupts.c: real-EPC delivery (psx_check_interrupts_at) is the
  primary entry; the legacy psx_check_interrupts wrapper keeps the
  0x80000048 sentinel fallback (ape tip restored it), so old-flavor
  generated code keeps working interrupt delivery. Master's bottom
  compat shim removed (would have recursed). Both observability paths
  kept (event-ring gate outcomes + ape irq-check ring; new
  GATE_CALL_DEPTH/GATE_BAD_RESUME kinds).
- Claimed-IRQ cooldown: master's per-source CLAIMED_PROGRESS_QUANTUM=8
  (SIO exempt) over ape's unconditional 0; ape's 2026-06-05 starvation
  note kept as a MERGE NOTE — first knob to A/B if title-LOAD regresses.
  (User-validated: Tomba title-LOAD slot1/slot2 renders and plays on
  this merge — the historical ape-fw blackout is gone.)
- dirty_ram_interp.c: ape's dispatch_until_impl structure (stop_addr,
  static-text fallback, exit-log ring) with master's Bug A call contract
  (psx_dispatch_game_compiled + overlay_loader_call_native at interp
  jal/jalr), overlay loader/fingerprint entry checks, freeze-gated flow
  ring, chain-target attribution, and interp-active bracketing on the
  public wrappers. Ape tip's final local-flow rule kept: dirty game-text
  NEVER chains locally (the tip reversed its own earlier widening).
- freeze_heartbeat.c: classifiers composed — per-frame storm threshold
  (master) + progress-counter gating and 6s window (ape) + slow_frames
  kind (master), thresholds rescaled to the window.
- code_generator.cpp: ape's continuation-entry switch + master's
  refactored scan_jr_tables.
- debug_server.c: ape originals win over master's ported copies
  (bioscall ring, card trace with the richer address set); master-only
  tools kept (dirty_*_dump_file, card_trace filter handler).
- dma.c/cdrom.c: master's overlay-capture path + ape's DMA/command
  history diagnostics, both wired.

Builds: recompiler + Tomba runtime (legacy gen x merged runtime).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-09 22:05:13 -07:00
Matthew Stanley 26d03e643a runtime: fix Bug A — interp jal/jalr must honor the call contract for native overlay callees
Root cause of the dwarf-village->overworld native-overlay blue screen
(deterministic, every run, at native call #70):

The dirty-RAM interpreter's jal/jalr handlers preserved the call's
return-address obligation for statically-compiled callees
(psx_dispatch_game_compiled -> *next_pc_out = return_pc) and for non-local
targets (dispatch_nonlocal_call passes return_pc as stop_addr), but a LOCAL
DIRTY target fell through to a bare `cpu->pc = target; return 1`. When that
target was a NATIVE overlay candidate, the dispatch loop ran it as a C
function; its pc==0 return made psx_dispatch_impl unwind to the C caller,
silently dropping the suspended interpreted caller's continuation.

Concrete case: main-EXE field dispatcher F=0x800338A8 (undiscovered entry —
discovery walked it from 0x338B0; reached via data-table jalr, so it runs
under the interpreter) does `jal 0x8011B0D4` per entity. Under native exec
F's epilogue never ran: guest sp leaked 0x18 per entity until the world
corrupted. Interp-only mode is immune (the callee's own jr ra is interpreted).
Evidence chain (full-register fingerprint diff ON vs OFF localized first
divergence to sp at exec #70; frozen dirty-block ring shows F re-entered at
sp-0x18 with no epilogue between) is in docs §9 and the session memory.

Fix: overlay_loader_call_native() — run a validated native candidate as a
unit and resume at return_pc — called from both jal and jalr handlers,
mirroring the psx_dispatch_game_compiled contract exactly. Side effect:
native overlay coverage rose from 86 calls per village->overworld run to
~112,000 (interpreted calls now route to native candidates).

Verified: dwarf->overworld renders and plays under native overlays.
Same-scene A/B: host CPU 7.31 (native) vs 7.83 (interp) CPU-s per 15s wall.
Timing hypotheses refuted by experiment before fixing: IRQ-check cadence
(suppress armed, identical failure) and device-event granularity
(conservative stepping, identical failure).

Also in this commit (the capture tooling that found it):
- fingerprint ring now stores full entry/exit register files + guest cycle
  (FP_CAP 2^20 crc-only -> 2^16 full-reg); overlay_fp_dump takes a path.
- insn_freeze <addr> <nth>: one-shot latch freezing insn/block/flow/fntrace
  rings immediately BEFORE the Nth dispatch of a watched candidate, so the
  pre-divergence window survives until dumped (ring-first, no arm-then-hope).
- dirty_insn_gate <lo> <hi>: runtime-settable extra PC range for the per-insn
  interp log; native entry/exit markers (transferred=200/201) in that ring.
- dirty_insn_dump_file: file dump for the insn ring — the 16MB inline TCP
  dump wedged the single-threaded debug server.
- freeze gating in the flow-log path gates ONLY the ring write, never flow.
- debug_client plumbing for the new commands + fntrace_arm/fntrace_dump.
- docs/overlay-recompilation-design.md §9: the dispatch call-contract
  invariant, the Bug A case study, verification numbers, and known residue
  (data-jump-table discovery-miss class; one unreproduced black-screen
  one-off; §8.3 tail-position caveat).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-09 12:25:03 -07:00
Matthew Stanley 20976b1993 overlay: mark OV-1 fixed (Inc1-D verified) + flag Inc2 reload-on-return next
OV-1 resolved by mechanism: invalidations=1 on the BIOS overlay-load write into
0xE7000, 88 funcs unregistered, stale_blocked into the tens of millions (every
one a prevented stale native call). Correctness only so far — native~122 vs
interp~31M — so Inc2 reload-on-return is flagged as the next/biggest benefit
lever.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 15:46:06 -07:00
Matthew Stanley 42246da229 overlay: status doc + OV-1 blue-screen root cause (stale registration)
Documents what works (capture, evidence-scoped --overlay, boundary gate, clean
88-func DLL, in-village native dispatch) and the open OV-1 issue: blue screen on
village->overworld transition.

OV-1 root-caused via live RAM-vs-captured comparison at the blue screen: the
front of the 0xE7000 region was overwritten (compiled-func addresses now hold
pointer tables: 0xE73C0/0xE7000/0xE8120 DIFFER, 0x100000/0x13B000 MATCH) while
the loader kept all 88 village functions registered (regions_checked=1, never
re-evaluated). The game dispatches into changed content, hits stale village
function pointers, runs wrong code -> blue (frame advancing, no crash, no
exception, 0 misses). The registration outlived its source bytes. Primary fix
is Inc1-D write-invalidation; A/B confirm = rerun with overlay_cache=false.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 14:29:46 -07:00
Matthew Stanley 00b1b02a07 overlay: evidence-scoped --overlay recompiler mode (Inc1-A) + design doc
Restores entry-scoped overlay analysis (analyze_exact_entries +
split_mid_function_targets=false) as the MANDATORY mode for overlay inputs,
renamed --overlay-exact -> --overlay: the ingester's contract, set
unconditionally by compile_overlays.py for every overlay, never a human toggle.
Evidence-scoped discovery compiles only proven seeds + reachable code and keeps
branch/jump-table targets as in-parent labels — no whole-byte sweep, so it no
longer decodes embedded data tables as code.

Effect on the captured Tomba overlay: the data-as-code explosion collapses from
thousands of UNSUPPORTED to 122 clean functions with 42 residual unsupported
opcodes + 8 bad targets. Those residuals are now a per-function-audit concern
(Inc1-B), not a wholesale region failure. Main-EXE codegen is unchanged (Tomba
still regenerates 2854 functions; split pre-pass still runs in non-overlay mode).

Also adds docs/overlay-recompilation-design.md — the converged design of record
(precision-over-recall; compiled!=registered!=callable; write-invalidation over
CRC; internal-labels-in-parent; the RAM-write-path audit result; the three
implementation increments).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 13:42:10 -07:00
Matthew Stanley b2929aa504 overlay: restore overlay DLL cache + capture (faithful port of codex/cache-discovery)
Reinstates the runtime-loaded overlay DLL cache (Layer A) and the offline
overlay compilation pipeline that were built on the pruned codex/cache-discovery
branch (tip 2bbafc2) and never merged. Ported onto current master; rebuilt
rather than history-restored.

Runtime (off by default, opt in via [runtime] overlay_cache = true):
- overlay_loader.c/.h, overlay_api.h: CRC32-keyed DLL cache at
  cache/<game_id>/<region>_<crc>.dll. On a dirty-RAM dispatch miss the loader
  LoadLibrary's the matching DLL, wires runtime callbacks via overlay_init(),
  enumerates func_XXXXXXXX exports, and dispatches them BEFORE the interpreter.
- overlay_capture.c/.h: write-once capture set; records overlay DMA bytes and
  writes overlay_captures.json (execution-verified seeds) at clean exit.
- dma.c: capture + cd_dma_log hook at CD DMA completion (async model).
- cdrom.c: cdrom_get_setloc_lba() for disc-position correlation.
- dirty_ram_interp.c: g_dirty_ram_exec_pc_table (exec-verified PCs), a
  decodability guard on JAL-into-dirty targets, and the overlay dispatch hooks.
- debug_server.c: cd_read_log + overlay_loader_status TCP commands.
- config_loader: [runtime] overlay_cache flag (default false).

Recompiler (compile_overlays.py / extract_overlays.py drive these):
- main_psx.cpp --overlay-exact: analyze only explicit entries + direct JAL
  targets and set split_mid_function_targets=false, so overlay basic-block
  targets are NOT split into bogus standalone functions (the mid-function-seed
  failure mode that produced black-on-load / blue-on-new-game).
- function_analysis.cpp: analyze_exact_entries(); code_generator.cpp: gate the
  mid-function split pre-pass behind split_mid_function_targets (default true,
  so normal main-EXE codegen is unchanged - Tomba still regens to 2854 funcs).

Deltas from the original: save states dropped (never worked); overlay enable is
runtime-config-gated (off by default) instead of a build-time define; loader
stderr logging converted to the overlay_loader_status TCP path (CLAUDE.md Rule 3).

Recovered from _backups/2026-06-06-consolidation/psxrecomp-all-PRE.bundle.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-07 12:32:19 -07:00
Matthew Stanley ec7e482691 Ape Escape support: framework WIP snapshot (safe haven)
Uncommitted framework work on codex/ape-escape-support reaching the
post-boot DualShock-required screen. Recompiler codegen (code_generator,
full_function_emitter, function_analysis, config_loader) + runtime
(cdrom, sio, dirty_ram_interp, dma, iso_reader, mdec, gpu, memory,
traps, debug_server, fntrace, freeze_heartbeat, psx_cycles) and headers.

Committed as a restore point before rebuilding Tomba against this
framework. Build dirs (recompiler/build-codex*) intentionally excluded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-04 12:48:38 -07:00
Matthew Stanley de4808d65e Move disc_speed docs to psxrecomp README; add B+A as #3 in bios-hardening
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 00:13:44 -07:00
Matthew Stanley 27860746b4 Add bios-hardening.md: outstanding work tracker
Documents all outstanding items from the overlay-discovery session:
- disc_speed "instant" hang root cause (binary search next step)
- dispatch hit/miss counters (not yet implemented)
- exception chain continuations 0x0DF8/0x0E08 (low priority)
- trampoline resolution logging gap
- Tomba game dispatch miss seeding (with mid-function safety warning)

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 00:04:22 -07:00
Matthew Stanley 2aab23968b Add design docs and Tomba miss dump tool
docs/disc-speed.md: design doc for per-game disc speed configuration.
docs/overlay-discovery.md: design doc for B+A overlay pre-compilation
  and runtime JIT caching (game-side, future work).
tools/dump_tomba_misses.py: one-shot dump of dirty_ram per_pc entries
  in the game text range, formatted as seed lines.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-05-30 00:02:33 -07:00
Matthew Stanley 4706680c1c README: add PSXRecomp logo at top
Tight-crop (1031x315) so the header isn't dominated by empty grey
above/below the logo.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 12:31:11 -07:00
Matthew Stanley 2f709ee542 Phase B: config-driven unified runtime + BIOS shell boot restored
- Config loader (recompiler/src/config_loader.{h,cpp}): single source of
  truth for both BIOS and game runtime settings (debug_port, window_title,
  memcard_dir, disc path). psxrecomp-bios and psxrecomp-game both accept
  --config <toml>; runtime takes --game <toml> (no more CLI overrides).
- runtime/src/main.cpp: replaced --disc/--memcard-dir/--game-root CLI
  overrides with --game <toml>; game.toml is the single source of truth.
  Discless launch (no args) goes to BIOS shell.
- runtime/src/dma.c: fixed update_master_irq() re-asserting I_STAT bit 3
  on every DICR read. BIOS shell DMA handler at 0x80058628 only clears
  DICR bit 27 (ch3 flag), so level-sensitive re-assert pinned the kernel
  in the exception loop. Bit 3 is now edge-triggered (one assert per DMA
  completion). Discless boot now reaches MAIN MENU / MEMORY CARD / CD
  PLAYER. With-disc Tomba boots through to Whoopee Camp splash.
- Audit tooling: codegen_audit.py (BIOS), codegen_audit_game.py (game),
  audit_config.py (config loader for tools), verify_audit_schema.py.
  Replaces older tools/_codegen_audit.py probe.
- Docs: CURRENT_STATE.md (project state), UNIFICATION_PLAN.md (Phase C
  translator merge plan), docs/config_schema.md (TOML schema reference),
  docs/audit_inventory.md (audit tier inventory).
- .gitignore: tools/_*.py convention for ad-hoc scratch probes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 01:31:27 -07:00
Matthew Stanley e88e9a0233 phase 4: fix mc_flag persistence so multi-sector card reads work
Before: only sector 0 read succeeded; sector ≥1 reads aborted at 2 bytes
because mc_flag was loaded from mc_slots[].flag on every 0x81 select but
never saved back. The clear in MC_CMD lived only in the working static, so
every reselect re-loaded the stale 0x08 ("new card / changed battery") init
value, and the BIOS bailed every multi-sector op as "card was just inserted,
re-init."

After: BIOS shell memcard menu reads the full 15-sector directory area for
both slots. Real save filenames decode (e.g. "BISLPS-01234SAVE00") and per
card icons render in the bottom panes — the Phase 4 success criterion.

Also bundled this session:
- debug_server: B9D0..B9F0 added to always-on wtrace ranges; fn_dump_parse
  default window scoped to 1M most-recent entries (filter scans against the
  full 128M ring used to freeze the debug server thread for many seconds).
- ISSUES.md + diagnostic tools/*.py written during the audit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-03 16:43:07 -07:00
Matthew Stanley 87d99d4f1d seeds: 5 dispatch_miss entries for chain writer PCs (vertical slice partial)
Adds the Beetle-discovered chain writer/init PCs as dispatch_miss seeds
so the recompiler emits standalone functions for them at their RAM
destination addresses (via the existing Phase 1e ROM↔RAM alias rule
RAM = ROM - 0xBFC10000 + 0x500):

  0xBFC14A54 → 0x00004F54 (chain BUSY-writer)
  0xBFC158D0 → 0x00005DD0 (chain init slot-0 BUSY)
  0xBFC158D8 → 0x00005DD8 (chain init slot-1 BUSY)
  0xBFC159F4 → 0x00005EF4 (chain step 0x02-OPEN writer — KEY)
  0xBFC15AA8 → 0x00005FA8 (chain step 0x04-OPEN writer — KEY)

Post-regen dispatch_table contains all 5 as standalone entries. e.g.
func_00005EF4 is a clean 3-instruction leaf (sb $t3, 0($a3); v0=1; jr ra).

Validation (full output in docs/CASE_A_AOT_GAP.md "Vertical slice result"):
- ✓ dispatch_table contains all 5 entries
- ✗ none of the 5 are reached at runtime — chain still doesn't dispatch
- ✗ [0x80007510..0x7570] all-zero — the chain init function FUN_bfc158a8
  (= func_00005DA8) never runs in our recomp during this session, gates
  haven't even reached the initial 0x01 BUSY state
- ✓ shell IS on memcard screen ([0x80066948] = 0x32)

The seed addition is correct and necessary work — it removes one of two
layers of "writer is unreachable". The remaining layer is upstream:
nothing in our recomp causes FUN_bfc14b00 / FUN_bfc158a8 to run during
the memcard flow. That's a different category from "chain runs but
fails on byte N" and is not chased here per Case A directive.

Beetle wtrace also revealed the writers at 0x5EF4 / 0x5FA8 are tiny
leaf utilities called from BIOS code in BFC08{B98,C94,D08,F28} — NOT
chain-step jump-table targets as initially assumed. The chain step
tables at RAM 0x6c70/0x6c98/0x6ccc point at 0x52xx / 0x56xx-0x5Axx /
0x5Bxx-0x5Dxx ranges. Memo updated accordingly.

Includes 4 new validation scripts under tools/ used during this audit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 14:26:19 -07:00
Matthew Stanley f8841ead30 case A: Beetle wtrace + recomp verification → memcard blocked on AOT gap
Adds a RAM-write trace hook to mednafen-PSX (SB/SH/SW handlers fire
g_psxrecomp_wtrace_cb after each write), a ring buffer + TCP commands
in psx-beetleoracle, and verification scripts that drive CROSS in
both Beetle and recomp and diff the gate-byte writers.

Conclusion (docs/CASE_A_AOT_GAP.md): the [0x7568+slot] LSB-clear
writers are at RAM PCs 0x5EF4 / 0x5FA8 (and 0x4F54 sets BUSY).
Recomp never dispatches any of those PCs, never executes the
0x4F00..0x4FFF or 0x5800..0x5FFF RAM windows, and never writes the
gate. Selective coverage pattern (interp reaches 0xCF0/0x52xx/0x6xxx
but not 0x4Fxx/0x5xxx) is install/dispatch path gap, not interpreter
bug. PLAN.md:941 already names this work as a hard gate.

Beetle-side modifications live inside the vendored beetle-psx/ git
clone; docs/beetle_wtrace_hook.patch captures the diff so they
survive a re-clone.

Memcard symptom debugging stops here. Next: trace-generated AOT
RAM-code manifest + recompiler ingestion (Phases B–F).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-30 13:10:05 -07:00
Matthew Stanley 5d4d4bd30b phase1.0e-a: psx_cycle_count scaffold + cycle-timing arch doc
Scaffold for guest-cycle-driven peripheral timing per
docs/CYCLE_TIMING_ARCH.md. The counter is defined and exposed via
freeze_check, but psx_advance_cycles has no callers yet — counter
stays 0. Validation: zero behavior change.

Files:
- runtime/include/psx_cycles.h: extern uint64_t psx_cycle_count;
  void psx_advance_cycles(uint32_t cycles); uint64_t psx_get_cycle_count(void);
- runtime/src/psx_cycles.c: scaffold body; peripheral hooks empty
  (1.0e-b will add sio_advance, 1.0e-c will add gpu_advance, etc.)
- runtime/CMakeLists.txt: psx_cycles.c added to PSX_SHARED_SOURCES
- runtime/src/debug_server.c: freeze_check exposes psx_cycle_count
- docs/CYCLE_TIMING_ARCH.md: design doc — single cycle clock,
  generic per-block emit (no card-specific paths), MMIO-split codegen
  per explicit user direction, IRQ-pending-vs-CPU-entry separation,
  five-step migration plan (1.0e-a..e)

Validation against 0d15e2b baseline:
- t=20s/30s tx_writes match (1160/5480)
- 4x CROSS succeeds; card txn distribution unchanged (10 reach READ_DATA)
- "MC" bytes still flow at txn 0 bytes 10-11
- psx_cycle_count = 0 throughout (no callers in this slice)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 19:04:11 -07:00
Matthew Stanley 8b0fc34714 phase4: SELECT-deassert protocol reset + slot-state instrumentation
Real card protocol restarts on every fresh 0x81 — SELECT deassert is
a bus-wide reset signal. Earlier model preserved mc_state across
deassert via mc_save_slot, which caused every subsequent 0x81 to land
on a stale non-IDLE saved state and trigger abort_reselect bookkeeping.
Audit confirmed the BIOS-visible RX was still correct (the abort path
itself reset mc_state), but the retained sector/cmd/counter context
across logically-independent transactions was wrong per no$psx.

Clear volatile protocol state on SELECT deassert (state, cmd, sector,
sector_msb, sector_lsb, data_idx, checksum on both slots). Preserve
mc_slots[].flag (persistent "new card" metadata) and mc_slots[].data
(sector buffer, harmless to retain).

Result on a 4-press boot run: 6 transactions reach READ_DATA (ts=10)
and successfully read the "MC" header magic (RX bytes 0x4D 0x43);
remaining txns continue as 4-byte presence pings. Pre-fix: 0 reads
of card data ever; post-fix: card protocol can complete the setup
phase and enter data phase.

The byte-15 abort that follows is a separate blocker — kernel chain
dispatcher interleaves pad polling into the card data read. Tracked
for next session via the chain dispatcher disassembly.

Also includes:
- Slot-state capture in SioTraceEntry + sio_trace JSON emission
- dirty_ram_interp: replace mid-block fprintf+fflush with a counter
  (CLAUDE.md §3 — synchronous Windows console I/O at high rate
  starves the dispatch loop)
- Passive Python TCP probes (raw_tcp, trace_audit, trace_range,
  txn_summary, txn_dist, txn_bytes, drs_filter, readram_test,
  step_audit{,2}, verify_card_buffer, wtrace_filter)
- docs/MEMCARD_AUDIT_2026_04_28.md (prior session's 7-layer audit)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 14:46:26 -07:00
Matthew Stanley 5b3c7dd08d phase4: freeze_check diagnostic + accumulated overnight infra
Today's session work
- runtime/src/debug_server.c: freeze_check TCP cmd snapshots all liveness
  counters (PC, exception entries, exception_reentry_blocks, dirty_ram
  blocks/insns, fn_entry total + top-K histogram, SIO IRQ pending +
  countdown, sio_stat/i_stat/i_mask, mc_max_state) for tight-loop
  diagnosis when the runtime hangs
- runtime/src/debug_server.c: wtrace slot 12 covering [0x72F0..0x72F4]
  (data-byte counter); measured peak=1 confirming setup_data_phase resets
  the counter mid-cycle
- runtime/src/interrupts.c: exception_entries_total + exception_reentry_blocks
  counters + psx_get_freeze_diag accessor
- runtime/src/sio.c: sio_get_freeze_diag accessor for IRQ pacing internals
- memory entries: phase4_card_gate_setup_writes_2026_04_28 (refutes the
  prior \"missing [0x7568+slot] clearer\" framing — setup functions write
  LSB=0 directly), phase4_72f0_one_byte_per_vblank_2026_04_28 (peak=1
  measurement)

Overnight session work bundled in
- runtime/src/dirty_ram_interp.{c,h}: small MIPS interpreter for
  install-at-runtime kernel RAM stubs (Rule 18; e.g., SIO data-byte
  handler the BIOS writes to RAM 0xCF0)
- recompiler/src/full_function_emitter.{cpp,h}: 3057 jal/jalr continuation
  return labels added to dispatch table (continuation fix); branch
  fall-through fix for conditional-branch delay slots
- recompiler/src/strict_translator.cpp: matching translator updates
- generated/: regenerated SCPH1001 dispatch + full C output reflecting
  the recompiler changes
- runtime/include/debug_server.h, src/memory.c: card_txn ring + SIO IRQ
  ring + EvCB walk ring + per-PC dirty_ram counters + always-on wtrace
  coverage for [0x755A], [0x7514], [0x7528], [0x7520], [0x74A4],
  [0x75C0..C8], [0x74B8..C0]
- recompiler/tests/, runtime/tests/: SIO/card protocol unit tests
- tools/: 60+ Python diagnostic scripts (chain_trace analysis, SIO trace
  diff, EvCB walker, wtrace dumpers, etc.)
- docs/: psx_bios_disasm reference + dynamic_handler_install rationale
- OVERNIGHT_2026_04_28.md, SUMMARY_20260412_2.md: session logs

Phase 4 status (still blocked, root cause better understood)
- BIOS card chain runs through R-handlers correctly (chain_trace shows
  R1..R11 cycling 26x); shell IS retriggering _card_read
- Install handler at ROM 0xBFC15F1C increments [0x72F0] correctly per
  SIO IRQ; gate at <0x80 / <0x7F is correct
- Bottleneck: SIO IRQ countdown is access-paced, fires ~1× per VBlank
  vs ~128× needed; setup_data_phase (called from outer coord on VBlank)
  zeroes [0x72F0] before counter reaches 128
- Both attempted fixes today caused freezes:
  * VBlank deferral on [0x75C0]==1 — pending VBlank already in i_stat
    delivered anyway; tighter dispatch gate also locked up
  * Cycle-paced sio_tick at rates 50, 200 (and gated on [0x75C0]==1) —
    runtime freezes once data phase activates, root cause unidentified
- delay=5 narrows the threshold (peak=2 vs peak=1, no freeze) but
  doesn't get to 0x80
- Real fix likely requires deeper instrumentation of the freeze (call
  stack / dispatch state at the moment of TCP starvation)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 21:29:19 -07:00