diff --git a/docs/RECURSION_BUG.md b/docs/RECURSION_BUG.md index 2e039e75..8601b976 100644 --- a/docs/RECURSION_BUG.md +++ b/docs/RECURSION_BUG.md @@ -931,3 +931,153 @@ contained interp/dispatch fix), or does it genuinely never return (→ §11 rede prior surface attempts (block-loop §20, JAL/JALR §22) are proven dead ends; stop surfacing single crossings. The §19 `ce_profile` flat-vs-climb remains the objective check; the self-driven screenshot-validated soak remains the harness. + +--- + +## 23. ORACLE DISAMBIGUATION DONE — VERDICT: contained return-contract fix, NOT §11 (2026-06-18) + +Ran the oracle disambiguation (only). Both sides traced **in the identical idle state** +(Tomba New Game, starting village, "Hey, you with the pink hair!" dialogue; object +`*(0x1F8001D4)=0x801FD950`, `state/scene/jumptable` all 0). Artifacts in +`_freeze_specimens/oracle/` (`oracle_chain_filtered.json`, `oracle_state.txt`, +`recomp_allfn_frame.json`, `recomp_ce_profile.json`). + +**Tooling built (CLAUDE.md §15 — the oracle had a real gap).** psxref has **no** +instruction/PC trace (`emu_step`/`emu_trace_addr` never existed — the handoff/§10 assumed +them). The right oracle is **`psx-beetle.exe`** (shares the wire protocol; Beetle PSX core +INTERPRETS, so its always-on `g_psxrecomp_fntrace_cb` records **every J/JAL/JR/JALR** +with caller/target/ra/a0/a1/kind). Loads Tomba via `--disc tomba.cue `. Added **SP +capture** to the fntrace ring on BOTH backends (read `GPR[29]` live) — `beetle_libretro.cpp` ++ `beetle_debug_server.c` and recomp `fntrace.c`/`fntrace.h`/`debug_server.c`; runtime-only, +no regen. Also moved psx-beetle's default debug port **4380 → 4382** (`--port N` override): +4380 collided with psxref AND `cdirecomp/CdiRuntime.exe`, and two LISTENers on one port +silently route connections to the wrong process — that dual-listener was the whole +"oracle keeps dying / unknown cmd" confusion early in the session, NOT a real crash. + +**ORACLE (real HW), one `func_8001A954` invocation — perfectly balanced, once/frame:** +``` +JAL 0x8001A630 -> 0x8001A954 ra=0x8001A5DC sp=0x801FE3E0 main_loop calls A954 (ret=0x8001A638) +JAL 0x8001A9B8 -> 0x8001AC00 ra=0x8001A638 sp=0x801FE3C8 ┐ +JAL 0x8001AC4C -> 0x8001B2B4 ra=0x8001A9C0 sp=0x801FE3A8 │ nested chain, +JAL 0x8001B3E4 -> 0x8001B5A8 ra=0x8001AC54 sp=0x801FE390 │ SP descends +JAL 0x8001B750 -> 0x80046264 ra=0x8001B73C sp=0x801FE378 │ +JAL 0x80046274 -> 0x80063864 ra=0x8001B758 sp=0x801FE360 ┘ (the dirty hop) +JR 0x8001A9E8 -> 0x8001A638 sp=0x801FE3E0 A954 RETURNS, SP fully restored +``` +- `func_8001A954` is entered **exactly once/frame** (`JAL` from `0x8001A630`) and **returns + exactly once/frame** (`jr ra` at `0x8001A9E8` → `0x8001A638`), with **SP back to its entry + value `0x801FE3E0`** ⇒ the whole subtree unwinds every frame. +- **NO re-entry of `func_8001A954` before its exit** (armed on `0x8001A954`; the only hit per + frame is the `0x8001A630` JAL). **The overlay `0x80063864` NEVER calls back into + `func_8001A954`** — §21's "interp overlay does `jalr func_8001A954`" is **REFUTED on real + HW**. There is **no guest call-cycle**: main_loop is a per-frame VSync callback that calls + A954 and A954 returns; §22's "`jal back to main_loop`" was a misread of the recomp's + forward-dispatched returns (below). + +**RECOMP (`build-recursion`, `PSX_MIXED_OWNER=0`), same idle — IDENTICAL guest flow, but +every boundary-crossing RETURN is a forward `psx_dispatch`:** +``` +... render leaf 0x8004DEE0 (ra=0x8004DEE8) ×5 ... +psx_dispatch 0x800462C4 sp=0x801FE360 (80046264 subtree returns) +psx_dispatch 0x8001B758 sp=0x801FE378 (80046264 -> B5A8 return) +psx_dispatch 0x8001B3EC sp=0x801FE390 (B5A8 -> B2B4 return) +psx_dispatch 0x8001AC54 sp=0x801FE3A8 (B2B4 -> AC00 return) +psx_dispatch 0x8001A9C0 sp=0x801FE3C8 (AC00 -> A954 return) +psx_dispatch 0x8001A638 sp=0x801FE3E0 (A954 -> main_loop return; SP restored) +``` +- The **guest** stack unwinds **identically** to the oracle (SP climbs back to `0x801FE3E0`, + same addresses, same RA). The down-chain calls (A954→AC00→…→80046264) are compiled DIRECT + calls; the render leaves are interpreted overlays (`0x8011xxxx`/`0x8012xxxx`/`0x8006Bxxx`, + ra `0x8001E0D8`). When those interpreted/dirty frames **return** to a compiled main-EXE + address, the dirty-RAM dispatch realizes the `jr ra` as a **forward `psx_dispatch(ret)`** + that NESTS `psx_dispatch_game_compiled` instead of surfacing to the trampoline owner. +- **Live leak confirmed** (`ce_profile`): `max_kb` 20099→22215 over 1807 frames = + **+1.17 KB/frame**, `entries/frame` constant ~1300, deepest func the render cluster + (`0x8004DEE0`/`0x8005FF60`/`0x8005FA60`). Identical to §20. + +**VERDICT (decisive):** +1. **No guest control-flow divergence.** Targets, RA, delay-slot results and SP are + bit-identical on both sides. ⇒ Hypothesis **C-as-interp-fabrication is RULED OUT** (the + interp does NOT mishandle a JAL/JALR/delay-slot/RA; it does not fabricate the re-entry). +2. **Beetle UNWINDS each invocation; the recomp leaves the chain SUSPENDED** (ChatGPT + determination **#1**), because the recomp **maps the guest's nonlocal-but-balanced returns + onto accumulating host calls** (determination **#2**). The guest **genuinely returns every + frame** (proven by SP on BOTH sides) → this is a **CONTAINED return/unwind-contract bug**, + **NOT** the §11 continuation-passing redesign. +3. **The leaking edge is the RETURN path, not the call path.** The boundary crossings that + leak are the up-chain `jr ra` returns from interpreted/dirty code into compiled main-EXE + functions: `0x800462C4 / 0x8001B758 / 0x8001B3EC / 0x8001AC54 / 0x8001A9C0 / 0x8001A638`. + This is **why §20 (block-loop tail-transfer) and §22 (interp JAL/JALR down-calls) failed** + — they surfaced the CALL path; the leak is the dirty→compiled RETURN dispatch nesting. + +**Next (the fix, contained — runtime-only, no regen):** at the dirty-RAM↔compiled boundary, +a guest **return** (`jr ra`, and any dirty-dispatch whose target is a compiled-game address +reached as a RETURN) must **surface `cpu->pc=ret` to the `psx_dispatch_impl` trampoline owner +and UNWIND**, not nest `psx_dispatch_game_compiled`. Single-owner model is §11's *boundary* +piece (NOT the full resumable-frame codegen). Regression-guard with: (a) §19 `ce_profile` +per-frame `max_kb` must go **FLAT** (currently +1.17 KB/frame); (b) the oracle trace above is +the gold reference — recomp `psx_dispatch` returns must vanish into host unwinds while the +guest JAL/JR/SP sequence stays bit-identical; (c) full dwarf→overworld playtest (the +overlay-native `pc==0` return contract is the regression risk — the surface must fire for +compiled-GAME return targets, not overlay-native DLL returns). + +--- + +## 24. native_stack capture CORRECTS §23's fix scope — it is the §11 boundary work, NOT a one-liner (2026-06-18) + +Before implementing the §23 fix I captured a clean early-trip `native_stack` +(`PSX_STACK_GUARD_KB=6144`, frame 7592, 6 MB; report decoded with `decode_report.py`). +**This revised the fix scope — §23's oracle VERDICT stands, but its "quick contained +runtime patch" framing was wrong.** Recorded honestly: + +**The leaking host cycle (native_stack histogram, ~5239× each, ≈1 chain/frame):** +``` +func_8001A954 → func_8001AC00 → func_8001B2B4 → func_8001B5A8 → func_80046264 + → psx_dispatch_impl → dirty_ram_dispatch → exec_one + → interp_enter_compiled → psx_dispatch_game_compiled → func_8001A954 (repeats) +``` +So the per-frame chain (the §8 chain, emitted as DIRECT compiled `func_X(cpu)` calls) +**re-enters itself**: `func_80046264`'s dirty hop (`psx_dispatch_impl` → `dirty_ram_dispatch` +→ interp) loops back up to `func_8001A954`, and the direct-call chain never unwinds. This is +exactly the native-stack-guard comment's "in-range guest transfer emitted as a DIRECT +`func_X(cpu)` call grows the native stack WITHOUT going through `psx_dispatch_impl`." + +**Instrument cross-check (the instruments DISAGREE — do not over-trust native_stack):** +- `ce_profile` (the ONE trustworthy gauge, §20): **+1.17 KB/frame, real, live-reconfirmed.** +- `xprobe` per-frame summary: `a954`≈1/frame, `cr`≈1844 crossings/frame, **`dmax` (mixed_depth) + BOUNDED at 6** (reset per frame); the xprobe *detail* ring shows the chain UNWINDING cleanly + at frame end (mixed_depth d=1, surfaced returns to 0x8001A638/0x8001A9C0/0x8001AC54/…). +- `dirty_insn_log` (65536 insns) and `dirty_block_log`: **NO transfer/dispatch targets + `0x8001A954`.** So the re-entry is NOT a dirty-interp instruction with a literal + `func_8001A954` target, and `dirty_ram_dispatch(0x8001A954)` is never called (matches §16). +- ⇒ The `interp_enter_compiled → psx_dispatch_game_compiled → func_8001A954` frames in the + native_stack are partly **the compiled chain's own direct calls** (and possibly some stale + return addresses the walker over-collects). The accumulating unit is the COMPILED per-frame + chain not unwinding — the interp boundary is the *enabler* (CD-DMA dirtiness routes the + chain through `dirty_ram_dispatch`, §12), not a simple nesting site. + +**Corrected fix scope.** §23's oracle proof (real HW's `func_8001A954` is balanced and returns +once/frame, guest SP bit-identical) RULES OUT a simple one-instruction return-contract bug — +but that means the fix is NOT the quick "surface the return" patch §23 sketched. The recomp's +**whole per-frame chain is emitted as direct nested C calls that re-enter via the CD-DMA-dirty +boundary and never unwind**; making them unwind is the **§11 single-mixed-dispatch-owner / +resumable-boundary** work, which spans the interp boundary (`dirty_ram_interp.c`, runtime) AND +the `psx_dispatch_impl` trampoline (`full_function_emitter.cpp`, **codegen → REGEN**). It is +delicate (the Bug A/C/D contract area) and regression-prone — the §14 watermark band-aid of +this already made the wedge "(Not Responding)". This is precisely the change ChatGPT's gate +flagged to undertake only deliberately. Two real, substantial options (NOT one-liners): +- **(§11) Boundary resumability / single owner** — the per-frame compiled chain entered from + the interp must surface tail-transfers/returns to ONE owner trampoline and re-dispatch flat. + Spans runtime + codegen (regen). Highest fidelity; highest risk. +- **(§12) CD-DMA dirty-marking precision** — `func_8001A954`'s page is only on the dirty/interp + path because CD-DMA marks its whole page dirty (textures/audio sharing the page). Make + dirtiness byte-range/content-precise so the compiled chain dispatches via the FLAT static + table (normal C returns) and the boundary cycle never forms. Removes the enabler at the + source; lower codegen risk but touches `dma.c` + `dirty_ram` granularity. + +**Validation gate for EITHER fix (unchanged):** §19 `ce_profile` per-frame `max_kb` FLAT +(not +1.17 KB/frame) + the §23 oracle JAL/JR/SP sequence stays identical + 15-min+ idle soak +with no freeze + dwarf→overworld playtest (regression). Artifacts this session in +`_freeze_specimens/`: `oracle/oracle_chain_filtered.json`, `oracle/recomp_allfn_frame.json`, +the early-trip `psx_last_run_report.json` (native_stack), `_xprobe.json`, `_dirty_insn.json`. + diff --git a/runtime/include/fntrace.h b/runtime/include/fntrace.h index 194328fe..0ac49a8e 100644 --- a/runtime/include/fntrace.h +++ b/runtime/include/fntrace.h @@ -41,6 +41,7 @@ typedef struct { uint32_t a2; /* cpu->gpr[6] */ uint32_t a3; /* cpu->gpr[7] */ uint32_t s3; /* cpu->gpr[19] — callee-saved, useful for arg-passing chains */ + uint32_t sp; /* cpu->gpr[29] — for func_8001A954 SP/RA-lifecycle oracle */ } FntraceEntry; extern FntraceEntry g_fntrace_ring[FNTRACE_RING_CAP]; diff --git a/runtime/src/beetle_debug_server.c b/runtime/src/beetle_debug_server.c index 94a53500..26659517 100644 --- a/runtime/src/beetle_debug_server.c +++ b/runtime/src/beetle_debug_server.c @@ -130,7 +130,8 @@ extern uint32_t beetle_fntrace_get(uint64_t *out_seq, uint32_t *out_caller, uint32_t *out_target, uint32_t *out_ra, uint32_t *out_a0, uint32_t *out_a1, uint32_t *out_frame, - uint8_t *out_kind, int max_count); + uint8_t *out_kind, uint32_t *out_sp, + int max_count); /* ---- Server state ---- */ static sock_t s_listen = SOCK_INVALID; @@ -663,13 +664,14 @@ static void h_fntrace_dump(int id, const char *json) { uint32_t *a1s = (uint32_t*)malloc(count * sizeof(uint32_t)); uint32_t *frames = (uint32_t*)malloc(count * sizeof(uint32_t)); uint8_t *kinds = (uint8_t*) malloc(count); - if (!seqs || !callers || !targets || !ras || !a0s || !a1s || !frames || !kinds) { + uint32_t *sps = (uint32_t*)malloc(count * sizeof(uint32_t)); + if (!seqs || !callers || !targets || !ras || !a0s || !a1s || !frames || !kinds || !sps) { free(seqs); free(callers); free(targets); free(ras); - free(a0s); free(a1s); free(frames); free(kinds); + free(a0s); free(a1s); free(frames); free(kinds); free(sps); send_err(id, "alloc"); return; } uint32_t got = beetle_fntrace_get(seqs, callers, targets, ras, - a0s, a1s, frames, kinds, count); + a0s, a1s, frames, kinds, sps, count); uint64_t total = beetle_fntrace_total(); send_fmt("{\"id\":%d,\"ok\":true,\"total\":%llu,\"count\":%u,\"entries\":[", @@ -678,14 +680,14 @@ static void h_fntrace_dump(int id, const char *json) { if (i > 0) send_fmt(","); send_fmt("{\"seq\":%llu,\"caller\":\"0x%08X\",\"target\":\"0x%08X\"," "\"ra\":\"0x%08X\",\"a0\":\"0x%08X\",\"a1\":\"0x%08X\"," - "\"frame\":%u,\"kind\":\"%s\"}", + "\"frame\":%u,\"kind\":\"%s\",\"sp\":\"0x%08X\"}", (unsigned long long)seqs[i], callers[i], targets[i], - ras[i], a0s[i], a1s[i], frames[i], fn_kind_str(kinds[i])); + ras[i], a0s[i], a1s[i], frames[i], fn_kind_str(kinds[i]), sps[i]); } send_fmt("]}\n"); free(seqs); free(callers); free(targets); free(ras); - free(a0s); free(a1s); free(frames); free(kinds); + free(a0s); free(a1s); free(frames); free(kinds); free(sps); } /* ---- spu_voices: Beetle oracle ground truth via PS_SPU::GetRegister ---- diff --git a/runtime/src/beetle_libretro.cpp b/runtime/src/beetle_libretro.cpp index f86c83a3..ff5e1d41 100644 --- a/runtime/src/beetle_libretro.cpp +++ b/runtime/src/beetle_libretro.cpp @@ -185,7 +185,7 @@ static void wtrace_callback(uint32_t addr, uint32_t value, #define BEETLE_FNTRACE_MAX_ARMS 64 struct BeetleFnTraceEntry { uint64_t seq; - uint32_t caller_pc, target_pc, parent_ra, a0, a1, frame; + uint32_t caller_pc, target_pc, parent_ra, a0, a1, frame, sp; uint8_t kind, pad[3]; }; static BeetleFnTraceEntry s_fntrace[BEETLE_FNTRACE_CAP]; @@ -216,6 +216,12 @@ static void fntrace_callback(uint32_t caller_pc, uint32_t target_pc, e->a1 = a1; e->frame = s_frame_count; e->kind = kind; + /* SP at retire — needed for the func_8001A954 SP/RA-lifecycle oracle + * (does the invocation's stack balance / return). GPR[29] read live. */ + { + char dummy[8] = {0}; + e->sp = PSX_CPU ? PSX_CPU->GetRegister(PS_CPU::GSREG_GPR + 29, dummy, sizeof(dummy)) : 0; + } s_fntrace_idx = (s_fntrace_idx + 1) % BEETLE_FNTRACE_CAP; } @@ -815,7 +821,8 @@ extern "C" uint32_t beetle_fntrace_get(uint64_t *out_seq, uint32_t *out_caller, uint32_t *out_target, uint32_t *out_ra, uint32_t *out_a0, uint32_t *out_a1, uint32_t *out_frame, - uint8_t *out_kind, int max_count) + uint8_t *out_kind, uint32_t *out_sp, + int max_count) { int avail = (int)(s_fntrace_seq < (uint64_t)BEETLE_FNTRACE_CAP ? s_fntrace_seq : BEETLE_FNTRACE_CAP); @@ -832,6 +839,7 @@ extern "C" uint32_t beetle_fntrace_get(uint64_t *out_seq, out_a1[i] = e->a1; out_frame[i] = e->frame; out_kind[i] = e->kind; + out_sp[i] = e->sp; } return (uint32_t)count; } diff --git a/runtime/src/beetle_main.cpp b/runtime/src/beetle_main.cpp index f8c163b8..0c193c9d 100644 --- a/runtime/src/beetle_main.cpp +++ b/runtime/src/beetle_main.cpp @@ -62,9 +62,16 @@ int main(int argc, char** argv) { const char* bios_path = "bios/SCPH1001.BIN"; const char* disc_path = NULL; + /* Default debug port 4382: 4380 collides with psxref and other recomp + * projects' oracles (e.g. cdirecomp's CdiRuntime), and two LISTENers on + * the same port silently route connections to the wrong process. Give + * psx-beetle its own port; override with --port N. */ + int dbg_port = 4382; for (int i = 1; i < argc; i++) { if (!std::strcmp(argv[i], "--disc") && i + 1 < argc) { disc_path = argv[++i]; + } else if (!std::strcmp(argv[i], "--port") && i + 1 < argc) { + dbg_port = std::atoi(argv[++i]); } else if (argv[i][0] != '-') { bios_path = argv[i]; } @@ -113,11 +120,8 @@ int main(int argc, char** argv) { } #ifndef PSX_NO_DEBUG_TOOLS -#ifdef DEFAULT_DEBUG_PORT - beetle_debug_server_init(DEFAULT_DEBUG_PORT); -#else - beetle_debug_server_init(4380); -#endif + beetle_debug_server_init(dbg_port); + std::fprintf(stdout, "psx-beetle: debug server on port %d\n", dbg_port); #endif static uint32_t pixels[640 * 512]; diff --git a/runtime/src/debug_server.c b/runtime/src/debug_server.c index 36ae2220..730f9262 100644 --- a/runtime/src/debug_server.c +++ b/runtime/src/debug_server.c @@ -2393,11 +2393,11 @@ static void handle_fntrace_dump(int id, const char *json) "%s{\"seq\":%llu,\"target\":\"0x%08X\"," "\"ra\":\"0x%08X\",\"a0\":\"0x%08X\",\"a1\":\"0x%08X\"," "\"a2\":\"0x%08X\",\"a3\":\"0x%08X\"," - "\"s3\":\"0x%08X\",\"frame\":%u}", + "\"s3\":\"0x%08X\",\"sp\":\"0x%08X\",\"frame\":%u}", emitted == 0 ? "" : ",", (unsigned long long)seq, e->target, e->ra, e->a0, e->a1, e->a2, e->a3, - e->s3, e->frame); + e->s3, e->sp, e->frame); emitted++; } pos += snprintf(out + pos, BUF_SZ - pos, "],\"emitted\":%d}\n", emitted); diff --git a/runtime/src/fntrace.c b/runtime/src/fntrace.c index fb1ebc81..a3ad1883 100644 --- a/runtime/src/fntrace.c +++ b/runtime/src/fntrace.c @@ -68,6 +68,7 @@ void fntrace_record(CPUState* cpu, uint32_t target) { e->a2 = cpu->gpr[6]; e->a3 = cpu->gpr[7]; e->s3 = cpu->gpr[19]; + e->sp = cpu->gpr[29]; } void fntrace_arm(uint32_t target) {