recursion(oracle): oracle disambiguation DONE + SP-fntrace instruments; verdict = §11 universal fix (docs §23–§24)

Oracle disambiguation (RECURSION_BUG.md §23–§24). Traced one func_8001A954
invocation on real HW (psx-beetle) vs the recomp in the IDENTICAL Tomba idle
state (New Game village dialogue; object 0x801FD950, state/scene/jumptable=0).

VERDICT: real HW's per-frame chain is BALANCED — func_8001A954 is entered once
per frame from main_loop (JAL 0x8001A630, sp=0x801FE3E0) and RETURNS once per
frame (jr ra @0x8001A9E8 -> 0x8001A638, SP fully restored); no re-entry, the
overlay 0x80063864 never calls back (refutes §21/§22's "call-cycle"). The recomp
runs the same guest flow and the guest SP unwinds identically, but the HOST stack
leaks +1.17 KB/frame (ce_profile, live-reconfirmed: 20099->22215 KB / 1807 frames)
because the per-frame compiled chain (func_8001A954->AC00->B2B4->B5A8->func_80046264,
emitted as DIRECT func_X(cpu) host calls) re-enters itself via func_80046264's
CD-DMA-dirty hop and never unwinds.

native_stack early-trip (PSX_STACK_GUARD_KB=6144, frame 7592, 6 MB) pins the
leaking host cycle. Cross-checked: xprobe mixed_depth bounded at 6, dirty-insn/
dirty-block show NO literal func_8001A954 transfer, and a954=1/frame is a SURFACED
guest tail-transfer (j/jr) to func_8001A954 that the oracle proves real HW never
makes — i.e. there is both (A) a correctness divergence (spurious per-frame
re-entry) and (B) the architectural host-mirroring limitation that amplifies it.

Fix scope CORRECTED from §23's "quick contained runtime patch": the UNIVERSAL fix
is §11 — route in-range guest calls through the depth-managed psx_dispatch_impl
trampoline instead of mirroring them as direct host calls, so the host stack stays
bounded regardless of guest re-entrancy (real or fabricated). §12 (CD-DMA dirty
precision) and fixing the specific divergence are surgical mitigations, not
universal. Performance loss accepted for now (static recomp: right before fast).

Instruments added (runtime-only, no regen):
- SP capture in the fntrace control-flow ring on BOTH backends (GPR[29] read live):
  psx-beetle (beetle_libretro.cpp + beetle_debug_server.c) and the recomp
  (fntrace.h/.c + debug_server.c) — enables the func_8001A954 SP/RA-lifecycle oracle.
- psx-beetle default debug port 4380 -> 4382 (+ --port N override): 4380 collided
  with psxref and cdirecomp/CdiRuntime.exe; two LISTENers on one port silently
  misroute connections (the phantom "oracle keeps dying" early in the session).

Oracle tooling: the control-flow oracle is psx-beetle (Beetle interprets -> records
every J/JAL/JR/JALR), NOT psxref (no PC trace; the assumed emu_step/emu_trace_addr
never existed). Loads Tomba via: psx-beetle.exe --disc tomba.cue <bios> --port 4382.

Next: implement the §11 universal fix (codegen + full regen).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Matthew Stanley
2026-06-18 08:03:04 -07:00
parent fd2570ff13
commit ae47e69902
7 changed files with 182 additions and 16 deletions
+150
View File
@@ -931,3 +931,153 @@ contained interp/dispatch fix), or does it genuinely never return (→ §11 rede
prior surface attempts (block-loop §20, JAL/JALR §22) are proven dead ends; stop
surfacing single crossings. The §19 `ce_profile` flat-vs-climb remains the objective
check; the self-driven screenshot-validated soak remains the harness.
---
## 23. ORACLE DISAMBIGUATION DONE — VERDICT: contained return-contract fix, NOT §11 (2026-06-18)
Ran the oracle disambiguation (only). Both sides traced **in the identical idle state**
(Tomba New Game, starting village, "Hey, you with the pink hair!" dialogue; object
`*(0x1F8001D4)=0x801FD950`, `state/scene/jumptable` all 0). Artifacts in
`_freeze_specimens/oracle/` (`oracle_chain_filtered.json`, `oracle_state.txt`,
`recomp_allfn_frame.json`, `recomp_ce_profile.json`).
**Tooling built (CLAUDE.md §15 — the oracle had a real gap).** psxref has **no**
instruction/PC trace (`emu_step`/`emu_trace_addr` never existed — the handoff/§10 assumed
them). The right oracle is **`psx-beetle.exe`** (shares the wire protocol; Beetle PSX core
INTERPRETS, so its always-on `g_psxrecomp_fntrace_cb` records **every J/JAL/JR/JALR**
with caller/target/ra/a0/a1/kind). Loads Tomba via `--disc tomba.cue <bios>`. Added **SP
capture** to the fntrace ring on BOTH backends (read `GPR[29]` live) — `beetle_libretro.cpp`
+ `beetle_debug_server.c` and recomp `fntrace.c`/`fntrace.h`/`debug_server.c`; runtime-only,
no regen. Also moved psx-beetle's default debug port **4380 → 4382** (`--port N` override):
4380 collided with psxref AND `cdirecomp/CdiRuntime.exe`, and two LISTENers on one port
silently route connections to the wrong process — that dual-listener was the whole
"oracle keeps dying / unknown cmd" confusion early in the session, NOT a real crash.
**ORACLE (real HW), one `func_8001A954` invocation — perfectly balanced, once/frame:**
```
JAL 0x8001A630 -> 0x8001A954 ra=0x8001A5DC sp=0x801FE3E0 main_loop calls A954 (ret=0x8001A638)
JAL 0x8001A9B8 -> 0x8001AC00 ra=0x8001A638 sp=0x801FE3C8 ┐
JAL 0x8001AC4C -> 0x8001B2B4 ra=0x8001A9C0 sp=0x801FE3A8 │ nested chain,
JAL 0x8001B3E4 -> 0x8001B5A8 ra=0x8001AC54 sp=0x801FE390 │ SP descends
JAL 0x8001B750 -> 0x80046264 ra=0x8001B73C sp=0x801FE378 │
JAL 0x80046274 -> 0x80063864 ra=0x8001B758 sp=0x801FE360 ┘ (the dirty hop)
JR 0x8001A9E8 -> 0x8001A638 sp=0x801FE3E0 A954 RETURNS, SP fully restored
```
- `func_8001A954` is entered **exactly once/frame** (`JAL` from `0x8001A630`) and **returns
exactly once/frame** (`jr ra` at `0x8001A9E8` → `0x8001A638`), with **SP back to its entry
value `0x801FE3E0`** ⇒ the whole subtree unwinds every frame.
- **NO re-entry of `func_8001A954` before its exit** (armed on `0x8001A954`; the only hit per
frame is the `0x8001A630` JAL). **The overlay `0x80063864` NEVER calls back into
`func_8001A954`** — §21's "interp overlay does `jalr func_8001A954`" is **REFUTED on real
HW**. There is **no guest call-cycle**: main_loop is a per-frame VSync callback that calls
A954 and A954 returns; §22's "`jal back to main_loop`" was a misread of the recomp's
forward-dispatched returns (below).
**RECOMP (`build-recursion`, `PSX_MIXED_OWNER=0`), same idle — IDENTICAL guest flow, but
every boundary-crossing RETURN is a forward `psx_dispatch`:**
```
... render leaf 0x8004DEE0 (ra=0x8004DEE8) ×5 ...
psx_dispatch 0x800462C4 sp=0x801FE360 (80046264 subtree returns)
psx_dispatch 0x8001B758 sp=0x801FE378 (80046264 -> B5A8 return)
psx_dispatch 0x8001B3EC sp=0x801FE390 (B5A8 -> B2B4 return)
psx_dispatch 0x8001AC54 sp=0x801FE3A8 (B2B4 -> AC00 return)
psx_dispatch 0x8001A9C0 sp=0x801FE3C8 (AC00 -> A954 return)
psx_dispatch 0x8001A638 sp=0x801FE3E0 (A954 -> main_loop return; SP restored)
```
- The **guest** stack unwinds **identically** to the oracle (SP climbs back to `0x801FE3E0`,
same addresses, same RA). The down-chain calls (A954→AC00→…→80046264) are compiled DIRECT
calls; the render leaves are interpreted overlays (`0x8011xxxx`/`0x8012xxxx`/`0x8006Bxxx`,
ra `0x8001E0D8`). When those interpreted/dirty frames **return** to a compiled main-EXE
address, the dirty-RAM dispatch realizes the `jr ra` as a **forward `psx_dispatch(ret)`**
that NESTS `psx_dispatch_game_compiled` instead of surfacing to the trampoline owner.
- **Live leak confirmed** (`ce_profile`): `max_kb` 20099→22215 over 1807 frames =
**+1.17 KB/frame**, `entries/frame` constant ~1300, deepest func the render cluster
(`0x8004DEE0`/`0x8005FF60`/`0x8005FA60`). Identical to §20.
**VERDICT (decisive):**
1. **No guest control-flow divergence.** Targets, RA, delay-slot results and SP are
bit-identical on both sides. ⇒ Hypothesis **C-as-interp-fabrication is RULED OUT** (the
interp does NOT mishandle a JAL/JALR/delay-slot/RA; it does not fabricate the re-entry).
2. **Beetle UNWINDS each invocation; the recomp leaves the chain SUSPENDED** (ChatGPT
determination **#1**), because the recomp **maps the guest's nonlocal-but-balanced returns
onto accumulating host calls** (determination **#2**). The guest **genuinely returns every
frame** (proven by SP on BOTH sides) → this is a **CONTAINED return/unwind-contract bug**,
**NOT** the §11 continuation-passing redesign.
3. **The leaking edge is the RETURN path, not the call path.** The boundary crossings that
leak are the up-chain `jr ra` returns from interpreted/dirty code into compiled main-EXE
functions: `0x800462C4 / 0x8001B758 / 0x8001B3EC / 0x8001AC54 / 0x8001A9C0 / 0x8001A638`.
This is **why §20 (block-loop tail-transfer) and §22 (interp JAL/JALR down-calls) failed**
— they surfaced the CALL path; the leak is the dirty→compiled RETURN dispatch nesting.
**Next (the fix, contained — runtime-only, no regen):** at the dirty-RAM↔compiled boundary,
a guest **return** (`jr ra`, and any dirty-dispatch whose target is a compiled-game address
reached as a RETURN) must **surface `cpu->pc=ret` to the `psx_dispatch_impl` trampoline owner
and UNWIND**, not nest `psx_dispatch_game_compiled`. Single-owner model is §11's *boundary*
piece (NOT the full resumable-frame codegen). Regression-guard with: (a) §19 `ce_profile`
per-frame `max_kb` must go **FLAT** (currently +1.17 KB/frame); (b) the oracle trace above is
the gold reference — recomp `psx_dispatch` returns must vanish into host unwinds while the
guest JAL/JR/SP sequence stays bit-identical; (c) full dwarf→overworld playtest (the
overlay-native `pc==0` return contract is the regression risk — the surface must fire for
compiled-GAME return targets, not overlay-native DLL returns).
---
## 24. native_stack capture CORRECTS §23's fix scope — it is the §11 boundary work, NOT a one-liner (2026-06-18)
Before implementing the §23 fix I captured a clean early-trip `native_stack`
(`PSX_STACK_GUARD_KB=6144`, frame 7592, 6 MB; report decoded with `decode_report.py`).
**This revised the fix scope — §23's oracle VERDICT stands, but its "quick contained
runtime patch" framing was wrong.** Recorded honestly:
**The leaking host cycle (native_stack histogram, ~5239× each, ≈1 chain/frame):**
```
func_8001A954 → func_8001AC00 → func_8001B2B4 → func_8001B5A8 → func_80046264
→ psx_dispatch_impl → dirty_ram_dispatch → exec_one
→ interp_enter_compiled → psx_dispatch_game_compiled → func_8001A954 (repeats)
```
So the per-frame chain (the §8 chain, emitted as DIRECT compiled `func_X(cpu)` calls)
**re-enters itself**: `func_80046264`'s dirty hop (`psx_dispatch_impl` → `dirty_ram_dispatch`
→ interp) loops back up to `func_8001A954`, and the direct-call chain never unwinds. This is
exactly the native-stack-guard comment's "in-range guest transfer emitted as a DIRECT
`func_X(cpu)` call grows the native stack WITHOUT going through `psx_dispatch_impl`."
**Instrument cross-check (the instruments DISAGREE — do not over-trust native_stack):**
- `ce_profile` (the ONE trustworthy gauge, §20): **+1.17 KB/frame, real, live-reconfirmed.**
- `xprobe` per-frame summary: `a954`≈1/frame, `cr`≈1844 crossings/frame, **`dmax` (mixed_depth)
BOUNDED at 6** (reset per frame); the xprobe *detail* ring shows the chain UNWINDING cleanly
at frame end (mixed_depth d=1, surfaced returns to 0x8001A638/0x8001A9C0/0x8001AC54/…).
- `dirty_insn_log` (65536 insns) and `dirty_block_log`: **NO transfer/dispatch targets
`0x8001A954`.** So the re-entry is NOT a dirty-interp instruction with a literal
`func_8001A954` target, and `dirty_ram_dispatch(0x8001A954)` is never called (matches §16).
- ⇒ The `interp_enter_compiled → psx_dispatch_game_compiled → func_8001A954` frames in the
native_stack are partly **the compiled chain's own direct calls** (and possibly some stale
return addresses the walker over-collects). The accumulating unit is the COMPILED per-frame
chain not unwinding — the interp boundary is the *enabler* (CD-DMA dirtiness routes the
chain through `dirty_ram_dispatch`, §12), not a simple nesting site.
**Corrected fix scope.** §23's oracle proof (real HW's `func_8001A954` is balanced and returns
once/frame, guest SP bit-identical) RULES OUT a simple one-instruction return-contract bug —
but that means the fix is NOT the quick "surface the return" patch §23 sketched. The recomp's
**whole per-frame chain is emitted as direct nested C calls that re-enter via the CD-DMA-dirty
boundary and never unwind**; making them unwind is the **§11 single-mixed-dispatch-owner /
resumable-boundary** work, which spans the interp boundary (`dirty_ram_interp.c`, runtime) AND
the `psx_dispatch_impl` trampoline (`full_function_emitter.cpp`, **codegen → REGEN**). It is
delicate (the Bug A/C/D contract area) and regression-prone — the §14 watermark band-aid of
this already made the wedge "(Not Responding)". This is precisely the change ChatGPT's gate
flagged to undertake only deliberately. Two real, substantial options (NOT one-liners):
- **(§11) Boundary resumability / single owner** — the per-frame compiled chain entered from
the interp must surface tail-transfers/returns to ONE owner trampoline and re-dispatch flat.
Spans runtime + codegen (regen). Highest fidelity; highest risk.
- **(§12) CD-DMA dirty-marking precision** — `func_8001A954`'s page is only on the dirty/interp
path because CD-DMA marks its whole page dirty (textures/audio sharing the page). Make
dirtiness byte-range/content-precise so the compiled chain dispatches via the FLAT static
table (normal C returns) and the boundary cycle never forms. Removes the enabler at the
source; lower codegen risk but touches `dma.c` + `dirty_ram` granularity.
**Validation gate for EITHER fix (unchanged):** §19 `ce_profile` per-frame `max_kb` FLAT
(not +1.17 KB/frame) + the §23 oracle JAL/JR/SP sequence stays identical + 15-min+ idle soak
with no freeze + dwarf→overworld playtest (regression). Artifacts this session in
`_freeze_specimens/`: `oracle/oracle_chain_filtered.json`, `oracle/recomp_allfn_frame.json`,
the early-trip `psx_last_run_report.json` (native_stack), `_xprobe.json`, `_dirty_insn.json`.
+1
View File
@@ -41,6 +41,7 @@ typedef struct {
uint32_t a2; /* cpu->gpr[6] */
uint32_t a3; /* cpu->gpr[7] */
uint32_t s3; /* cpu->gpr[19] — callee-saved, useful for arg-passing chains */
uint32_t sp; /* cpu->gpr[29] — for func_8001A954 SP/RA-lifecycle oracle */
} FntraceEntry;
extern FntraceEntry g_fntrace_ring[FNTRACE_RING_CAP];
+9 -7
View File
@@ -130,7 +130,8 @@ extern uint32_t beetle_fntrace_get(uint64_t *out_seq,
uint32_t *out_caller, uint32_t *out_target,
uint32_t *out_ra, uint32_t *out_a0,
uint32_t *out_a1, uint32_t *out_frame,
uint8_t *out_kind, int max_count);
uint8_t *out_kind, uint32_t *out_sp,
int max_count);
/* ---- Server state ---- */
static sock_t s_listen = SOCK_INVALID;
@@ -663,13 +664,14 @@ static void h_fntrace_dump(int id, const char *json) {
uint32_t *a1s = (uint32_t*)malloc(count * sizeof(uint32_t));
uint32_t *frames = (uint32_t*)malloc(count * sizeof(uint32_t));
uint8_t *kinds = (uint8_t*) malloc(count);
if (!seqs || !callers || !targets || !ras || !a0s || !a1s || !frames || !kinds) {
uint32_t *sps = (uint32_t*)malloc(count * sizeof(uint32_t));
if (!seqs || !callers || !targets || !ras || !a0s || !a1s || !frames || !kinds || !sps) {
free(seqs); free(callers); free(targets); free(ras);
free(a0s); free(a1s); free(frames); free(kinds);
free(a0s); free(a1s); free(frames); free(kinds); free(sps);
send_err(id, "alloc"); return;
}
uint32_t got = beetle_fntrace_get(seqs, callers, targets, ras,
a0s, a1s, frames, kinds, count);
a0s, a1s, frames, kinds, sps, count);
uint64_t total = beetle_fntrace_total();
send_fmt("{\"id\":%d,\"ok\":true,\"total\":%llu,\"count\":%u,\"entries\":[",
@@ -678,14 +680,14 @@ static void h_fntrace_dump(int id, const char *json) {
if (i > 0) send_fmt(",");
send_fmt("{\"seq\":%llu,\"caller\":\"0x%08X\",\"target\":\"0x%08X\","
"\"ra\":\"0x%08X\",\"a0\":\"0x%08X\",\"a1\":\"0x%08X\","
"\"frame\":%u,\"kind\":\"%s\"}",
"\"frame\":%u,\"kind\":\"%s\",\"sp\":\"0x%08X\"}",
(unsigned long long)seqs[i], callers[i], targets[i],
ras[i], a0s[i], a1s[i], frames[i], fn_kind_str(kinds[i]));
ras[i], a0s[i], a1s[i], frames[i], fn_kind_str(kinds[i]), sps[i]);
}
send_fmt("]}\n");
free(seqs); free(callers); free(targets); free(ras);
free(a0s); free(a1s); free(frames); free(kinds);
free(a0s); free(a1s); free(frames); free(kinds); free(sps);
}
/* ---- spu_voices: Beetle oracle ground truth via PS_SPU::GetRegister ----
+10 -2
View File
@@ -185,7 +185,7 @@ static void wtrace_callback(uint32_t addr, uint32_t value,
#define BEETLE_FNTRACE_MAX_ARMS 64
struct BeetleFnTraceEntry {
uint64_t seq;
uint32_t caller_pc, target_pc, parent_ra, a0, a1, frame;
uint32_t caller_pc, target_pc, parent_ra, a0, a1, frame, sp;
uint8_t kind, pad[3];
};
static BeetleFnTraceEntry s_fntrace[BEETLE_FNTRACE_CAP];
@@ -216,6 +216,12 @@ static void fntrace_callback(uint32_t caller_pc, uint32_t target_pc,
e->a1 = a1;
e->frame = s_frame_count;
e->kind = kind;
/* SP at retire — needed for the func_8001A954 SP/RA-lifecycle oracle
* (does the invocation's stack balance / return). GPR[29] read live. */
{
char dummy[8] = {0};
e->sp = PSX_CPU ? PSX_CPU->GetRegister(PS_CPU::GSREG_GPR + 29, dummy, sizeof(dummy)) : 0;
}
s_fntrace_idx = (s_fntrace_idx + 1) % BEETLE_FNTRACE_CAP;
}
@@ -815,7 +821,8 @@ extern "C" uint32_t beetle_fntrace_get(uint64_t *out_seq,
uint32_t *out_caller, uint32_t *out_target,
uint32_t *out_ra, uint32_t *out_a0,
uint32_t *out_a1, uint32_t *out_frame,
uint8_t *out_kind, int max_count)
uint8_t *out_kind, uint32_t *out_sp,
int max_count)
{
int avail = (int)(s_fntrace_seq < (uint64_t)BEETLE_FNTRACE_CAP
? s_fntrace_seq : BEETLE_FNTRACE_CAP);
@@ -832,6 +839,7 @@ extern "C" uint32_t beetle_fntrace_get(uint64_t *out_seq,
out_a1[i] = e->a1;
out_frame[i] = e->frame;
out_kind[i] = e->kind;
out_sp[i] = e->sp;
}
return (uint32_t)count;
}
+9 -5
View File
@@ -62,9 +62,16 @@ int main(int argc, char** argv) {
const char* bios_path = "bios/SCPH1001.BIN";
const char* disc_path = NULL;
/* Default debug port 4382: 4380 collides with psxref and other recomp
* projects' oracles (e.g. cdirecomp's CdiRuntime), and two LISTENers on
* the same port silently route connections to the wrong process. Give
* psx-beetle its own port; override with --port N. */
int dbg_port = 4382;
for (int i = 1; i < argc; i++) {
if (!std::strcmp(argv[i], "--disc") && i + 1 < argc) {
disc_path = argv[++i];
} else if (!std::strcmp(argv[i], "--port") && i + 1 < argc) {
dbg_port = std::atoi(argv[++i]);
} else if (argv[i][0] != '-') {
bios_path = argv[i];
}
@@ -113,11 +120,8 @@ int main(int argc, char** argv) {
}
#ifndef PSX_NO_DEBUG_TOOLS
#ifdef DEFAULT_DEBUG_PORT
beetle_debug_server_init(DEFAULT_DEBUG_PORT);
#else
beetle_debug_server_init(4380);
#endif
beetle_debug_server_init(dbg_port);
std::fprintf(stdout, "psx-beetle: debug server on port %d\n", dbg_port);
#endif
static uint32_t pixels[640 * 512];
+2 -2
View File
@@ -2393,11 +2393,11 @@ static void handle_fntrace_dump(int id, const char *json)
"%s{\"seq\":%llu,\"target\":\"0x%08X\","
"\"ra\":\"0x%08X\",\"a0\":\"0x%08X\",\"a1\":\"0x%08X\","
"\"a2\":\"0x%08X\",\"a3\":\"0x%08X\","
"\"s3\":\"0x%08X\",\"frame\":%u}",
"\"s3\":\"0x%08X\",\"sp\":\"0x%08X\",\"frame\":%u}",
emitted == 0 ? "" : ",",
(unsigned long long)seq,
e->target, e->ra, e->a0, e->a1, e->a2, e->a3,
e->s3, e->frame);
e->s3, e->sp, e->frame);
emitted++;
}
pos += snprintf(out + pos, BUF_SZ - pos, "],\"emitted\":%d}\n", emitted);
+1
View File
@@ -68,6 +68,7 @@ void fntrace_record(CPUState* cpu, uint32_t target) {
e->a2 = cpu->gpr[6];
e->a3 = cpu->gpr[7];
e->s3 = cpu->gpr[19];
e->sp = cpu->gpr[29];
}
void fntrace_arm(uint32_t target) {