From 0f7d0f16451a52e4f77825f7e25df007b6563d90 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Mon, 29 Jun 2026 15:52:51 -0700 Subject: [PATCH] spike(scheduler): deterministic TCB scheduler LIVE + HLE/LLE toggle (minimal A+) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the non-deterministic host-fiber thread-switch bridge with a deterministic TCB scheduler (HLE_SCHEDULER_CARVEOUT_PLAN.md steps 2-8, minimal "A+" scope): - psx_scheduler_run (traps.c): outer trampoline wrapping the top-level dispatch (main.cpp calls it instead of bare psx_dispatch). setjmp(g_scheduler_jmpbuf); restore current TCB / dispatch / on-escape reset g_psx_dispatch_depth + g_psx_call_bail; boot/pre-TCB runs raw cpu->pc; one-level switch-back safety net; fail-closed psx_is_dispatchable on every restored EPC. - psx_request_thread_switch: save yielder ctx (resume=$ra), repoint dword_108->entry, longjmp to the scheduler. No fibers, no cpu->pc=0 signal. Fires only with in_exception==0, so it never unwinds a live exception_jmpbuf frame (the in-exception RFE path stays LLE/intact). - sched_escape_ring (step 8) for always-on observability. - Hidden toggle PSX_HLE_SCHEDULER (default 1=HLE, 0=legacy fibers), read once; psx_change_thread branches on it; startup banner prints the mode. The trampoline is transparent in LLE mode. Fiber funcs kept compiled-but-dead. Validated live (HLE): thread1 trace shows ONLY the new TCB-switch events and ZERO fiber kinds 10/11/13 — the per-frame fiber recreate is eliminated. Builds clean. Freeze NOT resolved: same 0x800CD3F8 wedge persists (separate upstream root, per ChatGPT Q5). Keep HLE default until the upstream cause is fixed. Co-Authored-By: Claude Opus 4.8 --- HLE_SCHEDULER_CARVEOUT_PLAN.md | 37 +++++- runtime/include/psx_scheduler.h | 14 +++ runtime/src/main.cpp | 16 ++- runtime/src/traps.c | 197 +++++++++++++++++++++++++++++++- 4 files changed, 257 insertions(+), 7 deletions(-) diff --git a/HLE_SCHEDULER_CARVEOUT_PLAN.md b/HLE_SCHEDULER_CARVEOUT_PLAN.md index e3b66731..87e94940 100644 --- a/HLE_SCHEDULER_CARVEOUT_PLAN.md +++ b/HLE_SCHEDULER_CARVEOUT_PLAN.md @@ -245,7 +245,42 @@ diagnosis gate stays FIRST. g_sched_escape, psx_is_dispatchable fail-closed guard rejecting 0 + the exc sentinel). INERT (host-fiber bridge still authoritative), builds clean (traps.c only, no game-C regen), no behavior change. -- ⏭ **NEXT (steps 2-8)**: outer scheduler loop at the exception/syscall boundary; +- ✅ **Steps 2-8 (MINIMAL "A+") IMPLEMENTED + structurally validated** (2026-06-29, + runtime-only, UNCOMMITTED). Scope refined from the broad 8-step list to the minimal + bridge replacement ChatGPT endorsed: the Beetle-confirmed non-determinism lives ONLY + in the `!in_exception` host-fiber bridge; the in-exception RFE path already escapes via + `exception_jmpbuf` (no fibers) and was left intact. Implemented: + - `psx_scheduler_run` (traps.c) — outer trampoline wrapping the top-level psx_dispatch + (main.cpp now calls it instead of bare `psx_dispatch`); setjmp(g_scheduler_jmpbuf), + restore-current-TCB / dispatch / on-escape reset g_psx_dispatch_depth + g_psx_call_bail; + boot/pre-TCB runs raw cpu->pc; one-level g_sched_return_tcb switch-back safety net; + fail-closed psx_is_dispatchable on every restored EPC. + - `psx_request_thread_switch` replaces the fiber switch: save yielder ctx (resume=$ra), + set dword_108->entry=target, longjmp to the scheduler. No fibers, no cpu->pc=0 signal. + Only fires with in_exception==0 ⇒ never unwinds a live exception_jmpbuf frame. + - `sched_escape_ring` (step 8) added. Fiber funcs kept compiled-but-dead this pass. + - **HIDDEN TOGGLE**: env `PSX_HLE_SCHEDULER` (default 1=HLE; 0=legacy LLE fibers), + read once (mode can't flip mid-run); `psx_change_thread` branches on it; startup + banner prints the active mode. Trampoline is transparent in LLE mode. + - **VALIDATED LIVE (HLE)**: thread1 trace now shows ONLY kinds 3/8/20/1/2 (new + TCB-switch path, 50× cross-thread commits) and ZERO fiber kinds 10/11/13 — the + per-frame fiber recreate is ELIMINATED. Builds clean (no -Werror). +- ⛔ **Freeze NOT resolved — separate UPSTREAM root confirmed** (ChatGPT Q5 caution + realized). HLE build reaches the IDENTICAL pre-freeze state (gp0_draw frozen at 337079) + and wedges on the SAME `0x800CD3F8` dispatch miss, now a tight loop (~4000/s; render + thread stuck, gp0_writes still advancing + audio plays ⇒ black screen + music). The + scheduler swap removed the fiber-recreate class but the 0xCD3F8 wedge has an upstream + cause (the ~273-frames-early thread1 divergence). **Per user (2026-06-29): keep HLE + default until the upstream root is fixed.** +- ⏭ **NEXT — UPSTREAM ROOT**: thread1 first-divergence timeline diff vs Beetle (where + does 0x800CD3F8 land on thread1's stack/dispatch path, and where does our thread1 PC + first diverge from Beetle's). BLOCKER (Rule 15): Beetle's fntrace per-target filter is + broken (total is global, filter ignored) — FIX that tool before trusting Beetle + per-target traces. THEN diff from the last common milestone to the first 0xCD3F8. +- ⏭ **DEFERRED (post-fix)**: route in-exception RFE through structured escapes + unify + exception_jmpbuf + nested-exception guards (the broader original steps 4-7); delete the + now-dead fiber bridge; cross-title regress (Tomba1/Tomba2/BIOS) per pre-merge gates. +- ⏭ **(superseded sub-items of the original step list)**: outer scheduler loop at the exception/syscall boundary; `psx_request_thread_switch` (save current TCB + longjmp YIELD_TO_TCB); route RFE through structured escapes; remove all cpu->pc=0 signalling; runnable-thread dispatch pc=0 fail-fast; tighten psx_is_dispatchable to registered-entry/block-leader; add diff --git a/runtime/include/psx_scheduler.h b/runtime/include/psx_scheduler.h index e02c696b..c7666e2d 100644 --- a/runtime/include/psx_scheduler.h +++ b/runtime/include/psx_scheduler.h @@ -69,6 +69,20 @@ extern psx_sched_escape_t g_sched_escape; * pc=0 silent fiber-recreate pathology). */ int psx_is_dispatchable(uint32_t pc); +/* The outer scheduler trampoline. Replaces the bare top-level psx_dispatch() + * call in main.cpp: dispatch the current thread until it yields via a + * structured escape (longjmp to g_scheduler_jmpbuf), then run whatever the + * escape selected. Returns ONLY on an abnormal top-level exit (the outermost + * dispatch saw cpu->pc==0 with no escape pending) — main.cpp's existing + * exit-diagnostic dump then runs. Forward-declared for main.cpp (C++). */ +struct CPUState; +void psx_scheduler_run(struct CPUState* cpu); + +/* Hidden dev toggle (env PSX_HLE_SCHEDULER, default 1 = HLE). 1 = deterministic + * TCB scheduler (carve-out); 0 = legacy host-fiber bridge (LLE). Read once at + * first call; the mode cannot change mid-run. */ +int psx_hle_scheduler_enabled(void); + #ifdef __cplusplus } #endif diff --git a/runtime/src/main.cpp b/runtime/src/main.cpp index cba802b2..b6442eef 100644 --- a/runtime/src/main.cpp +++ b/runtime/src/main.cpp @@ -6,6 +6,7 @@ */ #include "cpu_state.h" +#include "psx_scheduler.h" /* psx_scheduler_run — deterministic TCB scheduler */ #include "psx_interpreter.h" #include "cdrom.h" #include "fntrace.h" @@ -2775,7 +2776,20 @@ int main(int argc, char** argv) { std::fprintf(stderr, "ORACLE: VSync incrementer hit count = %u\n", vsync_hits); std::fflush(stderr); #else - psx_dispatch(&cpu, cpu.pc); + /* Deterministic TCB-scheduler trampoline (carve-out): wraps the top-level + * dispatch so a cooperative thread switch unwinds here and re-dispatches the + * target thread, instead of the old per-frame host-fiber recreate. Returns + * only on the abnormal top-level pc==0 exit; the diagnostic dump below runs + * exactly as it did for the bare psx_dispatch. (HLE_SCHEDULER_CARVEOUT_PLAN.md) + * + * Hidden toggle: PSX_HLE_SCHEDULER=0 reverts to the legacy LLE host-fiber + * bridge (default 1 = HLE). The trampoline is transparent in LLE mode (the + * fiber path never longjmps to it). */ + std::fprintf(stdout, "psxrecomp: thread scheduler = %s (PSX_HLE_SCHEDULER)\n", + psx_hle_scheduler_enabled() ? "HLE (deterministic TCB)" + : "LLE (host fibers)"); + std::fflush(stdout); + psx_scheduler_run(&cpu); #endif /* If we reach here, all execution completed without MMIO abort. diff --git a/runtime/src/traps.c b/runtime/src/traps.c index cd5e42bd..93d12ca9 100644 --- a/runtime/src/traps.c +++ b/runtime/src/traps.c @@ -30,6 +30,51 @@ void psx_exception_longjmp(void); jmp_buf g_scheduler_jmpbuf; psx_sched_escape_t g_sched_escape; +/* Single-level "switch back to the yielder" safety net. The old fiber bridge + * remembered each thread's return_fiber so a target whose dispatch RETURNS + * (rather than yielding back) resumed its yielder instead of killing the run. + * In the redispatch model we keep the same one-level notion: the yielder's TCB + * latched at the last YIELD_TO_TCB, restored if the target's top-level dispatch + * returns. Zero = no pending return target (top-level pc==0 is a real exit). */ +static uint32_t g_sched_return_tcb = 0; + +/* sched_escape_ring (plan step 8) — always-on record of every structured + * scheduler escape, queryable via the `sched_escape_ring` debug command. This + * is the deterministic replacement for the fiber thread_event kinds 3-12. */ +typedef struct SchedEscapeEntry { + uint32_t seq; + uint32_t frame; + uint32_t reason; /* psx_run_reason_t */ + uint32_t current_tcb; + uint32_t target_tcb; + uint32_t resume_pc; + uint32_t pc; + uint32_t ra; + uint32_t sp; +} SchedEscapeEntry; +#define SCHED_ESCAPE_RING_CAP 256u +SchedEscapeEntry g_sched_escape_ring[SCHED_ESCAPE_RING_CAP]; +uint64_t g_sched_escape_seq = 0; + +static void sched_escape_ring_log(CPUState* cpu, uint32_t reason, + uint32_t current_tcb, uint32_t target_tcb, + uint32_t resume_pc) +{ + extern uint64_t s_frame_count; + SchedEscapeEntry* e = + &g_sched_escape_ring[g_sched_escape_seq & (SCHED_ESCAPE_RING_CAP - 1u)]; + e->seq = (uint32_t)g_sched_escape_seq; + e->frame = (uint32_t)s_frame_count; + e->reason = reason; + e->current_tcb = current_tcb; + e->target_tcb = target_tcb; + e->resume_pc = resume_pc; + e->pc = cpu->pc; + e->ra = cpu->gpr[31]; + e->sp = cpu->gpr[29]; + g_sched_escape_seq++; +} + int psx_is_dispatchable(uint32_t pc) { /* Fail closed on the known-bad resume PCs that the old pc=0 sentinel + @@ -523,16 +568,158 @@ static int psx_change_thread_fiber(CPUState* cpu, uint32_t target_tcb) return 1; } +/* ── Deterministic TCB scheduler — LIVE (carve-out, replaces the fiber bridge) ─ + * + * psx_request_thread_switch performs a cooperative thread switch by committing + * the yielding thread's full context to its TCB, repointing dword_108->entry at + * the target, and longjmp-ing to the outer scheduler loop (psx_scheduler_run), + * which restores the target TCB and re-dispatches its EPC. No host fibers, no + * per-thread native stacks, no cpu->pc=0 host signal — thread selection is a + * pure function of guest TCB state (HLE_SCHEDULER_CARVEOUT_PLAN.md §9). + * + * This only runs with in_exception==0 (psx_syscall case 3 gates it), so the + * longjmp never unwinds through a live setjmp(exception_jmpbuf) frame — the sole + * native state skipped is nested psx_dispatch_impl frames, whose only invariant + * (g_psx_dispatch_depth + g_psx_call_bail) the scheduler loop resets on landing. + * + * INVARIANT (psx_scheduler.h): all guest thread context lives in CPUState / RAM + * / the TCB reg array — never in a host stack. The yielding thread resumes by + * re-dispatching its saved EPC, which therefore MUST be a dispatchable block + * leader (CLASS-B every-leader re-entry guarantees this); psx_is_dispatchable + * fails loud if it ever is not. */ +static int psx_request_thread_switch(CPUState* cpu, uint32_t target_tcb) +{ + uint32_t current_tcb = psx_current_tcb_ptr(cpu); + debug_server_log_thread_event(3, cpu, current_tcb, target_tcb, 0); + if (!psx_is_valid_tcb(cpu, current_tcb) || !psx_is_valid_tcb(cpu, target_tcb)) { + debug_server_log_thread_event(4, cpu, current_tcb, target_tcb, 0); + return 0; /* invalid TCB(s): caller falls back to the manual-RFE path */ + } + if (current_tcb == target_tcb) { + /* Same thread — not a switch. The syscall wrapper returns 0 and falls + * through to its own jr $ra (CPS); the thread simply continues. No + * cpu->pc=0 host signal (removed with the fiber bridge). */ + debug_server_log_thread_event(5, cpu, current_tcb, target_tcb, cpu->gpr[31]); + return 1; + } + /* The target's entire context already lives in its TCB; it must be runnable + * (0x4000). A non-runnable target is not a cooperative yield — let the + * manual-RFE fallback handle it rather than dispatch a stale EPC. */ + if (psx_tcb_state(cpu, target_tcb) != 0x4000u) { + debug_server_log_thread_event(7, cpu, current_tcb, target_tcb, 0); + return 0; + } + /* Commit the yielding thread's context so it can be re-dispatched later. + * resume PC = its return address (resume right after the ChangeThread call). + * Only a still-running (0x4000) thread keeps a live context; a closed/free + * (0x1000) thread is being torn down and must not be saved. */ + if (psx_tcb_state(cpu, current_tcb) == 0x4000u) { + psx_save_context_to_tcb(cpu, current_tcb, cpu->gpr[31]); + } + psx_set_current_tcb(cpu, target_tcb); + g_sched_return_tcb = current_tcb; /* one-level switch-back safety net */ + g_sched_escape.target_tcb = target_tcb; + g_sched_escape.resume_pc = 0; + g_sched_escape.reason = PSX_RUN_YIELD_TO_TCB; + sched_escape_ring_log(cpu, PSX_RUN_YIELD_TO_TCB, current_tcb, target_tcb, 0); + debug_server_log_thread_event(8, cpu, current_tcb, target_tcb, 0); + longjmp(g_scheduler_jmpbuf, 1); /* unwind to psx_scheduler_run; never returns */ + return 1; /* unreachable */ +} + +/* Scheduler mode (hidden dev toggle). HLE = the deterministic TCB scheduler + * (psx_request_thread_switch, default); LLE = the legacy host-fiber bridge + * (psx_change_thread_fiber). Read ONCE from PSX_HLE_SCHEDULER (default 1 = HLE) + * so the mode can't flip mid-run — a thread suspended under one scheduler cannot + * be resumed under the other (fibers keep a native stack; the TCB scheduler keeps + * none). Keep HLE until the upstream 0x800CD3F8 wedge root is fixed. */ +int psx_hle_scheduler_enabled(void) +{ + static int s_mode = -1; + if (s_mode < 0) { + const char* e = getenv("PSX_HLE_SCHEDULER"); + s_mode = (e && e[0] == '0') ? 0 : 1; /* default HLE */ + } + return s_mode; +} + static int psx_change_thread(CPUState* cpu, uint32_t target_tcb) { - /* One scheduler on every platform now: host fibers (Win32 Fibers on - * Windows, ucontext on POSIX). The earlier setjmp/longjmp fallback was - * removed — it ran target threads nested on the caller's stack, which - * couldn't resume an arbitrary suspended thread (the BIOS CD-boot - * WaitEvent handoff hung on it). */ + if (psx_hle_scheduler_enabled()) + return psx_request_thread_switch(cpu, target_tcb); return psx_change_thread_fiber(cpu, target_tcb); } +void psx_scheduler_run(CPUState* cpu) +{ + extern int g_psx_dispatch_depth; + g_sched_escape.reason = PSX_RUN_CONTINUE; + for (;;) { + if (setjmp(g_scheduler_jmpbuf) != 0) { + /* A structured escape unwound the native stack to here. Reset the + * invariants the skipped psx_dispatch_impl frames would otherwise + * own. No exception_jmpbuf frame is ever skipped (switch fires only + * with in_exception==0), so interrupt state needs no fixup here. */ + g_psx_dispatch_depth = 0; + g_psx_call_bail = 0; + } + + switch (g_sched_escape.reason) { + case PSX_RUN_FATAL: + trap_crash("scheduler: structured FATAL escape"); + return; + case PSX_RUN_GUEST_EXIT: + return; /* abnormal top-level exit — main.cpp dumps diagnostics */ + default: + break; /* CONTINUE / YIELD_TO_TCB / RESUME_CURRENT */ + } + + uint32_t run_pc; + if (g_sched_escape.reason == PSX_RUN_RESUME_CURRENT && + g_sched_escape.resume_pc != 0u) { + /* Same-thread RFE: GPRs already committed by the RFE; just re-dispatch. */ + run_pc = g_sched_escape.resume_pc; + } else { + uint32_t cur = psx_current_tcb_ptr(cpu); + if (psx_is_valid_tcb(cpu, cur)) { + run_pc = psx_restore_context_from_tcb(cpu, cur); + if (!psx_is_dispatchable(run_pc)) { + char b[112]; + snprintf(b, sizeof(b), + "scheduler: TCB 0x%08X has non-dispatchable resume PC 0x%08X", + (unsigned)cur, (unsigned)run_pc); + trap_crash(b); + return; + } + } else { + run_pc = cpu->pc; /* boot / pre-TCB: run the raw PC */ + } + } + + cpu->pc = run_pc; + g_sched_escape.reason = PSX_RUN_CONTINUE; + psx_dispatch(cpu, run_pc); + + /* psx_dispatch returned with NO structured escape => the outermost + * dispatch saw cpu->pc==0. If a thread yielded to us and then ran to + * completion, hand control back to its yielder (one-level safety net) + * instead of tearing down the whole run. Otherwise it's the legacy + * top-level abnormal exit. */ + if (psx_is_valid_tcb(cpu, g_sched_return_tcb) && + g_sched_return_tcb != psx_current_tcb_ptr(cpu)) { + uint32_t yielder = g_sched_return_tcb; + g_sched_return_tcb = 0; + psx_set_current_tcb(cpu, yielder); + g_sched_escape.reason = PSX_RUN_YIELD_TO_TCB; + sched_escape_ring_log(cpu, PSX_RUN_YIELD_TO_TCB, + psx_current_tcb_ptr(cpu), yielder, 0); + continue; + } + g_sched_escape.reason = PSX_RUN_GUEST_EXIT; + return; + } +} + int psx_syscall(CPUState* cpu, uint32_t code) { /* * PS1 BIOS SYSCALL convention: