From 6917d4a620fc3b6fdf769850688b41da290ade38 Mon Sep 17 00:00:00 2001 From: Alex Vanderveen Date: Sun, 2 Aug 2026 17:49:36 -0400 Subject: [PATCH] scheduler optimization on WAN --- docs/ROLLBACK_MOTK_HOOKUP.md | 273 ++++++++++++++++++++++ lib/recomp-net | 2 +- runtime/include/netplay_snap_ring.h | 3 + runtime/include/psx_netplay_rb.h | 2 + runtime/src/netplay_snap_ring.c | 14 ++ runtime/src/psx_netplay.c | 112 ++++++++- runtime/src/psx_netplay_rb.c | 249 ++++++++++++++++++-- runtime/src/psx_netplay_sched.c | 348 +++++++++++++++++++++++++--- 8 files changed, 954 insertions(+), 49 deletions(-) diff --git a/docs/ROLLBACK_MOTK_HOOKUP.md b/docs/ROLLBACK_MOTK_HOOKUP.md index 7c5e1b25..7b1fccb9 100644 --- a/docs/ROLLBACK_MOTK_HOOKUP.md +++ b/docs/ROLLBACK_MOTK_HOOKUP.md @@ -2912,3 +2912,276 @@ re-delivers it" — that cadence never existed. baselines + ready but no `rb replay`; if `rb waiting peer seal rows` appears it should be followed by `rb seal rexmit` and a normal Replay entry, not an abort. + +## 53. Live ADVANCE cadence + choose_load max floor (2026-08-02) + +**WAN soak (§52 binary):** match-start countdown mispredict felt like a deep +rewind into the intro. Logs: + +```text +host: agreed HEAL 1069→2336 (hc=2347) → begin load=2336 mismatch=2461 +guest: agreed HEAL 1069→2448 (hc=2460) → follow load=2336 (host BEGIN) +``` + +Depth 125 with snaps through 2448 and guest HC at the mismatch — Layer 1 +violated again. Seal keepalive (§52) was healthy (`seal rexmit` → Replay). + +**Root cause:** +1. `advance_agreed_watermark_from_hc` / aged HEAL only ran at begin/follow, so + `agreed` froze at tip-hold commit **1069** for ~1400 Live ticks until the + snap aged out (`oldest=1440`), then emergency HEAL jumped to each peer's + *local* HC interval (asymmetric: 2336 vs 2448). +2. `choose_load` set `shared = agreed` and refused to raise via HC ("ADVANCE + already promotes") — so initiator BEGIN locked both peers to the worse + HEAL. Host HC itself lagged (~2347 vs guest ~2460) from invent-fork stalls + in the 128-slot ring; peer RESOLVED from a Live-ADVANCE'd guest would have + proven 2448, but raise-via-peer was not allowed. + +**Fix:** +1. **Live cadence** — every `psx_netplay_rb_pump`: `hc_heal_stale_gap` + + `advance_agreed_watermark_from_hc` (existing guards skip episodes / + TipHold / post-tip-hold holdoff). Agreed tracks HC through countdown. +2. **`choose_load` raise** — outside tip-hold holdoff, + `shared = max(agreed, hc_interval_floor, peer_resolved)`. + +**§53b (same day — regression from first cut):** Live pump also ran aged +`heal_agreed_watermark_if_aged_out`, and raise used raw `hc_floor`. Soak: + +```text +choose_load raise 752→765 → follow NACK (frontier=752 unconfirmed tip) +HEAL-FORCE 752→816→880→944→1008 (hc stuck) → baseline core mismatch forever +``` + +HEAL-FORCE does not advertise; under a live HC stall it invented unconfirmed +agreed tips every pump. Raw hc tip (765) is not mutual with follower's +ADVANCE'd interval frontier (752). + +**Corrected:** +1. Live pump: **ADVANCE only** — aged HEAL / HEAL-FORCE stay at begin/follow. +2. Raise / peer-RESOLVED clamp floor: **HC-confirmed interval snaps only** + (`hc_interval_floor`), plus peer RESOLVED — never raw tip `hc_floor`. + +**Re-soak watch:** periodic `agreed ADVANCE …` through Live; no Live +`HEAL-FORCE` cascades; no `choose_load raise …→` non-interval tip; no +follow NACK on load just above agreed interval; no repeating +`baseline core mismatch` after first menu edge. Countdown mispredict still +loads near mismatch when peer RESOLVED / HC interval is current. + + +## 54. Silent permanent fork: cooldown-swallowed correction + no HC recovery (2026-08-02) + +**WAN soak (§53b binary):** resim-purity abort at sim≈1363 (local pad edge +sampled mid-resim on the guest corrupted the sealed tip) → episode ABORT → +cooldown. At sim=1370 a *completed-tick* pad mispredict arrived during the +cooldown window; reconcile's `no_resim` branch promoted the wire row into +history and moved on. The live state had already simmed the wrong pad — both +peers then ran ~1000 ticks at a smooth 60 fps with permanently diverged +cores. No new pad mispredicts occurred, so no episode ever opened again; +`FIRST CORE DIVERGE` was log-only. + +**Two gaps, two fixes (both in `psx_netplay.c`):** + +1. **Deferred rewind for cooldown promotes.** The `no_resim` + (sweep / cooldown / fmv-lockstep) branch now records the earliest + `pads_differ && completed` tick it promote-only'd + (`s_deferred_rw_{valid,tick,slot}`). Once the window ends + (`!no_resim && !fmv_defer`), reconcile either drops it — if + `hc_confirm_through(t)` proves both peers matched after the tick anyway — + or opens the episode the cooldown suppressed + (`rb deferred rewind t=… slot=…`). Refusals retry next pump. +2. **HC-fork recovery episode.** `np_check_core_diverge` no longer stops at + logging: if the `hc_peek_mismatch` fork persists ≥16 ticks (well before + the 128-slot hc ring wraps and peek goes quiet) with no episode / TipHold / + pending load / cooldown / FMV window, the **slot-0 host** (single-initiator, + avoids dual-BEGIN collision) opens a recovery episode at the fork tick + (`rb hc-fork recovery begin t=…`), correction seat = remote slot. Retries + every 32 ticks while the fork persists; guest follows via the normal wire + SYNC. `choose_load` starts it at the newest provably-safe snap (HC is + confirmed through fork−1 by definition). + +Fix 1 should catch the divergence at its source; fix 2 is the backstop for +*any* fork that slips past pad-level reconciliation (invent races, purity +bugs, CD timing). + +**Re-soak watch:** after any `episode ABORT` + cooldown, a mispredict inside +the window should be followed by `rb deferred rewind …` → normal Replay (or +silence if HC confirmed through it); no repeat of the smooth-60fps silent +fork — if cores split without a pad edge, expect `rb hc-fork recovery begin` +within ~16 ticks and digests re-matching after the episode. + +## 55. Resim purity + dead-timeline evidence + fork bisect (2026-08-02) + +**WAN soak (§54 binary):** §54's hc-fork recovery healed two real silent forks +(t=2627 after a seal-timeout abort, t=4049 after a tip-extend abandon) — +matched baselines, clean replay, session continued. The session still died at +sim≈4160 to a three-stage failure: + +1. **Purity leak (root):** during the epoch=145 tip-extend chain the host + armed tick 4179 with the guest seat unsealed — `audit arm sim=4179 … + s1=----`. `rnet_rb_extend_target` zeroes remote-seat rows (peer SEAL_ROWS + fill them later) and `publish_sealed_sio` silently falls back to live hist + for any missing seat. The per-tick audit digest is CPU-only, so the RAM + fork was invisible until the FRAME_COMMIT abort at 4180 + (`resim core diverge`). +2. **Dead-timeline evidence:** both peers realigned to the matched 4160 snap, + then the guest ran `agreed ADVANCE 4160→4176 (hc=4179 live confirm)` — hash + confirms from the *aborted* timeline. Each peer re-simmed 4161..4176 with + its own history → permanently diverged 4176 snaps (`4ced9093` vs + `accdaaec`) that choose_load kept certifying as provably safe. +3. **Same-snap retry loop:** four recovery episodes all loaded 4176, all died + on `baseline core mismatch`, cooldowns escalated to the DESYNC + storm-breaker, the fork tick aged out of the 128-slot hc ring, peek went + quiet, and both sides ran silently forked at 60 fps until disconnect. + +**Fixes (one per stage):** + +1. **Seal authority (lib `rnet_rollback.c` + runtime arm gate).** + - `rnet_rb_fill_local_row`: remote seats now pre-seal from host history + when the row is wire-confirmed (`!is_predicted`) — that is the owner's + transmitted pad, byte-identical to what the owner seals, so it is + authoritative without waiting for SEAL_ROWS. Predicted rows stay + unsealed. + - `rnet_rb_resign_slot_range`: the peer-seal mask bit is only credited for + wire-confirmed rows on remote seats — invented pads no longer count as + authority (they used to satisfy `all_peer_seal_rows_complete`). + - New `rnet_rb_seat_row_authoritative()` + runtime `rb_arm_rows_ready()`: + Replay may not arm a sealed-span tick until every seat's row is + authoritative (local / peer-delivered / wire-confirmed). Stall sites: + `try_admit`, the `finish_frame` chain-arm, and `arm_rereplay_after_load` + (`rb arm wait rows sim=… slot=…`). Seal keepalive (§52) redelivers rows; + `RB_REPLAY_STALL_MS` aborts a genuinely dead episode. +2. **Realign evidence clamp** (`realign_invalidate_evidence`, at realign + queue AND apply): hash_confirm primed after the realign tick, agreed / + peer-RESOLVED / local-advertised watermarks clamped to it, and ring snaps + above it dropped (`netplay_snap_ring_drop_after`). Post-realign live + FRAME_COMMITs then re-prove (or immediately re-flag) the timeline from the + realign point instead of inheriting confirms for states that no longer + exist. Log: `rb realign evidence clamp tick=… (snaps_dropped=…)`. +3. **Fork bisect cap** (`g_bl_fork_cap`): a baseline core mismatch at load L + proves the fork predates L. choose_load now forces the next load strictly + below the failed one (`rb fork cap`, `rb choose_load bisect load=…`), + refusing episodes when nothing provably safe remains below. Cleared on any + verified commit or session reset. + +**Re-soak watch:** no `arm … s?=----` audits inside a sealed span (grep +`s0=----|s1=----` between `rb replay` and POST); occasional +`rb arm wait rows` immediately healed by `rb seal rexmit` is fine; after any +abort expect `rb realign evidence clamp` and NO `agreed ADVANCE` above the +realign tick until fresh live confirms; if a baseline mismatch does occur, +the next attempt should log `choose_load bisect` at a lower load instead of +retrying the same snap 4×. + +## 56. Scheduler equilibrium: cushion-rebuild exit, pipeline slack, scorecard (2026-08-02) + +**Post-§55 soak:** correctness held (no baseline mismatches, no desync, one +self-healed aux diverge) but the scheduler never reached its own contract. +`lead_avg` sat at 0…−1 the whole session instead of ≈ D=2, `invent_gap1` +climbed continuously (~225 by session end), and `cushion rebuilt` fired at +`remote_lead=1` with D=2 — one tick short of full. The user-reported symptom +is the consequence: episodes fire back-to-back because live play permanently +runs at the edge of the cushion, so every arrival wobble becomes an invent → +mispredict → episode. + +Reordered roadmap (user, 2026-08-02): 1) scheduler equilibrium, 2) continuous +replay ownership, 3) dense live snapshots, 4) jitter-driven adaptive D, +5) episode extension/reuse. This section is step 1's first slice: fix the one +provable exit bug and instrument the pipeline so the remaining phase loss is +measured, not guessed. + +**Changes:** + +1. **Cushion-rebuild exit (`np_sched_pre_admit`).** The old exit cleared at + `remote_lead >= D-1` — the rebuild "completed" one tick short and the next + miss invented into the missing tick. Now: `lead >= D` clears immediately + (`rb cushion rebuilt FULL`); `lead == D-1` only clears after holding + 400 ms (`rb cushion rebuilt NEAR … held=…ms`), giving the pipeline a real + chance to top up while still never deadlocking a link that genuinely + plateaus at D−1. Below D−1 the behavior is unchanged (wait; absurd-lead + guard intact). Rebuild-window start is stamped at + `np_sched_note_episode_boundary`. +2. **Pipeline slack instrumentation (lib + sched).** New + `rnet_session_remote_arrival_age_ms(s, slot, wire)`: every remote wire row + gets a monotonic arrival stamp at `store_remote_frame` (first-wins, + 128-deep, mirrors the ring); the scheduler samples, once per sim at + pre-admit, how long the row for the consumption wire had been sitting + there. That is the *time-domain* cushion the tick-domain `remote_lead` + quantizes away: healthy ≈ D·16 ms − transit; ≈0 ms means rows arrive + just-in-time and the cushion exists on paper only. Misses at need are + counted via `on_remote_miss` (`miss_need`). +3. **Equilibrium scorecard (~30 s cadence, both peers).** + `rb scorecard dt=… ep=+N depth_avg=… gap1=+… tip_stale=+… miss_need=… + lead avg/min/max slack avg/min/max n=… D=… cushion=…` — per-window deltas + (episode count from `psx_netplay_rb_episode_count`, replay depth from the + new cumulative `psx_netplay_rb_replay_ticks_total`, which is independent + of the FPS logger's take-and-reset counter). Scheduler changes should move + `ep`/`gap1`; snapshot changes should move `depth_avg` — the two are now + separable per soak window. + +**Re-soak watch:** compare `rb scorecard` lines across the session and +between peers. Expect `cushion rebuilt FULL` (or NEAR with held≈400ms) after +episodes instead of instant lead=1 exits. The `slack avg` value is the key +new datum: if it reads near zero while `lead avg` looks healthy, the produce +→consume pipeline is consuming the whole cushion in phase (production or +send timing), and that — not invent policy — is the next equilibrium fix; if +slack is healthy but `miss_need`/`gap1` still climb, the losses are arrival +jitter and step 4 (jitter-driven adaptive D) moves up. + +## 57. Arrival-driven adaptive D + transit-aware targets (2026-08-02) + +**§56 soak verdict:** the scorecard closed the "where did the cushion go?" +question. Slack was NOT near zero (24–36 ms of buffered arrival age), lead +matched the equilibrium the delay budget permits once transport is counted +(`lead ≈ D−1−transit`: predicted 0.6/2.1 for the two peers, measured +0.24/0.91), yet `miss_need` stayed at ~25% of ticks and episodes scaled with +`miss_need`, not with scheduler instability. Conclusion: the scheduler +converges to the equilibrium permitted by the current delay budget and the +measured transport latency — the budget itself is what's short. D=4 minus +1 pipeline tick minus ~2.5 ticks of one-way transit leaves ~0.5 ticks of +margin; every jitter excursion becomes a miss → invent → mispredict lottery. +Correctness layer held throughout (§54 hc-fork recovery healed the early +fork; two resim-diverge aborts realigned cleanly under the §55 evidence +clamp; zero baseline mismatches / desyncs). + +Roadmap consequence (user-confirmed): adaptive provisioning moves to the +front — it reduces how OFTEN rollback happens; dense snapshots then reduce +how MUCH each one costs; replay ownership the efficiency of the remainder. + +**Changes:** + +1. **Auto-delay signal replaced: RTT → arrival stream** (`np_auto_delay_tick` + §57 rewrite). The POST RTT EMA measured 80 ms on the host and 12 ms on the + guest for the same link — unusable. The controller now consumes exactly + what the scheduler consumes, per 5 s eval window of LIVE ticks only + (episode / tip-hold / |lead|≥32 samples excluded): + - `miss_rate` — rows absent at need (from `on_remote_miss`, once/wire); + - `lateness` — first-miss → arrival delta for misses we waited out + (timer armed in `note_miss`, resolved in `note_remote_hit`, invalidated + if the miss was invented instead; >2 s artifacts discarded); + - `transit_est` — EMA of `D−1−lead_avg` in 1/16-tick units, maintained on + BOTH peers (drives local targets; host alone proposes DELAY_SYNC). + Raise: `miss_rate > 2%` → `D += ceil(late_avg/tick)` (1..2/step), confirm + on 2 evals (10 s). Lower: `miss_rate < 0.2%` AND ≥2 spare ticks of lead, + confirm on 3 evals + 30 s cooldown + no pcap freeze in 30 s. Asymmetric + on purpose: a raise costs only input latency, a miss costs determinism. + Log: `rb auto delay X → Y (arrival: miss=… late avg/max … lead_avg=… + transit_est=…)`. The §22 reactive pcap-storm bump is unchanged. +2. **Cushion-rebuild target is now the achievable lead** (`FULL` unchanged at + `lead ≥ D`; the D−1 NEAR exit is replaced by + `achievable = clamp(D−1−round(transit_est), 1, D−1)` with the 400 ms + hold → `rb cushion rebuilt ACH …`). §56's NEAR exit spent 12.6 s parked + against a target the link physically cannot reach. +3. **Scorecard gains `transit_est=…`** so soak logs show the controller's + view of the link next to the raw lead/slack numbers. + +**Re-soak watch:** expect `rb auto delay ON (§57 …)` at session start and, +on the WAN link, a stepped climb (4→5/6 within ~20 s of fight start) with +each step logging its arrival evidence; after it settles, `rb scorecard` +should show `miss_need` collapsing toward zero, `gap1` deltas an order of +magnitude down, `ep=+N` per window dropping toward low single digits, and +`lead avg` rising to ≈ `D−1−transit_est`. `cushion rebuilt ACH` should +replace the long NEAR holds. If `miss_need` stays high at D=6–7 with +`transit_est` stable, the residue is loss/retransmit (wire holes), not +provisioning — investigate `rb wire hole` lines before raising D further. +Trade-off to feel for in play: each +1 D adds ~17 ms of local input latency; +if the WAN session feels laggy but smooth, that's the controller working — +step 3 (dense snapshots) then attacks replay cost so D can ride lower. diff --git a/lib/recomp-net b/lib/recomp-net index 1c41a137..ca18d96e 160000 --- a/lib/recomp-net +++ b/lib/recomp-net @@ -1 +1 @@ -Subproject commit 1c41a137055841ccef81bf33d3e32948e64e5882 +Subproject commit ca18d96ec2d47923c133eb1e5fa1a797d1aa51ba diff --git a/runtime/include/netplay_snap_ring.h b/runtime/include/netplay_snap_ring.h index e52fbebb..eb7ebb07 100644 --- a/runtime/include/netplay_snap_ring.h +++ b/runtime/include/netplay_snap_ring.h @@ -50,6 +50,9 @@ const uint8_t* netplay_snap_ring_peek(const NetplaySnapRing* r, uint32_t tick, /* Oldest/newest occupied tick, or 0 if empty (ambiguous with tick 0 — use * count() first). */ +/* Invalidate every snapshot with tick > tick (dead-timeline snaps after an + * abort realign). Returns the number of slots dropped. */ +uint32_t netplay_snap_ring_drop_after(NetplaySnapRing* r, uint32_t tick); uint32_t netplay_snap_ring_oldest_tick(const NetplaySnapRing* r); uint32_t netplay_snap_ring_newest_tick(const NetplaySnapRing* r); diff --git a/runtime/include/psx_netplay_rb.h b/runtime/include/psx_netplay_rb.h index a55445d3..2c6411b7 100644 --- a/runtime/include/psx_netplay_rb.h +++ b/runtime/include/psx_netplay_rb.h @@ -168,6 +168,8 @@ uint32_t psx_netplay_rb_snap_count(void); * by the host [FPS] line (PSX_NETPLAY_TIMING=1) to report what fraction of * the window was spent resimulating vs running Live. */ uint64_t psx_netplay_rb_take_replay_ticks(void); +/* §56: cumulative replay ticks (never reset) — scheduler scorecard deltas. */ +uint64_t psx_netplay_rb_replay_ticks_total(void); /* Live network-latency estimate (ms), EMA'd from the POST handshake's own * send/receive timestamps each episode commit; 0 = no episode has round- diff --git a/runtime/src/netplay_snap_ring.c b/runtime/src/netplay_snap_ring.c index 3716555d..55d7abc0 100644 --- a/runtime/src/netplay_snap_ring.c +++ b/runtime/src/netplay_snap_ring.c @@ -178,6 +178,20 @@ int netplay_snap_ring_load(NetplaySnapRing* r, uint32_t tick, CPUState* cpu, return boot_state_load_buffer(data, size, bios_checksum, entry_pc, cpu); } +uint32_t netplay_snap_ring_drop_after(NetplaySnapRing* r, uint32_t tick) { + uint32_t i, n = 0; + if (!r) return 0; + for (i = 0; i < r->depth; i++) { + if (!r->slots[i].valid) continue; + if (r->slots[i].tick > tick) { + slot_clear(&r->slots[i]); + if (r->count) r->count--; + n++; + } + } + return n; +} + uint32_t netplay_snap_ring_oldest_tick(const NetplaySnapRing* r) { uint32_t i, oldest = 0xffffffffu; int any = 0; diff --git a/runtime/src/psx_netplay.c b/runtime/src/psx_netplay.c index e4d670c6..84f8f9e5 100644 --- a/runtime/src/psx_netplay.c +++ b/runtime/src/psx_netplay.c @@ -358,12 +358,34 @@ typedef struct NpPartSlot { static NpPartSlot s_part_ring[NP_PART_RING]; static int s_core_diverge_logged; static uint32_t s_live_dig_last_tick = 0xffffffffu; +/* §54: HC-fork recovery — a live core fork with no pad mispredict never + * opened an episode (FIRST CORE DIVERGE was log-only), so a swallowed + * correction desynced the rest of the session silently at 60fps. Track how + * long the hash_confirm peek mismatch has persisted and open a recovery + * episode at the fork tick. */ +static uint32_t s_fork_tick = 0xffffffffu; +static uint32_t s_fork_first_sim; +static uint32_t s_fork_last_attempt_sim; +/* §54: completed-tick mispredict promoted during cooldown/sweep — the state + * already simmed with the wrong pad and nothing re-opens an episode after + * cooldown expires. Defer the rewind instead of dropping it. */ +static int s_deferred_rw_valid; +static uint32_t s_deferred_rw_tick; +static int s_deferred_rw_slot; +static int s_deferred_rw_logged; static void np_part_ring_reset(void) { memset(s_part_ring, 0, sizeof(s_part_ring)); s_core_diverge_logged = 0; s_live_dig_last_tick = 0xffffffffu; + s_fork_tick = 0xffffffffu; + s_fork_first_sim = 0; + s_fork_last_attempt_sim = 0; + s_deferred_rw_valid = 0; + s_deferred_rw_tick = 0; + s_deferred_rw_slot = -1; + s_deferred_rw_logged = 0; } static void np_part_ring_put(uint32_t tick, const NetplayCoreParts *parts, @@ -401,15 +423,63 @@ static void np_log_live_digest(uint32_t tick, const NetplayCoreParts *parts, fflush(stderr); } +/* §54: persistent live core fork with no pad mispredict → open a recovery + * episode at the fork tick. Only the slot-0 host initiates (avoids a dual + * simultaneous-BEGIN collision); the guest follows via the wire SYNC. The + * episode reloads from the newest provably-safe snap (≤ fork tick — HC is + * confirmed through fork-1 by definition) and reseals both seats from + * wire-authoritative history, converging the states. */ +static void np_try_hc_fork_recovery(uint32_t fork_tick) +{ + uint32_t sim; + int remote; + /* Fire before the 128-slot hc ring wraps and peek goes quiet. */ + const uint32_t persist_ticks = 16u; + const uint32_t retry_ticks = 32u; + + if (!g_np.rollback || !g_np.session) + return; + if (g_np.local_slot != 0) + return; /* initiator side only */ + sim = rnet_session_sim_tick(g_np.session); + if (fork_tick != s_fork_tick) { + s_fork_tick = fork_tick; + s_fork_first_sim = sim; + return; + } + if (sim < s_fork_first_sim || (sim - s_fork_first_sim) < persist_ticks) + return; + if (s_fork_last_attempt_sim != 0u && sim >= s_fork_last_attempt_sim && + (sim - s_fork_last_attempt_sim) < retry_ticks) + return; + if (psx_netplay_rb_active() || psx_netplay_rb_tip_holding() || + psx_netplay_rb_load_pending() || psx_netplay_rb_rewind_suppressed() || + psx_netplay_rb_fmv_defer_rewind() || psx_netplay_rb_lockstep_no_invent()) + return; + s_fork_last_attempt_sim = sim; + remote = (g_np.local_slot == 0) ? 1 : 0; + if (g_np.slot_count < 2) + remote = g_np.local_slot; + fprintf(stderr, + "psxrecomp: rb hc-fork recovery begin t=%u (persisted %u ticks, " + "no pad mispredict — state resync episode)\n", + (unsigned)fork_tick, (unsigned)(sim - s_fork_first_sim)); + fflush(stderr); + (void)psx_netplay_rb_begin_rewind(fork_tick, remote); +} + static void np_check_core_diverge(void) { uint32_t tick = 0, local_d = 0, peer_d = 0; const NpPartSlot *slot; - if (!netplay_hc_peek_mismatch(&g_np.hc, &tick, &local_d, &peer_d)) + if (!netplay_hc_peek_mismatch(&g_np.hc, &tick, &local_d, &peer_d)) { + s_fork_tick = 0xffffffffu; /* resolved / no complete pair pending */ return; + } /* Replay/Verify: never allow a false POST commit after mid-resim fork. */ if (psx_netplay_rb_abort_resim_core_mismatch(tick, local_d, peer_d)) return; + np_try_hc_fork_recovery(tick); if (s_core_diverge_logged) return; s_core_diverge_logged = 1; @@ -1669,6 +1739,34 @@ static void np_rollback_reconcile_wire(void) no_resim_why = "fmv-lockstep"; } sim = rnet_session_sim_tick(g_np.session); + + /* §54: deferred rewind for a completed-tick correction swallowed by + * cooldown/sweep. HC confirm through the tick proves both peers matched + * after it (mispredict was cosmetic / already healed) — drop. Otherwise + * open the episode the cooldown suppressed. */ + if (s_deferred_rw_valid && !no_resim && !fmv_defer && + !g_np.pending_rewind && g_np.rollback) { + if (netplay_hc_confirm_through(&g_np.hc, s_deferred_rw_tick)) { + s_deferred_rw_valid = 0; + } else if (!psx_netplay_rb_active() && !psx_netplay_rb_tip_holding() && + !psx_netplay_rb_load_pending()) { + if (!s_deferred_rw_logged) { + fprintf(stderr, + "psxrecomp: rb deferred rewind t=%u slot=%d " + "(completed-tick correction was promote-only in " + "cooldown)\n", + (unsigned)s_deferred_rw_tick, s_deferred_rw_slot); + fflush(stderr); + s_deferred_rw_logged = 1; + } + if (psx_netplay_rb_begin_rewind(s_deferred_rw_tick, + s_deferred_rw_slot)) { + g_np.needs_advance = 0; + s_deferred_rw_valid = 0; + } + /* refused (transient gate) — keep pending, retry next pump */ + } + } for (slot = 0; slot < g_np.slot_count; ++slot) { if (slot == g_np.local_slot) continue; for (t = (sim > 64u) ? (sim - 64u) : 0u; t <= sim; ++t) { @@ -1732,6 +1830,18 @@ static void np_rollback_reconcile_wire(void) if (no_resim) { (void)netplay_ih_promote(&g_np.ih, slot, &wire_frame); if (pads_differ) { + /* §54: a COMPLETED tick was simmed with the wrong pad and + * this promote repairs only history — the live state is + * now forked. Remember the earliest such tick and re-open + * an episode once the cooldown/sweep window ends (unless + * HC proves both peers matched through it anyway). */ + if (completed && + (!s_deferred_rw_valid || t < s_deferred_rw_tick)) { + s_deferred_rw_valid = 1; + s_deferred_rw_tick = t; + s_deferred_rw_slot = slot; + s_deferred_rw_logged = 0; + } if (n_no_resim == 0) { first_no_resim_t = t; first_no_resim_slot = slot; diff --git a/runtime/src/psx_netplay_rb.c b/runtime/src/psx_netplay_rb.c index 69f7ba7e..7b9b2d8a 100644 --- a/runtime/src/psx_netplay_rb.c +++ b/runtime/src/psx_netplay_rb.c @@ -79,6 +79,7 @@ void psx_netplay_rb_cpu_for_present_digest(struct CPUState *out, } uint32_t psx_netplay_rb_episode_count(void) { return 0; } uint64_t psx_netplay_rb_take_replay_ticks(void) { return 0; } +uint64_t psx_netplay_rb_replay_ticks_total(void) { return 0; } int psx_netplay_rb_phase(void) { return 0; } uint32_t psx_netplay_rb_snap_count(void) { return 0; } uint32_t psx_netplay_rb_sticky_bb_pc(void) { return 0; } @@ -128,6 +129,10 @@ static uint32_t g_episode_count; * "we spent NN% of this window resimulating" instead of inferring it from * episode density in the raw log. */ static uint64_t g_stat_replay_ticks; +/* §56 scorecard: cumulative (never reset on read) replay-tick counter so the + * scheduler scorecard can compute per-window deltas independently of the FPS + * logger's take-and-reset counter above. */ +static uint64_t g_stat_replay_ticks_total; static int g_pending_save_valid; static uint32_t g_pending_save_tick; static int g_pending_load_valid; @@ -430,6 +435,10 @@ static uint32_t g_peer_commit_tick; * keep Live running (no more rubber-band rewinds) and arm a long cooldown. */ static uint32_t g_fork_streak_tick; static uint32_t g_fork_streak; +/* §55: a baseline core mismatch at load L proves the fork predates L. Retrying + * L (2026-08-02 soak: 4× at 4176) can never heal — the next episode load must + * be strictly below it. 0 = no cap; cleared on verified commit / session reset. */ +static uint32_t g_bl_fork_cap; static RNetSession *sess(void); /* fwd — used by FMV lockstep window */ static void commit_episode(void); /* fwd — legacy immediate commit */ @@ -961,6 +970,35 @@ static void pin_baseline_from_ring(uint32_t tick) fflush(stderr); } +/* §55: an abort realign rewinds the state to `tick`, but every piece of sync + * evidence above it (hash confirms, agreed watermark, peer RESOLVED, ring + * snaps) describes the DISCARDED timeline. 2026-08-02 soak: right after + * realigning to 4160 the guest ran `agreed ADVANCE 4160→4176 (hc=4179)` on + * dead-timeline confirms; both peers then re-simmed 4161..4176 differently, + * and choose_load kept certifying the forked 4176 snap as provably safe — + * every recovery episode died on baseline core mismatch. Clamp it all. */ +static void realign_invalidate_evidence(uint32_t tick) +{ + uint32_t dropped = 0; + if (g_b.hc) + netplay_hc_prime_after(g_b.hc, tick); + if (g_agreed_valid && g_agreed_through > tick) { + g_agreed_through = tick; + if (g_agreed_span_lo > tick) + g_agreed_span_lo = tick; + } + if (g_peer_resolved_through > tick) + g_peer_resolved_through = tick; + if (g_local_advertised_through > tick) + g_local_advertised_through = tick; + if (g_snaps) + dropped = netplay_snap_ring_drop_after(g_snaps, tick); + fprintf(stderr, + "psxrecomp: rb realign evidence clamp tick=%u (snaps_dropped=%u)\n", + (unsigned)tick, (unsigned)dropped); + fflush(stderr); +} + static void schedule_live_realign(uint32_t tick, const char *why) { RNetSession *s = sess(); @@ -976,6 +1014,10 @@ static void schedule_live_realign(uint32_t tick, const char *why) g_pending_load_tick = tick; g_pending_load_valid = 1; g_pending_resume_valid = 0; + /* §55: clamp immediately (not only at apply) — the pump's Live ADVANCE + * runs between queue and apply and would promote agreed from the dead + * timeline's hash confirms. */ + realign_invalidate_evidence(tick); /* Snap is post-tick state; continue Live at tick+1 (same as episode commit). */ if (s) rnet_session_set_sim_tick(s, tick + 1u); @@ -1164,6 +1206,19 @@ static void abort_episode_realign(const char *why) * reopening into the doomed baseline (2026-08-02 soak: 986/944 fork, * ~40 abort-realign cycles). Any commit resets the streak. */ if (why && strstr(why, "baseline") != NULL && have) { + /* §55: remember the mismatched load — next choose_load bisects below. */ + { + uint32_t failed_load = g_rb ? rnet_rb_get_load_tick(g_rb) : 0u; + if (failed_load > 0u && + (g_bl_fork_cap == 0u || failed_load < g_bl_fork_cap)) { + g_bl_fork_cap = failed_load; + fprintf(stderr, + "psxrecomp: rb fork cap — next load must be < %u " + "(baseline mismatch there)\n", + (unsigned)g_bl_fork_cap); + fflush(stderr); + } + } if (tick == g_fork_streak_tick) g_fork_streak++; else { @@ -1900,10 +1955,15 @@ static void advance_agreed_watermark_from_hc(void) * choose_load returns the newest snapshot both peers can prove is safe * to replay from — independent of how Replay later reaches the frontier. * - * Provably-safe sources (any establishes the floor): + * Provably-safe sources (any establishes the floor — take the newest): * - tip-hold / episode commit watermark (g_agreed_through + dense span) * - hash_confirm resolved_through (both simmed, cores matched → same snaps) - * - peer RESOLVED advertise (soft; must NOT lower below HC-proven ticks) + * - peer RESOLVED advertise (raise-safe when advertised from HC ADVANCE/HEAL; + * must NOT lower below HC-proven ticks — §51) + * + * §53/§53b: Live ADVANCE (HC-confirmed interval only) runs every pump. + * choose_load may raise to max(agreed, hc_interval_floor, peer_resolved) — + * never raw hc tip ticks (soak: raise 752→765 → follow NACK). * * Without the shared-frontier path, mismatch one tick after a commit re-loaded * a full interval + slack deep (798 → 768) — main-menu "resim back to title". @@ -1924,7 +1984,37 @@ static uint32_t hc_provable_through(void) return rt; } -static int choose_load_tick(uint32_t mismatch, uint32_t *out_load) +/* Highest HC-confirmed *interval* snap in-ring at or below hc_floor. + * Follower frontiers / RESOLVED adverts are ADVANCE'd interval ticks only — + * raising choose_load to a raw tip (e.g. 765 with agreed=752) NACKs. */ +static uint32_t hc_interval_floor(uint32_t hc_floor, uint32_t iv) +{ + uint32_t t; + uint32_t oldest; + uint32_t newest; + if (hc_floor == 0u || !g_snaps || !g_b.hc) + return 0u; + if (iv < 1u) + iv = 1u; + oldest = netplay_snap_ring_oldest_tick(g_snaps); + newest = netplay_snap_ring_newest_tick(g_snaps); + t = hc_floor; + if (t > newest) + t = newest; + for (;;) { + if (t < oldest) + return 0u; + if (netplay_snap_ring_has(g_snaps, t) && + ((iv <= 1u) || ((t % iv) == 0u)) && + netplay_hc_confirm_through(g_b.hc, t)) + return t; + if (t == 0u) + return 0u; + t--; + } +} + +static int choose_load_tick_inner(uint32_t mismatch, uint32_t *out_load) { uint32_t t; uint32_t iv; @@ -1932,6 +2022,7 @@ static int choose_load_tick(uint32_t mismatch, uint32_t *out_load) uint32_t cap; uint32_t shared = 0; uint32_t hc_floor = 0; + uint32_t hc_iv = 0; int have_shared = 0; if (!out_load || !g_snaps || netplay_snap_ring_count(g_snaps) == 0) return 0; @@ -1945,17 +2036,47 @@ static int choose_load_tick(uint32_t mismatch, uint32_t *out_load) heal_agreed_watermark_if_aged_out(); advance_agreed_watermark_from_hc(); hc_floor = hc_provable_through(); + hc_iv = hc_interval_floor(hc_floor, iv); if (g_agreed_valid) { - /* Commit / ADVANCE watermark. Do not raise above agreed via HC here — - * ADVANCE already promotes interval HC ticks; TipHold invent historically - * false-confirmed tip snaps above the commit frontier. */ shared = g_agreed_through; have_shared = 1; + } else if (hc_iv > 0u) { + shared = hc_iv; + have_shared = 1; } else if (hc_floor > 0u) { + /* No interval snap yet — soft floor only when nothing else exists. */ shared = hc_floor; have_shared = 1; } + /* §53b: raise only to HC-confirmed interval snaps or peer-advertised + * RESOLVED — never raw hc tip (752→765 NACK). Peer RESOLVED is raise-safe + * (ADVANCE/HC-HEAL/commit advertise; never HEAL-FORCE). */ + if (!post_tip_hold_holdoff_active()) { + uint32_t raised = have_shared ? shared : 0u; + if (hc_iv > raised) + raised = hc_iv; + if (g_peer_resolved_through > raised) + raised = g_peer_resolved_through; + if (raised > 0u && (!have_shared || raised > shared)) { + if (have_shared && raised > shared) { + static uint32_t s_raise_log; + if (s_raise_log != raised) { + fprintf(stderr, + "psxrecomp: rb choose_load raise %u→%u " + "(agreed=%u hc_iv=%u peer_resolved=%u)\n", + (unsigned)shared, (unsigned)raised, + (unsigned)(g_agreed_valid ? g_agreed_through : 0u), + (unsigned)hc_iv, + (unsigned)g_peer_resolved_through); + fflush(stderr); + s_raise_log = raised; + } + } + shared = raised; + have_shared = 1; + } + } /* §38/§39: clamp ADVANCE→commit-frontier ONLY during post–tip-hold * holdoff. The always-on ≤runway+4 clamp (holdoff=0) undid healthy * ADVANCE 757→784 and forced SPAN CAP load=752 → tip-extend/verify @@ -1975,21 +2096,20 @@ static int choose_load_tick(uint32_t mismatch, uint32_t *out_load) shared = tip_rt; } } - /* §40/§41/§51: peer RESOLVED may limit unconfirmed tips above HC, but must - * never lower the load below HC-proven ticks (soak: ADVANCE 1189→1248 then - * clamp→1189 → interval load=1184 → 65-tick SPAN storm). */ + /* §40/§41/§51/§53b: peer RESOLVED may limit unconfirmed tips above HC, + * but must never lower the load below HC-proven *interval* ticks. */ if (have_shared && peer_resolved_gate_blocks(shared)) { uint32_t clamped = g_peer_resolved_through; - if (hc_floor > clamped) - clamped = hc_floor; + if (hc_iv > clamped) + clamped = hc_iv; if (clamped < shared) { static uint32_t s_peer_clamp_log_to; if (s_peer_clamp_log_to != clamped) { fprintf(stderr, "psxrecomp: rb choose_load clamp %u→%u " - "(peer RESOLVED; hc_floor=%u)\n", + "(peer RESOLVED; hc_iv=%u)\n", (unsigned)shared, (unsigned)clamped, - (unsigned)hc_floor); + (unsigned)hc_iv); fflush(stderr); s_peer_clamp_log_to = clamped; } @@ -2085,6 +2205,44 @@ static int choose_load_tick(uint32_t mismatch, uint32_t *out_load) return 0; } +static int choose_load_tick(uint32_t mismatch, uint32_t *out_load) +{ + /* §55: after a baseline core mismatch at load L the fork provably predates + * L — retrying the same snap can never heal (soak: 4 consecutive aborts at + * 4176). Bisect: force the next load strictly below the failed one. */ + if (g_bl_fork_cap > 0u) { + uint32_t m = mismatch; + if (m > g_bl_fork_cap) + m = g_bl_fork_cap; /* inner's shared walk returns load < m */ + if (!choose_load_tick_inner(m, out_load)) + return 0; + if (*out_load >= g_bl_fork_cap) { + static uint32_t s_refuse_log; + if (s_refuse_log != g_bl_fork_cap) { + fprintf(stderr, + "psxrecomp: rb choose_load refuse load=%u ≥ fork cap %u " + "(no provably-safe snap below the mismatched baseline)\n", + (unsigned)*out_load, (unsigned)g_bl_fork_cap); + fflush(stderr); + s_refuse_log = g_bl_fork_cap; + } + return 0; + } + { + static uint32_t s_bisect_log; + if (s_bisect_log != *out_load) { + fprintf(stderr, + "psxrecomp: rb choose_load bisect load=%u (< fork cap %u)\n", + (unsigned)*out_load, (unsigned)g_bl_fork_cap); + fflush(stderr); + s_bisect_log = *out_load; + } + } + return 1; + } + return choose_load_tick_inner(mismatch, out_load); +} + /* Follower's confirmed frontier for the NACK wire hint: the deepest tick this * peer can prove agreement through (agreed watermark, else mutually * hash-confirmed). 0 = unknown. The initiator demotes to this instead of @@ -2605,10 +2763,44 @@ static void log_resim_tick_audit(uint32_t sim, const char *tag) fflush(stderr); } +/* §55: resim purity — a sealed-span tick may only be armed once every seat's + * row is authoritative (local, peer-delivered, or wire-confirmed). The + * 2026-08-02 soak forked when the host armed 4179 with the guest seat + * unsealed (s1=----): publish fell back to live hist, RAM forked silently + * (per-tick audit digest is CPU-only), and the FRAME_COMMIT abort at 4180 + * cascaded into an unrecoverable baseline-mismatch spiral. Stalling here is + * safe: seal keepalive (§52) redelivers rows and the RB_REPLAY_STALL_MS + * watchdog aborts a genuinely dead episode. */ +static int rb_arm_rows_ready(uint32_t sim) +{ + static uint32_t s_wait_log_sim; + int slot; + int n = g_b.slot_count ? *g_b.slot_count : 0; + if (!g_rb || !rnet_rb_inputs_sealed(g_rb)) + return 1; + if (!rnet_rb_tick_in_sealed_span(g_rb, sim)) + return 1; + for (slot = 0; slot < n; ++slot) { + if (rnet_rb_seat_row_authoritative(g_rb, (int32_t)slot, sim)) + continue; + if (s_wait_log_sim != sim) { + fprintf(stderr, + "psxrecomp: rb arm wait rows sim=%u slot=%d (seat row not " + "authoritative — waiting for peer seals)\n", + (unsigned)sim, slot); + fflush(stderr); + s_wait_log_sim = sim; + } + return 0; + } + return 1; +} + static void arm_replay_tick(uint32_t sim) { publish_sealed_sio(sim); g_stat_replay_ticks++; + g_stat_replay_ticks_total++; g_needs_advance = 1; g_replay_progress_ms = rb_mono_ms(); fprintf(stderr, "psxrecomp: rb arm sim=%u (target=%u)\n", (unsigned)sim, @@ -2643,7 +2835,10 @@ static void arm_rereplay_after_load(uint32_t reload_tick) * that tip skipped the sealed span (870 snap → arm 873) and compared * against stale peer invent digests. */ rnet_session_set_sim_tick(s, first); - arm_replay_tick(first); + /* §55: rereplay first tick must also carry authoritative rows; if the + * peer's extend seals are still in flight, try_admit arms it later. */ + if (rb_arm_rows_ready(first)) + arm_replay_tick(first); } /* Enter Verify/POST at the restored tip (empty resim span, or finish_frame @@ -3085,6 +3280,7 @@ static void ownership_chain_next_span(uint32_t tip, uint32_t frontier) g_bl_mismatch_streak = 0; g_fork_streak = 0u; g_fork_streak_tick = 0u; + g_bl_fork_cap = 0u; /* §55: commit/reset clears the bisect cap */ /* §48: only the lower seat initiates chain BEGINs. Both peers calling * begin_rewind every chunk caused dual-init → tie-break abort → reload @@ -3328,6 +3524,7 @@ static void finalize_tip_hold(void) g_bl_mismatch_streak = 0; g_fork_streak = 0u; /* §42 P4: clean commit ends any fork suspicion */ g_fork_streak_tick = 0u; + g_bl_fork_cap = 0u; /* §55 */ /* §38: hold agreed at tip until peer tip-hold (SAFETY ~250ms) can exit. */ g_post_tip_hold_holdoff_t0_ms = rb_mono_ms(); if (g_post_tip_hold_holdoff_t0_ms == 0ull) @@ -3550,6 +3747,7 @@ static void commit_episode(void) g_bl_mismatch_streak = 0; g_fork_streak = 0u; /* §42 P4 */ g_fork_streak_tick = 0u; + g_bl_fork_cap = 0u; /* §55 */ /* No post-commit cooldown: promote-only after commit made char-select * D-pad feel rejected (hist updated, live sim not rolled). Abort/storm * cooldown still arms from live sim on failed episodes. */ @@ -3662,6 +3860,7 @@ void psx_netplay_rb_start(void) g_peer_commit_tick = 0; g_fork_streak = 0u; g_fork_streak_tick = 0u; + g_bl_fork_cap = 0u; /* §55: commit/reset clears the bisect cap */ clear_baseline_pin(); clear_episode_wire_state(); fprintf(stderr, "psxrecomp: rb snap ring ready depth=%u snap_interval=%u\n", @@ -3711,6 +3910,7 @@ void psx_netplay_rb_shutdown(void) g_peer_commit_tick = 0; g_fork_streak = 0u; g_fork_streak_tick = 0u; + g_bl_fork_cap = 0u; /* §55: commit/reset clears the bisect cap */ g_last_good_bb_pc = 0; g_agreed_valid = 0; g_agreed_through = 0; @@ -3901,6 +4101,7 @@ static int try_apply_pending_load(CPUState *cpu_in) * post-realign production fills need; prepare_local_tip still * emits. Keep absurd tip in the ring — cushion refuses invent * on miss (§45) until real rows land. */ + realign_invalidate_evidence(loaded_tick); fprintf(stderr, "psxrecomp: rb realign applied tick=%u pc=0x%08x core=%08x " "cd=%08x\n", @@ -4864,6 +5065,13 @@ void psx_netplay_rb_pump(void) while (rnet_session_take_rb_resolved(s, &resolved)) apply_peer_resolved(resolved); } + /* §53b: Live cadence — only HC-confirmed ADVANCE. Aged HEAL / HEAL-FORCE + * stay at begin/follow: running them every pump under a live HC stall + * invented unconfirmed agreed tips (752→816→…→1008 with hc stuck) and + * baseline-forked the session. */ + if (g_b.hc) + (void)netplay_hc_heal_stale_gap(g_b.hc); + advance_agreed_watermark_from_hc(); /* §41: retransmit confirmed RESOLVED while peer lagging. */ maybe_heartbeat_agreed_resolved(); @@ -5400,6 +5608,8 @@ int psx_netplay_rb_try_admit(void) } return 0; } + if (!rb_arm_rows_ready(sim)) + return 0; /* §55: stall until every seat row is authoritative */ arm_replay_tick(sim); s_stall_logged = 0; return 1; @@ -5442,7 +5652,11 @@ void psx_netplay_rb_finish_frame(void) uint32_t next = done + 1u; if (next <= rnet_rb_get_target_tick(g_rb)) { rnet_session_set_sim_tick(s, next); - arm_replay_tick(next); + /* §55: if the next tick's seat rows aren't authoritative yet, + * leave it un-armed — try_admit re-checks and arms once the + * peer's seal rows land (keepalive redelivers). */ + if (rb_arm_rows_ready(next)) + arm_replay_tick(next); } } } @@ -5459,6 +5673,11 @@ uint64_t psx_netplay_rb_take_replay_ticks(void) return n; } +uint64_t psx_netplay_rb_replay_ticks_total(void) +{ + return g_stat_replay_ticks_total; +} + /* Live POST-handshake latency EMA (ms), 0 = no episode has round-tripped * yet this session. See g_rb_rtt_ema_ms above. */ uint32_t psx_netplay_rb_rtt_estimate_ms(void) diff --git a/runtime/src/psx_netplay_sched.c b/runtime/src/psx_netplay_sched.c index e2481820..0f67295e 100644 --- a/runtime/src/psx_netplay_sched.c +++ b/runtime/src/psx_netplay_sched.c @@ -129,6 +129,36 @@ void np_sched_sync_delay_from_session(void) /* After episode commit: refuse invent until remote tip rebuilds most of D. */ static int g_cushion_rebuild; +/* §56: when the current rebuild window opened (episode boundary). A full + * cushion (lead >= D) clears immediately; a near-cushion lead (D-1) only + * clears after a hold-off so the rebuild actually reaches equilibrium + * instead of declaring victory one tick short and inventing into the gap. */ +static uint32_t g_cushion_rebuild_since_ms; + +/* ------------------------------------------------------------------ */ +/* §57: arrival-driven delay controller state */ +/* */ +/* The §56 soak proved the scheduler converges to the equilibrium the */ +/* delay budget permits: lead ≈ D − 1 − transit_ticks (measured 0.24 / */ +/* 0.91 with D=4 and ~2.1–2.8 ticks of transit), with ~25% of ticks */ +/* missing their row at need. D must therefore be provisioned from the */ +/* ARRIVAL stream (what the scheduler actually consumes), not from the */ +/* POST RTT EMA — the two peers' RTT estimates disagreed 80 vs 12 ms on */ +/* the same link. Accumulators reset each controller eval window. */ +/* ------------------------------------------------------------------ */ + +static uint32_t g_ad_ticks; /* live lead samples this window */ +static int64_t g_ad_lead_sum; +static uint32_t g_ad_miss; /* rows absent at need (once per wire) */ +static uint32_t g_ad_late_n; /* misses that later arrived (waited out) */ +static uint32_t g_ad_late_sum_ms; /* how late those rows were vs need */ +static uint32_t g_ad_late_max_ms; +static int g_ad_miss_pending; /* waiting to time the current miss */ +static uint32_t g_ad_miss_t0_ms; +/* Transit estimate EMA in 1/16-tick units: transit ≈ D − 1 − lead_avg. + * Also feeds the cushion-rebuild achievable-lead target (§57). */ +static uint32_t g_transit_x16; +static int g_transit_have; /* Invent-grace RTT: POST-handshake EMA is asymmetric (often 0 on the peer * that receives POST first / light-tip skips). Always keep a D-scaled synth @@ -434,6 +464,18 @@ static void np_gap1_note_expire_invent(void) void np_sched_note_remote_hit(void) { g_gap1_expire_invent_streak = 0u; + /* §57: a miss we chose to wait out just paid off — the row's lateness + * (first-miss → hit) is exactly the extra runway D was short by. */ + if (g_ad_miss_pending) { + uint32_t late = (uint32_t)(sched_mono_ms() - g_ad_miss_t0_ms); + g_ad_miss_pending = 0; + if (late < 2000u) { /* discard episode/pause artifacts */ + g_ad_late_sum_ms += late; + if (late > g_ad_late_max_ms) + g_ad_late_max_ms = late; + g_ad_late_n++; + } + } } /* ------------------------------------------------------------------ */ @@ -485,6 +527,10 @@ void np_sched_note_episode_boundary(void) * off-guard so the next live mispredicts can re-arm pacing. Keep debt — * the ahead peer may still need a few ms/tick shaves after TipHold. */ g_ts_pegged_streak = 0u; + if (!g_cushion_rebuild) { + uint32_t now = sched_mono_ms(); + g_cushion_rebuild_since_ms = now ? now : 1u; + } g_cushion_rebuild = 1; } @@ -813,6 +859,156 @@ static void np_admit_maybe_log_stats(uint32_t now) fflush(stderr); } +/* ------------------------------------------------------------------ */ +/* §56 equilibrium scorecard (per-window deltas, ~30 s) */ +/* */ +/* One line answers "is the scheduler converging?": episode rate and */ +/* replay depth move with scheduler changes; slack (ms between a remote */ +/* row ARRIVING and being NEEDED at admit) exposes the produce→consume */ +/* pipeline phase that the tick-domain lead quantizes away. Healthy */ +/* real-delay steady state: lead ≈ D, slack ≈ D·16ms − transit, */ +/* miss_need ≈ 0. slack ≈ 0 with "healthy" lead means rows arrive just */ +/* in time — the cushion exists on paper only. */ +/* ------------------------------------------------------------------ */ + +static uint32_t g_sc_window_t0_ms; +static uint32_t g_sc_ep0; +static uint64_t g_sc_rt0; +static uint32_t g_sc_gap1_0; +static uint32_t g_sc_tip_stale0; +static int64_t g_sc_lead_sum; +static uint32_t g_sc_lead_n; +static int g_sc_lead_min; +static int g_sc_lead_max; +static uint64_t g_sc_slack_sum; +static uint32_t g_sc_slack_n; +static uint32_t g_sc_slack_min; +static uint32_t g_sc_slack_max; +static uint32_t g_sc_miss_at_need; + +static void np_scorecard_window_reset(uint32_t now) +{ + g_sc_window_t0_ms = now ? now : 1u; + g_sc_ep0 = psx_netplay_rb_episode_count(); + g_sc_rt0 = psx_netplay_rb_replay_ticks_total(); + g_sc_gap1_0 = g_admit_invent_gap1; + g_sc_tip_stale0 = g_admit_invent_tip_stale; + g_sc_lead_sum = 0; + g_sc_lead_n = 0; + g_sc_lead_min = 0; + g_sc_lead_max = 0; + g_sc_slack_sum = 0; + g_sc_slack_n = 0; + g_sc_slack_min = 0; + g_sc_slack_max = 0; + g_sc_miss_at_need = 0; +} + +/* on_remote_miss: the row for `wire` was not there when needed. */ +static void np_scorecard_note_miss(rnet_u32 wire) +{ + static rnet_u32 s_counted_wire = 0xffffffffu; + if (s_counted_wire != wire) { + s_counted_wire = wire; + g_sc_miss_at_need++; + /* §57: feed the delay controller and arm the lateness timer — if we + * end up waiting this miss out, note_remote_hit measures how late + * the row was; if we invent instead, the pending flag is cleared at + * the invent exit (lateness then shows up as a mispredict age, not + * an arrival stat). */ + g_ad_miss++; + g_ad_miss_pending = 1; + g_ad_miss_t0_ms = sched_mono_ms(); + } +} + +static void np_scorecard_sample(uint32_t now, rnet_u32 sim, rnet_u32 wire, + const RNetSessionStats *st) +{ + static rnet_u32 s_last_sim = 0xffffffffu; + RNetSession *s = sched_session(); + + if (g_sc_window_t0_ms == 0u) + np_scorecard_window_reset(now); + + if (s_last_sim != sim) { + int slot; + s_last_sim = sim; + /* §57: controller lead samples — live steady state only. Episode / + * tip-hold leads (parked sim, lead 10+) and ring-age absurdities + * would poison the transit estimate. */ + if (!psx_netplay_rb_active() && !psx_netplay_rb_tip_holding() && + st->remote_lead > -32 && st->remote_lead < 32) { + g_ad_ticks++; + g_ad_lead_sum += st->remote_lead; + } + if (!g_sc_lead_n) { + g_sc_lead_min = st->remote_lead; + g_sc_lead_max = st->remote_lead; + } else { + if (st->remote_lead < g_sc_lead_min) + g_sc_lead_min = st->remote_lead; + if (st->remote_lead > g_sc_lead_max) + g_sc_lead_max = st->remote_lead; + } + g_sc_lead_sum += st->remote_lead; + g_sc_lead_n++; + if (s) { + for (slot = 0; slot < RNET_MAX_SLOTS; ++slot) { + uint32_t age = + rnet_session_remote_arrival_age_ms(s, slot, wire); + if (age == 0xffffffffu) + continue; + if (!g_sc_slack_n) { + g_sc_slack_min = age; + g_sc_slack_max = age; + } else { + if (age < g_sc_slack_min) + g_sc_slack_min = age; + if (age > g_sc_slack_max) + g_sc_slack_max = age; + } + g_sc_slack_sum += age; + g_sc_slack_n++; + } + } + } + + if ((uint32_t)(now - g_sc_window_t0_ms) >= 30000u) { + uint32_t ep = psx_netplay_rb_episode_count(); + uint64_t rt = psx_netplay_rb_replay_ticks_total(); + uint32_t d_ep = (ep >= g_sc_ep0) ? (ep - g_sc_ep0) : ep; + uint64_t d_rt = (rt >= g_sc_rt0) ? (rt - g_sc_rt0) : rt; + uint32_t d_gap1 = (g_admit_invent_gap1 >= g_sc_gap1_0) + ? (g_admit_invent_gap1 - g_sc_gap1_0) + : g_admit_invent_gap1; + uint32_t d_ts = (g_admit_invent_tip_stale >= g_sc_tip_stale0) + ? (g_admit_invent_tip_stale - g_sc_tip_stale0) + : g_admit_invent_tip_stale; + double dt_s = (double)(uint32_t)(now - g_sc_window_t0_ms) / 1000.0; + double lead_avg = + g_sc_lead_n ? (double)g_sc_lead_sum / (double)g_sc_lead_n : 0.0; + double depth_avg = d_ep ? (double)d_rt / (double)d_ep : 0.0; + double slack_avg = + g_sc_slack_n ? (double)g_sc_slack_sum / (double)g_sc_slack_n : 0.0; + fprintf(stderr, + "psxrecomp: rb scorecard dt=%.1fs ep=+%u depth_avg=%.1f " + "gap1=+%u tip_stale=+%u miss_need=%u " + "lead avg=%.2f min=%d max=%d " + "slack avg=%.0fms min=%u max=%u n=%u " + "transit_est=%.2f D=%d cushion=%d\n", + dt_s, (unsigned)d_ep, depth_avg, (unsigned)d_gap1, + (unsigned)d_ts, (unsigned)g_sc_miss_at_need, lead_avg, + g_sc_lead_min, g_sc_lead_max, slack_avg, + (unsigned)g_sc_slack_min, (unsigned)g_sc_slack_max, + (unsigned)g_sc_slack_n, + g_transit_have ? (double)g_transit_x16 / 16.0 : -1.0, + sched_delay(), g_cushion_rebuild); + fflush(stderr); + np_scorecard_window_reset(now); + } +} + static void np_adapt_delay_on_pcap_enter(uint32_t now) { const char *e; @@ -899,16 +1095,21 @@ static void np_pcap_freeze_exit(void) /* ------------------------------------------------------------------ */ /* Under real delay, D IS the cushion: a press costs D ticks of local - * latency and buys D ticks of transit budget. Resolve it from the measured - * link instead of paying phase error with prediction: - * target = ceil(one_way / tick) + 1, clamped [2 .. RB_ADAPT_DELAY_MAX]. - * Conservative by construction: only trusted POST RTT samples (>=4ms) are - * used; the target must repeat on 3 consecutive 5s evaluations before a - * change is proposed; 30s cooldown between changes; lowering additionally - * requires no pcap freeze in the last 30s (a freeze means prediction was - * needed at the CURRENT D — do not shrink the cushion). The reactive §22 - * bump (np_adapt_delay_on_pcap_enter) still raises D on freeze storms when - * RTT is untrusted. PSX_RB_AUTO_DELAY=0 disables. */ + * latency and buys D ticks of transit budget. + * + * §57 rewrite: D is resolved from the ARRIVAL stream — how the remote rows + * actually reach the consumption point — instead of the POST RTT EMA, which + * measured 80 ms on one peer and 12 ms on the other for the same link. Per + * 5 s eval window (live samples only, both peers accumulate, host decides): + * miss_rate = rows absent at need / live ticks; + * lateness = first-miss → arrival for misses we waited out; + * transit ≈ D − 1 − lead_avg (EMA'd; also drives the cushion-rebuild + * achievable-lead target). + * Raise: miss_rate > 2% → D += ceil(late_avg/tick) (1..2 per step). + * Lower: miss_rate < 0.2% AND ≥2 ticks of spare lead, must repeat on 3 + * consecutive evals + 30 s cooldown + no pcap freeze in the last 30 s. + * The reactive §22 bump (np_adapt_delay_on_pcap_enter) still catches freeze + * storms between eval windows. PSX_RB_AUTO_DELAY=0 disables. */ #define RB_AUTO_DELAY_EVAL_MS 5000u #define RB_AUTO_DELAY_AGREE 3u #define RB_AUTO_DELAY_COOLDOWN_MS 30000u @@ -921,9 +1122,12 @@ static void np_auto_delay_tick(uint32_t now) static uint32_t s_agree_streak; static int s_last_target = -1; RNetSession *s = sched_session(); - uint32_t rtt_raw = 0; uint32_t tick_ms = g_ts_tick_ema_ms ? g_ts_tick_ema_ms : 17u; - uint32_t one_way; + uint32_t ticks, miss, late_n, late_sum, late_max; + int64_t lead_sum; + int32_t lead_avg_x16; + int32_t transit_x16; + uint32_t miss_per_mille; int target; int d; @@ -932,33 +1136,73 @@ static void np_auto_delay_tick(uint32_t now) s_enabled = (e && e[0]) ? (atoi(e) != 0) : 1; if (s_enabled) fprintf(stderr, - "psxrecomp: rb auto delay ON (§44: D = ceil(one_way/tick)+1 " - "from trusted RTT, host-proposed; PSX_RB_AUTO_DELAY=0 off)\n"); + "psxrecomp: rb auto delay ON (§57: D provisioned from " + "arrival misses/lateness, host-proposed; " + "PSX_RB_AUTO_DELAY=0 off)\n"); fflush(stderr); } if (!s_enabled || !s) return; - if (sched_local_slot() != 0) - return; /* host proposes; guests follow DELAY_SYNC */ if (!np_sched_real_delay_enabled()) return; /* D is not a latency budget in legacy zero-delay mode */ if (s_last_eval_ms != 0u && (uint32_t)(now - s_last_eval_ms) < RB_AUTO_DELAY_EVAL_MS) return; s_last_eval_ms = now ? now : 1u; - (void)np_invent_rtt_ms(&rtt_raw); - if (rtt_raw < 4u) { - s_agree_streak = 0u; /* untrusted sample — keep current D */ + /* Harvest + reset the window (both peers, so guest transit stays live). */ + ticks = g_ad_ticks; + lead_sum = g_ad_lead_sum; + miss = g_ad_miss; + late_n = g_ad_late_n; + late_sum = g_ad_late_sum_ms; + late_max = g_ad_late_max_ms; + g_ad_ticks = 0; + g_ad_lead_sum = 0; + g_ad_miss = 0; + g_ad_late_n = 0; + g_ad_late_sum_ms = 0; + g_ad_late_max_ms = 0; + + d = sched_delay(); + if (ticks < 120u) { + s_agree_streak = 0u; /* <~2 s of live samples — episode/FMV window */ return; } - one_way = (rtt_raw + 1u) / 2u; - target = (int)((one_way + tick_ms - 1u) / tick_ms) + 1; + + lead_avg_x16 = (int32_t)((lead_sum * 16) / (int64_t)ticks); + transit_x16 = (int32_t)(d - 1) * 16 - lead_avg_x16; + if (transit_x16 < 0) + transit_x16 = 0; + if (transit_x16 > 12 * 16) + transit_x16 = 12 * 16; + if (g_transit_have) + g_transit_x16 = (g_transit_x16 * 7u + (uint32_t)transit_x16) / 8u; + else + g_transit_x16 = (uint32_t)transit_x16; + g_transit_have = 1; + + if (sched_local_slot() != 0) + return; /* host proposes; guests follow DELAY_SYNC */ + + miss_per_mille = (uint32_t)(((uint64_t)miss * 1000u) / ticks); + if (miss_per_mille > 20u) { + uint32_t late_avg = late_n ? (late_sum / late_n) : 0u; + int bump = (int)((late_avg + tick_ms - 1u) / tick_ms); + if (bump < 1) + bump = 1; + if (bump > 2) + bump = 2; + target = d + bump; + } else if (miss_per_mille < 2u && lead_avg_x16 >= 2 * 16) { + target = d - 1; /* ≥2 spare ticks sustained — cushion oversized */ + } else { + target = d; + } if (target < 2) target = 2; if (target > RB_ADAPT_DELAY_MAX) target = RB_ADAPT_DELAY_MAX; - d = sched_delay(); if (target == d) { s_agree_streak = 0u; s_last_target = target; @@ -969,7 +1213,9 @@ static void np_auto_delay_tick(uint32_t now) s_agree_streak = 1u; return; } - if (++s_agree_streak < RB_AUTO_DELAY_AGREE) + /* Asymmetric hysteresis: raises hurt only input latency, misses hurt + * determinism — confirm a raise on 2 evals (10 s), a lower on 3 (15 s). */ + if (++s_agree_streak < ((target > d) ? 2u : RB_AUTO_DELAY_AGREE)) return; if (s_last_change_ms != 0u && (uint32_t)(now - s_last_change_ms) < RB_AUTO_DELAY_COOLDOWN_MS) @@ -982,10 +1228,15 @@ static void np_auto_delay_tick(uint32_t now) s_last_change_ms = now ? now : 1u; s_agree_streak = 0u; fprintf(stderr, - "psxrecomp: rb auto delay %d → %d (rtt_raw=%u one_way=%u " - "tick=%u)\n", - d, target, (unsigned)rtt_raw, (unsigned)one_way, - (unsigned)tick_ms); + "psxrecomp: rb auto delay %d → %d (arrival: miss=%u/%u " + "(%u‰) late avg=%ums max=%ums n=%u lead_avg=%.2f " + "transit_est=%.2f ticks)\n", + d, target, (unsigned)miss, (unsigned)ticks, + (unsigned)miss_per_mille, + (unsigned)(late_n ? late_sum / late_n : 0u), + (unsigned)late_max, (unsigned)late_n, + (double)lead_avg_x16 / 16.0, + (double)g_transit_x16 / 16.0); fflush(stderr); } } @@ -1110,6 +1361,7 @@ int np_sched_pre_admit(uint32_t sim, uint32_t wire, const RNetSessionStats *st) st->remote_lead); np_phase_ctrl_maybe_log(now, sim, st->remote_lead); np_auto_delay_tick(now); + np_scorecard_sample(now, sim, wire, st); /* Rebuild D cushion after episode: do not invent until the remote tip * is nearly back at a full cushion (remote_lead >= D-1; §44: lead is @@ -1122,16 +1374,45 @@ int np_sched_pre_admit(uint32_t sim, uint32_t wire, const RNetSessionStats *st) if (g_cushion_rebuild && !psx_netplay_rb_active()) { int d = sched_delay(); int pred = g_sb.input_prediction ? *g_sb.input_prediction : 0; - int need = d > 0 ? d - 1 : 0; + int full = d > 0 ? d : 0; + int achievable; int max_ok = d + (pred > 0 ? pred : 4); - if (max_ok < need + 2) - max_ok = need + 2; - if (st->remote_lead >= need && st->remote_lead <= max_ok) { + if (max_ok < full + 2) + max_ok = full + 2; + /* §56: FULL cushion (lead >= D) clears immediately. + * §57: the physical steady-state ceiling is lead ≈ D−1−transit, so + * demanding D−1 on a WAN link just parked the rebuild against the + * timeout (soak: held=12665ms). Clear at the ACHIEVABLE lead for the + * measured transit, after a short hold so the rebuild really settles + * there rather than clipping through it. */ + achievable = d - 1; + if (g_transit_have) { + achievable = d - 1 - (int)((g_transit_x16 + 8u) / 16u); + if (achievable < 1) + achievable = 1; + if (achievable > d - 1) + achievable = d - 1; + } + if (st->remote_lead >= full && st->remote_lead <= max_ok) { g_cushion_rebuild = 0; fprintf(stderr, - "psxrecomp: rb cushion rebuilt remote_lead=%d D=%d\n", + "psxrecomp: rb cushion rebuilt FULL remote_lead=%d D=%d\n", st->remote_lead, d); fflush(stderr); + } else if (st->remote_lead >= achievable && st->remote_lead <= max_ok) { + uint32_t held = g_cushion_rebuild_since_ms + ? (uint32_t)(now - g_cushion_rebuild_since_ms) + : 0u; + if (held >= 400u) { + g_cushion_rebuild = 0; + fprintf(stderr, + "psxrecomp: rb cushion rebuilt ACH remote_lead=%d " + "achievable=%d D=%d transit_est=%.2f held=%ums\n", + st->remote_lead, achievable, d, + g_transit_have ? (double)g_transit_x16 / 16.0 : 0.0, + (unsigned)held); + fflush(stderr); + } } else if (st->remote_lead > max_ok) { static uint32_t s_last_absurd_ms; uint32_t now_a = sched_mono_ms(); @@ -1166,6 +1447,8 @@ int np_sched_on_remote_miss(int slot, uint32_t sim, uint32_t wire, if (!st) return 1; + np_scorecard_note_miss(wire); + /* FMV media + post-FMV settle: wait for remote wire (skip / title * Start). Invent idle opened tip episodes that hung. Tip-ahead with a * missing consumption wire is a ring hole — log before stalling. */ @@ -1355,6 +1638,7 @@ int np_sched_on_remote_miss(int slot, uint32_t sim, uint32_t wire, np_admit_note_invent_gap(wire, st->highest_remote_wire); np_admit_log_invent(sim, wire, st->highest_remote_wire, st->remote_lead, invent_reason); + g_ad_miss_pending = 0; /* §57: invented — lateness timer no longer valid */ if (reason_out) *reason_out = invent_reason; np_sched_clear_admit_stall(); /* inventing — not a barrier stall */