Two defects the last run exposed.
The oracle appeared to pause because every sample called walk_side,
which snapshots all 2 MB of RAM -- about 128 socket round trips serviced
on DuckStation's emulator thread. Nothing was paused; it was being read
to a standstill, repeatedly, and rarely landing in the animation as a
result. The list address is now cached after the first successful walk
and subsequent samples re-read only a 128 KB window around it.
Worse, the run reported 'signatures-agree' on 144 quads/155 lines while
psx-runtime had sampled the effect object ZERO times. Falling back to a
smaller object that both sides happened to see, and calling that
agreement, is the same failure as ranking 120 identical frames and
naming a winner. The biggest object either side saw is the effect; if
only one side sampled it, the verdict is now effect-object-one-sided and
says explicitly that smaller shared objects are not a substitute.
Also records that the oracle's effect object carries 3 distinct colours
at peak 3 AND at peak 220 -- colour count does not move with brightness,
so the 153-vs-3 difference is not a fade artifact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two bugs from the live run.
The crash: gpu_display_list's report() formats verts as the string
'(x,y) (x,y) ...' while the GP0 ring dump carries [[x, y], ...]. A tool
that reads psx-runtime from the ring and the oracle from a RAM walk
gets both, and signature() indexed the string shape as if it were the
list shape -- v[1] on a string. The 'has 3 vertices' guard did not stop
it because a 5-character string is longer than 3. Both shapes are now
parsed explicitly, with a test asserting the two produce an identical
signature.
The wedged oracle: a paused DuckStation serves its socket from a Qt idle
timer at about 1 Hz, so pc_unbreak can simply time out -- and __exit__
swallowed that DebugError, leaving the breakpoint live. Since the
breakpoint re-pauses on its next hit, every manual unpause bounced
straight back. Removal is now verified against pc_break_list and retried,
and if it still cannot clear it says so and names restarting DuckStation
rather than exiting quietly with the emulator wedged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
effect_palette's pc_break came back {'ok': False, 'slot': 1} and the
oracle was left paused. Both symptoms are one cause: phase_parity armed
a breakpoint per loop iteration and returned early without removing it,
so 0x8006844C stayed live in the oracle across runs. DuckStation's
AddBreakpointWithCallback refuses a duplicate address (ok:false, and
slot 1 because slot 0 was still taken), while the leaked breakpoint kept
firing -- so every resume immediately re-paused. The leak lives in the
oracle, not the tool, which is why it survived restarts of the script.
Adds OracleBreak, a context manager that clears stale breakpoints on
entry, arms once, and removes plus resumes on EVERY exit path. Tools in
this tree have leaked a parked oracle repeatedly; a guard is the fix,
not remembering to resume. phase_parity still parks deliberately when it
finds its phase, but disarms first -- parked is a result, armed is a leak.
Also fixes OracleBreak's own error path, which raised a message saying
the oracle 'has been left running' without resuming it. Caught by the
test, not by reading it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The fallback worked — psx-runtime produced 12 samples free-running. The oracle
still reported "PolyG4+semi never appeared within 120s. Drawing instead:
nothing", and the cost of the check is why.
class_on_screen() snapshots all 2 MB of RAM: 128 read_ram calls. Against a
PAUSED oracle, whose socket falls back to a ~1 Hz idle timer, that is 128
seconds for ONE poll. A 120-second wait therefore completes a single partial
read and reports "nothing" — not a statement about what is drawing, a statement
about a read that never finished. The oracle was very likely parked from an
earlier breakpoint run, which nothing in the wait undid.
So the wait now resumes first, pays for the full scan once, and then re-reads
only the span the display list occupies — a couple of KB, one or two reads
instead of 128. A cached root that stops walking means the list moved, and the
next poll pays for a rescan rather than sticking to a dead pointer.
A `check` seam lets a caller supply its own probe, which is also how this is
tested without standing up a synthetic display list in fake RAM.
Recorded for the next run: psx-runtime's series came from the free-running
fallback, so its steps (median 12, max 64) are ALIASED — samples ~90 frames
apart — and are not an animation increment. native_sampling says so in the
report, and the per-frame verdict is suppressed when the fallback ran.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"the effect is not on screen on psx-runtime, so this code never ran. Currently
drawing: PolyFT4 x584, PolyF4+semi x184, PolyG3+semi x176, PolyFT3 x151" — the
diagnostic added last commit, doing its job on the first run. The effect simply
was not up when the tool looked.
That is the real problem, and it is not a diagnostic problem. Every tool here
needs the effect RUNNING, and requiring it to be on screen at the instant a
button is clicked means racing a transient animation. Losing that race produces
a different-looking failure in each tool — no writes recorded, no candidate
fired, no table found, the block was not reached — four symptoms, one cause,
none of them naming it. Running lockstep beforehand appeared to help only
because it takes long enough for the effect to come round again.
So the tools wait. Press the button first, then trigger the effect; each polls
the display list until its class appears, up to two minutes, and each side is
waited for INDEPENDENTLY in its own thread. That last part matters: it removes
the need to hold two emulators inside the same effect simultaneously, which was
never realistic by hand.
Wired into scale_trace, packet_writers, range_writers and colour_inputs — every
tool that needs the effect. Timing out reports what WAS drawing, because "wrong
scene" and "tool is broken" look identical otherwise. A transient DebugError
mid-poll does not abort the wait: the emulator can be mid-pause, and one failed
poll is not a reason to stop waiting for something that has not happened yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The multi-line fix did not work, because I fixed the read loop and left the
parse. raw() kept reading until the JSON document was complete and then still
did split(b"\n", 1)[0] before parsing it — so pc_probe_dump was truncated in
exactly the same place, and the error still named the server. Verified against
the live runtime this time rather than reasoned about: 843 bytes, 6 newlines,
parses.
The whole buffer is parsed now (JSON treats the embedded newlines as
whitespace), with the first line kept as a fallback for a peer that sends more
than one document down a connection.
Underneath that, the probe was reporting the wrong cause. Every candidate came
back with count 0, and the message offered two possibilities: the addresses are
mid-block, or the code did not run. It was the second — the effect was not on
screen at all — and that is checkable rather than guessable. class_on_screen()
walks the display list and says which it is:
"PolyG4+semi is not being drawn, so this code never ran and no candidate
could fire. Currently drawing: PolyFT4 x576, PolyF4+semi x184, PolyFT3 x1."
Several tools depend on a routine actually running — a write trace over its
packets, a block probe at one of its instructions — and each reports its own
symptom when it is not: no writes recorded, no candidate fired, no table found.
All three are now one plain statement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Clicking a colour-only row aborted the Studio:
type_error.302: type must be string, but is null
nlohmann's value<T>() THROWS when a key exists but holds null, rather than
returning the default. colour_inputs wrote "block_leader": null for a probe
that had not fired, and reading it took the process down. A viewer must never
die on the content of a file it is shown, so this is fixed on both sides:
* the report omits absent keys instead of writing nulls — "missing" and
"present but null" should not be different things for it to express
* every string and integer read in the newer loaders goes through null-safe
accessors, so a malformed or unexpected document degrades to defaults
Underneath it was a real failure the crash was hiding:
pc_probe_dump: malformed reply (Expecting value: line 1 column 80)
That handler builds its JSON with repeated send_line() calls — header, one line
per slot, one per sample — so the first newline lands in the MIDDLE of the
document. DebugConn stopped at the first newline and got a truncated object,
and the error named the server rather than the reader. It now stops at a
newline only when what has arrived is a complete JSON document, counted by
braces with quotes and escapes tracked (a parse attempt per chunk would
dominate the read on a megabyte of hex).
So the probe never actually reported $s4; the whole run fell back to the
content search without that being visible as a failure.
frames_load_test now loads packet_writers/gte_check/colour_inputs/lockstep from
a real directory too. That mode would have caught this before it reached a
click: the crashing file loads cleanly under it now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
"pc_hit_last: server closed without replying", and every retry after it worse.
Two bugs, neither of which failed in a way that pointed at its cause.
A DuckStation breakpoint PAUSES the emulator when it fires. A paused oracle
pumps its debug socket only from a Qt idle timer -- 100 ms with a gamepad
attached, 1000 ms without -- so a headless oracle is a ~1 Hz server the moment
the breakpoint hits. Polling it every 100 ms stacks up connections it cannot
accept, and that surfaces as "server closed without replying": a
network-shaped error with a pacing-shaped cause. Polling is now 1.5 s, the
wait has headroom, and one dropped reply around the hit is survived rather
than treated as failure, because the hit parks the emulator mid-exchange.
Worse, the tool cleared the breakpoint and exited WITHOUT resuming. So the
first run left the oracle parked at 1 Hz permanently and every attempt after
it failed for a different reason than the first -- which is a miserable thing
to debug from the outside. It now resumes in the finally block, and resumes
BEFORE arming too, so a previously stranded oracle recovers on its own instead
of needing a restart.
ORACLE_PAUSED_POLL_S and oracle_resume() live in psx_gpu_frame because this is
the third time this behaviour has cost time: it also explains the 2 MB read
that could not finish while parked, and the earlier decision never to pause the
oracle for a snapshot.
Studio gets a Resume button next to the oracle's status for the same reason --
the symptom (everything times out) does not resemble the cause (it is parked).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three things stood between the fingerprint and being usable, all found by
running it against a live pair rather than by reading it.
DebugConn.frame() asked `ping` for the frame number. psx-runtime intercepts
ping in a lock-free fast path that never reaches the command dispatcher, so its
reply carries only {ok, pong, io_thread} -- no frame. Every native run
therefore reported frame 0, silently, and anything aligning two emulators by
frame number was comparing against zero. This is the "native frame 0" that has
been showing up in gpu_parity output all along. The oracle DOES put a frame in
its ping, which is what made it look like an oracle quirk rather than a
native-side blind spot. frame() now asks the `frame` command, which both
answer, and falls back to ping.
frame_fingerprint replies are now paged (64 frames per call). A whole ring in
one request is over a megabyte, and the oracle's socket truncates anything past
~16 KB, after which the read blocks forever waiting for a newline that never
comes. Paging here is not an optimisation; it is the difference between working
and hanging.
Added an alignment gate ahead of the coverage gate. A frame number means
"frames since THIS emulator booted", so overlapping frame numbers prove
nothing: start one thirty seconds after the other and both report a frame 500
that are half a minute apart. A diff across that names a divergence at the
first frame compared, every time, and it looks exactly like a real finding.
The gate compares guest cycle counts and refuses when they disagree. Measured
on a live pair, the two agree on cycles-per-frame to 0.3% (564480 vs 566204),
so this is a sharp test, and it correctly refused a mismatched pair on its
first live run.
The oracle now seeds ExecutionMode = Interpreter. The store hook lives in the
interpreter's write path; a recompiler oracle inlines stores, never reaches it,
and would compare clean against anything. A false "no divergence" is worse than
a slow reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two wrong answers the results made visible.
find_display_lists walked ONE BUCKET and called it the display list. An
ordering table is an array of link-only entries, each chaining to a run of
primitives; those links were excluded from the graph, so every bucket's first
primitive looked like an unpointed head. The output looked complete and was
not: a real walk came back with 946 primitives and no PolyG4+semi|B+F at all,
while the GP0 ring on the same runtime held 3008 vertices of exactly that
class. The bucket it picked simply did not contain the effect under
investigation. Link entries now join the graph (guarded so a zeroed word,
which reads as length 0 pointing at address 0, cannot flood it), so the scan
finds the table head and covers every bucket.
The colour verdict rested on `peak`, which is a maximum and therefore grows
with the number of samples drawn. Comparing 3008 vertices against 1024 it
reported "differ" for two distributions whose medians (13 vs 17), p90s (144 vs
126) and means (49.8 vs 42.0) all agreed -- a confident wrong answer, which is
worse than none. The verdict is now histogram intersection over normalised
bins, invariant to both sample count and animation phase, with an explicit
"inconclusive" band. On the reported data the old rule says differ and the new
one says inconclusive, which is what the evidence supports.
Studio: the Display list pane kept one dlist.json, so walking the oracle
destroyed the runtime's walk and the two could never be compared -- the
question worth asking. Each side now writes its own file, both are held at
once, and a class-count table shows the delta.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All of this existed and was lost to a `git pull --rebase` in this submodule
while uncommitted. Committing so it survives the next one.
Engine:
* pause / continue / step / run_to_frame are real again, debug-server only —
no key, button, or menu reaches them, so a shipped build cannot be frozen.
The wedge that got them removed the first time is fixed: the park loop
keeps calling debug_server_poll() (which delivers `continue` and feeds the
4-second starvation watchdog) and auto-resumes after s_pause_timeout_ms of
client silence, so a debugger that dies cannot take the game with it.
* present_hold_tick() / is_window_repaint_event(): a frame held on screen no
longer rots when another window crosses it. Previously nothing presented
while stalled, so the expose went unanswered and the frame smeared.
Tools:
* psx_gpu_frame.py: frame signatures (compare frames by opcode+blend rather
than by function — a game that DMAs an ordering table reports one submit PC
for every packet, so function attribution has one answer per frame), plus
reading display lists out of guest RAM.
* gpu_display_list.py: walk an ordering table and decode it. Finds the list
by scanning for chained nodes whose payload is a real GP0 command of
exactly the right length. Following DMA ch2's MADR does NOT work: a
finished linked-list transfer leaves the 0xFFFFFF terminator there, which
masks to 0x1FFFFF and looks like a valid address.
* gpu_colour_parity.py: compare vertex-colour distributions across emulators.
Distributions rather than a diff because two emulators cannot be aligned to
the same instant without deterministic replay, and an animating effect
sampled at two phases differs everywhere — which says nothing.
* read_ram is capped at 16 KB/call: the oracle advertises 65536 but replies
over a socket that truncates it, and the read then hangs. Measured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Alex Vanderveen