mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-28 02:56:36 -04:00
Matthew Stanley
e6ce5006b4
The repository root carried seven markdown files that were not onboarding
material, plus two that had been superseded by the code they described. This
sorts them: reader-facing docs to docs/, working notes to docs/internal/, and
the two superseded ones merged into the docs that replaced them.
Moved to docs/: WIDESCREEN.md, ENHANCEMENTS.md, TCP_COMMANDS.md
Moved to docs/internal/: PRINCIPLES.md, ISSUES.md, DEBUG.md, GTE_LM_FIX.md,
and from docs/: bios-hardening.md, overlay-status.md,
overlay-plan.md, overlay-recompilation-design.md,
overlay-discovery.md, TOMBA2_EXC_CONSULT_RESPONSE.md,
CASE_A_AOT_GAP.md, audit_inventory.md
Moved to docs/internal/upstream/: UPSTREAM_2026-08-20.md
The root keeps only what belongs there: README.md, CONTRIBUTING.md, LICENSE,
CLAUDE.md, THIRD_PARTY_ATTRIBUTION.md, RELEASE_NOTES.md, the dotfiles and
psxrecomp_cli.py.
MERGES
SPLITGEN_MIGRATION.md folded into docs/BUILDING.md as "Split generated C". Its
migration instructions were stale in a way that mattered: it told every game
repo to hand-edit its CMakeLists and add a file(GLOB ...) over
<serial>_full*.c, but runtime/runtime.cmake now does that itself. GEN_FULL_GLOB
is a multi-value argument of psxrecomp_add_game_runtime(); the loop at
runtime.cmake:1985 makes each pattern absolute against CMAKE_CURRENT_SOURCE_DIR,
expands it with file(GLOB), and concatenates the hits into
GAME_GENERATED_FULL_C. So a title names the pattern and gets however many shards
a regeneration produced. The rewritten section describes that, the two-step
fallback when the globs match nothing (GEN_FULL_FALLBACK, else <serial>_full.c
derived from the dispatch marker by substituting _dispatch.c), and the one
caveat the code actually has: that file(GLOB) is plain, not CONFIGURE_DEPENDS,
so a regeneration which changes the shard COUNT needs a re-configure to be seen.
psxrecomp-toml-readme.md folded into docs/GAME_PROJECT_SETUP.md as "EXE-only
autofill (psxrecomp-toml)", placed after "Disc autofill (probe_disc.py)" and
before "Project Studio" so the two autofill paths sit together. Its documented
flags were checked against the usage text in recompiler/src/main_toml.cpp:28-42
and against the argument parser at :150-165 -- all seven options match with no
drift, so the flag list is reproduced as-is. Added a pointer that probe_disc.py
is preferable when a cue exists, since that tool does the same EXE-header work
plus disc identity and netplay gates.
FUNCTIONAL FIXES
tools/gen_tcp_commands.py wrote to REPO/"TCP_COMMANDS.md". After the move that
path does not exist, so the generator would have created a new stray root file
and --check would have failed against it forever. Retargeted to
REPO/"docs"/"TCP_COMMANDS.md", along with the seven docstring and message
strings that name the path.
Running it surfaced two further problems, both fixed here:
* The doc was genuinely stale -- pre-existing drift, not introduced by this
commit (the moved file is byte-identical to origin/master, and no
debug-server command table is touched here). Four commands registered since
the last regeneration were missing from the index -- cdrom_timing_dump,
savestate_status, ws_ui_groups, xprobe_watch -- and scanline had gained a
backend. Regenerated: 307 -> 311 commands, and the change is confined to the
AUTOGEN block (the hand-written head and tail are byte-identical to master).
tools/gen_tcp_commands.py --check now exits 0.
* DOC.write_text() has no newline argument, so Python translated every newline
to os.linesep. On Windows that rewrote all 585 lines of the file to CRLF and
turned a four-line index update into a whole-file diff -- measured: 589 CRLF
/ 0 LF after a run, against 0 CRLF / 585 LF in the repo. Now written through
an explicit newline="" handle, verified to leave the file at 0 CRLF.
runtime/include/debug_server.h:9 and runtime/src/debug_server.c:8 cited "TCP.md
and DEBUG.md". TCP.md has never existed in this repo under that name; both now
name docs/TCP_COMMANDS.md and docs/internal/DEBUG.md.
LINK AND REFERENCE FIXES
Every inbound reference to a moved file was located by grepping each basename
across the whole tracked tree, not just the markdown. Markdown hrefs in
README.md, CONTRIBUTING.md, docs/ARCHITECTURE.md, docs/COMPILING_OVERLAYS.md,
docs/AOT_OVERLAY_PLAN.md, docs/RELOCATION_MANIFEST_FORMAT.md,
docs/config_schema.md, docs/OVERLAY_XREF.md, docs/TCP_COMMANDS.md and several
docs/internal/ files now point at the new locations; where two files became
siblings under docs/internal/ the path was shortened rather than lengthened
(bios-hardening.md's reference to overlay-discovery.md, for instance).
docs/COMPILING_OVERLAYS.md's "in this folder" phrasing was wrong once two of the
three files it listed moved, so it now names internal/ explicitly.
24 source-comment mentions of ENHANCEMENTS.md sections G1.x across 19 files
(recompiler, runtime, cmake) were retargeted to docs/ENHANCEMENTS.md with the
section numbers untouched. docs/SHADOW_ENHANCEMENTS.md is a DIFFERENT file and
was deliberately excluded.
20 rationale strings citing docs/CASE_A_AOT_GAP.md across the four
recompiler/seeds/*.json files were updated. Done as a byte-level substring
replacement rather than a JSON round-trip: two of those files are stored with
CRLF endings and two with LF, and re-serializing them reformatted 5,700 lines.
The byte-level version changes exactly 20 lines and each file still parses.
References that look similar but point elsewhere were deliberately left alone,
and are worth naming so a future pass does not "fix" them:
* "recomp-template PRINCIPLES.md" / "NES recomp template PRINCIPLES.md" in
CLAUDE.md:43, docs/SHADOW_ENHANCEMENTS.md:28,
docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md:5,
runtime/src/debug_server.c:4934 and tools/parity_diff.py:8 are a different
repository's file. Only the five that mean this repo's PRINCIPLES.md were
updated.
* Every remaining bare ISSUES.md mention is a GAME repo's tracker
(MegaManX6Recomp 2/7/8/9, TombaRecomp 6), not this repo's.
* docs/disc-speed.md:4,54,56 and docs/internal/bios-hardening.md:5 name
overlay-discovery as a BRANCH, not a file, so no path applies.
INDEXES
README.md's "New here?" table gains rows for docs/WIDESCREEN.md,
docs/ENHANCEMENTS.md and docs/TCP_COMMANDS.md. docs/internal/README.md gains
sections indexing all fourteen documents that moved in, plus an "Upstream
contributor reviews" section describing upstream/ and the new sweep file, plus a
"Removed" bullet for CYCLE_TIMING_ARCH.md (deleted in the previous commit)
pointing at FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md -- following the SLJIT_PERSIST_CACHE.md precedent already
in that file.
docs/ENHANCEMENTS.md:7 claimed framework bugs live in "game repos' ISSUES.md
until a framework tracker exists". A framework tracker does exist; it now says
so and names the Beads epic.
VERIFICATION
A link checker was written to scan every tracked *.md for relative links,
resolving each against the tree being scanned. Baseline at origin/master: 129
files, 137 relative links, 3 broken. After both commits: 127 files, 159 relative
links, 3 broken -- the SAME three, all pre-existing and untouched here:
docs/DATA_SHARDS.md -> INSTANT_LOADS_PARKED.md (never existed)
docs/DATA_SHARDS.md -> INSTANT_LOADS_SPIKE2.md (never existed)
docs/NETPLAY.md -> ../lib/recomp-net/README.md (inside a submodule)
Zero new broken links. The checker's own coverage was checked against a raw grep
for markdown link openers: the 3-link difference is C++ lambdas inside code
blocks, correctly not treated as links.
Also verified: py_compile over all 113 tracked tools/*.py plus psxrecomp_cli.py;
all four seed JSONs still parse; the test-registration guard still reports 123
test files all registered; recompiler/tests/test_coverage_vault.py still passes;
and no file in this commit had its line endings flipped (checked mechanically
across all 50 modified files, after two files were caught and redone).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
112 lines
6.4 KiB
Markdown
112 lines
6.4 KiB
Markdown
# Overlay DLL Cache — Status (features + issues)
|
||
|
||
Living status for the overlay capture/compile/cache feature on
|
||
`feat/overlay-jit-cache`. Design rationale lives in
|
||
`overlay-recompilation-design.md`; this file tracks *what works* and *what's
|
||
broken*, with evidence.
|
||
|
||
## Working (verified on Tomba)
|
||
|
||
- **Capture** — on CD-DMA into game RAM, the runtime records the overlay bytes
|
||
(full, data-inclusive) + executed PCs + dispatch-entry PCs, written to
|
||
`overlay_captures.json` at clean shutdown. One region captured for Tomba
|
||
village: `0x800E7000`, 356 KB, 15358 executed PCs, 66 dispatch seeds.
|
||
- **Evidence-scoped recompile (`--overlay`)** — `compile_overlays.py` runs the
|
||
recompiler in entry-scoped mode (no whole-byte sweep). Set unconditionally
|
||
for every overlay; not a human toggle.
|
||
- **Boundary-gated entry promotion** — a bare `DISPATCH_ENTRY` is promoted to a
|
||
registered function only with boundary evidence (`jr $ra`-preceded, prologue,
|
||
or region start). Dropped the data-as-code false entries: 122 → 88 functions,
|
||
42 → 0 unsupported, 8 → 0 bad targets.
|
||
- **DLL cache** — clean 274 KB DLL emitted (`000E7000_B476006F.dll`). Loader
|
||
lazily `LoadLibrary`s it on first dispatch into the dirty region, registers
|
||
the 88 `func_XXXX` exports, dispatches them before the interpreter.
|
||
- **Config gate** — `[runtime] overlay_cache` (default off).
|
||
- **In-village behavior** — with the cache on, the village scene renders and
|
||
plays correctly: the 88 registered functions execute natively, the deferred
|
||
34 fall back to the interpreter, 0 dispatch misses, 0 exceptions.
|
||
|
||
## Issues
|
||
|
||
### OV-1 — Blue screen on village → overworld transition (FIXED 2026-06-07, Inc1-D)
|
||
**Resolution verified by mechanism, not just symptom.** With Inc1-D write-
|
||
invalidation, the village→overworld transition no longer softlocks. Counters at
|
||
the overworld: `invalidations=1` (`last_msg: invalidated region 0x000E7000
|
||
(+16384) on write 0x000E7388 -> 88 funcs`, the write from BIOS pc 0xBFC12210),
|
||
`unregistered_funcs=88`, and **`stale_blocked` climbing into the tens of
|
||
millions** — i.e. the game dispatches into the reused region constantly, and
|
||
every one of those (which pre-fix executed stale village native code = the blue
|
||
screen) is now blocked and routed to the interpreter. A/B confirmed earlier:
|
||
`overlay_cache=false` transitions cleanly. Verified one transition + free
|
||
roaming; broader scenes/games still untested.
|
||
|
||
**Note on benefit:** correctness only, so far. `dispatch_native≈122` vs
|
||
interpreter ≈31M — near-0% native, because (a) only one overlay is compiled and
|
||
(b) an invalidated region is not re-registered this session. Speed payoff needs
|
||
broad overlay coverage + Inc2 reload-on-return (CRC-matched re-registration).
|
||
|
||
Original root-cause analysis below, retained for the record.
|
||
|
||
### OV-1 (original analysis) — stale registration
|
||
**Symptom:** leaving the village to the overworld lands on a solid blue screen.
|
||
Not a crash: frame counter still advancing (8358→8364), `dispatch_miss_total=0`,
|
||
`in_exception=0`, GPU enabled and drawing. Execution cycles low BIOS/kernel
|
||
(`current_func≈0x650`, `last_store_pc=0xBFC11BF0`) — i.e. running but down a
|
||
wrong path / stuck in a BIOS wait.
|
||
|
||
**Root cause — stale registration.** The `0x800E7000` region is reused after
|
||
the village. Direct RAM-vs-captured comparison at the blue screen:
|
||
|
||
| offset | captured (what we compiled) | live RAM | verdict |
|
||
|---|---|---|---|
|
||
| `0xE73C0` (compiled func) | `27bdffb8` (addiu sp,sp,-0x48 prologue) | `0x800E92C4` ×4 (pointer table) | DIFFER |
|
||
| `0xE7000` (region start) | zero-fill | `0x11111111` ×4 | DIFFER |
|
||
| `0xE8120` (compiled func) | real code | `0x801013F4` ×4 (pointers) | DIFFER |
|
||
| `0x100000` | — | — | MATCH |
|
||
| `0x13B000` | — | — | MATCH |
|
||
|
||
The **front of the region was overwritten** (now pointer tables, not our code)
|
||
while the **tail still matches** — and the loader **kept all 88 functions
|
||
registered** (`regions_checked=1`, loaded once, never re-evaluated; the
|
||
`already_checked` guard plus the absence of write-invalidation mean the
|
||
registration persists). The game then dispatches into `0xE7000` addresses whose
|
||
*content has changed*, hits our **stale village function pointers**, and runs
|
||
the wrong native code → wrong logic → blue.
|
||
|
||
**It is NOT:** a mid-function split (boundary-gated out; village renders fine),
|
||
nor a recompiler opcode gap (0 unsupported after the gate).
|
||
|
||
**It IS adjacent to** the DMA-time-vs-execution-time capture question
|
||
(design §2.2): the bytes changed after capture. But the correct fix is not
|
||
"capture different bytes" — it is "detect that the source bytes changed and
|
||
invalidate the registration."
|
||
|
||
**Fixes (in priority order):**
|
||
1. **Inc1-D — write-invalidation (primary).** Watch registered overlay pages;
|
||
on any write into them, invalidate (unregister) the affected functions so
|
||
the new content falls back to the interpreter / a re-evaluation. The
|
||
RAM-write-path audit confirms a single store chokepoint with no bypasses, so
|
||
this is sound; it needs the watched-page extension (current tracking is
|
||
kernel-only). This directly prevents OV-1.
|
||
2. **Inc2 — registration-time + manifest CRC guard.** Before registering, verify
|
||
the live region bytes still match the DLL's source CRC; refuse if not.
|
||
3. **Design §2.2 — execution-time capture.** Compile from the bytes that
|
||
actually executed, not DMA-time bytes (defense in depth).
|
||
4. **Design §2.6 — segmentation.** The 356 KB region almost certainly merges
|
||
the village overlay with other content; finer per-overlay segmentation +
|
||
per-overlay cache keying reduces the blast radius of a partial overwrite.
|
||
|
||
**Confirming A/B (cheap, do next):** set `overlay_cache=false`, repeat the exact
|
||
village→overworld navigation. If it transitions cleanly with the cache off,
|
||
OV-1 is confirmed as cache stale-registration. If it still blue-screens, the
|
||
cause is upstream (interpreter or game-level) and this diagnosis is wrong.
|
||
|
||
## Implementation increments (see design doc §7)
|
||
- Inc1-A ✅ evidence-scoped `--overlay`
|
||
- Inc1-B ◐ boundary gate done; per-function audit + manifest pending
|
||
- Inc1-C ☐ loader registers only manifest entries
|
||
- Inc1-D ✅ write-invalidation — fixes OV-1 (verified)
|
||
- **Inc2 (next, biggest benefit lever):** reload-on-return — re-register a
|
||
cached overlay when its region is reused and the bytes match, so a revisited
|
||
overlay runs native again instead of staying interpreted after first exit.
|