mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-27 18:52:11 -04:00
4b2fdfadc35858cc68614d50737e2320baef265c
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Shokunin
|
82c1f82164 |
docs: record rollout PR E (overlay shards keyed by segment)
docs/SEGMENT_AWARE_CODE.md: - §5.7 gains "Implemented in PR E": capture's per-segment bitmaps and schema v3; compile_overlays.py's segment views (including dispatch-less records and declared demands); the static fragment images; overlay_codegen_config() for config code sites; the cache layout and the S record; the loader's per-segment lookups and counters (unmapped segments included) and the overlay_candidates fields; the full-VA compares; tests. - §3.4, §5.3 and §5.4 point at it. - §7.1 describes the overlay acceptance cases and the overlay-segment-shards guard; the ledger stays at zero gaps. - §7.2 records the probe's new overlay phase against Beetle (both BIOSes, cold, warm and interpreter), the LLE boot after E, and R4's KUSEG and KSEG1 kernel shards with segment_alias_interp at 0 on a warm cache. - §8 E is recorded. AOT_OVERLAY_PLAN (the manifest contract and capture schema), AOT_SHARDING, OVERLAY_CACHE_V2, COMPILING_OVERLAYS (the cache layout) and OVERLAY_XREF (schema v3) follow the design, and MOD_PACKAGES notes that function-entry hooks reach overlay shards of every segment. FAITHFUL_TIMING_PLAN and ACCURACY_BURNDOWN get dated entries: overlay segments are closed, the last masked return compares (the overlay shadow run's) are now exact, mod hooks reach KUSEG/KSEG1 shards, and axis 5 records the probe's BIOS file read taking about half Beetle's cycles (found here, not caused here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Matthew Stanley
|
bc5ec35998 |
cli: generate builds the static overlay shard an AOT profile declares
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Matthew Stanley
|
de09f27d5c | tools: enrich runtime overlay captures with static roots | ||
|
Matthew Stanley
|
c3daaaa7f0 | Document disc-derived AOT sharding and preserve full producer extents | ||
|
Kevin
|
2fa3472a78 |
compile_overlays --static: isolated fragments per (entry, image), pooled, memoized
The static isolated-fragment pass resolved demands by bare entry address across every capture: once any part had a variant at an address, that address counted as served for every other capture of the band, and static_entry_sources kept only the last capture's bytes. The dispatcher validates each variant against the exact bytes it was compiled from, so in a band with N occupants exactly one occupant ever received fragments (BoF3: BOSS055 35/35, PLP678 19/19, SCENA19 20/20; the 128-occupant and 181-occupant bands got none). 23,728 (entry, image) demands were unserved; at the title screen the resident SCENA16 ran every observed entry on the interpreter while every spanning piece came from SCENA19/04/06. - Demands are keyed (entry, image crc); parts carry image_crc; select_static_fragment_demands / served_static_variant_keys are the pure helpers (unit-tested). - Fragment compiles run on the --jobs pool; the image table is sent once per worker (initializer) instead of once per demand; results merge in demand order and duplicates already served by an earlier-merged part are dropped, so the output equals the sequential loop's. - generate_interior_fragment_static returns (part, reason). Deterministic verdicts on the bytes (generated-C audit, requested-entry audit, and a walk that runs off the image, which the recompiler reports by throwing) are memoized in <out-dir>/interior_fail_memo.txt and counted as skipped, never as shard failures; --force-interior retries an entry. Only a non-deterministic recompiler failure is a shard failure. - Fragment parts are grouped one translation unit per image (BoF3: 779 units instead of 20,635 files). BoF3 full run: 20,250 fragments built, 3,570 memoized rejections, 238 s cold, nothing lost from the previous dispatcher (99,778 -> 138,148 served keys). Title screen: 12 interpreted SCENARIO-band entries -> 2, both unharvested PCs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Matthew Stanley
|
e6ce5006b4 |
docs: move root and stale docs into docs/ and docs/internal/
The repository root carried seven markdown files that were not onboarding
material, plus two that had been superseded by the code they described. This
sorts them: reader-facing docs to docs/, working notes to docs/internal/, and
the two superseded ones merged into the docs that replaced them.
Moved to docs/: WIDESCREEN.md, ENHANCEMENTS.md, TCP_COMMANDS.md
Moved to docs/internal/: PRINCIPLES.md, ISSUES.md, DEBUG.md, GTE_LM_FIX.md,
and from docs/: bios-hardening.md, overlay-status.md,
overlay-plan.md, overlay-recompilation-design.md,
overlay-discovery.md, TOMBA2_EXC_CONSULT_RESPONSE.md,
CASE_A_AOT_GAP.md, audit_inventory.md
Moved to docs/internal/upstream/: UPSTREAM_2026-08-20.md
The root keeps only what belongs there: README.md, CONTRIBUTING.md, LICENSE,
CLAUDE.md, THIRD_PARTY_ATTRIBUTION.md, RELEASE_NOTES.md, the dotfiles and
psxrecomp_cli.py.
MERGES
SPLITGEN_MIGRATION.md folded into docs/BUILDING.md as "Split generated C". Its
migration instructions were stale in a way that mattered: it told every game
repo to hand-edit its CMakeLists and add a file(GLOB ...) over
<serial>_full*.c, but runtime/runtime.cmake now does that itself. GEN_FULL_GLOB
is a multi-value argument of psxrecomp_add_game_runtime(); the loop at
runtime.cmake:1985 makes each pattern absolute against CMAKE_CURRENT_SOURCE_DIR,
expands it with file(GLOB), and concatenates the hits into
GAME_GENERATED_FULL_C. So a title names the pattern and gets however many shards
a regeneration produced. The rewritten section describes that, the two-step
fallback when the globs match nothing (GEN_FULL_FALLBACK, else <serial>_full.c
derived from the dispatch marker by substituting _dispatch.c), and the one
caveat the code actually has: that file(GLOB) is plain, not CONFIGURE_DEPENDS,
so a regeneration which changes the shard COUNT needs a re-configure to be seen.
psxrecomp-toml-readme.md folded into docs/GAME_PROJECT_SETUP.md as "EXE-only
autofill (psxrecomp-toml)", placed after "Disc autofill (probe_disc.py)" and
before "Project Studio" so the two autofill paths sit together. Its documented
flags were checked against the usage text in recompiler/src/main_toml.cpp:28-42
and against the argument parser at :150-165 -- all seven options match with no
drift, so the flag list is reproduced as-is. Added a pointer that probe_disc.py
is preferable when a cue exists, since that tool does the same EXE-header work
plus disc identity and netplay gates.
FUNCTIONAL FIXES
tools/gen_tcp_commands.py wrote to REPO/"TCP_COMMANDS.md". After the move that
path does not exist, so the generator would have created a new stray root file
and --check would have failed against it forever. Retargeted to
REPO/"docs"/"TCP_COMMANDS.md", along with the seven docstring and message
strings that name the path.
Running it surfaced two further problems, both fixed here:
* The doc was genuinely stale -- pre-existing drift, not introduced by this
commit (the moved file is byte-identical to origin/master, and no
debug-server command table is touched here). Four commands registered since
the last regeneration were missing from the index -- cdrom_timing_dump,
savestate_status, ws_ui_groups, xprobe_watch -- and scanline had gained a
backend. Regenerated: 307 -> 311 commands, and the change is confined to the
AUTOGEN block (the hand-written head and tail are byte-identical to master).
tools/gen_tcp_commands.py --check now exits 0.
* DOC.write_text() has no newline argument, so Python translated every newline
to os.linesep. On Windows that rewrote all 585 lines of the file to CRLF and
turned a four-line index update into a whole-file diff -- measured: 589 CRLF
/ 0 LF after a run, against 0 CRLF / 585 LF in the repo. Now written through
an explicit newline="" handle, verified to leave the file at 0 CRLF.
runtime/include/debug_server.h:9 and runtime/src/debug_server.c:8 cited "TCP.md
and DEBUG.md". TCP.md has never existed in this repo under that name; both now
name docs/TCP_COMMANDS.md and docs/internal/DEBUG.md.
LINK AND REFERENCE FIXES
Every inbound reference to a moved file was located by grepping each basename
across the whole tracked tree, not just the markdown. Markdown hrefs in
README.md, CONTRIBUTING.md, docs/ARCHITECTURE.md, docs/COMPILING_OVERLAYS.md,
docs/AOT_OVERLAY_PLAN.md, docs/RELOCATION_MANIFEST_FORMAT.md,
docs/config_schema.md, docs/OVERLAY_XREF.md, docs/TCP_COMMANDS.md and several
docs/internal/ files now point at the new locations; where two files became
siblings under docs/internal/ the path was shortened rather than lengthened
(bios-hardening.md's reference to overlay-discovery.md, for instance).
docs/COMPILING_OVERLAYS.md's "in this folder" phrasing was wrong once two of the
three files it listed moved, so it now names internal/ explicitly.
24 source-comment mentions of ENHANCEMENTS.md sections G1.x across 19 files
(recompiler, runtime, cmake) were retargeted to docs/ENHANCEMENTS.md with the
section numbers untouched. docs/SHADOW_ENHANCEMENTS.md is a DIFFERENT file and
was deliberately excluded.
20 rationale strings citing docs/CASE_A_AOT_GAP.md across the four
recompiler/seeds/*.json files were updated. Done as a byte-level substring
replacement rather than a JSON round-trip: two of those files are stored with
CRLF endings and two with LF, and re-serializing them reformatted 5,700 lines.
The byte-level version changes exactly 20 lines and each file still parses.
References that look similar but point elsewhere were deliberately left alone,
and are worth naming so a future pass does not "fix" them:
* "recomp-template PRINCIPLES.md" / "NES recomp template PRINCIPLES.md" in
CLAUDE.md:43, docs/SHADOW_ENHANCEMENTS.md:28,
docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md:5,
runtime/src/debug_server.c:4934 and tools/parity_diff.py:8 are a different
repository's file. Only the five that mean this repo's PRINCIPLES.md were
updated.
* Every remaining bare ISSUES.md mention is a GAME repo's tracker
(MegaManX6Recomp 2/7/8/9, TombaRecomp 6), not this repo's.
* docs/disc-speed.md:4,54,56 and docs/internal/bios-hardening.md:5 name
overlay-discovery as a BRANCH, not a file, so no path applies.
INDEXES
README.md's "New here?" table gains rows for docs/WIDESCREEN.md,
docs/ENHANCEMENTS.md and docs/TCP_COMMANDS.md. docs/internal/README.md gains
sections indexing all fourteen documents that moved in, plus an "Upstream
contributor reviews" section describing upstream/ and the new sweep file, plus a
"Removed" bullet for CYCLE_TIMING_ARCH.md (deleted in the previous commit)
pointing at FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md -- following the SLJIT_PERSIST_CACHE.md precedent already
in that file.
docs/ENHANCEMENTS.md:7 claimed framework bugs live in "game repos' ISSUES.md
until a framework tracker exists". A framework tracker does exist; it now says
so and names the Beads epic.
VERIFICATION
A link checker was written to scan every tracked *.md for relative links,
resolving each against the tree being scanned. Baseline at origin/master: 129
files, 137 relative links, 3 broken. After both commits: 127 files, 159 relative
links, 3 broken -- the SAME three, all pre-existing and untouched here:
docs/DATA_SHARDS.md -> INSTANT_LOADS_PARKED.md (never existed)
docs/DATA_SHARDS.md -> INSTANT_LOADS_SPIKE2.md (never existed)
docs/NETPLAY.md -> ../lib/recomp-net/README.md (inside a submodule)
Zero new broken links. The checker's own coverage was checked against a raw grep
for markdown link openers: the 3-link difference is C++ lambdas inside code
blocks, correctly not treated as links.
Also verified: py_compile over all 113 tracked tools/*.py plus psxrecomp_cli.py;
all four seed JSONs still parse; the test-registration guard still reports 123
test files all registered; recompiler/tests/test_coverage_vault.py still passes;
and no file in this commit had its line endings flipped (checked mechanically
across all 50 modified files, after two files were caught and redone).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
Kevin
|
7ab698caa5 |
overlays: parallel static compile, linear resume-wrapper pass, split output
Static mode (--static) was serial end to end and produced one translation unit, so a whole title compiled on one core twice: once in Python, once in gcc. Measured on BoF3 (339 captures, 79,688 identities) in a build-dbg loop: phase "compile all overlay bands" ~12 min, runtime build ~4 min at -O0. Three changes, each independently verified byte-identical: 1. static_capture_job(): the per-capture static work (recompiler subprocess, C audit, range audit, namespacing) is now a module-level, picklable job that returns its part + captured log. --jobs runs it in a ProcessPoolExecutor; results merge in capture order, so static_parts and entry sources are order-identical to the sequential loop. --jobs 1 and --jobs 3 on three bands: diff -r of the outputs is empty. 2. add_cps_resume_cases(): the universal CPS resume-wrapper pass called add_cps_resume_case() once per entry, and each call re-scanned and rebuilt the WHOLE part source. cProfile on three bands: 7,386 entries -> 22,159 re.search calls, 366 s of 424 s total. The batch form locates every signature once, edits each host's segment in isolation, rebuilds once. Same three bands: 418 s -> 5 s, outputs byte-identical (diff -r against the per-entry output). The per-entry function is kept; a unit test asserts the two agree on fresh-switch, existing-switch, missing label and missing host cases. 3. write_static_outputs(): overlays_static.c now holds the dispatcher plus an extern prototype per compiled entry, and every part becomes its own unit overlays_static_NNNN.c (each already carries its #includes and a private namespace). runtime.cmake globs the siblings with CONFIGURE_DEPENDS, so a changed part count needs no reconfigure. Unchanged units are not rewritten (mtime preserved -> not recompiled); stale units are removed. --static-single-file restores the monolithic file. Result, same BoF3 loop on a 12-core box: phase 5a 12 min -> 20 s, runtime build 4 min -> 69 s (358 units in parallel), whole loop ~16 min -> 90 s. Shard summary unchanged (332 ok / 7 benign audit fails), identity count unchanged. Headless boot of the rebuilt runtime: 485,650 static checks, 485,244 hits (99.92%), miss_total 0, 60 emu fps through the LOGO overlay. Note: tools/test_compile_overlays_additive.py already fails on master (it references removed functions: compile_and_publish_dll, load_additive_captures, validated_cached_bundle_ids); untouched here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Matthew Stanley
|
a3b7f47d17 |
tools: add stall_report.py, one-command interp/stall attribution
Diagnosing "why is this title spending its time in the interpreter" currently means issuing eight TCP commands by hand and correlating their JSON. That is enough friction that it does not get done, and when it is done by someone without the runtime's internals in their head the numbers get misread -- #72 is a live example: an empty phase_hot was read as a possible bad query when it in fact meant nothing native was executing at all. stall_report.py collapses that into one command and one pasteable report, usable against any title on any port: [1] Is the overlay cache actually BEING USED, or just present on disk? dispatch_native vs dispatch_interp_fallback, stale_blocked, the autocompile degraded channel, and shard ok/fail/skipped counts. [2] When code IS interpreted, is it chaining or thrashing? dirty_ram insns_run/blocks_run = instructions per dispatcher round-trip, with the kernel-window-is-per-block-by-design caveat stated inline so a low ratio there is not misread as a bug. [3] Where did the wall-clock actually go? The largest host_us gaps in the always-on PC-sample ring, each with the guest cycles that advanced across it. That ratio separates three failure shapes that look identical in a frame-time graph: CPU kept up (gap is GPU/present/vsync), CPU-bound but executing, and emu thread not running at all. [4]-[6] phase shares, hottest interpreted PCs tagged kernel vs above-floor, and phase_hot deltas for both the native and static sets. PASSIVE RING CONSUMER by construction, per the project's observability rule: it never arms a trace, never pauses, never single-steps. Everything is a cumulative counter (two snapshots subtract to a window) or a read of a ring that has been recording since process start, so attaching late still sees the event. `snap` reports cumulative; `run --secs N` reports deltas. Two caveats are printed in the output rather than left to be rediscovered: current_func is a STATIC dispatch stamp and localizes without proving interpretation, and an empty native phase_hot means nothing native ran rather than a malformed query. Validated against a live Tomba 2 debug build (Release + PSX_DEBUG_TOOLS=ON, port 4515), both modes and the --out JSON artifact. Representative real output: dispatch_native 3,253,132 (97.16%) vs interp_fallback 95,047 (2.84%), 16,329 instructions per dispatcher round-trip, and one 68 ms gap at 0.46x realtime flagged in-exception. Development of the tool caught two defects in itself that way -- snap mode subtracted a snapshot from itself and reported zeros for everything, and non-ASCII output mojibaked on a cp1252 console -- both fixed before commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Matthew Stanley
|
e3ecd67bd4 |
recompiler: resolve the BIOS profile against --project-root, not the CWD
compile_overlays.py spawns psxrecomp-game with cwd = dirname(game.toml), because the seeds and capture paths in the config are relative to it. That cwd is also where the recompiler probes for bios/SCPH1001.toml when the config carries no explicit [recompiler] bios_config -- and --ws-config, the only config channel overlay compiles use, deliberately does not adopt the config's paths, so bios_config is never consulted for them. For a PACKAGED game.toml (packaging/release/game.toml) that directory holds neither bios/SCPH1001.toml nor a vendored framework one level down. Every shard then died with "FATAL: no BIOS profile found", ok=0, and the overlay cache silently failed to build. The run still exits, so the only symptom is that the title ships with no cache and every player's first session runs interpreted. Reported as issue #72, and it is what the recipe printed by the game repos' package_release.ps1 told people to run. Fix: - psxrecomp-game takes --project-root <dir> and resolves the profile against it: the direct bios/SCPH1001.toml probe, the one-level vendored-framework probe, and a relative [recompiler] bios_config (documented as relative to the game project root, previously resolved against the CWD). Absent the flag, behaviour is unchanged. - Every "no BIOS profile found" / bios_config failure now names the absolute directory it searched. Without it the message reads as a config error when it is a CWD error, which is what cost two vault parses on the Tomba 2 v0.0.8 cut. - compile_overlays.py forwards --project-root to all three recompiler spawn sites, defaulting it to the framework root derived from the required --runtime-include. The derivation is used ONLY when it actually locates a profile, so it can repair a broken invocation but never alter one that already worked -- and the recipe the packager prints now works as written, with no game-repo change. Every attribute lookup degrades to "pass no flag" so the in-process callers that build a partial args object keep working. This also un-breaks the framework's own overlay test suite, which was RED on master for the same reason: test_aot_overlay_discovery.py compiles against tools/cycle_testrom/game.toml, so the recompiler probed tools/cycle_testrom/ -- which has no BIOS profile -- and every shard failed. Measured on this tree with one recompiler binary, changing only the tool and tests: master exits 1 at check_real_batched_fragment_publication; this commit reaches ALL PASS. Two test repairs were needed to get there, both pre-existing and both masked by the profile failure that stopped the suite before it could reach them: - check_full_candidate_cli_fastpath stages a SYNTHETIC runtime_include in a temp dir, so the framework root is not derivable from it and the root must be named explicitly. - the same test staged its cache prefill under cg<ver>_<hash>, missing the _gc<config-hash> component the cache path gained. A prefill one directory off is not found, so the run compiled for real instead of taking the fast path the test exists to assert. The cache tag is unaffected: main_psx.cpp is not in codegen_hash_sources.cmake (verified -- the baked hash is fbb44d49 before and after), and --overlay-config-hash returns before any of this code runs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Matthew Stanley
|
c94fcd5e1a |
docs: add COMPILING_OVERLAYS how-to (overlay_captures.json -> shards)
User-facing guide for turning overlay_captures.json into native code: developer-first framing (ship a full gcc cache), the manual gcc cache build, the --static bake-into-binary path, and the staleness guard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |