mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-27 10:38:46 -04:00
6fcfb8adbdcd666a2fc8ba419dd4e2aa7ee1faaa
348 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alex Vanderveen
|
155e269ba5 |
tools: DirectWrite/Direct2D/WIC are system DLLs, not payload
recomp-ui's Win32 emoji and font path imports DWrite.dll, d2d1.dll and the WIC/D3D family. None was in the system list, so the bundler went looking for them in the MinGW sysroot, did not find them (they only exist on Windows) and failed the packaging step — measured on the SNES side, where the same recomp-ui pulled the same imports and killed the release job. Brings the list to parity with snesrecomp's bundle_runtime_libs.sh, whose entries this one was missing as well (KERNELBASE, GDIPLUS, SHCORE, DWMAPI, UXTHEME, HID, XINPUT*, AVRT, MF*, EXT-MS-*, and the rest). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DYwgPRbwbXz8FhpVzDM7tA |
||
|
Alex Vanderveen
|
3d9930c5d3 |
release: Authenticode-sign Windows binaries when a certificate secret exists
Smart App Control blocks unsigned executables outright. The packager now runs tools/ci/sign_windows.sh over the staged tree (host exe, DLLs, emitters) with SHA-256 + RFC 3161 timestamp; no certificate in the environment means a notice and an unsigned package as before, while a configured certificate that fails to sign is fatal. The workflow template passes WINDOWS_SIGN_PFX_BASE64 / WINDOWS_SIGN_PFX_PASSWORD to the package step. Docs cover the secrets and the limit: the game exe the player builds locally after Generate is unsigned by nature. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V1dT6h1ZshMVUUVKTPxsCD |
||
|
Matt Stanley
|
b213858c4c |
Merge pull request #323 from Alexbeav/codex/sbi-setup
Preserve SBI companions and validate them before launch |
||
|
Matt Stanley
|
d485a751d8 |
Merge pull request #325 from kerokline/fix/static-fragments-per-variant
compile_overlays --static: isolated fragments per (entry, image), pooled, memoized |
||
|
Alex Vanderveen
|
d371d74421 |
tools: generate_ci -- the release workflow for an existing project, and nothing else
sh psxrecomp/tools/generate_ci.sh # in the project
powershell -File psxrecomp\tools\generate_ci.ps1
... [root] [--check] [--force] [--dry-run] [--zip-prefix X] [--title X]
Project Studio already knows how to emit the setup-host release workflow
(emit_ci_workflow, behind `apply` and `git install-ci`), but reaching it
meant a migration plan or a commit-and-push. This is the single-purpose
face of that operation: it resolves the template from the project's own
psxrecomp submodule, takes the zip prefix from
scripts/package_setup_release.sh so CI uploads the zips the packager
built, writes .github/workflows/release.yml, and stops.
--check reports missing / placeholder / stale (step names the template
has and the file lacks) with exit 1, so a fleet can be audited without
writing anything; only --force overwrites. The .sh and .ps1 are thin
launchers over the one Python implementation.
Verified read-only against two ports: ApeEscapeRecomp reports stale
(four emitter/ccache steps missing), TombaRecomp reports current.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0173EhspvAdRQc9DZdDr97Hg
|
||
|
Kevin
|
2fa3472a78 |
compile_overlays --static: isolated fragments per (entry, image), pooled, memoized
The static isolated-fragment pass resolved demands by bare entry address across every capture: once any part had a variant at an address, that address counted as served for every other capture of the band, and static_entry_sources kept only the last capture's bytes. The dispatcher validates each variant against the exact bytes it was compiled from, so in a band with N occupants exactly one occupant ever received fragments (BoF3: BOSS055 35/35, PLP678 19/19, SCENA19 20/20; the 128-occupant and 181-occupant bands got none). 23,728 (entry, image) demands were unserved; at the title screen the resident SCENA16 ran every observed entry on the interpreter while every spanning piece came from SCENA19/04/06. - Demands are keyed (entry, image crc); parts carry image_crc; select_static_fragment_demands / served_static_variant_keys are the pure helpers (unit-tested). - Fragment compiles run on the --jobs pool; the image table is sent once per worker (initializer) instead of once per demand; results merge in demand order and duplicates already served by an earlier-merged part are dropped, so the output equals the sequential loop's. - generate_interior_fragment_static returns (part, reason). Deterministic verdicts on the bytes (generated-C audit, requested-entry audit, and a walk that runs off the image, which the recompiler reports by throwing) are memoized in <out-dir>/interior_fail_memo.txt and counted as skipped, never as shard failures; --force-interior retries an entry. Only a non-deterministic recompiler failure is a shard failure. - Fragment parts are grouped one translation unit per image (BoF3: 779 units instead of 20,635 files). BoF3 full run: 20,250 fragments built, 3,570 memoized rejections, 238 s cold, nothing lost from the previous dispatcher (99,778 -> 138,148 served keys). Title screen: 12 interpreted SCENARIO-band entries -> 2, both unharvested PCs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Alexandros Mandravillis
|
2d92d67592 | Validate and import SBI companions in the native launcher | ||
|
Alexandros Mandravillis
|
1c60d340dc | Preserve and verify SBI companions during disc setup | ||
|
Matthew Stanley
|
240cff542e |
Merge remote-tracking branch 'origin/pr/312' into integrate/303-312
# Conflicts: # tools/compile_overlays.py |
||
|
Matthew Stanley
|
254a14f3d7 |
Merge remote-tracking branch 'origin/pr/303' into integrate/303-312
# Conflicts: # tools/compile_overlays.py |
||
|
Matthew Stanley
|
ea247cbbd4 |
chore: drop tools/_capture_button_press.py, orphaned by the scratch-tool purge
Its only referrer was one of the 204 scratch probes removed in 8e16acd8; after that commit a word-boundary grep over every tracked file returns zero hits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Matthew Stanley
|
e6ce5006b4 |
docs: move root and stale docs into docs/ and docs/internal/
The repository root carried seven markdown files that were not onboarding
material, plus two that had been superseded by the code they described. This
sorts them: reader-facing docs to docs/, working notes to docs/internal/, and
the two superseded ones merged into the docs that replaced them.
Moved to docs/: WIDESCREEN.md, ENHANCEMENTS.md, TCP_COMMANDS.md
Moved to docs/internal/: PRINCIPLES.md, ISSUES.md, DEBUG.md, GTE_LM_FIX.md,
and from docs/: bios-hardening.md, overlay-status.md,
overlay-plan.md, overlay-recompilation-design.md,
overlay-discovery.md, TOMBA2_EXC_CONSULT_RESPONSE.md,
CASE_A_AOT_GAP.md, audit_inventory.md
Moved to docs/internal/upstream/: UPSTREAM_2026-08-20.md
The root keeps only what belongs there: README.md, CONTRIBUTING.md, LICENSE,
CLAUDE.md, THIRD_PARTY_ATTRIBUTION.md, RELEASE_NOTES.md, the dotfiles and
psxrecomp_cli.py.
MERGES
SPLITGEN_MIGRATION.md folded into docs/BUILDING.md as "Split generated C". Its
migration instructions were stale in a way that mattered: it told every game
repo to hand-edit its CMakeLists and add a file(GLOB ...) over
<serial>_full*.c, but runtime/runtime.cmake now does that itself. GEN_FULL_GLOB
is a multi-value argument of psxrecomp_add_game_runtime(); the loop at
runtime.cmake:1985 makes each pattern absolute against CMAKE_CURRENT_SOURCE_DIR,
expands it with file(GLOB), and concatenates the hits into
GAME_GENERATED_FULL_C. So a title names the pattern and gets however many shards
a regeneration produced. The rewritten section describes that, the two-step
fallback when the globs match nothing (GEN_FULL_FALLBACK, else <serial>_full.c
derived from the dispatch marker by substituting _dispatch.c), and the one
caveat the code actually has: that file(GLOB) is plain, not CONFIGURE_DEPENDS,
so a regeneration which changes the shard COUNT needs a re-configure to be seen.
psxrecomp-toml-readme.md folded into docs/GAME_PROJECT_SETUP.md as "EXE-only
autofill (psxrecomp-toml)", placed after "Disc autofill (probe_disc.py)" and
before "Project Studio" so the two autofill paths sit together. Its documented
flags were checked against the usage text in recompiler/src/main_toml.cpp:28-42
and against the argument parser at :150-165 -- all seven options match with no
drift, so the flag list is reproduced as-is. Added a pointer that probe_disc.py
is preferable when a cue exists, since that tool does the same EXE-header work
plus disc identity and netplay gates.
FUNCTIONAL FIXES
tools/gen_tcp_commands.py wrote to REPO/"TCP_COMMANDS.md". After the move that
path does not exist, so the generator would have created a new stray root file
and --check would have failed against it forever. Retargeted to
REPO/"docs"/"TCP_COMMANDS.md", along with the seven docstring and message
strings that name the path.
Running it surfaced two further problems, both fixed here:
* The doc was genuinely stale -- pre-existing drift, not introduced by this
commit (the moved file is byte-identical to origin/master, and no
debug-server command table is touched here). Four commands registered since
the last regeneration were missing from the index -- cdrom_timing_dump,
savestate_status, ws_ui_groups, xprobe_watch -- and scanline had gained a
backend. Regenerated: 307 -> 311 commands, and the change is confined to the
AUTOGEN block (the hand-written head and tail are byte-identical to master).
tools/gen_tcp_commands.py --check now exits 0.
* DOC.write_text() has no newline argument, so Python translated every newline
to os.linesep. On Windows that rewrote all 585 lines of the file to CRLF and
turned a four-line index update into a whole-file diff -- measured: 589 CRLF
/ 0 LF after a run, against 0 CRLF / 585 LF in the repo. Now written through
an explicit newline="" handle, verified to leave the file at 0 CRLF.
runtime/include/debug_server.h:9 and runtime/src/debug_server.c:8 cited "TCP.md
and DEBUG.md". TCP.md has never existed in this repo under that name; both now
name docs/TCP_COMMANDS.md and docs/internal/DEBUG.md.
LINK AND REFERENCE FIXES
Every inbound reference to a moved file was located by grepping each basename
across the whole tracked tree, not just the markdown. Markdown hrefs in
README.md, CONTRIBUTING.md, docs/ARCHITECTURE.md, docs/COMPILING_OVERLAYS.md,
docs/AOT_OVERLAY_PLAN.md, docs/RELOCATION_MANIFEST_FORMAT.md,
docs/config_schema.md, docs/OVERLAY_XREF.md, docs/TCP_COMMANDS.md and several
docs/internal/ files now point at the new locations; where two files became
siblings under docs/internal/ the path was shortened rather than lengthened
(bios-hardening.md's reference to overlay-discovery.md, for instance).
docs/COMPILING_OVERLAYS.md's "in this folder" phrasing was wrong once two of the
three files it listed moved, so it now names internal/ explicitly.
24 source-comment mentions of ENHANCEMENTS.md sections G1.x across 19 files
(recompiler, runtime, cmake) were retargeted to docs/ENHANCEMENTS.md with the
section numbers untouched. docs/SHADOW_ENHANCEMENTS.md is a DIFFERENT file and
was deliberately excluded.
20 rationale strings citing docs/CASE_A_AOT_GAP.md across the four
recompiler/seeds/*.json files were updated. Done as a byte-level substring
replacement rather than a JSON round-trip: two of those files are stored with
CRLF endings and two with LF, and re-serializing them reformatted 5,700 lines.
The byte-level version changes exactly 20 lines and each file still parses.
References that look similar but point elsewhere were deliberately left alone,
and are worth naming so a future pass does not "fix" them:
* "recomp-template PRINCIPLES.md" / "NES recomp template PRINCIPLES.md" in
CLAUDE.md:43, docs/SHADOW_ENHANCEMENTS.md:28,
docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md:5,
runtime/src/debug_server.c:4934 and tools/parity_diff.py:8 are a different
repository's file. Only the five that mean this repo's PRINCIPLES.md were
updated.
* Every remaining bare ISSUES.md mention is a GAME repo's tracker
(MegaManX6Recomp 2/7/8/9, TombaRecomp 6), not this repo's.
* docs/disc-speed.md:4,54,56 and docs/internal/bios-hardening.md:5 name
overlay-discovery as a BRANCH, not a file, so no path applies.
INDEXES
README.md's "New here?" table gains rows for docs/WIDESCREEN.md,
docs/ENHANCEMENTS.md and docs/TCP_COMMANDS.md. docs/internal/README.md gains
sections indexing all fourteen documents that moved in, plus an "Upstream
contributor reviews" section describing upstream/ and the new sweep file, plus a
"Removed" bullet for CYCLE_TIMING_ARCH.md (deleted in the previous commit)
pointing at FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md -- following the SLJIT_PERSIST_CACHE.md precedent already
in that file.
docs/ENHANCEMENTS.md:7 claimed framework bugs live in "game repos' ISSUES.md
until a framework tracker exists". A framework tracker does exist; it now says
so and names the Beads epic.
VERIFICATION
A link checker was written to scan every tracked *.md for relative links,
resolving each against the tree being scanned. Baseline at origin/master: 129
files, 137 relative links, 3 broken. After both commits: 127 files, 159 relative
links, 3 broken -- the SAME three, all pre-existing and untouched here:
docs/DATA_SHARDS.md -> INSTANT_LOADS_PARKED.md (never existed)
docs/DATA_SHARDS.md -> INSTANT_LOADS_SPIKE2.md (never existed)
docs/NETPLAY.md -> ../lib/recomp-net/README.md (inside a submodule)
Zero new broken links. The checker's own coverage was checked against a raw grep
for markdown link openers: the 3-link difference is C++ lambdas inside code
blocks, correctly not treated as links.
Also verified: py_compile over all 113 tracked tools/*.py plus psxrecomp_cli.py;
all four seed JSONs still parse; the test-registration guard still reports 123
test files all registered; recompiler/tests/test_coverage_vault.py still passes;
and no file in this commit had its line endings flipped (checked mechanically
across all 50 modified files, after two files were caught and redone).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
Matthew Stanley
|
f55c731c5f |
chore: remove dead scratch tooling and stale root files
Removes 204 tools/ scripts (203 .py/.sh plus tools/_missing_funcs.json) that
no tracked file references. Verified mechanically, not by eye: for each of the
204 stems, a word-boundary grep across every tracked text file in the repo,
excluding files that are themselves being deleted, returns zero hits. So
nothing kept imports, execs, or documents any of them. Most are the
underscore-prefixed one-shot probes the .gitignore convention already treats
as scratch (tools/_*.py is ignored; these predate that rule and were force-
added), plus a long tail of session-specific dump/decode/poll scripts with
hardcoded local paths and addresses.
Also removed at the root:
* card2.mcd.bak.before_beetle_copy -- a stray backup of a memcard.
* .mcp.json -- local MCP server wiring, not repo configuration.
* SCPH1001.toml -- a root duplicate of bios/SCPH1001.toml, still on the
pre-refactor [[recompiler.bios_vectors]] schema while the bios/ copy has
moved to [recompiler.address_model]. Every reference in the tree already
names bios/SCPH1001.toml, so the root copy was a stale trap: editing it
would have had no effect on any build.
* docs/CYCLE_TIMING_ARCH.md -- superseded by
docs/internal/FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md; nothing linked to it.
Memcard fixtures move out of the repo root into
runtime/tests/fixtures/memcards/ with a README recording what each of the
four images actually contains (directory entries read out of the images, not
guessed). Nothing in the build read them from a fixed path -- the four
hardcoded consumers (tools/check_card.py, inspect_card.py, peek_cards.py,
verify_card_buffer.py) are all in the deletion above.
.gitignore is corrected to match: runtime memcards a session creates stay
ignored (*.mcr anywhere, /*.mcd at the root, which is where the runtime drops
card1.mcd/card2.mcd when no path is configured), while the tracked fixtures
are re-included. The old !dummy.*.mcr negation existed only to keep the two
root oracle cards tracked and is replaced by
!runtime/tests/fixtures/memcards/*. Verified with git check-ignore --no-index
on each moved file, on the pre-existing runtime/tests/test_card_synth.mcd and
runtime/tests/test_dir/card1.mcd fixtures (all not-ignored), and on
hypothetical root card1.mcd / card2.mcd / saves/card1.mcr (all ignored).
beetle_libretro.cpp's warn-only memcard sanity check keeps its behaviour; its
WARNING now names where populated fixtures can be copied from. The cards are
opened relative to the process's working directory because the libretro
GET_SAVE_DIRECTORY environment callback answers "." (same file, line 419), so
the message says to copy them into the CWD rather than implying a repo path.
BIOS profiles SCPH101.toml and SCPH5552.toml move from the root into bios/,
joining OpenBIOS.toml and SCPH1001.toml so that bios/*.toml is the whole set.
Behaviour-preserving: config paths inside a profile resolve against
find_project_root() (config_loader.cpp:821), which walks up from the profile's
directory for .gitignore/.git/CMakeLists.txt -- bios/ has none of those, so it
lands on the same repo root as before, and bios/SCPH1001.toml already relied
on exactly that. tools/build_cli.py's staging map is updated to the new
sources; it already copied them to framework/bios/, so the packaged layout is
unchanged. recompiler/src/main_cli.cpp's bios/-first lookup keeps its root
fallback (harmless, and it is what makes older layouts still resolve).
Fixes a real latent break while in bios/SCPH5552.toml: seeds pointed at
"seeds/phase2_ghidra_seeds_SCPH5552.json", but there is no seeds/ directory at
the repo root -- the file lives at
recompiler/seeds/phase2_ghidra_seeds_SCPH5552.json, which is what SCPH101.toml
already named. Porting these two profiles to the address_model schema is
deliberately NOT done here; it is a separate tracked follow-up.
Test files that no build system knew about:
* tools/test_coverage_vault.py PASSES (8/8). Moved to
recompiler/tests/test_coverage_vault.py and registered as the ctest
coverage_vault, which puts it inside runtime/check_test_registration.cmake's
glob so it can never silently rot again. Its ROOT is adjusted from
parents[1] to parents[2] for the new depth, matching every other python
test in that directory. Re-run from the new location and from an unrelated
CWD: 8/8. The registration guard was run standalone before and after, and
with the add_test() deliberately broken as a negative control -- it does
name recompiler/tests/test_coverage_vault.py as an orphan when
unregistered, so the registration is load-bearing rather than decorative.
* tools/test_compile_overlays_additive.py FAILS, 10 of 14 tests erroring
against an API that has moved on, so it is removed rather than wired up.
It calls four module attributes that no longer exist in
tools/compile_overlays.py -- load_additive_captures,
validated_cached_bundle_ids, cached_bundle_pairs, compile_and_publish_dll
(zero occurrences each) -- and two signatures have changed under it:
parse_overlay_func_ids() no longer takes require_stored_crc, and
load_region_coverage() now takes 2-3 positional arguments where the test
passes 5. Reviving it means rewriting it against the current additive
capture API, which is a change to overlay-cache test coverage and needs its
own commit and its own evidence, not a rename.
Verification: py_compile over all 113 remaining tracked tools/*.py plus
psxrecomp_cli.py passes; bash -n over every remaining tracked tools/*.sh
passes; importing sdk_progress, pad_delivery and cursor_writers from tools/
succeeds.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
||
|
Matt Stanley
|
d08d84a3df |
Merge pull request #296 from kerokline/perf/static-overlay-parallel
overlays: parallel static compile, linear resume-wrapper pass, split output |
||
|
theanch92
|
71a4db4cf5 | compile_overlays: support --target-os and auto-detect mingw cross-compiler | ||
|
Matt Stanley
|
a7459cddbe | Make packaged overlay toolchain self-contained (#311) | ||
|
Matt Stanley
|
6cb7a6b8c2 |
Stage overlay dispatch preamble with toolchain (#310)
Co-authored-by: Codex <codex@openai.com> |
||
|
Matt Stanley
|
c6aa071a47 |
Merge pull request #297 from mstan/tools/cpu-throttle-harness
tools: throttle a live title's CPU to reproduce a weaker host |
||
|
Matthew Stanley
|
36e57fbb8a |
release: stop overstating the staged toolchain size by following symlinks
os.path.getsize() follows symlinks. A relocatable CPython is dense with them -- the staged Linux overlay toolchain carries over a thousand -- so the size line counted every link target a second time and reported ~110 MB for a tree that is 80 MB on disk, a 38% overstatement in a release log. lstat, and skip links. The line now also reports the file and symlink counts, because "80 MB in 4103 files + 1051 symlinks" is a number someone can compare against an extracted artifact, which is how everything else here is verified. Log output only: no staged byte changes, and an artifact built before and after this commit is identical apart from mtimes. Bead: beads-eio.3.102 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Matthew Stanley
|
37b32bdd0c |
release: refuse to derive a release tag from an unbuilt runtime include tree
compile_overlays.codegen_hash() returns 0 when runtime/include/
overlay_codegen_hash.h is absent. That fallback is correct for its purpose and
should stay: overlay_api.h has the same __has_include fallback, so a runtime
built without the header and a compiler run without the header agree on
cg<N>_00000000_..., and the cache still loads.
It is never correct for a RELEASE. The header is generated by the runtime
BUILD, in runtime.cmake's "Hashing recompiler codegen" step, so a packager that
reads a tree which has not been built yet silently produces a tag with a zeroed
codegen hash: a plausible-looking string naming a namespace the shipped binary
does not read.
MEASURED 2026-09-02 while building the Tomba 2 Linux cache. Deriving the tag
before the runtime build produced
cg10_00000000_gc530dff78_f0
where the correct tag for that same tree, after the build, is
cg10_ecd487f7_gc530dff78_f0
Note what it looks like from the outside: a well-formed tag, the right number
of fields, the right config hash, the right flavor. A packager would happily
stage a whole cache under it and report a shard count. The runtime would ignore
every one of them. Nothing in a build, a configure, a link or a log would say
so -- which is the exact failure shape this bead exists to remove, one field
over from the field that started it.
So `release_stage.py cg-tag` now fails when the codegen hash reads as 0, and
says which header is missing and that the runtime target has to be built first
(the same build that publishes the flavor file the tag also needs). There is no
override: for a release the condition has no correct outcome.
test_release_stage_cache.py gains CodegenHashGuardTest -- refused on an unbuilt
tree, accepted on a built one. 18 cases, all passing.
Bead: beads-eio.3.102
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Matthew Stanley
|
d655e47168 |
release: never stage a producer-internal cache file, even one ending in .so
Found by looking at a REAL cache tree instead of a synthetic one. A live Tomba 2
Linux cache build, sampled mid-run 2026-09-02, holds this beside its 184
finished shards:
184 *.so.pair-lock producer concurrency locks
185 *.c generated C the shard was compiled from
1 .overlay-candidate-capacity.lock
1 .overlay-candidate-capacity.lock.invocation
2 .00096000_407B1780.so.tmp.ryt4jqca.so
.00096000_407B1780.so.tmp.ryt4jqca.ranges
The last pair is the problem. compile_overlays writes each shard to a hidden
temp name and renames it into place, so a cache directory that is being written
-- or that was written by an INTERRUPTED run and still holds leftovers --
contains files that end in .so and .ranges and are not shards.
A suffix allowlist alone therefore ships them. So did the shell packagers'
`find ... -name '*.so' -o -name '*.ranges'`, which is a latent defect in all
three forks that nobody had hit yet. It is not loud when it happens: an extra
dlopen-able file with a hidden name lands in the release, the loader tries it,
and the outcome depends on how far the interrupted compile got.
FIX: the forbidden-artifact denylist is checked BEFORE the suffix allowlist, and
gains two rules -- a leading dot, and `.tmp.` anywhere in the name. Every
producer-internal file observed in a real cache is hidden and no shippable
artifact ever is, so the leading dot is the reliable discriminator; the `.tmp.`
rule is belt and braces for a future producer that stops hiding them.
Exclusions are now reported as a grouped tally rather than one line each. A real
cache holds one .pair-lock and one .c PER SHARD, so per-file notes would bury
the shard count under ~370 lines in every release log -- the same "the important
line scrolled past" failure this bead is about, one level down.
test_release_stage_cache.py gains test_real_producer_artifacts_never_ship, which
models that exact file set verbatim, including the temp pair, and asserts the
staged directory contains exactly the 11 shippable files and nothing hidden.
16 cases, all passing.
Also corrects the ApeEscapeRecomp line count in the header comment: 416 on
origin/master, not 403. 403 is what its parked working tree reports, and reading
a working tree instead of origin/master is the mistake that produced the wrong
first version of this bead.
Bead: beads-eio.3.102
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Matthew Stanley
|
94e3ef610d |
release: ONE staging implementation for both platforms (2 titles had no Linux release path)
tools/release_overlay_stage.ps1 exists because a packager that re-derived the
overlay cache tag in PowerShell went stale the moment the framework appended an
_f<flavor> field to it: the packager's filter matched nothing, a perfectly
regenerated cache staged ZERO shards, and the release shipped with no native
overlays (beads-eio.2.6). The fix was correct -- expose cache_tag() and route
the packager through it -- and it fixed Windows only.
Linux was never consolidated at all. Three title repositories each carry a
forked tools/package_appimage.sh, and each hand-builds the same tag string,
inside a python heredoc that has just imported the module that owns it:
spec = importlib.util.spec_from_file_location('co', mod_path)
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
print(<the tag format, reproduced by hand> % (m.codegen_ver(inc),
m.codegen_hash(inc), m.overlay_config_hash(exe, gt)))
MEASURED 2026-09-02, reading each repository's origin/master (not a working
tree -- the Ape checkout is parked on codex/ape-issue3-diagnostic-prerelease,
behind master, and reading it gives the wrong answer, including a wrong line
count):
repo lines calls cache_tag() composes _f stages toolchain
ApeEscapeRecomp 416 no yes no
Tomba2Recomp 436 no NO no
MegaManX6Recomp 408 no NO no
ApeEscapeRecomp 4a17272 (2026-08-28, "release: stage flavored Linux overlay
cache") hand-appended the field to ONE of the three forks. It did not adopt
cache_tag(); it wrote cache_tag="${cg_tag}_f${overlay_flavor}" and switched the
filters to it. So Ape is correct today and one character from the same failure
on the next tag-shape change, and the other two are broken right now:
find -path "*/$cg_tag/*" requires a separator immediately after the tag, the
real directory is named ..._f0, so the pattern cannot match. shards evaluates
to 0 and the packager exits 1.
That is not hypothetical. Tomba 2 v0.0.9, released 2026-09-02, has exactly two
assets: Tomba2Recomp-v0.0.9-windows-x64.zip and its .sha256. There is no
AppImage in that release at all, which is what a packager that exits 1 looks
like from the outside.
Two implementations, one per platform, have to be kept in step by review. This
bug is the proof that review does not hold: the identical defect had ALREADY
been fixed on the other platform, and the reason cache_tag() exists is that
fix. So there is now ONE implementation.
tools/release_stage.py the implementation (new)
tools/release_overlay_stage.ps1 thin wrapper, surface UNCHANGED
tools/release_overlay_stage.sh thin wrapper, new (Linux had none)
Both wrappers are argument marshalling. Neither contains a tag format string, a
shard filter, an extension list, an arch-abi segment or a catalog rule. Every
sh function forwards "$@" verbatim, so a new option needs no edit there at all.
No title's package_release.ps1 needs a change: every parameter, switch and
return value of Get-OverlayCgTag / Add-OverlayCache / Add-OverlayToolchain /
Add-ModCatalog / Get-PinnedArchive is preserved, verified by dot-sourcing the
module in PowerShell 5.1 and exercising all five.
WHAT THE SHARED IMPLEMENTATION DOES DIFFERENTLY
* The tag is cache_tag(). Not "a call that produces the same string" -- the
call. release_stage.py does not know the tag's shape and must never learn it.
It also takes the arch-abi and the shared-library suffix from
compile_overlays.cache_arch_abi()/overlay_ext() rather than naming them, so
".so under linux-x64" and ".dll under win-x64" are one decision, made in the
module the loader is kept in lockstep with, not two.
* Shards are selected by the tag DIRECTORY'S IDENTITY, not by a path substring.
-path "*/<tag>/*" is simultaneously too loose (it accepts a foreign arch-abi)
and too strict (it rejected the real _f0 directory and staged nothing). The
staged layout is then asserted -- exactly one tag directory, and it must be
this build's -- because the loader scans cache/<id>/<tier>/<arch-abi>/<tag>/
literally, and a shard anywhere else is worth exactly as much as no shard,
which no count can distinguish.
* Selection is an ALLOWLIST of {<ext>, .ranges, .resident}, so a new artifact
kind in a producer's cache is excluded by default rather than shipped by
default, and the staged tree is re-scanned afterwards to prove that
.abi_<tag>.ok, *.c and *.pair-lock did not ship. The sweep memo is the one
that matters: it tells a player's first launch that an ABI sweep it never ran
already passed, so a stale shard gets loaded instead of rejected.
* --allow-no-cache is gone from the Linux surface entirely. It is the
relaxation that let a packager staging nothing read as a successful build.
What remains is --ship-without-overlay-cache-because '<reason>', which is
unwieldy on purpose, prints the reason, and is not the documented path in any
recipe. PowerShell's -AllowNoCache is kept (unmigrated title packagers still
pass it) and maps onto that flag, so even the escape hatch is one
implementation.
* The mod catalog is verified against the manifest the BUILD published.
runtime.cmake's _psxrt_stage_mod_catalog already writes
psx_mod_catalog_<target>.txt -- the sorted, deduplicated id list that build
staged -- which is a strictly better authority than re-globbing source trees,
and it handles Tomba 2's Italian catalog overriding framework psx.* packages
at the same ids without a per-variant number.
LINUX GETS AN OVERLAY TOOLCHAIN, AND THE RUNTIME CAN NOW SEE IT
No AppImage packager has ever staged one (grep -c overlay_toolchain = 0 in all
three forks), so on Linux the shipped AOT cache was all a player ever got: the
capture-and-compile fail-safe could not fire, and an overlay outside the cache
was interpreted forever.
Staging one alone would not have worked. runtime/src/main.cpp gated autocompile
on overlay_toolchain/python/python.exe, and built the command from
psxrecomp-game.exe and tcc/tcc.exe -- hardcoded Windows spellings, so on Linux
the gate could never be true no matter what a packager staged. A bundled Linux
toolchain would have been dead weight the runtime never looked at. Those three
names are now per-platform; the layout is identical, only the spellings differ,
and they are kept in lockstep with release_stage.py's TOOLCHAIN_PY_REL /
TOOLCHAIN_RECOMPILER.
The Linux interpreter pin is a python-build-standalone CPython 3.13.1 --
deliberately the same version as the Windows embeddable-zip pin -- fetched and
SHA256-verified on every use by the same code path as the Windows one, so the
platforms cannot drift on retry, cache or verification behaviour either. The
_stripped asset (21 MB download, 78 MB unpacked) is used; compile_overlays.py
needs only the stdlib, verified by import. tinycc publishes no prebuilt Linux
binary, so there is no pinned tcc there and the bundle drives a gcc from PATH;
main.cpp already says so on stdout rather than silently doing nothing. The
SHA256 was measured by downloading the asset and matching the publisher's own
.sha256 sidecar, not copied from a listing.
THE FLAVOR IS NO LONGER GUESSED
PSX_OVERLAY_FLAVOR is the high half of overlay_abi() and the _f<n> field of the
tag. It is a property of the BINARY, decided in runtime.cmake (0 base, 2 for a
PGXP target), and it is NOT platform-dependent -- Windows and Linux builds of
the same target share it, so nobody should ever add a platform branch for it.
Every packager simply assumed 0, and that assumption is invisible when wrong: a
PGXP runtime reads ..._f2/ while the packager stages ..._f0/, the package ships
a cache the binary ignores completely, and nothing fails.
runtime.cmake now publishes psxrecomp_overlay_flavor-<target>.txt next to the
exe-name publication it already writes, for the same stated reason: so nothing
downstream re-derives it. release_stage.py cg-tag takes either --flavor or
--flavor-from-build, and there is no default -- a packager either states the
value or reads what the build published.
TESTS
runtime/tests/test_packagers_never_format_cache_tag.py asserts the rule the two
fixes above were supposed to establish: the tag format appears only in
tools/compile_overlays.py and runtime/src/overlay_loader.c (measured: 5
occurrences, all inside those two files -- note overlay_loader.c has four, at
1745, 1773, 1924 and 1930, the last two being the actual read path) and nowhere
else, and a release packager that touches the overlay cache must reference the
shared staging surface. It carries a positive control -- each owner must
actually contain the pattern -- because the investigation of this very bug once
reported "no cache present" from a probe that could not have found a cache at
all. It is also the SHARED check a title runs against its own tree
(--root <repo>), so the rule about not duplicating logic is not itself
duplicated per repository.
Run against Tomba2Recomp origin/master it reports, pre-fix:
tools\package_appimage.sh:292 formats the overlay cache tag itself
tools\package_appimage.sh is a release packager but does not reference the
framework shared staging surface
runtime/tests/test_release_stage_cache.py pins the behaviour, hermetically, on
synthetic cache trees -- no recompiler, no captures, no build, which is the
point: this is the assertion that could have run in CI on the day the tag grew
a field. 15 cases, including that a tag one field short selects nothing AND
raises rather than warns, that a foreign tag namespace / arch-abi / the removed
sljit tier are not shipped, that the three forbidden artifact kinds never ship,
and that developer state (mods/installed, mods/state.toml) never ships.
Both are registered in runtime/CMakeLists.txt, so the configure-time
test-registration guard covers them and ctest runs them.
NOT DONE, deliberately: tools/package_release.ps1,
tools/package_release_macos.sh and tools/package_setup_host.sh each carry their
own hand-written mod-catalog staging. That is the same duplication defect one
layer over and they should route through stage-mods too, but it is a separate
change needing separate proof (macOS cannot be built or verified here), so the
new test scopes its packager assertion to packagers that touch the overlay
cache rather than pretending to cover them.
Bead: beads-eio.3.102
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Matthew Stanley
|
57648745a3 |
mods: the framework stages the title's catalog too (5 titles could not package)
|
||
|
Matthew Stanley
|
80ff96780a |
recompiler: the overlay guard word is a delay-slot source, never a block leader
The overlay capture format DELIBERATELY appends one coherent guard instruction
past the end of a dirty-page run (overlay_capture.c write_json_window, `size
+= 4u`, pinned by runtime/tests/test_interpreter_perf_guards.py:47): a MIPS
branch sitting at the run's final word (...FFC) always executes its delay slot
in the next page (...000), and the shard must be able to emit and hash that
word. So a capture whose size is a page multiple PLUS 4 is a CORRECT capture --
N bytes of overlay plus one guard word. Measured: 36 of 36 Ape Escape
(SCUS-94423) vault records and 846 of 946 Tomba 2 (SCUS-94454) records are that
shape.
The recompiler was never taught the asymmetry: the guard word is a legal
DELAY-SLOT SOURCE but an ILLEGAL BLOCK LEADER, because the word after it does
not exist in the image. It had exactly one bound -- end_address() -- used both
to read words and to decide what is code.
MEASURED on Ape Escape, its own single-source vault at framework
v0.3.2-alpha-139-g1bf70960, 36 variants:
PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0
! [recompiler] overlay 0x80136000 crc 4BAB23C9: what(): cannot emit control
flow at 0x80138000: mandatory delay slot 0x80138004 is outside the input
image
+ 16 x [fragment] interior 0x801360.. @region 0x00136000, identical message
One root cause, seventeen shards. The whole region then falls back to the
dirty-RAM interpreter forever, and nothing at runtime says so.
Causal chain, every step read in source:
1. overlay_capture.c:349-361 appends the guard word; region 0x80136000 gets
size 0x2004, so the image is [0x80136000, 0x80138004).
2. compile_overlays.py make_psxexe wraps those bytes verbatim with text size
= len(data), so PS1Executable::end_address() = 0x80138004.
3. function_analysis.cpp in_exe / in_function admit 0x80138000 as ordinary
code, and the walk reaches it by fall-through from 0x80137FFC
(0x24022000, addiu -- not control flow), so func.end_addr becomes the
image end.
4. control_flow.cpp makes the final block end_addr = func.end_addr - 4 =
0x80138000: the block LEADER is the guard word.
5. control_flow.cpp's "Also check last instruction itself" promotion finds
0x80137FFC is not control flow and therefore promotes read_word(end_addr)
-- the guard word 0x1062005A, a beq -- to block.exit_instr, with
has_delay_slot = true and address == end_addr. THIS IS THE DEFECT.
6. code_generator.cpp computes delay_slot_addr = 0x80138004, read_word
returns nullopt, and it throws.
THE THROW AT STEP 6 IS CORRECT AND IS PRESERVED. Emitting a control transfer
without its architectural delay slot silently corrupts guest state, which is
what the code did before that throw was added. The gap is step 5.
Fix: give the image view two distinct bounds.
end_address() READ bound -- every byte the image supplies, so a
mandatory delay slot living at a guard word resolves.
analysis_end_address() ANALYSIS bound -- one past the last byte that may be
discovered, made a block leader, or made a block's
exit instruction.
read_word() keeps the READ bound. Discovery and block construction move to the
ANALYSIS bound: function_analysis.cpp in_exe / in_exe_p3 / add_forced_entry /
every hard_cap that used to be the image end / the jr-table producer_hi
defaults / the static jr-ra and JAL sweeps; control_flow.cpp gains
analysis_walk_hi(func) and uses it for cfg.function_end, alias entries,
add_boundary, the scan loop, the final block's end_addr and in_func;
code_generator.cpp's mid-function split pre-pass; main_psx.cpp seed admission
and the data-scan pointer promotion; analysis_db.cpp, main_toml.cpp,
main_cli.cpp target/seed admission. The clamp is applied only when it leaves a
non-empty walk, so a degenerate function starting at or past the bound keeps
failing closed rather than silently producing an empty CFG.
HOW THE BOUND IS DERIVED -- the producer STATES it; the recompiler is told.
The alternative was for the recompiler to infer the count from `size % 4096 ==
4`. That fits every capture the current writer has ever produced, but it is a
consumer guessing at a producer's intent, and this tree already has the
counter-example: 52 of the 946 SCUS-94454 records have residues from 112 to
3996 bytes (one not even 4-byte aligned) because they come from the STATIC
extractor, which records an exact file extent and appends no guard word. A rule
keyed to the page-run shape is a rule that shifts the moment a producer
changes.
So the count travels with the image, in a magic-tagged field in the otherwise
zero-filled tail of the 2048-byte PS-EXE header (exe_tag in ps1_exe_parser.h:
"PSXRGRD1" at 0x7E0, uint32 count at 0x7E8). Putting it in the header rather
than on the command line means every consumer gets it automatically and no
future invocation path can forget to pass it -- compile_overlays has three
make_psxexe sites and five recompiler spawn sites. An absent or malformed tag
means ZERO guard bytes, which is exactly what every genuine PS-X EXE is: the
BIOS and every retail main executable have no guard word, their analysis bound
equals their read bound, and their analysis is bit-for-bit unchanged.
Producer side:
overlay_capture.c writes "guard_bytes": 4|0 per region -- the writer
stating its own intent, additive to the v2 schema and
preserved through coverage_vault.py compaction.
compile_overlays.py capture_guard_bytes(): prefer that field; for captures
recorded before it existed, reconstruct the single
shape write_json_window can produce (page multiple + 4)
and return 0 for everything else, which is the
fail-closed answer -- an unrecognised guard word is
analysed as code and the step-6 throw still fires
loudly at generation time.
make_psxexe guard_bytes is keyword-only with NO default, so a new
call site is a TypeError, not a silently mis-analysed
shard. The count is threaded through
compile_fragment_batch / compile_interior_fragment /
generate_interior_fragment_static and into the fragment
fail-memo key (bumped to v3, so a memo recorded under
the old analysis cannot suppress a retry under the
corrected one).
SECOND-ORDER DEFECT FIXED IN THE SAME PASS. generate_ranges_manifest extends a
range by +4 for any block whose exit instruction IS its last word, so the
cloned delay slot joins the candidate CRC and the page-generation watch. That
extension was unclamped. Reachable TODAY, independent of the guard word: a
branch-likely opcode (0x14-0x17, RESERVED on the R3000A) as the image's very
last word is short-circuited by translate_basic_block into an inline RI raise
BEFORE any delay slot is read, so generation succeeds and the manifest claims 4
bytes past the image end. Measured on origin/master with a 28-byte synthetic
image: `R 80010000 20` for an image ending at 0x8001001C. The cache would then
validate the shard against bytes the shard never saw. Now clamped to the READ
bound -- not the analysis bound, because a guard word genuinely IS compiled in
when it serves as a delay slot and must participate in the CRC.
DEAD CODE, documented rather than trusted: basic_block.cpp's BasicBlockAnalyzer
bounds its walk correctly but nothing in the tree constructs it. It is still
compiled and still hashed into the codegen tag, so a change there invalidates
caches without moving an emitted byte. Its bounds move to the analysis bound
too -- wiring it up must not reintroduce this defect -- and a header comment now
says where the live walker actually is.
TESTS. Both registered, so ctest runs them and the configure-time
registration guard is satisfied.
recompiler/tests/test_overlay_guard_word_analysis_bound.py drives
psxrecomp-game over five synthetic images:
A tagged, 2 pages + guard word, last real word not control flow, guard word
IS a beq -- the measured Ape shape. Must generate; the guard word must not
be emitted as an instruction; the block must publish 0x80012000 for the
runtime to dispatch; no range may claim it.
B byte-for-byte the same payload with NO tag. Must still fail closed with
the mandatory-delay-slot diagnostic naming 0x80012000. A and B differ only
in the header tag, which proves the tag is what changed behaviour and that
the throw was not weakened.
B2 a misaligned declaration (6). Must be rejected and behave like B.
C tagged, with a real branch at the last analysable word. The guard word
must be emitted AS the delay slot and must be covered by the manifest.
D branch-likely as the final word. Generation must succeed and no range may
exceed the image end.
Against origin/master's psxrecomp-game the same file reports exactly the two
things this commit changes, and passes B, B2 and C:
FAIL: tagged guard word still fails generation: what(): cannot emit control
flow at 0x80012000: mandatory delay slot 0x80012004 is outside the
input image
FAIL: range [0x80010000,0x80010020) claims 4 byte(s) past the image end
0x8001001C
tools/tests/test_overlay_guard_bytes_declaration.py pins the producer half:
field precedence, the reconstruction table, rejection of nonsense
declarations, the tag's byte offsets, that make_psxexe with guard_bytes=0 is
byte-identical to the pre-fix wrapper, and that overlay_capture.c both appends
the guard word and now declares it.
MEASURED OUTCOME, Ape Escape, same vault, same flags, fresh cache:
before PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0
after PSX_SHARD_RESULT ok=31 failed=0 skipped=57 capacity_fastpath=0
ok falls because 19 isolated island fragment shards at 0x00136000 are no longer
needed once the region variant compiles as one shard. Coverage is what matters,
and it is measured from the DLL exports: 349 -> 361 unique func_XXXXXXXX
entries, ZERO lost, 12 gained -- 0x80136140, 0x8013614C, 0x80136154,
0x80136238, 0x801362D4, 0x801362DC, 0x801362F0, 0x80136324, 0x80136370,
0x80136400, 0x80136480, 0x80137950, all from the failure list. The other four
failing PCs were already served by a sibling variant's shard.
INERTNESS.
BIOS (SCPH1001, psxrecomp-bios --config bios/SCPH1001.toml): all four
generated artifacts BYTE-IDENTICAL, verified by sha256 rather than by exit
code, because tools/regen_bios.sh exits 0 on failure.
Tomba 2 main EXE (psxrecomp-game --config game.toml): all nine generated
artifacts BYTE-IDENTICAL, and the console log is identical too.
Tomba 2 overlays, 82 variants over 6 regions from its own vault: identical
result line (ok=98 failed=0 skipped=103) and identical shard identity set,
4901 unique func exports on both sides -- zero gained, zero lost. 97 of 98
range manifests are byte-identical. The one that differs is region
0x8010A000 variant A32DB188, size 0x2004, whose last function's range goes
from 0x218 to 0x214 bytes: its guard word (0x24840008, addiu) had been
emitted as an ordinary instruction and hashed, and now is not. That is
precisely the intended change and nothing else moved.
Bead beads-eio.3.100.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Matthew Stanley
|
aa0a285c86 |
tools: compute the affinity mask by shifting, not by Pow on a double
[math]::Pow(2, $n) - 1 builds the processor-affinity mask in DOUBLE precision, and a double carries only a 53-bit mantissa. From n = 54 the subtraction is lost to rounding: 2^n - 1 rounds straight back up to 2^n, so the mask comes out with exactly ONE bit set. Measured across the range, old math vs shifting: n = 50..53 both select n CPUs ok n = 54..62 old selects 1 CPU of n SILENTLY WRONG n = 63..64 old throws converting to Int64 visible failure The silent band is by far the worse half, and it is worst precisely in this tool. On a 54-to-62-processor host, -Off would pin the title to a single logical CPU while printing "affinity restored to all N processors", and -Cores N would pin it to one CPU while reporting N. A throttling tool that reports the opposite of what it did does not just fail, it invalidates every measurement taken through it -- which is the exact failure this tool exists to prevent. Get-AffinityMask shifts a uint64 instead, and special-cases a full 64 because .NET masks a 64-bit shift count to 63 (shifting by 64 quietly yields 1, not 0). Verified equivalent to the old expression for every n where the old expression was correct (1..53), correct by popcount for 1..64, and saturating above 64. ProcessorAffinity addresses a single processor group, so a host with more than 64 processors is inherently limited to the group the process already runs in; the mask now saturates there rather than throwing. No behaviour change on any host where the tool previously worked -- this box has 16 processors, where old and new masks are byte-identical (0xFFFF). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Matthew Stanley
|
b7ca2f99ac |
tools: throttle a live title's CPU to reproduce a weaker host
"Runs fine here, players report slowdown there" is not reproducible on a fast dev box, and guessing at the budget has cost us whole investigations. This throttles a RUNNING title until the reported symptom appears, so the CPU budget is read off a dial instead of argued about. Uses the Windows Job Object CPU rate control -- the same hard cap the scheduler applies to containers -- so it needs no launcher or runtime support and attaches to a live process. That matters: an observability tool that requires relaunching the session destroys the state you were trying to observe, and the reported hitch often takes minutes of play to reach. Two independent knobs, both reversible while the game runs: -CoresWorth <n> hard CPU-rate cap as cores' worth of compute (slower clock) -Cores <n> processor affinity: how many logical CPUs are available -Measure sample the current draw, change nothing -Off release the cap and restore affinity The job is NAMED, so the cap can be re-dialed from a fresh shell without re-assigning the process, and the script can exit while the cap stays on. Job reuse is probed with CreateJobObject + ERROR_ALREADY_EXISTS rather than OpenJobObject, because nesting a second job would be silently unrecoverable: nested CPU caps take the MINIMUM, so a loosened dial would appear to do nothing. Releasing a cap sets it to the full machine (CpuRate 10000) rather than clearing the control block, because the API validates the union against the flags and rejects an all-zero block with ERROR_INVALID_PARAMETER. Read it as a reproduction tool, not a profiler: the hard cap is enforced by withholding scheduling quanta, so a heavily throttled process stutters rather than degrading smoothly. Good for "does the reported hitch appear", wrong for "what is the real frame time on that CPU". The header says so too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Alex Vanderveen
|
4cc04be3e6 | rebump | ||
|
Kevin
|
7ab698caa5 |
overlays: parallel static compile, linear resume-wrapper pass, split output
Static mode (--static) was serial end to end and produced one translation unit, so a whole title compiled on one core twice: once in Python, once in gcc. Measured on BoF3 (339 captures, 79,688 identities) in a build-dbg loop: phase "compile all overlay bands" ~12 min, runtime build ~4 min at -O0. Three changes, each independently verified byte-identical: 1. static_capture_job(): the per-capture static work (recompiler subprocess, C audit, range audit, namespacing) is now a module-level, picklable job that returns its part + captured log. --jobs runs it in a ProcessPoolExecutor; results merge in capture order, so static_parts and entry sources are order-identical to the sequential loop. --jobs 1 and --jobs 3 on three bands: diff -r of the outputs is empty. 2. add_cps_resume_cases(): the universal CPS resume-wrapper pass called add_cps_resume_case() once per entry, and each call re-scanned and rebuilt the WHOLE part source. cProfile on three bands: 7,386 entries -> 22,159 re.search calls, 366 s of 424 s total. The batch form locates every signature once, edits each host's segment in isolation, rebuilds once. Same three bands: 418 s -> 5 s, outputs byte-identical (diff -r against the per-entry output). The per-entry function is kept; a unit test asserts the two agree on fresh-switch, existing-switch, missing label and missing host cases. 3. write_static_outputs(): overlays_static.c now holds the dispatcher plus an extern prototype per compiled entry, and every part becomes its own unit overlays_static_NNNN.c (each already carries its #includes and a private namespace). runtime.cmake globs the siblings with CONFIGURE_DEPENDS, so a changed part count needs no reconfigure. Unchanged units are not rewritten (mtime preserved -> not recompiled); stale units are removed. --static-single-file restores the monolithic file. Result, same BoF3 loop on a 12-core box: phase 5a 12 min -> 20 s, runtime build 4 min -> 69 s (358 units in parallel), whole loop ~16 min -> 90 s. Shard summary unchanged (332 ok / 7 benign audit fails), identity count unchanged. Headless boot of the rebuilt runtime: 485,650 static checks, 485,244 hits (99.92%), miss_total 0, 60 emu fps through the LOGO overlay. Note: tools/test_compile_overlays_additive.py already fails on master (it references removed functions: compile_and_publish_dll, load_additive_captures, validated_cached_bundle_ids); untouched here. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Matthew Stanley
|
900591543b |
tools: one shared overlay release staging for every title
Each title carried its own hand-copied release packager and they drifted three
ways (measured 2026-09-01): Tomba 2 538 lines, MegaManX6 375, Ape Escape 192.
overlay cache staging Tomba2 yes MMX6 yes Ape NO
overlay_toolchain staging Tomba2 yes MMX6 yes Ape NO
SHA256-pinned downloads Tomba2 yes MMX6 NO Ape n/a
mod catalog guard three different magic numbers: -ne 5, -lt 16, -ne 4
Ape's packager was created by copying MegaManX6's and trimming it; git log -S
shows it has NEVER contained overlay_toolchain or AllowNoCache in any commit.
So every Ape release ever made shipped with no shard cache and no toolchain and
ran 100% of its overlay dispatches on the dirty-RAM interpreter, forever, for
every player (measured: disp_native=0, disp_interp=4,480,307 in one session).
The copies drifted because improvements never flow back between forks AND
because nothing FAILS when a title lacks the feature -- the runtime silently
falls back to interpretation, so a stripped packager looks perfectly healthy.
release_overlay_stage.ps1 is the single implementation, based on Tomba 2 (the
most complete) plus the good patterns from MegaManX6:
Get-PinnedArchive verify SHA256 on EVERY use including cache hits, retry
Get-OverlayCgTag derive from the STAGED game.toml
Add-OverlayCache tag-filtered, throws unless -AllowNoCache, and asserts
the staged LAYOUT (MMX6): a shard the loader cannot
find is worth exactly as much as no shard at all
Add-OverlayToolchain pinned python + tcc + recompiler + headers, asserts
python/python.exe landed since that exact path is the
runtime's autocompile gate
Add-ModCatalog derived staged-vs-source invariant replacing all three
magic numbers -- everything the sources define must
survive into the package, which cannot go stale when a
mod is added but still catches a mod silently not
shipping
Pins live in the module so a title cannot ship unpinned by forgetting a hash.
compile_overlays.py: expose cache_tag() as the single source of truth for the
cache-namespace string, and use it for the writer's own cache dir.
The packagers rebuilt that string themselves in PowerShell. That parallel
implementation went stale when the flavor suffix was added: the packager derived
cg<N>_<hash>_gc<hash> while this tool writes cg<N>_<hash>_gc<hash>_f<flavor>, so
the tag filter matched nothing and a correctly regenerated cache staged ZERO
shards. Measured on Tomba 2. Now the writer, the packagers and
overlay_loader.c scan_cache_dir all agree on one definition.
Validated on Tomba 2: packages exit 0 with no relaxations, artifact verified by
extraction (5882 entries, 2799 cache DLLs under cg10_33a7813e_gcff2b6643_f0,
246 toolchain entries, 7 mod manifests), and a 10-minute attract soak on a
debug-tools build reported 99.841% native dispatch (125,294,633 native vs
199,533 interp) across 52 overlay regions with frame period locked at 59.94.
Faithfulness was proven before migrating: Tomba 2's inline cache filter and the
hoisted one selected 1264 identical files with byte-identical copies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Matt Stanley
|
1bf7096032 |
Merge pull request #289 from kerokline/fix/static-overlay-residency-signal
EMI to overlay extract, cached/mapped lookup table, "resident-occupant" memo |
||
|
Alex Vanderveen
|
209970cfe5 |
self-build: loop breaker, install-folder preflight, boot-EXE name lint
Follow-ups to the marker-skew hardening, closing the remaining legs of
the generate+build loop class:
- Loop breaker: host_prepare_generate stamps a successful generate into
.psxrecomp_last_generate at the project root. When apply() is about to
reopen the wizard because sources are missing and the stamp is recent,
the wizard's prepare note becomes a diagnostic naming exactly which
gate is unmet (the expected *_dispatch.c and/or the BIOS backends) and
what generated/ actually contains, instead of silently asking for
another round. The stamp is dropped once sources are visible so a much
later manual wipe of generated/ reads as a fresh first run.
- Preflight: generate now refuses up front when the project root sits
under the Windows temp directory (Explorer's run-from-inside-the-ZIP
extraction, which evaporates) or is not writable, with instructions,
instead of failing downstream in ways that present as the same loop.
- CI lint: tools/ci/check_boot_exe.sh cross-checks the three baked
copies of the boot-EXE name (game.toml boot_exe/exe, CMakeLists
GEN_MARKER, codegen_setup.c gen_marker_relpath) and the setup-release
template gains a fail-fast "Lint boot-EXE name consistency" step, so a
skew fails the maintainer's release instead of the player's install.
op_emit_ci_workflow's step-name drift detection flags older titles
automatically. Script is bash-3.2-clean for macOS runners; verified
against fixture repos (consistent / skewed / boot_exe override /
single-source) and real ports - where it immediately caught
Tomba2Recomp's placeholder gen_marker_relpath ("SLUS_01234" vs
SCUS_944.54), a live instance of the reported loop.
Both gitignore templates learn the new sidecar names.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KfPHUBEYiSCwaGiwRc8ea7
|
||
|
Kevin
|
ecc0de16a9 | Merge remote-tracking branch 'upstream/master' into fix/static-overlay-residency-signal | ||
|
Kevin
|
7015317566 |
tools: remember the resident occupant per dispatch address
On hardware the game CD-reads a file to a fixed address and jal's straight into it: identity is implicit in control flow and costs nothing. We rediscover it by walking a band's occupants and CRC-gating each in turn. On a deep band that walk dominates -- the memory-card screen measured 41.25 checks per hit, roughly 41 failed gates before the resident one. Remember which occupant last satisfied each address and try it first. The memo is a hint, never an authority: every call still passes through psx_overlay_static_code_matches(), so a band that swapped occupants fails the memo and falls into the full walk, which then re-seeds it. Correctness is therefore unchanged -- the memo only reorders candidates. Measured headless over an identical 140 s boot window, all ten bands compiled: build throughput chk/hit variant misses all-bands 99.0 1.479 368,479 all-bands + memo 131.4 1.069 81,505 +33% throughput, 4.5x fewer wasted gate calls. It is neutral where chains are already shallow (a savestate-loaded combat workload measured 1.019 chk/hit before the memo and showed no gain), so it pays exactly where it was aimed. Together with the O(1) lookup this overturns the earlier "do not compile all bands" result: all ten bands now beat a three-band build on both workloads measured -- 131.4 vs 107.9 at boot, and parity on the savestate workload -- because more compiled code means more native execution once dispatch is cheap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Kevin
|
69d783f51f |
tools: O(1) overlay dispatch instead of a sparse switch
psx_overlay_dispatch is consulted on every interpreted entry, and the overwhelming majority of those calls are for addresses that are compiled nowhere. Measured on Breath of Fire III during a battle transition: ~264,000 calls/sec, of which roughly 1,160 out of every 1,161 found nothing. Emitting one `case` per address makes that miss path a binary search tree over tens of thousands of sparse cases, plus a jump table far larger than cache. So the cost scaled with how much overlay code was compiled even on screens where none of it ran, and frame rate tracked dispatch-case COUNT rather than variant chain depth -- which is why adding bands cost frame rate for nothing. Replace the switch with a compile-time open-addressed hash table. The miss path becomes hash, one load, compare, return. Two parallel uint32 arrays keep a miss inside a single table: the address array answers "is this mine?" without touching the entry or variant arrays. Load factor is held at <= 0.5. Behaviour is unchanged: on identical captures the emitted address set and variant count match the switch exactly (11,913 addresses / 12,522 variants for a three-band build; 10,545 / 10,545 for two-band), and per-address variant order is preserved, so the CRC gate still chooses the resident occupant. Verified offline: the C and Python hash functions agree on all 524,288 word-aligned addresses in the 2 MB RAM window; every table entry resolves to its own index; variant runs tile the variant array exactly; and no non-entry address produces a false hit. Measured headless, three bands, same savestate and same 200 s protocol: throughput 106.5 -> 113.2 emulated fps (+6.3%), p1 93.9 -> 105.2. At the transition the switch build absorbed 160,970 misses/sec at 63.9 fps while the table build absorbed 307,978/sec at 131.3 fps. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Matt Stanley
|
0cbd435349 |
Merge pull request #271 from Alexbeav/pr/stale-static-guards
codegen+runtime: stop executing stale static code when a game overlays its own text |
||
|
Matt Stanley
|
90b02420b2 |
Merge pull request #274 from Alexbeav/pr/psx-exe-boot-fallback
tools: support SYSTEM.CNF-less discs via the BIOS PSX.EXE boot path |
||
|
Matt Stanley
|
cab6143ac0 |
Merge pull request #276 from Alexbeav/pr/toolchain-unpack-modes
tools: restore Unix modes when unpacking the toolchain zip |
||
|
Matt Stanley
|
ff012589c7 |
Merge pull request #275 from Alexbeav/pr/python39-tools-compat
tools: keep prepare_disc working on Python 3.9 |
||
|
Matt Stanley
|
68a2b3819a |
Merge pull request #270 from Alexbeav/pr/cosim-icount-checkpoints
cosim: key checkpoints on retired-instruction count, not guest cycles |
||
|
Alexandros Mandravillis
|
d795fef42a |
tools: restore Unix modes when unpacking the toolchain zip
zipfile.extractall() ignores the mode stored in ZipInfo.external_attr, so on POSIX hosts every file in cmake-clang-v1 is extracted 0644. The pack is then unusable and ensure-toolchain fails with: Toolchain cmake probe failed (.../cmake-clang-v1/1.0.14/bin/cmake): [Errno 13] Permission denied Restore the recorded mode after extraction. Zips built on Windows record no Unix mode, in which case the extracted default already applies, and the whole block is skipped on nt. Verified against a zip recording 0755: extractall alone yields 0644 and os.access(X_OK) False; with this change 0755 and True. |
||
|
Alexandros Mandravillis
|
da9094c2a3 |
tools: keep prepare_disc importable on Python 3.9
Path.write_text(newline=) is Python 3.10+. RHEL/Rocky 9, Debian 11 and Ubuntu 20.04 all ship 3.9 as the system python3, where staging died with 'TypeError: write_text() got an unexpected keyword argument newline' after writing the .bin but before the .cue. open(..., newline=) is behaviour-identical and available everywhere. No other tool in the repo needs 3.10: the rest already rely on 'from __future__ import annotations'. Verified by running the real prepare_disc under 3.9 end to end (digest check, PSX.EXE fallback, staging, RESULT_CUE). |
||
|
Alexandros Mandravillis
|
8268553aab |
tools: prepare_disc falls back to PSX.EXE boot when SYSTEM.CNF is absent
Mirrors the probe_disc.py fallback from 5ab7a053 into the staging step. probe_disc.py accepts SYSTEM.CNF-less discs (King's Field, Dec 1994), but prepare_disc.py still required the file, so 'generate' failed on any clean worktree where disc/ was not already staged. Windows never hit this because the staged boot EXE made psxrecomp_cli skip prepare_disc entirely. Tools-only: runtime and recompiler sources are identical to 54fe30bf. |
||
|
Alexandros Mandravillis
|
cabe566d01 | tools: probe falls back to PSX.EXE boot when SYSTEM.CNF is absent | ||
|
Alexandros Mandravillis
|
4d9060a54e |
codegen: stale-static guard on inter-piece host transfers
A game that overlays its own text at runtime (CMR2 streams transform-
loop variants with patched projection constants over the boot EXE)
kept executing the stale static translation through inter-piece host
transfers: dispatch validates an entry's emitted ranges, but split
function pieces chain via direct host calls (fallthrough to split
piece / next function, legacy non-CPS split branch/jump/continuation
transfers, non-CPS direct jal) with no revalidation. A clean entry
piece falls through natively into a stale sibling whose live bytes an
overlay replaced -- the CMR2 attract/gameplay wedge corruption
(first divergence: one ADDI immediate, screen-center 160 static vs
256 live, at 0x800199D4, retirement 3,203,596,289, frame ~11963).
Emit at every such transfer:
if (!psx_game_text_native_ok(T)) { cpu->pc = T; return; }
publishing the PC and unwinding to the trampoline, whose dispatch
takes the sanctioned dirty-RAM-interpreter fallback over live bytes.
The non-CPS direct jal falls back to call_by_address instead.
dirty_ram_text_native_ok_ranges_from gains a page-clean fast path
(skip the memcmp when no touched page is guard-modified or runtime-
dirty) so the guards are near-free on pristine pages.
cosim.py: print full A/B cpu dumps in cpudiff mode (values, not just
diffs).
|
||
|
Alexandros Mandravillis
|
4aebb349da |
cosim: key checkpoints on retired-instruction count, not guest cycles
The cycle-keyed design hashed+parked at the first cosim_instr boundary AFTER the stride cycle was crossed; the two backends can reach that crossing at different retirement positions, so the oracle compared different architectural states (observed as a +5-cycle park skew and phantom CPU-register divergences mid-CPS). Checkpoints are now taken inside cosim_instr, keyed on the retirement count itself (a branch and its delay slot retire as one event in both backends). Checkpoint k = state after k*stride retirements in BOTH instances by construction. The guest-cycle clock stays inside the state hash, so real cycle-accounting drift is still detected -- now pinned to an exact instruction window instead of corrupting the alignment. cosim.py: stride/--max/--start-cycle now in retirement units; the cycle skew warning is now reported as a real timing divergence. launch() gives instance B its own exe (the mods root is exe-dir-relative, so a shared exe races on mods/state.toml publish regardless of cwd separation) and refuses a stale COSIM_CWD_B exe. Gates on CMR2 (SLUS-01222): compiled/compiled and interp/interp both clean over 100M retirements with identical cross-pair chains; injected 1-bit RAM fault detected at the next checkpoint. |
||
|
Alex Vanderveen
|
72b17fd095 |
ci: carry the mod-catalog gate in the template, and detect stale workflows
Two gaps, one cause: a title installs the CI workflow once and then never
learns the template moved on.
- docs/ci/templates/setup-release.yml gains the release gate that asserts the
zip actually ships a mod catalog, carries no per-machine state.toml, and
carries no developer-channel package. It existed only in one title's
workflow, so every project generated before now would publish an empty Mods
page with a green build -- the exact failure the gate was written for.
- op_emit_ci_workflow stopped at "release.yml already filled" and reported OK.
Filled is not the same as current. It now compares STEP NAMES against the
template and reports which are missing, so Studio's migration tab surfaces
the drift instead of silently blessing it.
Step names rather than a text diff, deliberately: a filled workflow always
differs from the template (tokens, and titles customise theirs), so a textual
comparison would report stale on every project forever and be ignored within a
week. A step the template defines and the installed file does not is
unambiguous. Names containing a token are skipped -- the installed copy has
them substituted, so they are evidence of nothing.
It reports rather than rewrites: re-emitting overwrites a customised workflow
wholesale, so that stays behind --force.
Checked for a version-stamp gap while here and found none: both the template
and the installed workflow already configure with
-DPSX_GAME_VERSION="${RELEASE_VERSION}" and verify the lobby pin against it,
and the CI cache covers only build-recompiler, never the runtime. The
mismatch that prompted the look was a stale LOCAL build directory, not CI.
Verified both directions: the current workflow reports "already filled and
current"; the same workflow with the gate stripped reports the missing step
by name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0194TYJAqgyG9a9PJe9uUcgQ
|
||
|
Alex Vanderveen
|
86e9c40b34 |
package: stage assets/ so a rebuilt zip can configure and keeps its icon
The CMakeLists gate added in
|
||
|
Alex Vanderveen
|
29cba084fc |
package: keep src/ staging conditional after the merge
Both merged branches added src/ to the packaging template: feat/setup-host- multi-disc as an unconditional --project-dir in the static argument list, and fix/package-src-mods as a guarded EXTRA_PROJECT entry. git merged them cleanly because they touch different lines, leaving src/ passed twice. The duplicate copy is harmless, but the unconditional entry is not: copy_proj() hard-fails on a missing path, so a scaffolded project without src/ would stop packaging with "error: missing src". Every single-disc title that has never added game-owned C is in that position. Drops the static entry and keeps the guarded one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1TpXRJve6WvwyZbSJAbCL |
||
|
Alex Vanderveen
|
081bb79084 |
Merge fix/package-src-mods (#267) into feat/multi-disc-launcher
Brings the packaging gates and Project Studio audit for allowlist drift:
- template stages src/, mods/ and the generated symbol headers
- package_setup_host.sh refuses to zip a stage whose CMakeLists.txt names
an unstaged path, or whose project sources include one
- Studio Migrate gains packager_stages_sources / packager_stages_optional
and op_sync_packager_project_dirs
Both branches touch tools/package_setup_host.sh and the packaging template,
so this is merged here rather than left to race on master.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1TpXRJve6WvwyZbSJAbCL
|
||
|
Alex Vanderveen
|
61d23ba777 |
package: stage the symbol headers, and gate on what sources include
FF7 v0.1.3 shipped src/ff7_mods.c (the previous commit's fix) and then died at the first object file: src/ff7_mods.c:20:10: fatal error: '../psx_symbols.h' file not found psx_symbols.h is a tracked 82 KB header that game-owned C in src/ includes by relative path. Nothing in CMakeLists.txt names it, so the CMakeLists gate added in the previous commit could not see it — it is a compile input of the shipped tree that only the source knows about. Two changes: 1. The template stages psx_symbols.h and psx_symbols_overlays.h when present. 2. package_setup_host.sh gains a second pre-zip gate: every relative #include in the project's own staged sources must resolve inside the stage. Only project-owned paths (PROJECT_FILES / PROJECT_DIRS) are scanned — framework trees resolve their headers through -I and are not ours to police. The include check accepts three resolutions: relative to the including file, relative to the stage root, and finally any file of that basename anywhere in the stage. That last one covers -I resolution out of the staged framework trees (a bare "mod_plugins.h" comes from psxrecomp/runtime/include). It is deliberately loose: a false pass only forfeits a warning, while a false failure would block a good release. Verified against the real v0.1.3 tree: the stage as shipped fails naming exactly ../psx_symbols.h, and passes once the header is staged. mod_plugins.h was a false positive in a minimal stage and is correctly cleared once the framework tree is present. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1TpXRJve6WvwyZbSJAbCL |