Files
psxrecomp/docs/RELOCATION_MANIFEST_FORMAT.md
T
Matthew Stanley e6ce5006b4 docs: move root and stale docs into docs/ and docs/internal/
The repository root carried seven markdown files that were not onboarding
material, plus two that had been superseded by the code they described. This
sorts them: reader-facing docs to docs/, working notes to docs/internal/, and
the two superseded ones merged into the docs that replaced them.

Moved to docs/:            WIDESCREEN.md, ENHANCEMENTS.md, TCP_COMMANDS.md
Moved to docs/internal/:   PRINCIPLES.md, ISSUES.md, DEBUG.md, GTE_LM_FIX.md,
                           and from docs/: bios-hardening.md, overlay-status.md,
                           overlay-plan.md, overlay-recompilation-design.md,
                           overlay-discovery.md, TOMBA2_EXC_CONSULT_RESPONSE.md,
                           CASE_A_AOT_GAP.md, audit_inventory.md
Moved to docs/internal/upstream/: UPSTREAM_2026-08-20.md

The root keeps only what belongs there: README.md, CONTRIBUTING.md, LICENSE,
CLAUDE.md, THIRD_PARTY_ATTRIBUTION.md, RELEASE_NOTES.md, the dotfiles and
psxrecomp_cli.py.

MERGES

SPLITGEN_MIGRATION.md folded into docs/BUILDING.md as "Split generated C". Its
migration instructions were stale in a way that mattered: it told every game
repo to hand-edit its CMakeLists and add a file(GLOB ...) over
<serial>_full*.c, but runtime/runtime.cmake now does that itself. GEN_FULL_GLOB
is a multi-value argument of psxrecomp_add_game_runtime(); the loop at
runtime.cmake:1985 makes each pattern absolute against CMAKE_CURRENT_SOURCE_DIR,
expands it with file(GLOB), and concatenates the hits into
GAME_GENERATED_FULL_C. So a title names the pattern and gets however many shards
a regeneration produced. The rewritten section describes that, the two-step
fallback when the globs match nothing (GEN_FULL_FALLBACK, else <serial>_full.c
derived from the dispatch marker by substituting _dispatch.c), and the one
caveat the code actually has: that file(GLOB) is plain, not CONFIGURE_DEPENDS,
so a regeneration which changes the shard COUNT needs a re-configure to be seen.

psxrecomp-toml-readme.md folded into docs/GAME_PROJECT_SETUP.md as "EXE-only
autofill (psxrecomp-toml)", placed after "Disc autofill (probe_disc.py)" and
before "Project Studio" so the two autofill paths sit together. Its documented
flags were checked against the usage text in recompiler/src/main_toml.cpp:28-42
and against the argument parser at :150-165 -- all seven options match with no
drift, so the flag list is reproduced as-is. Added a pointer that probe_disc.py
is preferable when a cue exists, since that tool does the same EXE-header work
plus disc identity and netplay gates.

FUNCTIONAL FIXES

tools/gen_tcp_commands.py wrote to REPO/"TCP_COMMANDS.md". After the move that
path does not exist, so the generator would have created a new stray root file
and --check would have failed against it forever. Retargeted to
REPO/"docs"/"TCP_COMMANDS.md", along with the seven docstring and message
strings that name the path.

Running it surfaced two further problems, both fixed here:

  * The doc was genuinely stale -- pre-existing drift, not introduced by this
    commit (the moved file is byte-identical to origin/master, and no
    debug-server command table is touched here). Four commands registered since
    the last regeneration were missing from the index -- cdrom_timing_dump,
    savestate_status, ws_ui_groups, xprobe_watch -- and scanline had gained a
    backend. Regenerated: 307 -> 311 commands, and the change is confined to the
    AUTOGEN block (the hand-written head and tail are byte-identical to master).
    tools/gen_tcp_commands.py --check now exits 0.
  * DOC.write_text() has no newline argument, so Python translated every newline
    to os.linesep. On Windows that rewrote all 585 lines of the file to CRLF and
    turned a four-line index update into a whole-file diff -- measured: 589 CRLF
    / 0 LF after a run, against 0 CRLF / 585 LF in the repo. Now written through
    an explicit newline="" handle, verified to leave the file at 0 CRLF.

runtime/include/debug_server.h:9 and runtime/src/debug_server.c:8 cited "TCP.md
and DEBUG.md". TCP.md has never existed in this repo under that name; both now
name docs/TCP_COMMANDS.md and docs/internal/DEBUG.md.

LINK AND REFERENCE FIXES

Every inbound reference to a moved file was located by grepping each basename
across the whole tracked tree, not just the markdown. Markdown hrefs in
README.md, CONTRIBUTING.md, docs/ARCHITECTURE.md, docs/COMPILING_OVERLAYS.md,
docs/AOT_OVERLAY_PLAN.md, docs/RELOCATION_MANIFEST_FORMAT.md,
docs/config_schema.md, docs/OVERLAY_XREF.md, docs/TCP_COMMANDS.md and several
docs/internal/ files now point at the new locations; where two files became
siblings under docs/internal/ the path was shortened rather than lengthened
(bios-hardening.md's reference to overlay-discovery.md, for instance).
docs/COMPILING_OVERLAYS.md's "in this folder" phrasing was wrong once two of the
three files it listed moved, so it now names internal/ explicitly.

24 source-comment mentions of ENHANCEMENTS.md sections G1.x across 19 files
(recompiler, runtime, cmake) were retargeted to docs/ENHANCEMENTS.md with the
section numbers untouched. docs/SHADOW_ENHANCEMENTS.md is a DIFFERENT file and
was deliberately excluded.

20 rationale strings citing docs/CASE_A_AOT_GAP.md across the four
recompiler/seeds/*.json files were updated. Done as a byte-level substring
replacement rather than a JSON round-trip: two of those files are stored with
CRLF endings and two with LF, and re-serializing them reformatted 5,700 lines.
The byte-level version changes exactly 20 lines and each file still parses.

References that look similar but point elsewhere were deliberately left alone,
and are worth naming so a future pass does not "fix" them:

  * "recomp-template PRINCIPLES.md" / "NES recomp template PRINCIPLES.md" in
    CLAUDE.md:43, docs/SHADOW_ENHANCEMENTS.md:28,
    docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md:5,
    runtime/src/debug_server.c:4934 and tools/parity_diff.py:8 are a different
    repository's file. Only the five that mean this repo's PRINCIPLES.md were
    updated.
  * Every remaining bare ISSUES.md mention is a GAME repo's tracker
    (MegaManX6Recomp 2/7/8/9, TombaRecomp 6), not this repo's.
  * docs/disc-speed.md:4,54,56 and docs/internal/bios-hardening.md:5 name
    overlay-discovery as a BRANCH, not a file, so no path applies.

INDEXES

README.md's "New here?" table gains rows for docs/WIDESCREEN.md,
docs/ENHANCEMENTS.md and docs/TCP_COMMANDS.md. docs/internal/README.md gains
sections indexing all fourteen documents that moved in, plus an "Upstream
contributor reviews" section describing upstream/ and the new sweep file, plus a
"Removed" bullet for CYCLE_TIMING_ARCH.md (deleted in the previous commit)
pointing at FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md -- following the SLJIT_PERSIST_CACHE.md precedent already
in that file.

docs/ENHANCEMENTS.md:7 claimed framework bugs live in "game repos' ISSUES.md
until a framework tracker exists". A framework tracker does exist; it now says
so and names the Beads epic.

VERIFICATION

A link checker was written to scan every tracked *.md for relative links,
resolving each against the tree being scanned. Baseline at origin/master: 129
files, 137 relative links, 3 broken. After both commits: 127 files, 159 relative
links, 3 broken -- the SAME three, all pre-existing and untouched here:

    docs/DATA_SHARDS.md -> INSTANT_LOADS_PARKED.md   (never existed)
    docs/DATA_SHARDS.md -> INSTANT_LOADS_SPIKE2.md   (never existed)
    docs/NETPLAY.md     -> ../lib/recomp-net/README.md  (inside a submodule)

Zero new broken links. The checker's own coverage was checked against a raw grep
for markdown link openers: the 3-link difference is C++ lambdas inside code
blocks, correctly not treated as links.

Also verified: py_compile over all 113 tracked tools/*.py plus psxrecomp_cli.py;
all four seed JSONs still parse; the test-registration guard still reports 123
test files all registered; recompiler/tests/test_coverage_vault.py still passes;
and no file in this commit had its line endings flipped (checked mechanically
across all 50 modified files, after two files were caught and redone).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-03 07:39:07 -07:00

7.2 KiB

Relocation Manifest Format

STATUS AMENDMENT (2026-07, BIOS profiles). The bulk boot-time ROM-to-RAM copy case this document originally targeted is now carried by the BIOS profile instead: [[recompiler.address_model.copy]] in bios/<STEM>.toml (single source of truth for the recompiler, the emitted C, and the runtime via psx_bios_image; see recompiler/src/bios_address_model.h). The kernel-bless mechanism in memory.c remains the runtime enforcement of the "verify live RAM against the ROM source before dispatching AOT code" contract described here. This document stays authoritative ONLY for the oracle-observed case — regions with src_rom_addr: null that cannot be seen statically — which remains unimplemented.

Status: Phase B (design + first iteration), 2026-04-30. File on disk: generated/address_aliases.json. Authority: PLAN.md:941-967 ("BIOS Relocation Is a Hard Gate"), CLAUDE.md:333-365 (Rule 18). Companion files: generated/normalization_rule.md, generated/relocation_proofs/<name>/.


Purpose

Records every region of code that lives at a RAM address at runtime but that the static recompiler cannot see in SCPH1001.BIN byte-for-byte at compile time. Two flavors:

  1. ROM↔RAM aliases. BIOS copies bytes from 0xBFCxxxxx to a kernel-RAM address during init and runs the RAM copy. Same bytes, two addresses.
  2. Trace-discovered RAM regions. BIOS installs code at a RAM address via a mechanism we may not have statically identified (relocated copy, patched copy, runtime-assembled, etc.). We have observed bytes via the oracle.

Both flavors share one record shape so the recompiler can ingest them uniformly. The discovered_by field records how the entry was proven.


File shape

generated/address_aliases.json:

{
  "version": 1,
  "normalization_rule": "addr & 0x1FFFFFFF",
  "entries": [ /* AliasEntry[] */ ]
}

version increments on incompatible schema changes. normalization_rule mirrors generated/normalization_rule.md so the recompiler can check agreement at load time.

AliasEntry

Field Type Required Notes
name string yes Slug, used as proof-directory name and as part of emitted function names. [a-z0-9_]+.
dst_ram_addr hex string yes Physical RAM address (post-normalization). E.g. "0x00005800". Recompiled function dispatch keys are derived from this.
length int yes Byte length of the region. Must be a multiple of 4 (MIPS instruction size).
entrypoints array of hex string yes RAM addresses at which psx_dispatch may enter this region. The recompiler emits a separate compiled function per entrypoint. At least one entry, must include dst_ram_addr if the region's first instruction is callable.
bytes_hash string yes sha256:<64 hex chars> of the region bytes. Runtime verifies against live RAM before dispatching the AOT function.
bytes_file string yes Path (relative to repo root) to the raw bytes file under generated/relocation_proofs/<name>/bytes.bin. Recompiler reads from here when src_rom_addr is null.
src_rom_addr hex string optional Physical ROM address the bytes were copied from, if known. E.g. "0x1FC2D800". When present, recompiler reads from ROM and additionally checks ROM[src..src+length] hash matches bytes_hash.
discovered_by object yes Provenance. See below.

discovered_by

Field Type Required Notes
tool string yes Identifier of the oracle / static tool. E.g. "beetle_psx_oracle", "ghidra_static_copy_loop", "recomp_install_hook_log".
writer_pc hex string | null when known PC of the instruction that wrote the LAST byte of this region during install. null if not observed.
first_exec_pc hex string | null when known First PC inside this region that the oracle executed. Useful for cross-validating entrypoints[0].
reason string yes One-line human explanation of why this region matters and what evidence drove its inclusion.
captured_at ISO-8601 date yes When the proof was captured.
captured_session string optional Free-form session/commit reference.

Proof directory

Per PLAN.md every entry must have a directory generated/relocation_proofs/<name>/ containing:

File Required Purpose
proof.json yes Canonical record of the entry (full AliasEntry plus extra evidence fields).
bytes.bin yes Raw bytes of the region, exactly length bytes.
wtrace_excerpt.json when discovered_by.tool=oracle Slice of the Beetle wtrace ring covering the writer events.
dispatch_excerpt.json optional Slice of recomp dispatch_tail showing the gap (we never dispatched these PCs).
ghidra_*.{json,png} when discovered_by.tool starts with ghidra Static analysis artifact.

address_aliases.json is GENERATED from the proof directories — never hand-edited. A small tool (tools/build_address_aliases.py, Phase C) walks generated/relocation_proofs/, validates each proof.json + bytes.bin, and emits the top-level manifest.


Normalization

Working rule (per PLAN.md Phase 1e draft): normalized = addr & 0x1FFFFFFF.

Handles the four PSX address-space windows:

Region Virtual base Normalized
KUSEG (raw) 0x00000000 self
KSEG0 (cached) 0x80000000 strip top 3 bits
KSEG1 (uncached) 0xA0000000 strip top 3 bits
BIOS via KSEG1 0xBFC00000 0x1FC00000 (distinct from RAM)

So 0x00007568, 0x80007568, 0xA0007568 all normalize to 0x00007568 and refer to the same RAM byte. 0xBFC07568 stays at 0x1FC07568 — a distinct ROM byte. The recompiler stores all dispatch keys in normalized form.


First-iteration scope

Today's first manifest entries cover only the two dark RAM windows identified in docs/internal/CASE_A_AOT_GAP.md:

name dst_ram_addr length entrypoints
memcard_handler_4f00 0x00004F00 0x100 0x4F0C, 0x4F54
memcard_handler_5800 0x00005800 0x800 0x5800, 0x5DB8, 0x5DD0, 0x5DD8, 0x5EF4, 0x5FA8

Both are flagged discovered_by.tool="beetle_psx_oracle", both have non-null writer_pc (from the wtrace) and first_exec_pc. Both have src_rom_addr=null in this iteration — the BIOS source these were copied from is not yet identified statically. When we identify the source (e.g. via tracing the BIOS init copy loops in Beetle and matching them against ROM), we'll set src_rom_addr and drop bytes.bin from the proof directory.

Future iterations expand coverage and add static src identification.


Failure modes the manifest is designed to catch

  • Stale AOT dispatch. If live RAM bytes drift from bytes_hash, runtime aborts loudly rather than running the wrong code.
  • Duplicate logical function. If two entries normalize to the same dst_ram_addr, the recompiler refuses to build.
  • Unproven alias. If an entry exists in address_aliases.json but the matching relocation_proofs/<name>/ directory is missing or has the wrong hash, the recompiler refuses to build.
  • Speculative addition. Every entry must point to a proof. No "I think this region exists" entries.