mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-25 18:05:12 -04:00
Matthew Stanley
e6ce5006b4
The repository root carried seven markdown files that were not onboarding
material, plus two that had been superseded by the code they described. This
sorts them: reader-facing docs to docs/, working notes to docs/internal/, and
the two superseded ones merged into the docs that replaced them.
Moved to docs/: WIDESCREEN.md, ENHANCEMENTS.md, TCP_COMMANDS.md
Moved to docs/internal/: PRINCIPLES.md, ISSUES.md, DEBUG.md, GTE_LM_FIX.md,
and from docs/: bios-hardening.md, overlay-status.md,
overlay-plan.md, overlay-recompilation-design.md,
overlay-discovery.md, TOMBA2_EXC_CONSULT_RESPONSE.md,
CASE_A_AOT_GAP.md, audit_inventory.md
Moved to docs/internal/upstream/: UPSTREAM_2026-08-20.md
The root keeps only what belongs there: README.md, CONTRIBUTING.md, LICENSE,
CLAUDE.md, THIRD_PARTY_ATTRIBUTION.md, RELEASE_NOTES.md, the dotfiles and
psxrecomp_cli.py.
MERGES
SPLITGEN_MIGRATION.md folded into docs/BUILDING.md as "Split generated C". Its
migration instructions were stale in a way that mattered: it told every game
repo to hand-edit its CMakeLists and add a file(GLOB ...) over
<serial>_full*.c, but runtime/runtime.cmake now does that itself. GEN_FULL_GLOB
is a multi-value argument of psxrecomp_add_game_runtime(); the loop at
runtime.cmake:1985 makes each pattern absolute against CMAKE_CURRENT_SOURCE_DIR,
expands it with file(GLOB), and concatenates the hits into
GAME_GENERATED_FULL_C. So a title names the pattern and gets however many shards
a regeneration produced. The rewritten section describes that, the two-step
fallback when the globs match nothing (GEN_FULL_FALLBACK, else <serial>_full.c
derived from the dispatch marker by substituting _dispatch.c), and the one
caveat the code actually has: that file(GLOB) is plain, not CONFIGURE_DEPENDS,
so a regeneration which changes the shard COUNT needs a re-configure to be seen.
psxrecomp-toml-readme.md folded into docs/GAME_PROJECT_SETUP.md as "EXE-only
autofill (psxrecomp-toml)", placed after "Disc autofill (probe_disc.py)" and
before "Project Studio" so the two autofill paths sit together. Its documented
flags were checked against the usage text in recompiler/src/main_toml.cpp:28-42
and against the argument parser at :150-165 -- all seven options match with no
drift, so the flag list is reproduced as-is. Added a pointer that probe_disc.py
is preferable when a cue exists, since that tool does the same EXE-header work
plus disc identity and netplay gates.
FUNCTIONAL FIXES
tools/gen_tcp_commands.py wrote to REPO/"TCP_COMMANDS.md". After the move that
path does not exist, so the generator would have created a new stray root file
and --check would have failed against it forever. Retargeted to
REPO/"docs"/"TCP_COMMANDS.md", along with the seven docstring and message
strings that name the path.
Running it surfaced two further problems, both fixed here:
* The doc was genuinely stale -- pre-existing drift, not introduced by this
commit (the moved file is byte-identical to origin/master, and no
debug-server command table is touched here). Four commands registered since
the last regeneration were missing from the index -- cdrom_timing_dump,
savestate_status, ws_ui_groups, xprobe_watch -- and scanline had gained a
backend. Regenerated: 307 -> 311 commands, and the change is confined to the
AUTOGEN block (the hand-written head and tail are byte-identical to master).
tools/gen_tcp_commands.py --check now exits 0.
* DOC.write_text() has no newline argument, so Python translated every newline
to os.linesep. On Windows that rewrote all 585 lines of the file to CRLF and
turned a four-line index update into a whole-file diff -- measured: 589 CRLF
/ 0 LF after a run, against 0 CRLF / 585 LF in the repo. Now written through
an explicit newline="" handle, verified to leave the file at 0 CRLF.
runtime/include/debug_server.h:9 and runtime/src/debug_server.c:8 cited "TCP.md
and DEBUG.md". TCP.md has never existed in this repo under that name; both now
name docs/TCP_COMMANDS.md and docs/internal/DEBUG.md.
LINK AND REFERENCE FIXES
Every inbound reference to a moved file was located by grepping each basename
across the whole tracked tree, not just the markdown. Markdown hrefs in
README.md, CONTRIBUTING.md, docs/ARCHITECTURE.md, docs/COMPILING_OVERLAYS.md,
docs/AOT_OVERLAY_PLAN.md, docs/RELOCATION_MANIFEST_FORMAT.md,
docs/config_schema.md, docs/OVERLAY_XREF.md, docs/TCP_COMMANDS.md and several
docs/internal/ files now point at the new locations; where two files became
siblings under docs/internal/ the path was shortened rather than lengthened
(bios-hardening.md's reference to overlay-discovery.md, for instance).
docs/COMPILING_OVERLAYS.md's "in this folder" phrasing was wrong once two of the
three files it listed moved, so it now names internal/ explicitly.
24 source-comment mentions of ENHANCEMENTS.md sections G1.x across 19 files
(recompiler, runtime, cmake) were retargeted to docs/ENHANCEMENTS.md with the
section numbers untouched. docs/SHADOW_ENHANCEMENTS.md is a DIFFERENT file and
was deliberately excluded.
20 rationale strings citing docs/CASE_A_AOT_GAP.md across the four
recompiler/seeds/*.json files were updated. Done as a byte-level substring
replacement rather than a JSON round-trip: two of those files are stored with
CRLF endings and two with LF, and re-serializing them reformatted 5,700 lines.
The byte-level version changes exactly 20 lines and each file still parses.
References that look similar but point elsewhere were deliberately left alone,
and are worth naming so a future pass does not "fix" them:
* "recomp-template PRINCIPLES.md" / "NES recomp template PRINCIPLES.md" in
CLAUDE.md:43, docs/SHADOW_ENHANCEMENTS.md:28,
docs/internal/HLE_SCHEDULER_CARVEOUT_PLAN.md:5,
runtime/src/debug_server.c:4934 and tools/parity_diff.py:8 are a different
repository's file. Only the five that mean this repo's PRINCIPLES.md were
updated.
* Every remaining bare ISSUES.md mention is a GAME repo's tracker
(MegaManX6Recomp 2/7/8/9, TombaRecomp 6), not this repo's.
* docs/disc-speed.md:4,54,56 and docs/internal/bios-hardening.md:5 name
overlay-discovery as a BRANCH, not a file, so no path applies.
INDEXES
README.md's "New here?" table gains rows for docs/WIDESCREEN.md,
docs/ENHANCEMENTS.md and docs/TCP_COMMANDS.md. docs/internal/README.md gains
sections indexing all fourteen documents that moved in, plus an "Upstream
contributor reviews" section describing upstream/ and the new sweep file, plus a
"Removed" bullet for CYCLE_TIMING_ARCH.md (deleted in the previous commit)
pointing at FAITHFUL_TIMING_PLAN.md, CYCLE_MODEL_BEETLE.md and
PRECISE_IRQ_SLICE.md -- following the SLJIT_PERSIST_CACHE.md precedent already
in that file.
docs/ENHANCEMENTS.md:7 claimed framework bugs live in "game repos' ISSUES.md
until a framework tracker exists". A framework tracker does exist; it now says
so and names the Beads epic.
VERIFICATION
A link checker was written to scan every tracked *.md for relative links,
resolving each against the tree being scanned. Baseline at origin/master: 129
files, 137 relative links, 3 broken. After both commits: 127 files, 159 relative
links, 3 broken -- the SAME three, all pre-existing and untouched here:
docs/DATA_SHARDS.md -> INSTANT_LOADS_PARKED.md (never existed)
docs/DATA_SHARDS.md -> INSTANT_LOADS_SPIKE2.md (never existed)
docs/NETPLAY.md -> ../lib/recomp-net/README.md (inside a submodule)
Zero new broken links. The checker's own coverage was checked against a raw grep
for markdown link openers: the 3-link difference is C++ lambdas inside code
blocks, correctly not treated as links.
Also verified: py_compile over all 113 tracked tools/*.py plus psxrecomp_cli.py;
all four seed JSONs still parse; the test-registration guard still reports 123
test files all registered; recompiler/tests/test_coverage_vault.py still passes;
and no file in this commit had its line endings flipped (checked mechanically
across all 50 modified files, after two files were caught and redone).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
156 lines
7.2 KiB
Markdown
156 lines
7.2 KiB
Markdown
# Relocation Manifest Format
|
|
|
|
> **STATUS AMENDMENT (2026-07, BIOS profiles).** The bulk boot-time
|
|
> ROM-to-RAM copy case this document originally targeted is now carried by
|
|
> the BIOS profile instead: `[[recompiler.address_model.copy]]` in
|
|
> `bios/<STEM>.toml` (single source of truth for the recompiler, the
|
|
> emitted C, and the runtime via `psx_bios_image`; see
|
|
> `recompiler/src/bios_address_model.h`). The kernel-bless mechanism in
|
|
> `memory.c` remains the runtime enforcement of the "verify live RAM
|
|
> against the ROM source before dispatching AOT code" contract described
|
|
> here. This document stays authoritative ONLY for the oracle-observed
|
|
> case — regions with `src_rom_addr: null` that cannot be seen statically
|
|
> — which remains unimplemented.
|
|
|
|
|
|
**Status:** Phase B (design + first iteration), 2026-04-30.
|
|
**File on disk:** `generated/address_aliases.json`.
|
|
**Authority:** `PLAN.md:941-967` ("BIOS Relocation Is a Hard Gate"), `CLAUDE.md:333-365` (Rule 18).
|
|
**Companion files:** `generated/normalization_rule.md`, `generated/relocation_proofs/<name>/`.
|
|
|
|
---
|
|
|
|
## Purpose
|
|
|
|
Records every region of code that lives at a RAM address at runtime but
|
|
that the static recompiler cannot see in `SCPH1001.BIN` byte-for-byte
|
|
at compile time. Two flavors:
|
|
|
|
1. **ROM↔RAM aliases.** BIOS copies bytes from `0xBFCxxxxx` to a kernel-RAM
|
|
address during init and runs the RAM copy. Same bytes, two addresses.
|
|
2. **Trace-discovered RAM regions.** BIOS installs code at a RAM address
|
|
via a mechanism we may not have statically identified (relocated copy,
|
|
patched copy, runtime-assembled, etc.). We have observed bytes via the
|
|
oracle.
|
|
|
|
Both flavors share one record shape so the recompiler can ingest them
|
|
uniformly. The `discovered_by` field records *how* the entry was proven.
|
|
|
|
---
|
|
|
|
## File shape
|
|
|
|
`generated/address_aliases.json`:
|
|
|
|
```json
|
|
{
|
|
"version": 1,
|
|
"normalization_rule": "addr & 0x1FFFFFFF",
|
|
"entries": [ /* AliasEntry[] */ ]
|
|
}
|
|
```
|
|
|
|
`version` increments on incompatible schema changes. `normalization_rule`
|
|
mirrors `generated/normalization_rule.md` so the recompiler can check
|
|
agreement at load time.
|
|
|
|
### AliasEntry
|
|
|
|
| Field | Type | Required | Notes |
|
|
|---|---|---|---|
|
|
| `name` | string | yes | Slug, used as proof-directory name and as part of emitted function names. `[a-z0-9_]+`. |
|
|
| `dst_ram_addr` | hex string | yes | Physical RAM address (post-normalization). E.g. `"0x00005800"`. Recompiled function dispatch keys are derived from this. |
|
|
| `length` | int | yes | Byte length of the region. Must be a multiple of 4 (MIPS instruction size). |
|
|
| `entrypoints` | array of hex string | yes | RAM addresses at which `psx_dispatch` may enter this region. The recompiler emits a separate compiled function per entrypoint. At least one entry, must include `dst_ram_addr` if the region's first instruction is callable. |
|
|
| `bytes_hash` | string | yes | `sha256:<64 hex chars>` of the region bytes. Runtime verifies against live RAM before dispatching the AOT function. |
|
|
| `bytes_file` | string | yes | Path (relative to repo root) to the raw bytes file under `generated/relocation_proofs/<name>/bytes.bin`. Recompiler reads from here when `src_rom_addr` is null. |
|
|
| `src_rom_addr` | hex string | optional | Physical ROM address the bytes were copied from, if known. E.g. `"0x1FC2D800"`. When present, recompiler reads from ROM and additionally checks ROM[src..src+length] hash matches `bytes_hash`. |
|
|
| `discovered_by` | object | yes | Provenance. See below. |
|
|
|
|
### `discovered_by`
|
|
|
|
| Field | Type | Required | Notes |
|
|
|---|---|---|---|
|
|
| `tool` | string | yes | Identifier of the oracle / static tool. E.g. `"beetle_psx_oracle"`, `"ghidra_static_copy_loop"`, `"recomp_install_hook_log"`. |
|
|
| `writer_pc` | hex string \| null | when known | PC of the instruction that wrote the LAST byte of this region during install. `null` if not observed. |
|
|
| `first_exec_pc` | hex string \| null | when known | First PC inside this region that the oracle executed. Useful for cross-validating `entrypoints[0]`. |
|
|
| `reason` | string | yes | One-line human explanation of why this region matters and what evidence drove its inclusion. |
|
|
| `captured_at` | ISO-8601 date | yes | When the proof was captured. |
|
|
| `captured_session` | string | optional | Free-form session/commit reference. |
|
|
|
|
---
|
|
|
|
## Proof directory
|
|
|
|
Per PLAN.md every entry must have a directory `generated/relocation_proofs/<name>/`
|
|
containing:
|
|
|
|
| File | Required | Purpose |
|
|
|---|---|---|
|
|
| `proof.json` | yes | Canonical record of the entry (full AliasEntry plus extra evidence fields). |
|
|
| `bytes.bin` | yes | Raw bytes of the region, exactly `length` bytes. |
|
|
| `wtrace_excerpt.json` | when discovered_by.tool=oracle | Slice of the Beetle wtrace ring covering the writer events. |
|
|
| `dispatch_excerpt.json` | optional | Slice of recomp dispatch_tail showing the gap (we never dispatched these PCs). |
|
|
| `ghidra_*.{json,png}` | when discovered_by.tool starts with ghidra | Static analysis artifact. |
|
|
|
|
`address_aliases.json` is GENERATED from the proof directories — never
|
|
hand-edited. A small tool (`tools/build_address_aliases.py`, Phase C)
|
|
walks `generated/relocation_proofs/`, validates each `proof.json` +
|
|
`bytes.bin`, and emits the top-level manifest.
|
|
|
|
---
|
|
|
|
## Normalization
|
|
|
|
Working rule (per PLAN.md Phase 1e draft): `normalized = addr & 0x1FFFFFFF`.
|
|
|
|
Handles the four PSX address-space windows:
|
|
|
|
| Region | Virtual base | Normalized |
|
|
|---|---|---|
|
|
| KUSEG (raw) | 0x00000000 | self |
|
|
| KSEG0 (cached) | 0x80000000 | strip top 3 bits |
|
|
| KSEG1 (uncached) | 0xA0000000 | strip top 3 bits |
|
|
| BIOS via KSEG1 | 0xBFC00000 | 0x1FC00000 (distinct from RAM) |
|
|
|
|
So `0x00007568`, `0x80007568`, `0xA0007568` all normalize to `0x00007568`
|
|
and refer to the same RAM byte. `0xBFC07568` stays at `0x1FC07568` — a
|
|
distinct ROM byte. The recompiler stores all dispatch keys in normalized
|
|
form.
|
|
|
|
---
|
|
|
|
## First-iteration scope
|
|
|
|
Today's first manifest entries cover only the two dark RAM windows
|
|
identified in `docs/internal/CASE_A_AOT_GAP.md`:
|
|
|
|
| name | dst_ram_addr | length | entrypoints |
|
|
|---|---|---|---|
|
|
| `memcard_handler_4f00` | 0x00004F00 | 0x100 | 0x4F0C, 0x4F54 |
|
|
| `memcard_handler_5800` | 0x00005800 | 0x800 | 0x5800, 0x5DB8, 0x5DD0, 0x5DD8, 0x5EF4, 0x5FA8 |
|
|
|
|
Both are flagged `discovered_by.tool="beetle_psx_oracle"`, both have
|
|
non-null `writer_pc` (from the wtrace) and `first_exec_pc`. Both have
|
|
`src_rom_addr=null` in this iteration — the BIOS source these were
|
|
copied from is not yet identified statically. When we identify the
|
|
source (e.g. via tracing the BIOS init copy loops in Beetle and
|
|
matching them against ROM), we'll set `src_rom_addr` and drop
|
|
`bytes.bin` from the proof directory.
|
|
|
|
Future iterations expand coverage and add static src identification.
|
|
|
|
---
|
|
|
|
## Failure modes the manifest is designed to catch
|
|
|
|
- **Stale AOT dispatch.** If live RAM bytes drift from `bytes_hash`,
|
|
runtime aborts loudly rather than running the wrong code.
|
|
- **Duplicate logical function.** If two entries normalize to the
|
|
same `dst_ram_addr`, the recompiler refuses to build.
|
|
- **Unproven alias.** If an entry exists in `address_aliases.json` but
|
|
the matching `relocation_proofs/<name>/` directory is missing or
|
|
has the wrong hash, the recompiler refuses to build.
|
|
- **Speculative addition.** Every entry must point to a proof. No
|
|
"I think this region exists" entries.
|