Files
psxrecomp/recompiler/include/ps1_exe_parser.h
Matthew Stanley 80ff96780a recompiler: the overlay guard word is a delay-slot source, never a block leader
The overlay capture format DELIBERATELY appends one coherent guard instruction
past the end of a dirty-page run (overlay_capture.c write_json_window, `size
+= 4u`, pinned by runtime/tests/test_interpreter_perf_guards.py:47): a MIPS
branch sitting at the run's final word (...FFC) always executes its delay slot
in the next page (...000), and the shard must be able to emit and hash that
word. So a capture whose size is a page multiple PLUS 4 is a CORRECT capture --
N bytes of overlay plus one guard word. Measured: 36 of 36 Ape Escape
(SCUS-94423) vault records and 846 of 946 Tomba 2 (SCUS-94454) records are that
shape.

The recompiler was never taught the asymmetry: the guard word is a legal
DELAY-SLOT SOURCE but an ILLEGAL BLOCK LEADER, because the word after it does
not exist in the image. It had exactly one bound -- end_address() -- used both
to read words and to decide what is code.

MEASURED on Ape Escape, its own single-source vault at framework
v0.3.2-alpha-139-g1bf70960, 36 variants:

  PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0

  ! [recompiler] overlay 0x80136000 crc 4BAB23C9: what():  cannot emit control
    flow at 0x80138000: mandatory delay slot 0x80138004 is outside the input
    image
  + 16 x [fragment] interior 0x801360.. @region 0x00136000, identical message

One root cause, seventeen shards. The whole region then falls back to the
dirty-RAM interpreter forever, and nothing at runtime says so.

Causal chain, every step read in source:

  1. overlay_capture.c:349-361 appends the guard word; region 0x80136000 gets
     size 0x2004, so the image is [0x80136000, 0x80138004).
  2. compile_overlays.py make_psxexe wraps those bytes verbatim with text size
     = len(data), so PS1Executable::end_address() = 0x80138004.
  3. function_analysis.cpp in_exe / in_function admit 0x80138000 as ordinary
     code, and the walk reaches it by fall-through from 0x80137FFC
     (0x24022000, addiu -- not control flow), so func.end_addr becomes the
     image end.
  4. control_flow.cpp makes the final block end_addr = func.end_addr - 4 =
     0x80138000: the block LEADER is the guard word.
  5. control_flow.cpp's "Also check last instruction itself" promotion finds
     0x80137FFC is not control flow and therefore promotes read_word(end_addr)
     -- the guard word 0x1062005A, a beq -- to block.exit_instr, with
     has_delay_slot = true and address == end_addr. THIS IS THE DEFECT.
  6. code_generator.cpp computes delay_slot_addr = 0x80138004, read_word
     returns nullopt, and it throws.

THE THROW AT STEP 6 IS CORRECT AND IS PRESERVED. Emitting a control transfer
without its architectural delay slot silently corrupts guest state, which is
what the code did before that throw was added. The gap is step 5.

Fix: give the image view two distinct bounds.

  end_address()           READ bound -- every byte the image supplies, so a
                          mandatory delay slot living at a guard word resolves.
  analysis_end_address()  ANALYSIS bound -- one past the last byte that may be
                          discovered, made a block leader, or made a block's
                          exit instruction.

read_word() keeps the READ bound. Discovery and block construction move to the
ANALYSIS bound: function_analysis.cpp in_exe / in_exe_p3 / add_forced_entry /
every hard_cap that used to be the image end / the jr-table producer_hi
defaults / the static jr-ra and JAL sweeps; control_flow.cpp gains
analysis_walk_hi(func) and uses it for cfg.function_end, alias entries,
add_boundary, the scan loop, the final block's end_addr and in_func;
code_generator.cpp's mid-function split pre-pass; main_psx.cpp seed admission
and the data-scan pointer promotion; analysis_db.cpp, main_toml.cpp,
main_cli.cpp target/seed admission. The clamp is applied only when it leaves a
non-empty walk, so a degenerate function starting at or past the bound keeps
failing closed rather than silently producing an empty CFG.

HOW THE BOUND IS DERIVED -- the producer STATES it; the recompiler is told.

The alternative was for the recompiler to infer the count from `size % 4096 ==
4`. That fits every capture the current writer has ever produced, but it is a
consumer guessing at a producer's intent, and this tree already has the
counter-example: 52 of the 946 SCUS-94454 records have residues from 112 to
3996 bytes (one not even 4-byte aligned) because they come from the STATIC
extractor, which records an exact file extent and appends no guard word. A rule
keyed to the page-run shape is a rule that shifts the moment a producer
changes.

So the count travels with the image, in a magic-tagged field in the otherwise
zero-filled tail of the 2048-byte PS-EXE header (exe_tag in ps1_exe_parser.h:
"PSXRGRD1" at 0x7E0, uint32 count at 0x7E8). Putting it in the header rather
than on the command line means every consumer gets it automatically and no
future invocation path can forget to pass it -- compile_overlays has three
make_psxexe sites and five recompiler spawn sites. An absent or malformed tag
means ZERO guard bytes, which is exactly what every genuine PS-X EXE is: the
BIOS and every retail main executable have no guard word, their analysis bound
equals their read bound, and their analysis is bit-for-bit unchanged.

Producer side:
  overlay_capture.c    writes "guard_bytes": 4|0 per region -- the writer
                       stating its own intent, additive to the v2 schema and
                       preserved through coverage_vault.py compaction.
  compile_overlays.py  capture_guard_bytes(): prefer that field; for captures
                       recorded before it existed, reconstruct the single
                       shape write_json_window can produce (page multiple + 4)
                       and return 0 for everything else, which is the
                       fail-closed answer -- an unrecognised guard word is
                       analysed as code and the step-6 throw still fires
                       loudly at generation time.
  make_psxexe          guard_bytes is keyword-only with NO default, so a new
                       call site is a TypeError, not a silently mis-analysed
                       shard. The count is threaded through
                       compile_fragment_batch / compile_interior_fragment /
                       generate_interior_fragment_static and into the fragment
                       fail-memo key (bumped to v3, so a memo recorded under
                       the old analysis cannot suppress a retry under the
                       corrected one).

SECOND-ORDER DEFECT FIXED IN THE SAME PASS. generate_ranges_manifest extends a
range by +4 for any block whose exit instruction IS its last word, so the
cloned delay slot joins the candidate CRC and the page-generation watch. That
extension was unclamped. Reachable TODAY, independent of the guard word: a
branch-likely opcode (0x14-0x17, RESERVED on the R3000A) as the image's very
last word is short-circuited by translate_basic_block into an inline RI raise
BEFORE any delay slot is read, so generation succeeds and the manifest claims 4
bytes past the image end. Measured on origin/master with a 28-byte synthetic
image: `R 80010000 20` for an image ending at 0x8001001C. The cache would then
validate the shard against bytes the shard never saw. Now clamped to the READ
bound -- not the analysis bound, because a guard word genuinely IS compiled in
when it serves as a delay slot and must participate in the CRC.

DEAD CODE, documented rather than trusted: basic_block.cpp's BasicBlockAnalyzer
bounds its walk correctly but nothing in the tree constructs it. It is still
compiled and still hashed into the codegen tag, so a change there invalidates
caches without moving an emitted byte. Its bounds move to the analysis bound
too -- wiring it up must not reintroduce this defect -- and a header comment now
says where the live walker actually is.

TESTS. Both registered, so ctest runs them and the configure-time
registration guard is satisfied.

recompiler/tests/test_overlay_guard_word_analysis_bound.py drives
psxrecomp-game over five synthetic images:
  A  tagged, 2 pages + guard word, last real word not control flow, guard word
     IS a beq -- the measured Ape shape. Must generate; the guard word must not
     be emitted as an instruction; the block must publish 0x80012000 for the
     runtime to dispatch; no range may claim it.
  B  byte-for-byte the same payload with NO tag. Must still fail closed with
     the mandatory-delay-slot diagnostic naming 0x80012000. A and B differ only
     in the header tag, which proves the tag is what changed behaviour and that
     the throw was not weakened.
  B2 a misaligned declaration (6). Must be rejected and behave like B.
  C  tagged, with a real branch at the last analysable word. The guard word
     must be emitted AS the delay slot and must be covered by the manifest.
  D  branch-likely as the final word. Generation must succeed and no range may
     exceed the image end.

Against origin/master's psxrecomp-game the same file reports exactly the two
things this commit changes, and passes B, B2 and C:
  FAIL: tagged guard word still fails generation: what():  cannot emit control
        flow at 0x80012000: mandatory delay slot 0x80012004 is outside the
        input image
  FAIL: range [0x80010000,0x80010020) claims 4 byte(s) past the image end
        0x8001001C

tools/tests/test_overlay_guard_bytes_declaration.py pins the producer half:
field precedence, the reconstruction table, rejection of nonsense
declarations, the tag's byte offsets, that make_psxexe with guard_bytes=0 is
byte-identical to the pre-fix wrapper, and that overlay_capture.c both appends
the guard word and now declares it.

MEASURED OUTCOME, Ape Escape, same vault, same flags, fresh cache:
  before  PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0
  after   PSX_SHARD_RESULT ok=31 failed=0  skipped=57 capacity_fastpath=0

ok falls because 19 isolated island fragment shards at 0x00136000 are no longer
needed once the region variant compiles as one shard. Coverage is what matters,
and it is measured from the DLL exports: 349 -> 361 unique func_XXXXXXXX
entries, ZERO lost, 12 gained -- 0x80136140, 0x8013614C, 0x80136154,
0x80136238, 0x801362D4, 0x801362DC, 0x801362F0, 0x80136324, 0x80136370,
0x80136400, 0x80136480, 0x80137950, all from the failure list. The other four
failing PCs were already served by a sibling variant's shard.

INERTNESS.
  BIOS (SCPH1001, psxrecomp-bios --config bios/SCPH1001.toml): all four
  generated artifacts BYTE-IDENTICAL, verified by sha256 rather than by exit
  code, because tools/regen_bios.sh exits 0 on failure.
  Tomba 2 main EXE (psxrecomp-game --config game.toml): all nine generated
  artifacts BYTE-IDENTICAL, and the console log is identical too.
  Tomba 2 overlays, 82 variants over 6 regions from its own vault: identical
  result line (ok=98 failed=0 skipped=103) and identical shard identity set,
  4901 unique func exports on both sides -- zero gained, zero lost. 97 of 98
  range manifests are byte-identical. The one that differs is region
  0x8010A000 variant A32DB188, size 0x2004, whose last function's range goes
  from 0x218 to 0x214 bytes: its guard word (0x24840008, addiu) had been
  emitted as an ordinary instruction and hashed, and now is not. That is
  precisely the intended change and nothing else moved.

Bead beads-eio.3.100.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-02 15:35:24 -07:00

196 lines
8.0 KiB
C++

#pragma once
#include <cstdint>
#include <vector>
#include <string>
#include <optional>
#include <filesystem>
namespace PSXRecomp {
// PS-X EXE header structure (2048 bytes)
#pragma pack(push, 1)
struct PS1ExeHeader {
// Offset 0x00-0x0F: Magic + padding
char magic[8]; // "PS-X EXE"
uint32_t pad0;
uint32_t pad1;
// Offset 0x10-0x1F: Core addresses
uint32_t initial_pc; // Entry point
uint32_t initial_gp; // Global pointer
uint32_t load_address; // RAM load address
uint32_t file_size; // Executable size (bytes)
// Offset 0x20-0x2F: Memfill (BSS section)
uint32_t unknown0;
uint32_t unknown1;
uint32_t memfill_start;
uint32_t memfill_size;
// Offset 0x30-0x3F: Stack setup
uint32_t initial_sp; // Stack pointer
uint32_t initial_fp; // Frame pointer
uint32_t stack_base;
uint32_t stack_offset;
// Offset 0x40-0x7FF: Reserved (2048 - 64 bytes already used = 1984 bytes)
uint8_t reserved[1984];
// Helper methods
uint32_t end_address() const {
return load_address + file_size;
}
bool entry_in_range() const {
return initial_pc >= load_address &&
initial_pc < end_address();
}
uint32_t bss_end() const {
return memfill_start + memfill_size;
}
};
#pragma pack(pop)
static_assert(sizeof(PS1ExeHeader) == 2048, "PS1ExeHeader must be exactly 2048 bytes");
// --------------------------------------------------------------------------
// Analysis-bound tag: how many TRAILING bytes of the payload are guard words
// --------------------------------------------------------------------------
//
// Overlay capture DELIBERATELY appends a coherent guard instruction past the
// end of a dirty-page run (runtime/src/overlay_capture.c, write_json_window:
// `size += 4u`, pinned by runtime/tests/test_interpreter_perf_guards.py). A
// MIPS branch sitting at the run's final word (...FFC) always executes its
// delay slot in the NEXT page (...000); supplying that one word lets overlay
// codegen emit exact semantics and lets the range manifest hash/watch it.
//
// So the guard word is a legal DELAY-SLOT SOURCE but an ILLEGAL BLOCK LEADER:
// the word AFTER it does not exist in the image, so a control transfer AT the
// guard word could never have its mandatory delay slot emitted. The
// recompiler must therefore carry two distinct bounds — a READ bound (the
// whole payload, so a delay slot at the guard word resolves) and an ANALYSIS
// bound (short of the guard words, so discovery and block construction can
// never lead there).
//
// The producer that wrapped the capture states the count explicitly, in a
// magic-tagged field in the otherwise-zero tail of the 2048-byte PS-X EXE
// header (tools/compile_overlays.py, make_psxexe). The recompiler is TOLD
// rather than inferring the count from the payload size: an inference from
// `size % 4096 == 4` fits every capture written by the current format but
// would silently shift underneath any future change to capture granularity,
// and pre-2026-07-25 captures are page-exact with no guard word at all.
//
// An absent or malformed tag means ZERO guard bytes, which is exactly what
// every genuine PS-X EXE (the BIOS, a retail main executable) is: those images
// have no guard word, their analysis bound equals their read bound, and their
// analysis is bit-for-bit unchanged by this mechanism.
namespace exe_tag {
// Offsets are into the 2048-byte header. Retail headers use only
// [0x00, 0x4C) plus an ASCII region string ending well before 0x100; the tail
// is zero-filled. The 8-byte magic makes a false positive on a real EXE
// effectively impossible even if some image did carry bytes there.
inline constexpr uint32_t kGuardMagicOffset = 0x7E0u;
inline constexpr uint32_t kGuardCountOffset = 0x7E8u; // uint32, little-endian
inline constexpr char kGuardMagic[8] =
{'P', 'S', 'X', 'R', 'G', 'R', 'D', '1'};
// A guard run exists to cover branch delay slots at a page edge; one word is
// what the capture format appends. Cap the accepted value at a page so a
// corrupt tag can never shrink analysis to nothing.
inline constexpr uint32_t kMaxGuardBytes = 4096u;
} // namespace exe_tag
// Decode the analysis-bound tag. Returns 0 when the tag is absent, when the
// magic does not match, or when the declared count is not a sane, strictly
// interior, instruction-aligned trailer of `file_size` bytes of payload.
uint32_t decode_analysis_guard_bytes(const PS1ExeHeader& header,
uint32_t file_size);
// Parsed PS1 executable with code data
class PS1Executable {
public:
PS1ExeHeader header;
std::vector<uint8_t> code_data; // Raw binary (file_size bytes)
// Trailing bytes of code_data that exist ONLY to supply architectural
// delay slots to instructions inside the analysis bound. Read-legal,
// analysis-illegal. See the exe_tag comment above. Zero for every image
// that is not an overlay capture wrapped by compile_overlays.
uint32_t analysis_guard_bytes = 0;
// Computed properties
uint32_t load_address() const { return header.load_address; }
uint32_t entry_point() const { return header.initial_pc; }
uint32_t code_size() const { return header.file_size; }
// READ bound: one past the last byte the image physically supplies. Used
// by read_word so a mandatory delay slot living at a guard word resolves.
uint32_t end_address() const { return header.end_address(); }
// ANALYSIS bound: one past the last byte that may be DISCOVERED, made a
// block leader, or made a block's exit instruction. Equal to
// end_address() unless the producer declared trailing guard words.
uint32_t analysis_end_address() const {
const uint32_t hi = end_address();
return (analysis_guard_bytes < hi - load_address())
? hi - analysis_guard_bytes
: hi;
}
bool has_analysis_guard() const { return analysis_guard_bytes != 0; }
// Access code as 32-bit words (for MIPS disassembly).
//
// Deliberately bounded by end_address(), NOT analysis_end_address(): the
// entire point of a guard word is that it is readable so that a transfer
// at the last analysable word can emit its delay slot.
std::optional<uint32_t> read_word(uint32_t address) const {
if (address < load_address() || address >= end_address()) {
return std::nullopt; // Out of range
}
uint32_t offset = address - load_address();
if (offset + 3 >= code_data.size()) {
return std::nullopt; // Partial word
}
// Little-endian read
return (uint32_t)code_data[offset] |
((uint32_t)code_data[offset+1] << 8) |
((uint32_t)code_data[offset+2] << 16) |
((uint32_t)code_data[offset+3] << 24);
}
// Validate executable is well-formed
bool validate(std::string& error_msg) const;
};
// Parser class
class PS1ExeParser {
public:
// Parse PS1-EXE from file
static std::optional<PS1Executable> parse_file(
const std::filesystem::path& path,
std::string& error_msg
);
// Parse PS1-EXE from memory buffer
static std::optional<PS1Executable> parse_buffer(
const std::vector<uint8_t>& buffer,
std::string& error_msg
);
// Validate header (public for PS1Executable::validate)
static bool validate_header(const PS1ExeHeader& header, std::string& error_msg);
};
// Restrict a parsed main executable to a title-verified static-analysis size.
// The original disc image is not modified; only this in-memory analysis view
// is shortened. Returns false without mutation when the configured bound is
// invalid. A canonical page-rounded size above the header size is accepted as
// an existing runtime reservation and leaves the analysis view unchanged.
bool apply_static_analysis_bound(PS1Executable& exe,
uint32_t configured_size,
std::string& error_msg);
} // namespace PSXRecomp