From 00b1b02a07c8e7bf5e6a2e87455a397b4b6f10b2 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Sun, 7 Jun 2026 13:42:10 -0700 Subject: [PATCH] overlay: evidence-scoped --overlay recompiler mode (Inc1-A) + design doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restores entry-scoped overlay analysis (analyze_exact_entries + split_mid_function_targets=false) as the MANDATORY mode for overlay inputs, renamed --overlay-exact -> --overlay: the ingester's contract, set unconditionally by compile_overlays.py for every overlay, never a human toggle. Evidence-scoped discovery compiles only proven seeds + reachable code and keeps branch/jump-table targets as in-parent labels — no whole-byte sweep, so it no longer decodes embedded data tables as code. Effect on the captured Tomba overlay: the data-as-code explosion collapses from thousands of UNSUPPORTED to 122 clean functions with 42 residual unsupported opcodes + 8 bad targets. Those residuals are now a per-function-audit concern (Inc1-B), not a wholesale region failure. Main-EXE codegen is unchanged (Tomba still regenerates 2854 functions; split pre-pass still runs in non-overlay mode). Also adds docs/overlay-recompilation-design.md — the converged design of record (precision-over-recall; compiled!=registered!=callable; write-invalidation over CRC; internal-labels-in-parent; the RAM-write-path audit result; the three implementation increments). Co-Authored-By: Claude Opus 4.8 --- docs/overlay-recompilation-design.md | 302 ++++++++++++++++++++ recompiler/include/code_generator.h | 2 + recompiler/include/function_analysis.h | 5 + recompiler/src/code_generator.cpp | 6 +- recompiler/src/function_analysis.cpp | 376 +++++++++++++++++++++++++ recompiler/src/main_psx.cpp | 24 +- tools/compile_overlays.py | 15 +- 7 files changed, 719 insertions(+), 11 deletions(-) create mode 100644 docs/overlay-recompilation-design.md diff --git a/docs/overlay-recompilation-design.md b/docs/overlay-recompilation-design.md new file mode 100644 index 00000000..2e77f494 --- /dev/null +++ b/docs/overlay-recompilation-design.md @@ -0,0 +1,302 @@ +# Overlay Recompilation — Design Review + +**Status:** CONVERGED (post-review). The open questions below were taken to an +external reviewer; the resolutions are recorded in §5–§7 at the end, which +supersede any "OPEN" markers above. §0–§4 are preserved as the reasoning trail. + +--- + +## 0. Context (for a reader unfamiliar with the project) + +PSXRecomp is a **static recompiler** for PlayStation 1 games: it translates the +game's MIPS machine code to native C ahead of time, compiles that to a real +binary, and runs it natively (no CPU interpreter on the hot path). This is the +N64Recomp / SuperMarioWorldRecomp model. The project premise — non-negotiable — +is that **the game runs as statically recompiled native code**, not emulation. + +The hard part: PS1 games stream **overlays** — code segments DMA'd from disc +into RAM at runtime, executed, then overwritten by the next overlay. A pure +static recompiler can't see overlay code at build time (it isn't in the main +executable; it arrives at runtime). Two existing mechanisms: + +1. **Dirty-RAM interpreter** (`dirty_ram_interp.c`): a small MIPS interpreter + that runs any code in RAM pages written at runtime. This is the *failover* — + correct but slow, and explicitly **not** the goal. +2. **Overlay DLL cache** (the subject of this doc): capture overlays as they + load, recompile them offline to native DLLs keyed by content hash, and on + later runs `LoadLibrary` the DLL and dispatch its functions **before** the + interpreter. Coverage grows as people play. + +### The pipeline as built +``` +runtime capture ──► overlay_captures.json (full bytes + executed PCs + dispatch PCs) + │ │ + │ compile_overlays.py (offline) + │ │ - classify seeds (which PCs are real function entries) + │ │ - run the recompiler on the overlay bytes + │ │ - audit generated C, compile to .dll + ▼ ▼ + dirty-RAM page cache//_.dll + │ │ + └──► dispatch: overlay DLL func (if registered) ──► else interpreter (failover) +``` + +--- + +## 1. The governing principle we've converged on + +**Asymmetric failure ⇒ bias for precision over recall.** + +- A **missed** overlay function is *safe*: the loader only claims the specific + `func_XXXXXXXX` addresses it registered from the DLL. An unregistered address + returns "not handled" and **falls through to the interpreter**. The seed is + then captured and the overlay is regenerated next time with that seed + included. Under-coverage self-heals. +- A **mis-compiled** function is *fatal*: it gets registered, the game calls it, + and it executes wrong native code against shared CPU state → corruption → + black/blue screen. There is no failover once we've claimed an address. + +Therefore every design choice should **minimize false registrations**, even at +the cost of compiling fewer functions per pass. Recall is recovered for free by +the interpreter-failover-and-recapture loop; precision is not recoverable. + +**Question for review:** Is this asymmetry actually total? Are there cases where +a *missed* function does NOT safely fall back — e.g., the game's control flow +assumes the function executes within some timing budget that the interpreter +blows, or the missed function is on a path where the interpreter itself can't +handle some instruction? If "miss is always safe" has exceptions, the whole +precision-first thesis weakens. + +--- + +## 2. Converged positions (please attack) + +### 2.1 Capture is complete and data-inclusive; the DLL is code-only +- The on-disk capture artifact contains the **full overlay image, including + data** (jump tables, constant pools). Required because (a) re-recompiling + later with better heuristics needs the original bytes, (b) the recompiler must + *read* data to resolve jump tables, (c) the cache key is a CRC over the image. +- The compiled **DLL contains code only**. The overlay's data is never emitted; + it lives in PSX RAM (the game DMA'd it there), and recompiled functions read + it through normal memory callbacks at runtime. +- Consequence: "how much benign data surrounds the code" is **not** a + correctness question for the DLL. The only correctness question is "is this + word reached as code?" Data is recognized as not-code and left alone. + +### 2.2 Capture bytes: DMA-time vs execution-time (OPEN — leaning a direction) +The current capture stores **DMA-time ("unpatched")** bytes — the overlay as +delivered from disc, *before* any load-time relocation/fixups the game applies. +If a game patches the overlay after load (relocation, jump-table base fixups, +self-modification), the bytes we recompile ≠ the bytes that actually executed, +and the DLL is subtly wrong. This is a candidate root cause for past black/blue +failures that has nothing to do with seed classification. + +- **What's correct to recompile from:** execution-time bytes (what the CPU + actually ran). +- **Why not just snapshot RAM at exit:** by exit, that RAM range may already be + contaminated by a *later* overlay reusing the address. +- **The right moment is first-execution-time:** snapshot `RAM[region]` the first + time we dispatch into it — after fixups, before reuse. +- **Proposed:** in dev builds, dump **both** (`_dma` and `_exec`) and + diff them. Identical ⇒ no runtime patching, capture-time is irrelevant. + Different ⇒ the diff reveals the fixup pattern and likely the bug. + +**Question for review:** Is "recompile from execution-time bytes" correct, or +does it break the content-hash cache model (the patched bytes may be +load-address-specific — see §2.5 relocation)? Should the cache key be over +unpatched or patched bytes? + +### 2.3 Function discovery is one evidence hierarchy; no whole-byte sweep +Rank evidence that an address is a real function entry: +1. Known entries (initial PC, exported symbols, declared entries) +2. **Executed PCs** — we literally watched them run (strongest signal) +3. Direct `JAL` targets (statically provable callers) +4. Jump-table targets we resolved by reading the table +5. ~~Whole-byte sweep: decode every byte hoping to find functions~~ + +Tier 5 is the *only* unsound tier for overlays — it invents entries from data, +which is what produces the data-as-code failures. We have tiers 1–4 for free on +overlays (we captured them by running them). So discovery should use 1–4 and +**never** tier 5 for overlays. (The cold main EXE, with no execution evidence, +is the only place tier 5 has ever been justified — and even there it should be +the flagged, lowest-confidence tier.) + +This was previously implemented as an opt-in recompiler flag +(`--overlay-exact`). We believe a flag is the wrong abstraction: evidence-driven +discovery is *implied by whether execution evidence exists for the input*, not a +human toggle. + +### 2.4 Mid-function seeds: capture them, merge at encode time +A runtime "dispatch entry" can be a **jump-table case label mid-function**, not +a callable function boundary. These are dangerous: the game *does* reach them +(so we'd register them), but compiling them as standalone functions yields wrong +boundaries/stack assumptions. + +- Don't filter at capture time (the runtime can't tell). **Capture everything.** +- At encode time, with the full bytes, classify each seed: a seed that falls + strictly inside another function's `[start,end)` is **mid-function → merge + into the parent** (it becomes a label in the parent's CFG, not its own func). +- The discriminators are local and don't need external callers: preceded by + `jr $ra`+delay slot (real boundary), has a prologue, is a direct `JAL` target. + +### 2.5 Confidence flagging ("jump-risk") +The ambiguous residue: a function reachable **only** via a computed/indirect +jump (jump table or function pointer), never via a direct `JAL`. It looks +identical to a jump-table case label. Proposal: tag each emitted entry with a +confidence (`strong` / `mid-function` / `jump-risk` / `forced`) in a +machine-readable sidecar (the recompiler already emits `skipped_functions.json`, +so the pattern exists). + +**Open sub-question:** for `jump-risk` entries, do we (a) emit them with the +flag, or (b) *exclude* them from compilation initially and let them +interp-fallback until a direct-JAL or execution proof confirms them? Given the +precision-first thesis, (b) is tempting. + +### 2.6 Overlay segmentation and cache keying (OPEN) +The runtime currently captures **one region = all contiguous dirty pages** +(observed: a single 356 KB region for Tomba). That is almost certainly *not one +overlay* — it likely merges several disc loads plus unrelated dirtied RAM. +Problems: the cache is CRC-keyed per region, so if a later run loads a different +overlay into part of that range, the CRC changes and the whole region misses. +Proposal: re-segment using the CD-DMA transfer log (each forward CH3 DMA records +LBA→dest→size) into real per-load overlays, and key the cache per overlay. + +**Question for review:** can disc-load boundaries (CD-DMA log) reliably bound +overlays, given overlapping/reused address ranges and overlays assembled from +multiple DMA blocks? Is per-(load_addr, crc) keying sufficient, or does +relocation (§2.2) force something else? + +### 2.7 Code hygiene (DRY) +The same low-level MIPS primitives (`jr $ra` detect, `addiu $sp` prologue, +valid-MIPS-word, classify-control-flow) are currently reimplemented in at least +three places (the Python seed vetter, the Python overlay classifier, the C +runtime decodability guard). The *policies* differ legitimately (single-address +verdict vs. whole-overlay CFG classification) and should stay separate, but the +primitives should be one shared module. + +--- + +## 3. The specific failure we just observed (concrete grounding) +- Captured one overlay: load `0x800E7000`, 356 KB, 15358 executed PCs, 66 + dispatch-entry seeds, full bytes present. +- The offline classifier selected 122 function entries + 59 direct-JAL targets + and excluded 3096 branch targets + 14221 observed-only PCs (mid-function + filtering already works). +- But the recompiler, run **without** evidence-scoping (whole-byte sweep), walked + into embedded data tables and emitted `UNKNOWN_BAD` jump targets + + `UNSUPPORTED_INSTRUCTION` runs. The offline audit then refused to emit the DLL + (all-or-nothing per region) → **zero DLLs produced**. +- So the observed failure is at the **discovery strategy** (tier-5 sweep + decoding data), not at seed classification. This is the direct evidence behind + §2.3. + +--- + +## 4. The specific questions we most want challenged +1. Is the precision-over-recall asymmetry (§1) truly total, or are there + "missed function does not safely interp" cases? +2. Execution-time vs DMA-time capture (§2.2): which should we recompile from, + and which should key the cache, especially under relocation? +3. Does relocation/self-modification break the static-overlay premise for some + overlays — i.e., is there a class that *must* stay interpreted forever, and + how do we detect it cheaply? +4. Jump-risk entries (§2.5): emit-with-flag vs exclude-until-proven? +5. Segmentation (§2.6): can the CD-DMA log reliably carve real overlay + boundaries out of the dirty-page blob? + +--- + +## 5. Resolved — converged decisions (post-review) + +Accepted refinements (now design-of-record, not open): + +- **(A) Runtime guard = write-invalidation, not CRC-per-dispatch.** On DLL + load: validate code + dependency ranges once. On any write to a *watched* + RAM page: invalidate the affected registered entries / DLL range. On + dispatch: registration-table check only, no hot-path CRC. Coarse page-level + invalidation first; false positives (over-invalidation → interp fallback) + are acceptable. Do NOT start with fine interval invalidation. +- **(D) Internal labels are compiled inside a proven parent, never + registered.** A jump-table / branch target is *not* a callable function + entry; it is compiled as a label inside a call-proven parent function and + reached via the parent's native switch. This is what keeps precision-first + from degenerating into "interpret all switch code." +- **(B) Edge capture, minimal schema** — at the control-flow classifier, not + the dispatcher. `{src_pc, dst_pc, edge_type ∈ {Jal,Jalr,Jump,Branch,JrRa, + JrOther}, writes_ra}`. Defer `sp_before/after`, `delay_slot_pc` until a + concrete classifier failure demands them. Share the edge enum with the + recompiler's static walk. +- **(C) `UNSUPPORTED_INSTRUCTION` is a boundary/CFG suspicion first, real + opcode gap last.** Classify in order: (1) reachable from entry via valid CF? + (2) fell through past `jr $ra`/terminal branch into data? (3) in a reachable + branch/jump delay slot? (4) impossible target from bad discovery? (5) only + then a true opcode gap. Trim only when justified by an actual terminal + control-flow edge — never "trim until the audit passes." +- **(E) Native-vs-interp differential testing** is the empirical backbone for + the parity assumption — but it is increment-2, not an increment-1 blocker. + +### The three-bucket entry rule +``` +direct JAL target / jalr-with-link / declared / initial entry + -> compile as function, REGISTER as callable +jump-table or branch target inside a known parent CFG + -> compile as label inside the parent, DO NOT register +observed dispatch target with no call-edge / parent proof + -> DO NOT register; interp fallback; capture better edge evidence +``` + +### The load-bearing invariant +``` +Compiled != Registered. +Emitted != Callable. +Only REGISTERED entries may bypass the interpreter. +``` +The DLL may physically contain functions for diagnostics/future use; what the +runtime registration table claims is the only thing that affects correctness. + +### Precision-over-recall — refined (answer to §4 Q1) +The asymmetry is not a law; "a miss is safe" holds only if native↔interp state +preservation, instruction coverage, delay/load-slot and IRQ/DMA-timing +equivalence, and stale-registration invalidation all hold. But the thesis +survives in its correct form: **false registration is immediately corrupting; +missed registration is recoverable if interpreter parity holds; therefore +optimize for registration precision.** Parity itself is to be *measured* (E), +not assumed. + +## 6. RAM-write-path audit result (gates the §5-A guard) + +- All CPU stores (generated + interpreter) and DMA writes funnel through + `psx_write_{word,half,byte}` in `runtime/src/memory.c`. The only `memcpy`/ + `memset` into memory target device-local buffers (`param_fifo`, `vram`, + `spu_ram`), never main `ram`. **No native path mutates main RAM behind the + store functions** ⇒ write-invalidation has no hidden false-negative source. +- Existing dirty tracking on the store path is **kernel-only** + (`dirty_ram_mark_kernel_write` bails at `phys >= 0x10000`). Overlay pages are + marked only by DMA (`dirty_ram_mark_executable_range`). So overlay CPU + self-modification is currently untracked; the bitmap covers all 2 MB, the + guard simply doesn't set it there. +- **Required change for §5-A:** a watched-overlay-pages bitmap set at + registration; one extra bitmap test in the RAM-store branch of all three + `psx_write_*`; on hit → `overlay_loader_invalidate_page()`. Coarse, cheap. + +## 7. Implementation increments + +**Increment 1 — make DLLs safe and loadable (current)** +1. No whole-byte sweep for overlay inputs (mandatory, ingester-set `--overlay` + contract; not a human flag). +2. Split classification: code-reachable / callable entry / internal label. +3. Per-function (or SCC) audit, not per-region. +4. Register only proven callable entries. +5. Jump-table/case labels compiled only inside their proven parent. +6. Coarse write-invalidation for registered overlay pages (per §6). + Success: Tomba emits a partial DLL; runtime loads it; only strong entries + registered; unproven code interp-falls-back; no black/blue from false + registration. + +**Increment 2 — make it measurable** +Edge-capture schema (B); DLL manifest with code/dependency ranges; +registration-time validation; native-vs-interp differential harness (E). + +**Increment 3 — make it efficient and stable** +Execution-time vs DMA-time dual capture; source-grouped + exec-specialized +cache key; executed-code-envelope segmentation; CD-DMA as provenance only. diff --git a/recompiler/include/code_generator.h b/recompiler/include/code_generator.h index 184dc3e7..78cf0836 100644 --- a/recompiler/include/code_generator.h +++ b/recompiler/include/code_generator.h @@ -34,6 +34,7 @@ struct CodeGenConfig { bool emit_line_numbers; // Include original address as comments bool optimize_zero_reg; // Optimize away $zero assignments bool use_switch_for_blocks; // Use switch instead of goto labels + bool split_mid_function_targets; // Split branch targets into funcs for legacy main-EXE analysis std::string indent; // Indentation string (default: " ") CodeGenConfig() @@ -41,6 +42,7 @@ struct CodeGenConfig { , emit_line_numbers(true) , optimize_zero_reg(true) , use_switch_for_blocks(false) + , split_mid_function_targets(true) , indent(" ") {} }; diff --git a/recompiler/include/function_analysis.h b/recompiler/include/function_analysis.h index c6f7fd49..1b7b4469 100644 --- a/recompiler/include/function_analysis.h +++ b/recompiler/include/function_analysis.h @@ -35,6 +35,11 @@ public: // Scan entire executable for function boundaries FunctionAnalysisResult analyze(); + // Analyze only explicit entry points and direct JAL targets reachable from + // them. This is for runtime-loaded overlays where scanning the whole blob + // treats data and basic-block targets as standalone functions. + FunctionAnalysisResult analyze_exact_entries(const std::vector& entries); + // Add a forced entry point address that is treated as a function start // even if it has no standard ADDIU $sp prologue. The function will be // included in the analysis result with has_prologue = false. diff --git a/recompiler/src/code_generator.cpp b/recompiler/src/code_generator.cpp index 96edab42..2f212505 100644 --- a/recompiler/src/code_generator.cpp +++ b/recompiler/src/code_generator.cpp @@ -1573,7 +1573,11 @@ std::string CodeGenerator::generate_file( // hit `psx_unknown_dispatch`. Surfaced by Phase B2 audit. The 16-pass // cap is a safety limit; in practice each pass strictly reduces the // remaining target set so convergence is fast. - { + // + // Overlay exact mode disables this: overlay branch targets are basic-block + // labels, not callable function entries, so splitting them into standalone + // functions produces broken dispatch — the mid-function-seed failure mode. + if (config_.split_mid_function_targets) { std::set cfgs_to_scan; // Which CFGs to scan (empty = all) uint32_t exe_start = exe_.header.load_address; uint32_t exe_end = exe_.end_address(); diff --git a/recompiler/src/function_analysis.cpp b/recompiler/src/function_analysis.cpp index 838c4022..859e2591 100644 --- a/recompiler/src/function_analysis.cpp +++ b/recompiler/src/function_analysis.cpp @@ -1,6 +1,7 @@ #include "function_analysis.h" #include #include +#include #include #include @@ -217,6 +218,381 @@ bool FunctionAnalyzer::is_likely_data_section(uint32_t start_addr, uint32_t end_ return false; } +namespace { + +enum class ExactCfKind { + Normal, + Branch, + Jump, + Jal, + JrRa, + JrOther, + Jalr, +}; + +struct ExactCf { + ExactCfKind kind = ExactCfKind::Normal; + uint32_t target = 0; +}; + +struct ExactWalkResult { + std::set visited; + std::set direct_jal_targets; + std::set jump_table_targets; + uint32_t jr_ra_count = 0; +}; + +static bool exact_is_jr_ra_word(uint32_t instr) { + return instr == 0x03E00008u; +} + +static bool exact_is_addiu_sp_neg(uint32_t instr) { + uint32_t opcode = (instr >> 26) & 0x3Fu; + uint32_t rs = (instr >> 21) & 0x1Fu; + uint32_t rt = (instr >> 16) & 0x1Fu; + int16_t imm = static_cast(instr & 0xFFFFu); + return opcode == 0x09u && rs == 29u && rt == 29u && imm < 0; +} + +static bool exact_is_valid_mips_word(uint32_t instr) { + if (instr == 0xFFFFFFFFu || instr == 0xFFFFFFFDu) return false; + + uint32_t opcode = (instr >> 26) & 0x3Fu; + uint32_t funct = instr & 0x3Fu; + uint32_t rt = (instr >> 16) & 0x1Fu; + + if (opcode == 0x00u) { + switch (funct) { + case 0x00u: case 0x02u: case 0x03u: case 0x04u: + case 0x06u: case 0x07u: case 0x08u: case 0x09u: + case 0x0Cu: case 0x0Du: + case 0x10u: case 0x11u: case 0x12u: case 0x13u: + case 0x18u: case 0x19u: case 0x1Au: case 0x1Bu: + case 0x20u: case 0x21u: case 0x22u: case 0x23u: + case 0x24u: case 0x25u: case 0x26u: case 0x27u: + case 0x2Au: case 0x2Bu: + return true; + default: + return false; + } + } + if (opcode == 0x01u) { + return rt == 0x00u || rt == 0x01u || rt == 0x10u || rt == 0x11u; + } + + switch (opcode) { + case 0x02u: case 0x03u: case 0x04u: case 0x05u: + case 0x06u: case 0x07u: + case 0x08u: case 0x09u: case 0x0Au: case 0x0Bu: + case 0x0Cu: case 0x0Du: case 0x0Eu: case 0x0Fu: + case 0x10u: case 0x12u: + case 0x20u: case 0x21u: case 0x22u: case 0x23u: + case 0x24u: case 0x25u: case 0x26u: + case 0x28u: case 0x29u: case 0x2Au: case 0x2Bu: + case 0x2Eu: + case 0x30u: case 0x32u: case 0x38u: case 0x3Au: + return true; + default: + return false; + } +} + +static uint32_t exact_branch_target(uint32_t pc, uint32_t instr) { + int16_t imm = static_cast(instr & 0xFFFFu); + return pc + 4u + (static_cast(imm) << 2); +} + +static uint32_t exact_jump_target(uint32_t pc, uint32_t instr) { + return ((pc + 4u) & 0xF0000000u) | ((instr & 0x03FFFFFFu) << 2); +} + +static ExactCf exact_classify_cf(uint32_t pc, uint32_t instr) { + ExactCf cf; + uint32_t opcode = (instr >> 26) & 0x3Fu; + uint32_t funct = instr & 0x3Fu; + uint32_t rs = (instr >> 21) & 0x1Fu; + + if (opcode == 0x00u && funct == 0x08u) { + cf.kind = (rs == 31u) ? ExactCfKind::JrRa : ExactCfKind::JrOther; + return cf; + } + if (opcode == 0x00u && funct == 0x09u) { + cf.kind = ExactCfKind::Jalr; + return cf; + } + if (opcode == 0x02u) { + cf.kind = ExactCfKind::Jump; + cf.target = exact_jump_target(pc, instr); + return cf; + } + if (opcode == 0x03u) { + cf.kind = ExactCfKind::Jal; + cf.target = exact_jump_target(pc, instr); + return cf; + } + if (opcode == 0x01u || (opcode >= 0x04u && opcode <= 0x07u) || + (opcode >= 0x14u && opcode <= 0x17u)) { + cf.kind = ExactCfKind::Branch; + cf.target = exact_branch_target(pc, instr); + return cf; + } + return cf; +} + +} // namespace + +FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(const std::vector& entries) { + FunctionAnalysisResult result; + result.total_instructions = 0; + result.jr_ra_count = 0; + result.prologue_count = 0; + result.call_discovered_count = 0; + result.state_continuation_count = 0; + + fmt::print("\n=== Exact Overlay Function Analysis ===\n\n"); + + auto in_exe = [&](uint32_t addr) { + return addr >= exe_.header.load_address && addr < exe_.end_address() && (addr & 3u) == 0; + }; + + auto callable_direct_jal_target = [&](uint32_t addr) { + auto word_opt = exe_.read_word(addr); + if (!word_opt.has_value() || !exact_is_valid_mips_word(*word_opt)) return false; + if (addr == exe_.header.load_address) return true; + + if (addr >= exe_.header.load_address + 8u) { + auto prev8_opt = exe_.read_word(addr - 8u); + if (prev8_opt.has_value() && exact_is_jr_ra_word(*prev8_opt)) return true; + } + + if (exact_is_addiu_sp_neg(*word_opt)) { + if (addr < exe_.header.load_address + 4u) return true; + auto prev_opt = exe_.read_word(addr - 4u); + if (!prev_opt.has_value()) return true; + return exact_classify_cf(addr - 4u, *prev_opt).kind == ExactCfKind::Normal; + } + + return false; + }; + + auto find_jump_table_targets = [&](uint32_t entry, uint32_t hard_cap, + uint32_t jr_pc, uint32_t jr_rs) { + std::vector targets; + uint32_t lw_base = 0xFFu; + int32_t lw_offset = 0; + uint32_t addu_cand[2] = {0xFFu, 0xFFu}; + uint32_t lui_val = 0; + int16_t addiu_val[2] = {0, 0}; + bool found_addiu[2] = {false, false}; + bool found_lui = false; + uint32_t table_count = 0; + + for (int back = 1; back <= 40; back++) { + uint32_t scan_addr = jr_pc - static_cast(back * 4); + if (scan_addr < entry) break; + auto scan_opt = exe_.read_word(scan_addr); + if (!scan_opt.has_value()) break; + + uint32_t instr = *scan_opt; + uint32_t op = (instr >> 26) & 0x3Fu; + uint32_t rs = (instr >> 21) & 0x1Fu; + uint32_t rt = (instr >> 16) & 0x1Fu; + uint32_t rd = (instr >> 11) & 0x1Fu; + uint32_t fn = instr & 0x3Fu; + + if (op == 0x23u && rt == jr_rs && lw_base == 0xFFu) { + lw_base = rs; + lw_offset = static_cast(static_cast(instr & 0xFFFFu)); + continue; + } + if (op == 0x00u && fn == 0x21u && rd == lw_base && lw_base != 0xFFu && + addu_cand[0] == 0xFFu) { + addu_cand[0] = rs; + addu_cand[1] = rt; + continue; + } + if (op == 0x09u && addu_cand[0] != 0xFFu) { + for (int c = 0; c < 2; c++) { + if (!found_addiu[c] && addu_cand[c] != 0xFFu && + rs == addu_cand[c] && rt == addu_cand[c]) { + addiu_val[c] = static_cast(instr & 0xFFFFu); + found_addiu[c] = true; + break; + } + } + continue; + } + if (op == 0x0Fu && addu_cand[0] != 0xFFu && !found_lui) { + for (int c = 0; c < 2; c++) { + if (addu_cand[c] != 0xFFu && rt == addu_cand[c]) { + lui_val = (instr & 0xFFFFu) << 16; + if (!found_addiu[c]) addiu_val[c] = 0; + addiu_val[0] = addiu_val[c]; + found_addiu[0] = found_addiu[c]; + found_lui = true; + break; + } + } + continue; + } + if (op == 0x0Bu && table_count == 0) { + table_count = instr & 0xFFFFu; + continue; + } + if (found_lui && table_count != 0) break; + } + + if (!found_lui || table_count == 0 || table_count >= 512) return targets; + + uint32_t table_base = lui_val + + (found_addiu[0] ? static_cast(static_cast(addiu_val[0])) : 0u) + + static_cast(lw_offset); + for (uint32_t i = 0; i < table_count; i++) { + auto entry_opt = exe_.read_word(table_base + i * 4u); + if (!entry_opt.has_value()) continue; + uint32_t target = *entry_opt; + if (target >= entry && target < hard_cap && in_exe(target)) { + targets.push_back(target); + } + } + return targets; + }; + + auto walk = [&](uint32_t entry, uint32_t hard_cap) { + ExactWalkResult wr; + std::queue work; + work.push(entry); + + auto in_function = [&](uint32_t addr) { + return in_exe(addr) && addr >= entry && addr < hard_cap; + }; + + while (!work.empty()) { + uint32_t pc = work.front(); + work.pop(); + + if (wr.visited.count(pc) || !in_function(pc)) continue; + auto word_opt = exe_.read_word(pc); + if (!word_opt.has_value()) continue; + + uint32_t instr = *word_opt; + wr.visited.insert(pc); + ExactCf cf = exact_classify_cf(pc, instr); + uint32_t delay = pc + 4u; + + switch (cf.kind) { + case ExactCfKind::Normal: + if (in_function(pc + 4u)) work.push(pc + 4u); + break; + case ExactCfKind::Branch: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(pc + 8u)) work.push(pc + 8u); + if (in_function(cf.target)) work.push(cf.target); + break; + case ExactCfKind::Jump: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(cf.target)) work.push(cf.target); + break; + case ExactCfKind::Jal: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(pc + 8u)) work.push(pc + 8u); + if (in_exe(cf.target) && callable_direct_jal_target(cf.target)) { + wr.direct_jal_targets.insert(cf.target); + } + break; + case ExactCfKind::Jalr: + if (in_function(delay)) wr.visited.insert(delay); + if (in_function(pc + 8u)) work.push(pc + 8u); + break; + case ExactCfKind::JrRa: + if (in_function(delay)) wr.visited.insert(delay); + wr.jr_ra_count++; + break; + case ExactCfKind::JrOther: + if (in_function(delay)) wr.visited.insert(delay); + { + uint32_t jr_rs = (instr >> 21) & 0x1Fu; + for (uint32_t target : find_jump_table_targets(entry, hard_cap, pc, jr_rs)) { + wr.jump_table_targets.insert(target); + if (in_function(target)) work.push(target); + } + } + break; + } + } + return wr; + }; + + std::set known_entries; + std::queue pending; + for (uint32_t entry : entries) { + if (!in_exe(entry)) continue; + if (known_entries.insert(entry).second) pending.push(entry); + } + + size_t explicit_count = known_entries.size(); + fmt::print("Explicit entries: {}\n", explicit_count); + + std::set processed; + while (!pending.empty()) { + uint32_t entry = pending.front(); + pending.pop(); + if (processed.count(entry)) continue; + processed.insert(entry); + + auto next_it = known_entries.upper_bound(entry); + uint32_t hard_cap = (next_it != known_entries.end()) ? *next_it : exe_.end_address(); + ExactWalkResult wr = walk(entry, hard_cap); + + for (uint32_t target : wr.direct_jal_targets) { + if (known_entries.insert(target).second) pending.push(target); + } + } + + result.call_discovered_count = static_cast(known_entries.size() - explicit_count); + fmt::print("Direct-JAL entries: {}\n", result.call_discovered_count); + fmt::print("Total exact entries: {}\n\n", known_entries.size()); + + std::vector starts_vec(known_entries.begin(), known_entries.end()); + std::sort(starts_vec.begin(), starts_vec.end()); + + for (size_t i = 0; i < starts_vec.size(); i++) { + uint32_t entry = starts_vec[i]; + uint32_t hard_cap = (i + 1 < starts_vec.size()) ? starts_vec[i + 1] : exe_.end_address(); + ExactWalkResult wr = walk(entry, hard_cap); + if (wr.visited.empty()) continue; + + Function func; + func.start_addr = entry; + func.end_addr = *wr.visited.rbegin() + 4u; + if (func.end_addr > hard_cap) func.end_addr = hard_cap; + if (func.end_addr <= func.start_addr) func.end_addr = func.start_addr + 4u; + func.size = func.end_addr - func.start_addr; + func.name = fmt::format("func_{:08X}", func.start_addr); + + auto first_instr = exe_.read_word(func.start_addr); + if (first_instr.has_value()) { + int32_t stack_size; + func.has_prologue = is_prologue(*first_instr, stack_size); + func.stack_frame_size = func.has_prologue ? stack_size : 0; + if (func.has_prologue) result.prologue_count++; + } else { + func.has_prologue = false; + func.stack_frame_size = 0; + } + + func.has_epilogue = wr.jr_ra_count > 0; + func.is_data_section = false; + + result.jr_ra_count += static_cast(wr.jr_ra_count); + result.total_instructions += static_cast(wr.visited.size()); + result.functions.push_back(func); + } + + return result; +} + + FunctionAnalysisResult FunctionAnalyzer::analyze() { FunctionAnalysisResult result; result.total_instructions = 0; diff --git a/recompiler/src/main_psx.cpp b/recompiler/src/main_psx.cpp index dde3ae91..9494ab7f 100644 --- a/recompiler/src/main_psx.cpp +++ b/recompiler/src/main_psx.cpp @@ -40,6 +40,7 @@ int main(int argc, char** argv) { std::string extra_funcs_storage; // lifetime anchor for the .c_str() below const char* extra_funcs_path = nullptr; bool inspect_mode = false; + bool overlay_mode = false; std::filesystem::path out_dir = "generated"; if (!config_path.empty()) { @@ -72,6 +73,13 @@ int main(int argc, char** argv) { * with psxrecomp-bios and project scripts. */ } else if (arg == "--inspect") { inspect_mode = true; + } else if (arg == "--overlay") { + /* Overlay-compilation contract: this input is a runtime-captured + * overlay with execution evidence, so discovery is evidence-scoped + * (no whole-byte sweep) and branch/jump-table targets stay as + * in-parent labels. Set unconditionally by compile_overlays.py for + * every overlay input — never a human toggle. */ + overlay_mode = true; } } } @@ -197,7 +205,11 @@ int main(int argc, char** argv) { fmt::print("Performing function analysis...\n"); PSXRecomp::FunctionAnalyzer analyzer(*exe); - analyzer.add_forced_entry(exe->header.initial_pc); + std::vector exact_entries; + exact_entries.push_back(exe->header.initial_pc); + if (!overlay_mode) { + analyzer.add_forced_entry(exe->header.initial_pc); + } // Forced entry points: functions called from the dispatch table but not // automatically detected by the heuristic-based function analysis. @@ -213,7 +225,10 @@ int main(int argc, char** argv) { if (line.empty() || line[0] == '#') continue; uint32_t addr = (uint32_t)std::strtoul(line.c_str(), nullptr, 16); if (addr >= 0x80010000u && addr < 0x80200000u) { - analyzer.add_forced_entry(addr); + exact_entries.push_back(addr); + if (!overlay_mode) { + analyzer.add_forced_entry(addr); + } extra_count++; } } @@ -223,7 +238,9 @@ int main(int argc, char** argv) { } } - auto analysis_result = analyzer.analyze(); + auto analysis_result = overlay_mode + ? analyzer.analyze_exact_entries(exact_entries) + : analyzer.analyze(); // Print summary statistics fmt::print("\n=== Function Analysis Summary ===\n\n"); @@ -344,6 +361,7 @@ int main(int argc, char** argv) { PSXRecomp::CodeGenConfig codegen_config; codegen_config.emit_comments = true; codegen_config.emit_line_numbers = true; + codegen_config.split_mid_function_targets = !overlay_mode; // Load per-game annotations: annotations/_annotations.csv // Silently skipped if the file doesn't exist. diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index 4c878263..446039b2 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -956,15 +956,16 @@ def main(): out_dir_tmp = os.path.join(tmp, 'out') os.makedirs(out_dir_tmp) - # Run psxrecomp-game. - # NOTE: --overlay-exact was removed (the pivot). Overlays now compile - # via the normal whole-blob analysis, which splits mid-function branch - # targets into standalone functions. That is the original (buggy) - # behavior we are reproducing; the fix moves to ingester-side - # parent-merge of mid-function seeds, not scan-time exclusion. + # Run psxrecomp-game in --overlay mode (always, for every overlay + # input). Evidence-scoped discovery: compile only the proven entry + # seeds and the code reachable from them; never whole-byte sweep + # (which decodes embedded data tables as code). Branch/jump-table + # targets stay as in-parent labels, not standalone functions. This + # is the overlay-compilation contract, not a tunable. cmd = [args.recompiler, psx_path, '--seeds', seeds_path, - '--out-dir', out_dir_tmp] + '--out-dir', out_dir_tmp, + '--overlay'] print(f' recompile: {args.recompiler} ...') toml_dir = os.path.dirname(os.path.abspath(args.game_toml)) r = subprocess.run(cmd, capture_output=True, text=True,