5 Commits

Author	SHA1	Message	Date
Shokunin	6a76eb452b	recompiler: compile segment variants of game code from segment-qualified seeds ... A seed that names the image's bytes in another segment than its link segment (0xA00100F0 for a KUSEG-linked EXE, 0x00012345 for a KSEG0 one) now compiles a segment variant (docs/SEGMENT_AWARE_CODE.md §5.4): the home functions the seed reaches through direct edges (calls, tail jumps, split pieces, fall-throughs, call returns; a seed at a CPS continuation roots its owner, an alias entry its alias group) are compiled again through a view of the image in that segment. The variant's names, rows and every PC it bakes are that segment's, and a KSEG1 variant charges a fetch per instruction (#429's rule through B's runtime_pc()). A cached variant is the home C with the segment swapped. Exact-match config sites move into the variant's segment, so a variant gets every hook the home body gets. A seed that names no home dispatch entry stops the build, and so does a seed in a segment that does not map physical memory (0x20000000-0x7FFFFFFF or KSEG2: Beetle's addr_mask leaves them unmasked, so they address no RAM and are no alias of the image's bytes). tools/collect_game_misses.py reports such a segment miss instead of writing it as a seed. The dispatch table holds a row per compiled segment of a word, adjacent and in PC order; when any word has more than one row the lookup (indexed and binary search) searches the word's rows for the exact PC. Two rows for one PC remain a build error. A title without variants gets the same dispatch source as before. A jump-table case whose value is in another segment than its body is no longer a local case: the default tail-transfer dispatches it with its full PC. This applies to every compile. In a variant it is the common case (a KSEG0 variant of KUSEG-linked code). A home or overlay compile has one only when its table holds another segment's PCs of its own bytes, for example a KSEG0-compiled overlay shard of code that runs at KUSEG. Such a case used to continue in the KSEG0 body with KSEG0 links and fetch tags. Now it dispatches: a game-text PC with no row in its segment is a recorded segment miss, and #417's gate sends an overlay PC to the interpreter; both run it at its own PC (PR E returns overlay code to native per-segment shards). R4's game C and its 26 overlay shard sources are unchanged. Ledger: segment-variants and kseg1-fetch-charge close (KNOWN_GAPS is empty) and become regression guards; resume-pc-segment now requires each row's key and resume PC to be in its body's segment, and segment-variants checks every PC a variant body bakes. New segment_variants_test; kuseg_dispatch_lookup and emitter_runtime_pc_test cover variant rows, closure, equality with the home C, per-instruction KSEG1 charging, foreign jump-table cases at home and in variants, data stubs (no row), alias groups declared for every shard and seeds outside the physical segments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-30 00:24:07 -07:00
Shokunin	d67f0f9396	recompiler: compile a KUSEG-linked EXE for its link segment; exact dispatch ... A PS-X EXE header may carry KUSEG addresses (Alien Resurrection SLUS-00633, Kula World SCES-01000); the BIOS then runs the program at KUSEG PCs, so its links, EPCs and I-cache tags are KUSEG. The parser folded such a header to KSEG0, so the whole program compiled for a segment it never runs in, and the dispatch lookup matched any segment by physical address (docs/SEGMENT_AWARE_CODE.md §3.2, §5.3, §5.5). - EXE parser: keeps the header's addresses. link_segment() is the load address's segment; the load address and entry must share it and lie in the 8 MiB RAM window of KUSEG, KSEG0 or KSEG1 (KSEG1 compiles as an uncached home segment). phys_load_address()/contains_phys() do byte addressing; the BSS overlap check and the text_size bound are physical. - function_analysis: the three JAL scans take the target's top bits from the delay slot's PC instead of assuming KSEG0 (unchanged for KSEG0). - Seeds are range-checked by physical address. A seed in the link segment is an entry (0x000100E4 was silently dropped before). A seed for the same bytes in another segment asks for a segment variant (§5.4): it is listed in a WARNING, never folded into the home body. Seeds outside the image are counted. The producer_range, cross_call_allow and hosted_interior directives have no variant form: a foreign spelling is refused with the link-segment one instead of a bare "invalid" error. - Config code sites that are matched exactly (widescreen and cull sites, mod hooks, hot/data-shard funcs, the VSync-query function and its event-horizon return PCs, persisted options) and name the image's bytes in another segment are refused with the spelling that matches, since they would now match nothing. Kinds matched by physical address (byte patches, the full-word-guarded cull keep/angle/aspect-cone and signed-X-bound sites, dome call sites) reach the same bytes in any spelling and are not refused. Overlay compiles are exempt until PR E. - tools/collect_game_misses.py wrote every dirty-RAM miss as a KSEG0 seed, which a KUSEG-linked image now reports instead of compiling. It writes them in the link segment (--segment, --game-toml, or a segmented text range; it refuses to guess from a physical one) and adds the segment_misses rows at their full PC. - Dispatch source moves to game_dispatch_emitter.cpp. The lookup indexes the physical word, then requires row.addr == addr; a second row for one word is an error until variants share the table. psx_game_address_in_text stays physical, so a segment miss takes the clean-text-miss path (the previous commit records it). For a KSEG0 title the game C is byte-identical; its dispatch file changes only in psx_game_find_entry (the exact-PC compare) and its comment. Tests: - segment_aware_codegen closes link-segment, fetch-tag-segment, irq-resume-segment, resume-pc-segment, store-pc-segment, home-seed-accepted, alias-fetch-coherence and segment-miss; they stay as regression guards, and resume-pc-segment now also checks the dispatch rows' keys and resume PCs. - kuseg_dispatch_lookup: KUSEG rows, names and baked PCs; aliases miss while text stays physical; variant seeds reported; a function reached only by jal and a SaveState-style continuation are discovered in a KUSEG image (kills the two discovery scans reverted to KSEG0; the third, the invalid-JAL data heuristic, reads the same bits either way); mixed-segment headers rejected; a KSEG1 image charged per instruction; foreign config sites refused; the RAM-mirror contract as before. - kuseg_dispatch_lookup also compiles the lookup's binary-search form (a table spanning 2 MiB or more) and queries every segment's alias of each row; refuses a KSEG0 event-horizon site and KSEG0 seeds directives with their KUSEG spelling; accepts a KSEG0 signed_x_bound site and applies it. - emitter_runtime_pc_test: the dispatch rows follow runtime_pc() and keep the compile identity in func_ names (kills B's two surviving mutants: 76 and 12 failing checks); two rows on one physical word are an error. - collect_game_misses (new, tools/tests): link-segment seeds from each source, segment misses at their full PC, no guessing, no duplicates, against a fake server using the runtime's own summary row format. - reachable_discovery_test: the parser keeps KUSEG, accepts KSEG1, rejects mixed segments and non-RAM load addresses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-09-29 21:57:34 -07:00
Matthew Stanley	80ff96780a	recompiler: the overlay guard word is a delay-slot source, never a block leader ... The overlay capture format DELIBERATELY appends one coherent guard instruction past the end of a dirty-page run (overlay_capture.c write_json_window, `size += 4u`, pinned by runtime/tests/test_interpreter_perf_guards.py:47): a MIPS branch sitting at the run's final word (...FFC) always executes its delay slot in the next page (...000), and the shard must be able to emit and hash that word. So a capture whose size is a page multiple PLUS 4 is a CORRECT capture -- N bytes of overlay plus one guard word. Measured: 36 of 36 Ape Escape (SCUS-94423) vault records and 846 of 946 Tomba 2 (SCUS-94454) records are that shape. The recompiler was never taught the asymmetry: the guard word is a legal DELAY-SLOT SOURCE but an ILLEGAL BLOCK LEADER, because the word after it does not exist in the image. It had exactly one bound -- end_address() -- used both to read words and to decide what is code. MEASURED on Ape Escape, its own single-source vault at framework v0.3.2-alpha-139-g1bf70960, 36 variants: PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0 ! [recompiler] overlay 0x80136000 crc 4BAB23C9: what(): cannot emit control flow at 0x80138000: mandatory delay slot 0x80138004 is outside the input image + 16 x [fragment] interior 0x801360.. @region 0x00136000, identical message One root cause, seventeen shards. The whole region then falls back to the dirty-RAM interpreter forever, and nothing at runtime says so. Causal chain, every step read in source: 1. overlay_capture.c:349-361 appends the guard word; region 0x80136000 gets size 0x2004, so the image is [0x80136000, 0x80138004). 2. compile_overlays.py make_psxexe wraps those bytes verbatim with text size = len(data), so PS1Executable::end_address() = 0x80138004. 3. function_analysis.cpp in_exe / in_function admit 0x80138000 as ordinary code, and the walk reaches it by fall-through from 0x80137FFC (0x24022000, addiu -- not control flow), so func.end_addr becomes the image end. 4. control_flow.cpp makes the final block end_addr = func.end_addr - 4 = 0x80138000: the block LEADER is the guard word. 5. control_flow.cpp's "Also check last instruction itself" promotion finds 0x80137FFC is not control flow and therefore promotes read_word(end_addr) -- the guard word 0x1062005A, a beq -- to block.exit_instr, with has_delay_slot = true and address == end_addr. THIS IS THE DEFECT. 6. code_generator.cpp computes delay_slot_addr = 0x80138004, read_word returns nullopt, and it throws. THE THROW AT STEP 6 IS CORRECT AND IS PRESERVED. Emitting a control transfer without its architectural delay slot silently corrupts guest state, which is what the code did before that throw was added. The gap is step 5. Fix: give the image view two distinct bounds. end_address() READ bound -- every byte the image supplies, so a mandatory delay slot living at a guard word resolves. analysis_end_address() ANALYSIS bound -- one past the last byte that may be discovered, made a block leader, or made a block's exit instruction. read_word() keeps the READ bound. Discovery and block construction move to the ANALYSIS bound: function_analysis.cpp in_exe / in_exe_p3 / add_forced_entry / every hard_cap that used to be the image end / the jr-table producer_hi defaults / the static jr-ra and JAL sweeps; control_flow.cpp gains analysis_walk_hi(func) and uses it for cfg.function_end, alias entries, add_boundary, the scan loop, the final block's end_addr and in_func; code_generator.cpp's mid-function split pre-pass; main_psx.cpp seed admission and the data-scan pointer promotion; analysis_db.cpp, main_toml.cpp, main_cli.cpp target/seed admission. The clamp is applied only when it leaves a non-empty walk, so a degenerate function starting at or past the bound keeps failing closed rather than silently producing an empty CFG. HOW THE BOUND IS DERIVED -- the producer STATES it; the recompiler is told. The alternative was for the recompiler to infer the count from `size % 4096 == 4`. That fits every capture the current writer has ever produced, but it is a consumer guessing at a producer's intent, and this tree already has the counter-example: 52 of the 946 SCUS-94454 records have residues from 112 to 3996 bytes (one not even 4-byte aligned) because they come from the STATIC extractor, which records an exact file extent and appends no guard word. A rule keyed to the page-run shape is a rule that shifts the moment a producer changes. So the count travels with the image, in a magic-tagged field in the otherwise zero-filled tail of the 2048-byte PS-EXE header (exe_tag in ps1_exe_parser.h: "PSXRGRD1" at 0x7E0, uint32 count at 0x7E8). Putting it in the header rather than on the command line means every consumer gets it automatically and no future invocation path can forget to pass it -- compile_overlays has three make_psxexe sites and five recompiler spawn sites. An absent or malformed tag means ZERO guard bytes, which is exactly what every genuine PS-X EXE is: the BIOS and every retail main executable have no guard word, their analysis bound equals their read bound, and their analysis is bit-for-bit unchanged. Producer side: overlay_capture.c writes "guard_bytes": 4|0 per region -- the writer stating its own intent, additive to the v2 schema and preserved through coverage_vault.py compaction. compile_overlays.py capture_guard_bytes(): prefer that field; for captures recorded before it existed, reconstruct the single shape write_json_window can produce (page multiple + 4) and return 0 for everything else, which is the fail-closed answer -- an unrecognised guard word is analysed as code and the step-6 throw still fires loudly at generation time. make_psxexe guard_bytes is keyword-only with NO default, so a new call site is a TypeError, not a silently mis-analysed shard. The count is threaded through compile_fragment_batch / compile_interior_fragment / generate_interior_fragment_static and into the fragment fail-memo key (bumped to v3, so a memo recorded under the old analysis cannot suppress a retry under the corrected one). SECOND-ORDER DEFECT FIXED IN THE SAME PASS. generate_ranges_manifest extends a range by +4 for any block whose exit instruction IS its last word, so the cloned delay slot joins the candidate CRC and the page-generation watch. That extension was unclamped. Reachable TODAY, independent of the guard word: a branch-likely opcode (0x14-0x17, RESERVED on the R3000A) as the image's very last word is short-circuited by translate_basic_block into an inline RI raise BEFORE any delay slot is read, so generation succeeds and the manifest claims 4 bytes past the image end. Measured on origin/master with a 28-byte synthetic image: `R 80010000 20` for an image ending at 0x8001001C. The cache would then validate the shard against bytes the shard never saw. Now clamped to the READ bound -- not the analysis bound, because a guard word genuinely IS compiled in when it serves as a delay slot and must participate in the CRC. DEAD CODE, documented rather than trusted: basic_block.cpp's BasicBlockAnalyzer bounds its walk correctly but nothing in the tree constructs it. It is still compiled and still hashed into the codegen tag, so a change there invalidates caches without moving an emitted byte. Its bounds move to the analysis bound too -- wiring it up must not reintroduce this defect -- and a header comment now says where the live walker actually is. TESTS. Both registered, so ctest runs them and the configure-time registration guard is satisfied. recompiler/tests/test_overlay_guard_word_analysis_bound.py drives psxrecomp-game over five synthetic images: A tagged, 2 pages + guard word, last real word not control flow, guard word IS a beq -- the measured Ape shape. Must generate; the guard word must not be emitted as an instruction; the block must publish 0x80012000 for the runtime to dispatch; no range may claim it. B byte-for-byte the same payload with NO tag. Must still fail closed with the mandatory-delay-slot diagnostic naming 0x80012000. A and B differ only in the header tag, which proves the tag is what changed behaviour and that the throw was not weakened. B2 a misaligned declaration (6). Must be rejected and behave like B. C tagged, with a real branch at the last analysable word. The guard word must be emitted AS the delay slot and must be covered by the manifest. D branch-likely as the final word. Generation must succeed and no range may exceed the image end. Against origin/master's psxrecomp-game the same file reports exactly the two things this commit changes, and passes B, B2 and C: FAIL: tagged guard word still fails generation: what(): cannot emit control flow at 0x80012000: mandatory delay slot 0x80012004 is outside the input image FAIL: range [0x80010000,0x80010020) claims 4 byte(s) past the image end 0x8001001C tools/tests/test_overlay_guard_bytes_declaration.py pins the producer half: field precedence, the reconstruction table, rejection of nonsense declarations, the tag's byte offsets, that make_psxexe with guard_bytes=0 is byte-identical to the pre-fix wrapper, and that overlay_capture.c both appends the guard word and now declares it. MEASURED OUTCOME, Ape Escape, same vault, same flags, fresh cache: before PSX_SHARD_RESULT ok=50 failed=17 skipped=32 capacity_fastpath=0 after PSX_SHARD_RESULT ok=31 failed=0 skipped=57 capacity_fastpath=0 ok falls because 19 isolated island fragment shards at 0x00136000 are no longer needed once the region variant compiles as one shard. Coverage is what matters, and it is measured from the DLL exports: 349 -> 361 unique func_XXXXXXXX entries, ZERO lost, 12 gained -- 0x80136140, 0x8013614C, 0x80136154, 0x80136238, 0x801362D4, 0x801362DC, 0x801362F0, 0x80136324, 0x80136370, 0x80136400, 0x80136480, 0x80137950, all from the failure list. The other four failing PCs were already served by a sibling variant's shard. INERTNESS. BIOS (SCPH1001, psxrecomp-bios --config bios/SCPH1001.toml): all four generated artifacts BYTE-IDENTICAL, verified by sha256 rather than by exit code, because tools/regen_bios.sh exits 0 on failure. Tomba 2 main EXE (psxrecomp-game --config game.toml): all nine generated artifacts BYTE-IDENTICAL, and the console log is identical too. Tomba 2 overlays, 82 variants over 6 regions from its own vault: identical result line (ok=98 failed=0 skipped=103) and identical shard identity set, 4901 unique func exports on both sides -- zero gained, zero lost. 97 of 98 range manifests are byte-identical. The one that differs is region 0x8010A000 variant A32DB188, size 0x2004, whose last function's range goes from 0x218 to 0x214 bytes: its guard word (0x24840008, addiu) had been emitted as an ordinary instruction and hashed, and now is not. That is precisely the intended change and nothing else moved. Bead beads-eio.3.100. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-02 15:35:24 -07:00
Matthew Stanley	bd705bfcec	Add bounded reachable main-EXE discovery ... Adapt Douglas's opt-in reachable discovery onto current exact/static dispatch. Validate configured static-analysis bounds before truncation, preserve the whole-image default and canonical page reservations, and leave unresolved indirect or out-of-bound targets to runtime fallback. Source: https://github.com/douglasjv/mm8 Source-commit: https://github.com/douglasjv/mm8/commit/8415f3a95458c41c7f48fbe36caaf0ee82730720 Co-authored-by: douglasjv <vanderveerdouglas@gmail.com>	2026-07-13 21:56:43 -07:00
Matthew Stanley	7c40d33db3	Phase 1 complete: recompiler proven correct on SCPH1001.BIN ... Phase 1a (boot slice): 30 instructions at 0xBFC00000, compiles clean. Phase 1b (instruction coverage): 54 opcodes implemented in strict_translator, 44,629 walker-reachable instructions covered, 0 missing. Phase 1c (function discovery): 76 functions from 2 seeds, 2,988 instructions, 199 call edges, 27 indirect jumps. All validated through StrictTranslator. Phase 1d (indirect control flow): 27 sites classified — 21 A0/B0/C0 dispatch stubs, 4 computed calls, 2 computed tail calls. Ghidra cross-referenced with 0 disagreements. Phase 1e (relocation): ROM-to-RAM copy at FUN_bfc00420 proven with both Ghidra (static) and DuckStation (runtime). 69 address aliases, 68/69 function entries match byte-for-byte in RAM. Normalization rule documented. Includes PRINCIPLES.md and DEBUG.md adapted from recomp-template. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-11 21:26:04 -07:00