From 752b2e2634b55b21b89714bc7a543ada6394c1a5 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 13:02:44 -0700 Subject: [PATCH 01/13] docs: plan for overlay root enrichment after #386 Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/internal/OVERLAY_ROOT_ENRICHMENT.md | 84 ++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 docs/internal/OVERLAY_ROOT_ENRICHMENT.md diff --git a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md new file mode 100644 index 00000000..77310569 --- /dev/null +++ b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md @@ -0,0 +1,84 @@ +# Overlay root enrichment: next steps after #386 + +#386 added `tools/enrich_overlay_captures.py`. It scans each captured overlay +image for likely function starts and writes them to +`static_discovery_entry_pcs`. Nothing runs it automatically yet. This page is +the plan for making it the default and getting more coverage from it, without +any risk of softlocks. + +## Where we are + +- **Measured on Ace Combat 3** (x6568tank's captures, compile tooling fixed): + rootless images went from 250 to 1, and capacity skips from 3,028 to 0. + There were 8 shard failures, all `fragment: no-func-ids`, and all of them + also appear in the comparison tool. So no new failures. +- **A richer tool finds 47% more starts** (15,008 vs 10,191). That gave +31% + entries served and +19% covered bytes. It also gave 19 failures instead of + 8, a 38% larger cache and 39% more compile time. +- **Our titles' captures** have no `function_entry_pcs`, only + `dispatch_entry_pcs`. Enrichment would add: + + | Title | Dispatch entries | Starts added | + |---|---|---| + | Tomba | 38 | 764 | + | Tomba 2 | 178 | 3,316 | + | Mega Man X6 | 325 | 901 | + | Ape Escape | 149 | 459 | + +## The rule that matters + +A shard **failure** is safe. The compiler refuses the piece and it runs +interpreted. The real risk is a wrong start point that **passes** every check. +A walk root is a hard cap, so a false root in the middle of a function splits +that function. The output is valid C that passes every audit, but it behaves +wrong. That is the mid-function-seed softlock class, and it never shows up in +the failure count. + +So the failure count is not the safety metric. The goal is **zero +mis-splits**, then as much coverage as possible. + +## Plan + +1. **No-split guard** (in `compile_overlays.py`). + - Walk each candidate root without hard caps. + - A candidate that another root reaches by fallthrough or branch (not by + `jal`) is inside that function. Demote it to `DISPATCH_INTERIOR`, which + makes it an alias entry and never a cap. + - This extends the protection dispatch entries already get to every + static root. + - With this guard, a wrong guess can only cost speed, never correctness. +2. **Move enrichment into `compile_overlays.py`.** + - Call the same root derivation when building `pre_roots`, instead of + running a separate pipeline step. + - Add it to the overlay cache/recipe key, so shards built without it are + never reused. + - Keep the script for inspecting captures. +3. **Add more root categories, behind the guard.** These are the sources the + richer tool uses and #386 does not: + - `jal` edges from the main EXE and other images into the overlay. These + are real call evidence, and likely the safest win. + - Frameless starts right after `jr ra` + delay slot. These are riskiest; + `compile_overlays` itself says this spot can be data. + - Pointer tables with fewer than 3 entries, and `lui`/`addiu`-built + pointers. + + Measure each category on its own: roots added, entries served, failures. +4. **Explain the `no-func-ids` failures.** + - Find which requested entries produce an empty ranges manifest, and why + they were requested. + - Fix the request side so nothing unprovable is asked for. Do not + reclassify these as skips just to reach zero. +5. **Validate on Tomba, Tomba 2, Mega Man X6 and Ape Escape.** + - Run interpreted-only and enriched-native to the same frame, and compare + `frame_fingerprint` rings. They must match. + - Then play and explore areas that load new overlays: levels, menus, + cutscenes, save and load. + +## Done when + +- The no-split guard is in, with a test that a mid-function candidate is + demoted, not rooted. +- Enrichment is on by default in `compile_overlays.py`, and part of the + cache key. +- The four titles match interpreted-only fingerprints and pass a play check. +- The `no-func-ids` cause is known and fixed at the source. From 77086423426ebc045b1b519c0bd4ab08fe221531 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 19:13:03 -0700 Subject: [PATCH 02/13] overlays: no-split guard on every static root; enrichment default-on and in the cache key Co-Authored-By: Claude Opus 5.5 (1M context) --- recompiler/CMakeLists.txt | 12 + recompiler/include/function_analysis.h | 14 +- recompiler/src/function_analysis.cpp | 86 ++- recompiler/src/main_psx.cpp | 9 +- recompiler/tests/reachable_discovery_test.cpp | 77 +++ runtime/codegen_hash_sources.cmake | 10 + tools/compile_overlays.py | 498 +++++++++++++++--- tools/enrich_overlay_captures.py | 105 ++-- tools/tests/test_overlay_no_split_guard.py | 374 +++++++++++++ 9 files changed, 1046 insertions(+), 139 deletions(-) create mode 100644 tools/tests/test_overlay_no_split_guard.py diff --git a/recompiler/CMakeLists.txt b/recompiler/CMakeLists.txt index 179df0ae..06d01e26 100644 --- a/recompiler/CMakeLists.txt +++ b/recompiler/CMakeLists.txt @@ -581,6 +581,18 @@ if(BUILD_TESTING) COMMAND ${Python3_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/../tools/tests/test_enrich_overlay_captures.py WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..) + # A walk root is a hard cap: a root inside another function splits it + # and still passes every audit. The classifier's no-split guard, the + # image-local enrichment proof, the recompiler's own guard on explicit + # roots, and the root policy's place in the cache namespace + # (beads-eio.3.177). + add_test( + NAME overlay_no_split_guard + COMMAND ${Python3_EXECUTABLE} + ${CMAKE_CURRENT_SOURCE_DIR}/../tools/tests/test_overlay_no_split_guard.py + --cmake ${CMAKE_COMMAND} + --recompiler $ + WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..) # The runtime exe name must come from CMake, not be re-derived. add_test( diff --git a/recompiler/include/function_analysis.h b/recompiler/include/function_analysis.h index b006c712..1a8acec0 100644 --- a/recompiler/include/function_analysis.h +++ b/recompiler/include/function_analysis.h @@ -172,10 +172,22 @@ public: // reachable from them. Used by runtime-loaded overlays and opt-in main-EXE // reachable discovery. Unresolved jalr/indirect targets do not mint // functions; evidence-backed entries must be supplied explicitly. + // + // no_split_guard (overlay mode): an EXPLICIT entry that the entry below + // it reaches by fallthrough or branch -- walked with the entry's own cap + // removed -- is inside that function. It is absorbed like a derived + // call target and exported in absorbed_entries (an alias, never a cap). + // `trusted_entries` (classifier-promoted dispatch roots) are exempt. + // compile_overlays.py applies the same rule before writing seeds; this is + // the enforcement point for walk proofs only this analyzer can make + // (computed-stride and self-limited jump tables). Off by default so the + // main-EXE reachable-discovery partition is unchanged. FunctionAnalysisResult analyze_exact_entries( const std::vector& entries, const std::vector>& producer_ranges = {}, - const std::set& cross_call_allow = {}); + const std::set& cross_call_allow = {}, + bool no_split_guard = false, + const std::set& trusted_entries = {}); // Add a forced entry point address that is treated as a function start // even if it has no standard ADDIU $sp prologue. The function will be diff --git a/recompiler/src/function_analysis.cpp b/recompiler/src/function_analysis.cpp index 5b1564db..0705d6ed 100644 --- a/recompiler/src/function_analysis.cpp +++ b/recompiler/src/function_analysis.cpp @@ -1416,7 +1416,9 @@ bool resolve_self_limited_jump_table( FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries( const std::vector& entries, const std::vector>& producer_ranges, - const std::set& cross_call_allow) { + const std::set& cross_call_allow, + bool no_split_guard, + const std::set& trusted_entries) { FunctionAnalysisResult result; result.total_instructions = 0; result.jr_ra_count = 0; @@ -1626,6 +1628,54 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries( std::map>> derived_evidence; fmt::print("Explicit entries: {}\n", explicit_count); + // No-split guard: fixed point over EVERY non-trusted entry (explicit and + // derived); same rule as compile_overlays.no_split_partition. For each + // host, the run of following entries its UNCAPPED walk reaches (stopping + // at the first it does not reach, or a trusted entry) may be inside it; + // walk the host capped at the end of the run, shrink the run to the first + // entry that walk misses, and absorb the rest together. Absorbing the run + // at once lets a switch whose table targets several entries resolve + // (every target must lie inside the walk); stopping at the first + // unreached entry keeps a forward tail call over an unrelated function + // from orphaning its target. Absorbing only removes caps, so repeat. + auto absorb_split_entries = [&](std::set& entries) { + const uint32_t analysis_end = exe_.analysis_end_address(); + bool changed = true; + while (changed) { + changed = false; + std::vector roots(entries.begin(), entries.end()); + std::vector kept; + size_t index = 0; + while (index < roots.size()) { + const uint32_t host = roots[index]; + kept.push_back(host); + const ExactWalkResult reach = walk(host, analysis_end); + size_t end = index + 1; + while (end < roots.size() && reach.visited.count(roots[end]) && + !trusted_entries.count(roots[end])) + end++; + while (end > index + 1) { + const uint32_t cap = + end < roots.size() ? roots[end] : analysis_end; + const ExactWalkResult inner = walk(host, cap); + size_t blocked = end; + for (size_t k = index + 1; k < end; k++) { + if (!inner.visited.count(roots[k])) { + blocked = k; + break; + } + } + if (blocked == end) break; + end = blocked; + } + if (end > index + 1) changed = true; + index = end; + } + entries = std::set(kept.begin(), kept.end()); + } + }; + if (no_split_guard) absorb_split_entries(known_entries); + // Discover callees in rounds against one stable entry partition. Processing // roots sequentially let an early caller insert a JAL target that actually // lay inside a later explicit root. That new entry hard-capped the later @@ -1641,6 +1691,7 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries( fmt::print("WARNING: exact-entry ownership did not converge; " "falling back to explicit roots only\n"); known_entries = explicit_entries; + if (no_split_guard) absorb_split_entries(known_entries); derived_entries.clear(); derived_evidence.clear(); break; @@ -1678,7 +1729,9 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries( // "absorbed forever" set therefore loses reachable code. known_entries = explicit_entries; known_entries.insert(derived_entries.begin(), derived_entries.end()); - for (uint32_t target : derived_entries) { + if (no_split_guard) { + absorb_split_entries(known_entries); + } else for (uint32_t target : derived_entries) { auto target_it = known_entries.find(target); if (target_it == known_entries.end() || explicit_entries.count(target) || @@ -1699,8 +1752,15 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries( derived_entries == previous_derived) break; } - result.call_discovered_count = static_cast(known_entries.size() - explicit_count); + size_t explicit_kept = 0; + for (uint32_t entry : known_entries) + explicit_kept += explicit_entries.count(entry); + result.call_discovered_count = + static_cast(known_entries.size() - explicit_kept); fmt::print("Direct-JAL entries: {}\n", result.call_discovered_count); + if (no_split_guard) + fmt::print("No-split guard absorbed explicit entries: {}\n", + explicit_count - explicit_kept); fmt::print("Total exact entries: {}\n\n", known_entries.size()); std::vector starts_vec(known_entries.begin(), known_entries.end()); @@ -1782,6 +1842,26 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries( break; } } + // Explicit entries the no-split guard absorbed: the proof is the host's + // own walk (fallthrough/branch), so the host start is the source. A + // delay-slot target is not exported: an alias there would make the slot + // a block leader. It stays with the interpreter. + if (no_split_guard) { + for (uint32_t target : explicit_entries) { + if (final_starts.count(target)) continue; + if (target >= exe_.header.load_address + 4u) { + auto prev = exe_.read_word(target - 4u); + if (prev.has_value() && is_branch_or_jump(*prev)) continue; + } + for (const auto& [host, wr] : final_walks) { + if (!wr.visited.count(target)) continue; + result.absorbed_entries.push_back( + {target, host.start_addr, host.end_addr, + host.start_addr, false}); + break; + } + } + } return result; } diff --git a/recompiler/src/main_psx.cpp b/recompiler/src/main_psx.cpp index 25e7f72c..0286d93c 100644 --- a/recompiler/src/main_psx.cpp +++ b/recompiler/src/main_psx.cpp @@ -912,8 +912,15 @@ int main(int argc, char** argv) { { PSXRecomp::FunctionAnalyzer analyzer(*exe); std::vector roots_vec(roots.begin(), roots.end()); + // Overlay mode enforces the no-split guard on explicit roots too + // (compile_overlays.py applies the same rule first; this catches + // walk proofs only this analyzer makes). Classifier-promoted + // dispatch roots are exempt. Main-EXE reachable discovery keeps + // its existing partition. analysis_result = analyzer.analyze_exact_entries( - roots_vec, producer_ranges, cross_call_allow); + roots_vec, producer_ranges, cross_call_allow, + /*no_split_guard=*/overlay_mode, + overlay_mode ? trusted_root_seeds : std::set{}); } std::vector alias_entries; diff --git a/recompiler/tests/reachable_discovery_test.cpp b/recompiler/tests/reachable_discovery_test.cpp index 972a629c..8df90102 100644 --- a/recompiler/tests/reachable_discovery_test.cpp +++ b/recompiler/tests/reachable_discovery_test.cpp @@ -760,6 +760,83 @@ int main() { thunk_starts.count(kLoad + 0x610), "packed BIOS thunk bounds the following frameless leaf"); + // ---- No-split guard on EXPLICIT entries (overlay mode) ---------------- + // A root is a hard cap, so an explicit entry that the entry below it + // reaches by fallthrough splits that function (beads-eio.3.177: Tomba + // X00 root 0x8011B1CC, a subu mid-way through host 0x8011AC10). + { + auto split_image = make_exe_buffer(0x1000); + put32(split_image, text + 0x100, 0x27BDFFE0u); // addiu sp,sp,-32 + put32(split_image, text + 0x104, 0x24420001u); + put32(split_image, text + 0x108, 0x00431023u); // subu v0,v0,v1 + put32(split_image, text + 0x10C, 0x03E00008u); + put32(split_image, text + 0x110, 0x27BD0020u); + auto split_exe = parse(split_image); + PSXRecomp::FunctionAnalyzer unguarded(split_exe); + CHECK(starts(unguarded.analyze_exact_entries( + {kLoad + 0x100, kLoad + 0x108})) == + std::set({kLoad + 0x100, kLoad + 0x108}), + "without the guard an explicit mid-function entry still caps"); + PSXRecomp::FunctionAnalyzer guarded(split_exe); + const auto guarded_result = guarded.analyze_exact_entries( + {kLoad + 0x100, kLoad + 0x108}, {}, {}, true); + CHECK(starts(guarded_result) == std::set({kLoad + 0x100}), + "no-split guard absorbs an explicit entry reached by fallthrough"); + bool aliased = false; + for (const auto& absorbed : guarded_result.absorbed_entries) + aliased |= absorbed.addr == kLoad + 0x108 && + absorbed.host_start == kLoad + 0x100 && + absorbed.addr < absorbed.host_end; + CHECK(aliased, "absorbed explicit entry is exported as a hosted alias"); + + PSXRecomp::FunctionAnalyzer trusted(split_exe); + CHECK(starts(trusted.analyze_exact_entries( + {kLoad + 0x100, kLoad + 0x108}, {}, {}, true, + {kLoad + 0x108})) == + std::set({kLoad + 0x100, kLoad + 0x108}), + "trusted (promoted dispatch) entries are exempt from the guard"); + } + { + // Every case label of a switch is an explicit entry. Each one caps + // the host below its siblings, so the table (which resolves only when + // all of its targets are inside the walk) is only visible when the + // whole run of case labels is absorbed together. + auto sw = make_ape_switch(); + auto sw_exe = parse(sw.image); + std::vector entries = {sw.entry}; + entries.insert(entries.end(), sw.cases.begin(), sw.cases.end()); + PSXRecomp::FunctionAnalyzer guarded(sw_exe); + const auto result = guarded.analyze_exact_entries(entries, {}, {}, true); + CHECK(starts(result) == std::set({sw.entry}), + "no-split guard absorbs every case-label entry of a switch host"); + size_t hosted = 0; + for (const auto& absorbed : result.absorbed_entries) + hosted += absorbed.host_start == sw.entry; + CHECK(hosted == sw.cases.size(), + "every absorbed case label is hosted by the switch function"); + } + { + // A forward tail call over an unrelated function must not demote the + // tail-call target: the intervening root would leave it hostless. + auto tail_image = make_exe_buffer(0x1000); + put32(tail_image, text + 0x700, 0x08000000u | + (((kLoad + 0x740) >> 2) & 0x03FFFFFFu)); + put32(tail_image, text + 0x704, 0x00000000u); + put32(tail_image, text + 0x720, 0x03E00008u); + put32(tail_image, text + 0x724, 0x00000000u); + put32(tail_image, text + 0x740, 0x24020001u); + put32(tail_image, text + 0x744, 0x03E00008u); + put32(tail_image, text + 0x748, 0x00000000u); + auto tail_exe = parse(tail_image); + PSXRecomp::FunctionAnalyzer guarded(tail_exe); + CHECK(starts(guarded.analyze_exact_entries( + {kLoad + 0x700, kLoad + 0x720, kLoad + 0x740}, + {}, {}, true)) == + std::set( + {kLoad + 0x700, kLoad + 0x720, kLoad + 0x740}), + "tail call over an unrelated root keeps its target a root"); + } + PSXRecomp::FunctionAnalyzer seeded_analyzer(exe); const auto seeded = starts( seeded_analyzer.analyze_exact_entries({kLoad, kLoad + 0x200})); diff --git a/runtime/codegen_hash_sources.cmake b/runtime/codegen_hash_sources.cmake index 61ce3601..90b3ea35 100644 --- a/runtime/codegen_hash_sources.cmake +++ b/runtime/codegen_hash_sources.cmake @@ -54,6 +54,16 @@ set(PSXRECOMP_CODEGEN_HASH_SRCS ${PSXRECOMP_CODEGEN_HASH_ROOT}/recompiler/include/basic_block.h ${PSXRECOMP_CODEGEN_HASH_ROOT}/recompiler/include/code_generator.h + # --- Root policy ------------------------------------------------------------ + # compile_overlays.py decides which captured PCs become walk roots, alias + # entries or interpreter-only PCs (the seed classifier, the no-split guard + # and default-on root enrichment all live there). Roots are hard caps, so + # the same bytes and the same emitter produce DIFFERENT shards under a + # different root policy: a shard built before the no-split guard can carry + # a mis-split host (beads-eio.3.177). Hashing the classifier moves the cache + # namespace whenever the policy changes, so such shards are never reused. + ${PSXRECOMP_CODEGEN_HASH_ROOT}/tools/compile_overlays.py + # --- Shard COMPILE surface ------------------------------------------------- # The cg tag must invalidate on anything a shard compiles/links against, not # just what the emitter WRITES. A change to the overlay ABI header, the diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index 2230de2d..edf7a124 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -717,6 +717,26 @@ def optional_enrichment_fallback_capture(cap: dict) -> dict | None: return fallback +def conservative_retry_capture(cap: dict, seed_audit: dict) -> dict | None: + """Recipe to retry after the enriched recipe's generated C was rejected. + + Prefer the extractor's declared conservative recipe; either way derived + enrichment roots are dropped from the retry (ROOT_ENRICHMENT_OFF_KEY), so + a data-as-code root that enrichment introduced can only cost the roots it + added, never the region's whole shard. + """ + fallback = optional_enrichment_fallback_capture(cap) + if fallback is not None: + fallback[ROOT_ENRICHMENT_OFF_KEY] = True + return fallback + if seed_audit.get('derived_static_roots') and \ + not cap.get(ROOT_ENRICHMENT_OFF_KEY): + fallback = dict(cap) + fallback[ROOT_ENRICHMENT_OFF_KEY] = True + return fallback + return None + + def _word_at(data: bytes, load_addr: int, addr: int): off = addr - load_addr if off < 0 or off + 4 > len(data): @@ -1298,6 +1318,224 @@ def _frameless_dispatch_root_proven(data: bytes, load_addr: int, size: int, return plausible_callable_target(data, load_addr, size, addr, producer_hi) +# --------------------------------------------------------------------------- +# Static root derivation (enrichment) and the no-split guard +# --------------------------------------------------------------------------- +# +# A walk root is a hard cap: the walk of the root below it stops there. So a +# root that sits inside another function's body splits that function. The +# generated C still passes every audit (a cross-cap fallthrough becomes a call +# into the next "function"), which is why a mis-split never shows up as a shard +# failure. On Tomba it changed guest timing by one cycle against the +# interpreter (bead beads-eio.3.177, root 0x8011B1CC in X00.BIN). +# +# Two rules keep that from happening, for every root source: +# +# 1. The no-split guard (no_split_partition, applied inside +# classify_overlay_seeds). A candidate that a root below it reaches by +# fallthrough or branch (never by jal) -- walked without the caps of the +# candidates in between -- is inside that function. It is demoted to +# DISPATCH_INTERIOR, an alias entry that never caps. Applied to dispatch entries, captured function +# entries, static discovery roots, TOML entries, derived call targets and +# enrichment alike; only promoted kernel orphans (DISPATCH_ROOT) are +# exempt, because they are promoted precisely when no walk covers them. +# A candidate that sits in a delay slot is never a root and never an +# alias (entering a host at a delay slot would make the slot a block +# leader); it is left to the interpreter. +# +# 2. Image-local entry proof for enrichment (image_local_entry_proven). A +# jal or a pointer table is call evidence for the image that is resident +# when it executes, and overlay regions are swapped: shared engine code +# at 0x80100984 is byte-identical in 12 Tomba area images and calls +# 0x8011B1CC, which is a function start in exactly one of them. So a jal +# or table target becomes a root only when THIS image's bytes also prove +# a boundary there: a non-delay-slot stack-frame prologue, or a preceding +# `jr $ra` (plus alignment padding) together with the bounded CFG probe. +# A frameless target that only the CFG probe accepts is not rooted from +# enrichment; if a rooted walk calls it, the classifier's own derived +# DIRECT_JAL_TARGET path still finds it, under the guard. +# +# With both in place a wrong guess can only cost speed (an alias entry or an +# interpreted PC), never correctness. + +# Tool-owned capture key: set on a conservative retry recipe so enrichment is +# not derived again after the enriched recipe was rejected. +ROOT_ENRICHMENT_OFF_KEY = '_root_enrichment_disabled' + + +def root_enrichment_default() -> bool: + """Enrichment is on unless PSX_OVERLAY_ROOT_ENRICHMENT=0 (diagnostic). + + Read from the environment so process-pool workers (static mode) inherit + the same policy as the parent without extra plumbing.""" + return os.environ.get('PSX_OVERLAY_ROOT_ENRICHMENT', '1').strip() != '0' + + +def _in_delay_slot(data: bytes, load_addr: int, addr: int) -> bool: + """True when the word before ``addr`` is a delayed control transfer.""" + if addr - 4 < load_addr: + return False + prev = _word_at(data, load_addr, addr - 4) + return _is_valid_mips_word(prev) and _is_control_flow(prev) + + +def _return_precedes(data: bytes, load_addr: int, addr: int) -> bool: + """`jr $ra` + delay slot immediately before ``addr``, allowing only zero + alignment padding between them (the recompiler's callable_boundary rule: + Psy-Q pads leaf routines with one to six NOPs).""" + if _is_jr_ra(_word_at(data, load_addr, addr - 8)): + return True + for back in range(12, 36, 4): + if addr - back < load_addr: + return False + if _word_at(data, load_addr, addr - back + 8) != 0: + return False + if _is_jr_ra(_word_at(data, load_addr, addr - back)): + return True + return False + + +def image_local_entry_proven(data: bytes, load_addr: int, size: int, + addr: int, producer_hi: int, + analysis_hi: int | None = None) -> bool: + """Does THIS image's own byte content prove a function boundary at addr? + + Used for every enrichment source (prologue scan, jal targets, pointer + tables). A call or pointer elsewhere in the image is not accepted as proof + by itself; see the section comment above for why. + """ + hi = load_addr + size if analysis_hi is None else analysis_hi + if addr & 3 or not (load_addr <= addr < hi): + return False + word = _word_at(data, load_addr, addr) + if not word or not _is_valid_mips_word(word): + return False + if _dense_local_pointer_table(data, load_addr, addr): + return False + if _in_delay_slot(data, load_addr, addr): + return False + if _is_addiu_sp_neg(word): + return True + if _return_precedes(data, load_addr, addr): + return plausible_callable_target(data, load_addr, size, addr, + min(producer_hi, hi)) + return False + + +def derive_static_roots(data: bytes, load_addr: int, size: int, + producer_ranges=(), analysis_hi: int | None = None, + sources: dict | None = None) -> set[int]: + """Enrichment: independently proven function starts in one image. + + Candidates come from three generic sources -- stack-frame prologues, + direct jal targets, and runs of three or more in-image pointers -- and + every candidate must pass image_local_entry_proven. ``sources`` (optional) + receives {addr: set of source names} for inspection. + """ + hi = load_addr + size if analysis_hi is None else analysis_hi + ranges = sorted(producer_ranges or ()) + + def producer_hi_for(addr: int): + if not ranges: + return hi + for range_lo, range_hi in ranges: + if range_lo <= addr < range_hi: + return range_hi + return None + + candidates: dict[int, set] = {} + + def note(addr: int, source: str) -> None: + candidates.setdefault(addr, set()).add(source) + + pointers = [] + for addr in range(load_addr, hi, 4): + word = _word_at(data, load_addr, addr) + if _is_addiu_sp_neg(word): + note(addr, 'prologue') + if word is not None and (word >> 26) == 0x03: + note(_jump_target(addr, word), 'jal') + pointers.append(word if word is not None and not (word & 3) and + load_addr <= word < hi else None) + start = 0 + while start < len(pointers): + if pointers[start] is None: + start += 1 + continue + end = start + 1 + while end < len(pointers) and pointers[end] is not None: + end += 1 + if end - start >= 3: + for target in pointers[start:end]: + note(target, 'pointer_table') + start = end + + roots = set() + for addr, why in candidates.items(): + producer_hi = producer_hi_for(addr) + if producer_hi is None: + continue + if image_local_entry_proven(data, load_addr, size, addr, + producer_hi, hi): + roots.add(addr) + if sources is not None: + sources[addr] = set(why) + return roots + + +def no_split_partition(candidates, exempt, walk, hi: int): + """Fixed point of the no-split guard over one root partition. + + For each root h, ascending: + 1. Walk h with no cap at all (to the image / producer end). The run of + following candidates that this uncapped walk reaches -- stopping at + the first one it does not reach, or at an exempt root -- are the + candidates that may be inside h. + 2. Walk h capped at the end of that run. If some candidate in the run + is not reached (it is reachable only through a path beyond the + run), it stays a root and becomes the cap; shrink the run to it and + walk again. + 3. Every candidate left in the run is reached by h's walk with none of + their caps in the way: absorb them all. + Absorbing a whole run at once is what lets a switch host whose jump table + targets several candidates resolve that table (a table resolves only when + all of its targets lie inside the walk), and step 1 keeps a forward tail + call over an unrelated function from demoting the tail-call target, which + that intervening root would leave hostless. Absorption only removes caps, + so whole passes repeat until nothing more is absorbed. + + Returns (roots, {absorbed: owner}). + """ + roots = sorted(candidates) + absorbed = {} + changed = True + while changed: + changed = False + index = 0 + while index < len(roots): + host = roots[index] + reach = walk(host, hi)['visited'] + end = index + 1 + while (end < len(roots) and roots[end] in reach and + roots[end] not in exempt): + end += 1 + while end > index + 1: + cap = roots[end] if end < len(roots) else hi + inner = walk(host, cap)['visited'] + blocked = next((k for k in range(index + 1, end) + if roots[k] not in inner), None) + if blocked is None: + break + end = blocked + if end > index + 1: + for addr in roots[index + 1:end]: + absorbed[addr] = host + del roots[index + 1:end] + changed = True + index += 1 + return set(roots), absorbed + + def _walk_overlay_function(data: bytes, load_addr: int, size: int, entry: int, hard_cap: int, producer_ranges=(), @@ -1525,10 +1763,22 @@ def _collect_toml_overlay_entries(toml_doc: dict, load_addr: int, crc32: int, def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, - crc32: int, toml_doc: dict) -> tuple[list[str], dict]: + crc32: int, toml_doc: dict, + root_enrichment: bool | None = None + ) -> tuple[list[str], dict]: + """Classify one capture's entry evidence into recompiler seed records. + + ``root_enrichment`` None means root_enrichment_default(). A capture that + carries ROOT_ENRICHMENT_OFF_KEY (a conservative retry recipe) is never + enriched. The no-split guard is not optional. + """ lo = load_addr hi = load_addr + size region = lambda a: lo <= a < hi and (a & 3) == 0 + if root_enrichment is None: + root_enrichment = root_enrichment_default() + if cap.get(ROOT_ENRICHMENT_OFF_KEY): + root_enrichment = False producer_ranges = [] for raw_range in cap.get('producer_ranges', []) or []: if not isinstance(raw_range, dict): @@ -1621,21 +1871,53 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, fragment_hi = hi - capture_guard_bytes(cap, size) game_text = _game_text_range(toml_doc) + # Every walk in this classification is a pure function of (entry, cap): + # memoize so the no-split guard's owner walks cost nothing extra. + walk_cache = {} + + def walk(entry: int, hard_cap: int) -> dict: + key = (entry, hard_cap) + result = walk_cache.get(key) + if result is None: + result = _walk_overlay_function( + data, load_addr, size, entry, hard_cap, producer_ranges, + allow_cross_producer_calls) + walk_cache[key] = result + return result + + # Enrichment (default on): generic, image-local root evidence. It only + # ADDS addresses that no capture evidence classifies; it never upgrades a + # captured address's reason (a dispatch entry keeps the dispatch gate that + # demotes jump-table case labels and unproven frameless heads). Every + # derived root still passes include() and the no-split guard below. + derived_static_roots = set() + if root_enrichment: + derived_static_roots = derive_static_roots( + data, load_addr, size, producer_ranges, fragment_hi) + derived_static_roots -= (static_discovery_entries | + captured_function_entries | + dispatch_entry_pcs | toml_entries | + legacy_seeds) + # A dispatch into dirty RAM can land on a jump-table case label. That # proves coverage, not a callable function boundary, so discover those - # labels before promoting dispatch entries to seeds. + # labels before promoting dispatch entries to seeds. The prepass walks the + # guarded partition, so a candidate inside a switch host cannot cap the + # host before its jr and hide the table. pre_roots = (set(legacy_callable_seeds) | set(captured_function_entries) | - set(static_discovery_entries) | set(toml_entries)) + set(static_discovery_entries) | set(toml_entries) | + derived_static_roots) pre_roots.update(a for a in dispatch_entry_pcs if region(a) and _callable_legacy_seed(data, load_addr, a)) jump_table_targets = set() - pre_roots_sorted = sorted(a for a in pre_roots if region(a)) + pre_roots_guarded, _ = no_split_partition( + {a for a in pre_roots + if region(a) and not _in_delay_slot(data, load_addr, a)}, + set(), walk, hi) + pre_roots_sorted = sorted(pre_roots_guarded) for i, entry in enumerate(pre_roots_sorted): hard_cap = pre_roots_sorted[i + 1] if i + 1 < len(pre_roots_sorted) else hi - walk = _walk_overlay_function( - data, load_addr, size, entry, hard_cap, producer_ranges, - allow_cross_producer_calls) - jump_table_targets.update(walk['jump_table_targets']) + jump_table_targets.update(walk(entry, hard_cap)['jump_table_targets']) def impossible_entry_start(addr: int) -> bool: word = _word_at(data, load_addr, addr) @@ -1726,6 +2008,12 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, include(addr, 'FUNCTION_POINTER_TARGET') for addr in toml_entries: include(addr, 'TOML_DECLARED_ENTRY') + # Enrichment roots: proven by this image's own bytes. A known jump-table + # case label is not a function start, whatever its neighbourhood says. + for addr in sorted(derived_static_roots): + if addr in included or addr in excluded or addr in jump_table_targets: + continue + include(addr, 'STATIC_DISCOVERY_ROOT') legacy_seed_mode = bool(legacy_seeds) and not cap.get('schema') if legacy_seed_mode: @@ -1745,8 +2033,21 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, # Walk roots: callable entries only. DISPATCH_INTERIOR addresses are NOT # roots — as roots they would hard-cap (truncate) the sibling walk that # owns them. - known = {a for a, r in included.items() if r != 'DISPATCH_INTERIOR'} - initial_known = set(known) + explicit_candidates = {a for a, r in included.items() + if r != 'DISPATCH_INTERIOR'} + # A candidate in a delay slot can neither cap (it would separate a + # transfer from its slot's owner) nor alias (the slot would become a block + # leader). Leave it to the interpreter. + delay_slot_rejected = {} + for addr in sorted(explicit_candidates): + if _in_delay_slot(data, load_addr, addr): + delay_slot_rejected[addr] = included.pop(addr) + excluded[addr] = 'DELAY_SLOT' + explicit_candidates -= set(delay_slot_rejected) + # The explicit universe is fixed; the no-split guard decides, each round + # and from scratch, which of it (plus derived targets) actually roots. + initial_known = explicit_candidates + known, _ = no_split_partition(initial_known, set(), walk, hi) derived_reasons = {} promoted_roots = set() suppressed_derived = set() @@ -1786,7 +2087,9 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, print(' WARNING: ownership discovery cycle; suppressing ' f'{len(unstable)} unstable derived roots' + (f' ({unstable_text})' if unstable_text else '')) - known = initial_known | promoted_roots + known, _ = no_split_partition( + initial_known | promoted_roots, promoted_roots, + walk, hi) derived_reasons = {} discovery_round = 0 seen_states = {} @@ -1797,7 +2100,8 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, + (f' (cycle={len(state_history) - cycle_start}, ' f'unstable={unstable_text})' if cycle_start is not None else ' (round limit)')) - known = initial_known | promoted_roots + known, _ = no_split_partition( + initial_known | promoted_roots, promoted_roots, walk, hi) derived_reasons = {} break seen_states[state] = len(state_history) @@ -1805,23 +2109,28 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, next_reasons = {} round_branch_targets = set() sorted_known = sorted(known) + + def derivable(target: int) -> bool: + if target in initial_known or target in suppressed_derived: + return False + if _in_delay_slot(data, load_addr, target): + delay_slot_rejected.setdefault(target, 'DERIVED') + return False + return True + for index, entry in enumerate(sorted_known): hard_cap = (sorted_known[index + 1] if index + 1 < len(sorted_known) else hi) - walk = _walk_overlay_function( - data, load_addr, size, entry, hard_cap, producer_ranges, - allow_cross_producer_calls) - round_branch_targets.update(walk['branch_targets']) - round_branch_targets.update(walk['jump_table_targets']) - for target in walk['direct_jals']: - if (target not in initial_known and - target not in suppressed_derived): + entry_walk = walk(entry, hard_cap) + round_branch_targets.update(entry_walk['branch_targets']) + round_branch_targets.update(entry_walk['jump_table_targets']) + for target in entry_walk['direct_jals']: + if derivable(target): next_reasons[target] = 'DIRECT_JAL_TARGET' - for target in walk['static_indirect_targets']: - if (target not in initial_known and - target not in suppressed_derived): + for target in entry_walk['static_indirect_targets']: + if derivable(target): next_reasons.setdefault(target, 'STATIC_INDIRECT_TARGET') - for target in walk['forward_branch_targets']: + for target in entry_walk['forward_branch_targets']: source_reason = (included.get(entry) or derived_reasons.get(entry) or ('DISPATCH_ROOT' @@ -1845,26 +2154,16 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, producer_hi = target_range[1] if target_range else hi if plausible_callable_target( data, load_addr, size, target, producer_hi): - if (target not in initial_known and - target not in suppressed_derived): + if derivable(target): next_reasons.setdefault(target, 'STATIC_BRANCH_ROOT') - normalized = initial_known | promoted_roots | set(next_reasons) - roots = sorted(normalized) - for target in sorted(next_reasons): - if target not in normalized: - continue - index = bisect_left(roots, target) - if index == len(roots) or roots[index] != target or index == 0: - continue - owner = roots[index - 1] - hard_cap = roots[index + 1] if index + 1 < len(roots) else hi - owner_walk = _walk_overlay_function( - data, load_addr, size, owner, hard_cap, producer_ranges, - allow_cross_producer_calls) - if target in owner_walk['visited']: - normalized.remove(target) - roots.pop(index) + # The no-split guard over the whole candidate universe: explicit + # roots and derived targets alike are absorbed when the root below + # reaches them. Promoted kernel orphans are exempt (promoted + # precisely because no walk covered them). + normalized, _ = no_split_partition( + initial_known | promoted_roots | set(next_reasons), + promoted_roots, walk, hi) if normalized == known and next_reasons == derived_reasons: all_branch_targets = round_branch_targets @@ -1883,10 +2182,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, sorted_known = sorted(known) for i, entry in enumerate(sorted_known): hard_cap = sorted_known[i + 1] if i + 1 < len(sorted_known) else hi - walk = _walk_overlay_function( - data, load_addr, size, entry, hard_cap, producer_ranges, - allow_cross_producer_calls) - covered |= walk['visited'] + covered |= walk(entry, hard_cap)['visited'] promoted = sorted(a for a, r in included.items() if r == 'DISPATCH_INTERIOR' and a not in covered and _is_valid_mips_word(_word_at(data, load_addr, a))) @@ -1908,19 +2204,36 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, accepted_cross_producer_calls = set() final_covered = set() sorted_known = sorted(known) + final_walks = {} for i, entry in enumerate(sorted_known): hard_cap = sorted_known[i + 1] if i + 1 < len(sorted_known) else hi - walk = _walk_overlay_function( - data, load_addr, size, entry, hard_cap, producer_ranges, - allow_cross_producer_calls) - all_branch_targets.update(walk['branch_targets']) - all_branch_targets.update(walk['jump_table_targets']) - all_branch_targets.update(walk['forward_branch_targets']) - final_covered.update(walk['visited']) + entry_walk = walk(entry, hard_cap) + final_walks[entry] = entry_walk + all_branch_targets.update(entry_walk['branch_targets']) + all_branch_targets.update(entry_walk['jump_table_targets']) + all_branch_targets.update(entry_walk['forward_branch_targets']) + final_covered.update(entry_walk['visited']) rejected_cross_producer_calls.update( - walk['rejected_cross_producer_calls']) + entry_walk['rejected_cross_producer_calls']) accepted_cross_producer_calls.update( - walk['accepted_cross_producer_calls']) + entry_walk['accepted_cross_producer_calls']) + + # No-split guard outcome for the explicit universe: every candidate the + # final partition absorbed becomes an alias entry of its final host. At + # the guard's fixed point that host exists by construction; the fallback + # branch only keeps a broken invariant from minting a hostless alias. + guard_demoted = {} + for addr in sorted(initial_known - known): + prior = included.get(addr) + index = bisect_left(sorted_known, addr) + host = sorted_known[index - 1] if index > 0 else None + if host is not None and addr in final_walks[host]['visited']: + included[addr] = 'DISPATCH_INTERIOR' + guard_demoted[addr] = (prior, host) + else: + included.pop(addr, None) + excluded[addr] = ('OBSERVED_PC_ONLY' + if addr in executed_pcs else 'UNKNOWN') # A DISPATCH_INTERIOR is emitted only when the final partition has a real # CFG host for that exact PC. Reconsider every derived target after all @@ -1940,11 +2253,14 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, excluded[addr] = ('OBSERVED_PC_ONLY' if addr in executed_pcs else 'UNKNOWN') + for addr in delay_slot_rejected: + if addr not in included: + excluded[addr] = 'DELAY_SLOT' candidates = {a for a in (executed_pcs | dispatch_entry_pcs | captured_function_entries | static_discovery_entries | legacy_seeds | toml_entries) if region(a)} - for addr in sorted(candidates - set(included)): + for addr in sorted(candidates - set(included) - set(delay_slot_rejected)): if addr in all_branch_targets or addr in jump_table_targets: excluded[addr] = 'BRANCH_TARGET_ONLY' elif addr in executed_pcs or addr in legacy_seeds: @@ -2004,6 +2320,11 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, 'static_jump_table_proofs': cap.get('static_jump_table_proofs', []), 'static_call_continuation_pcs': _parse_addr_list( cap.get('static_call_continuation_pcs', [])), + # No-split guard / enrichment provenance, for audits and retries. + 'root_enrichment': bool(root_enrichment), + 'derived_static_roots': derived_static_roots, + 'guard_demoted': guard_demoted, + 'delay_slot_rejected': delay_slot_rejected, } # Interior entries carry the 'interior' marker so the recompiler emits # them as overlapping aliases, never as walk roots. Promoted kernel @@ -2019,6 +2340,25 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, return seeds, audit +RECOMPILER_GUARD_LINE = re.compile( + r'^No-split guard absorbed explicit entries: (\d+)\s*$', re.M) + + +def report_recompiler_guard_absorptions(stdout: str) -> int: + """Surface explicit roots the recompiler's own no-split guard absorbed. + + Nonzero means the recompiler's walk reached a root this classifier's walk + did not (e.g. a jump table only the C++ resolvers prove). The shard is + still split-free -- the recompiler aliased those entries -- but the + classifier's view diverged, so say so in the compile log.""" + total = sum(int(m.group(1)) for m in + RECOMPILER_GUARD_LINE.finditer(stdout or '')) + if total: + print(f' recompiler no-split guard absorbed {total} explicit ' + f'root(s) the classifier kept') + return total + + def print_seed_audit(audit: dict) -> None: print(f'Overlay {audit["load_addr"]:08X}_{audit["crc32"]:08X}') print(f'Region: {audit["lo"]:08X}..{audit["hi"] - 1:08X}') @@ -2035,7 +2375,15 @@ def print_seed_audit(audit: dict) -> None: print(f'toml_entries_included: {audit["counts"].get("TOML_DECLARED_ENTRY", 0)}') print(f'dispatch_interior_included: {audit["counts"].get("DISPATCH_INTERIOR", 0)}') print(f'dispatch_roots_promoted: {audit["counts"].get("DISPATCH_ROOT", 0)}') - unhosted_dispatch = (audit.get('dispatch_fragment_demands', set()) & + print(f'root_enrichment: {"on" if audit.get("root_enrichment") else "off"}' + f' (derived {len(audit.get("derived_static_roots", ()))})') + guard_demoted = audit.get('guard_demoted', {}) + print(f'no_split_guard_demoted: {len(guard_demoted)}') + print(f'delay_slot_roots_rejected: ' + f'{len(audit.get("delay_slot_rejected", {}))}') + for addr, (prior, host) in sorted(guard_demoted.items()): + print(f' {addr:08X} no-split: {prior} inside 0x{host:08X}') + unhosted_dispatch =(audit.get('dispatch_fragment_demands', set()) & audit['executed_pcs']) - set(audit['included_reasons']) print(f'unhosted_executed_dispatch_fragment_demands: {len(unhosted_dispatch)}') print(f'cross_producer_calls_rejected: ' @@ -5850,8 +6198,9 @@ def _static_capture_job(cap, args, toml, forced_interiors, static_out, result): result['entry_sources'][key] = ( data, load_addr, size, phys_addr, guard_bytes) - seeds, seed_audit = classify_overlay_seeds(cap, data, load_addr, size, - crc32, toml) + seeds, seed_audit = classify_overlay_seeds( + cap, data, load_addr, size, crc32, toml, + root_enrichment=getattr(args, 'root_enrichment', None)) print(f'Overlay load=0x{load_addr:08X} size={size} crc32=0x{crc32:08X}' + (f' guard={guard_bytes}B (delay-slot only, not analysed)' if guard_bytes else '')) @@ -5903,6 +6252,7 @@ def _static_capture_job(cap, args, toml, forced_interiors, static_out, result): print(f' RECOMPILER ERROR:\n{r.stderr or r.stdout}') fail('recompiler', r.stderr or r.stdout) return + report_recompiler_guard_absorptions(r.stdout) stem = os.path.basename(psx_path) full_c = os.path.join(out_dir_tmp, stem + '_full.c') @@ -5926,6 +6276,16 @@ def _static_capture_job(cap, args, toml, forced_interiors, static_out, result): c_audit = audit_generated_c(src, load_addr, size, crc32, toml) print_generated_c_audit(load_addr, size, crc32, c_audit) if c_audit['unknown_bad'] or c_audit['unsupported_todo_addrs']: + retry = conservative_retry_capture(cap, seed_audit) + if retry is not None: + print(' ENRICHED RECIPE AUDIT REJECTED; ' + 'retrying conservative recipe\n') + # Reset the per-capture result; the retry owns the outcome. + result['requested_entries'] = set() + result['entry_sources'] = {} + _static_capture_job(retry, args, toml, forced_interiors, + static_out, result) + return print(' GENERATED-C AUDIT FAILED\n') fail('audit', f'{len(c_audit["unknown_bad"])} unknown_bad, ' f'{len(c_audit["unsupported_todo_addrs"])} unsupported') @@ -6163,7 +6523,17 @@ def main(): 'to the sequential path).') ap.add_argument('--target-os', choices=['auto', 'win', 'linux', 'macos'], default='auto', help='target operating system for compiled overlay shards (default: auto)') + ap.add_argument('--no-root-enrichment', dest='root_enrichment', + action='store_false', default=root_enrichment_default(), + help='DIAGNOSTIC: do not derive image-local static roots ' + '(also PSX_OVERLAY_ROOT_ENRICHMENT=0). The no-split ' + 'guard stays on. Both policies are split-safe, so ' + 'their shards may share a cache namespace; a default ' + 'run over an unenriched cache rebuilds any region ' + 'whose new root demands the cache does not serve.') args = ap.parse_args() + print(f'root policy: no-split guard on, enrichment ' + f'{"on" if args.root_enrichment else "OFF (diagnostic)"}') target_os = args.target_os if target_os == 'auto': if 'mingw' in args.gcc.lower() or 'w64' in args.gcc.lower() or args.gcc.lower().endswith('.exe'): @@ -6372,8 +6742,9 @@ def main(): print(f' merged {len(prior_entries)} prior-manifest entries ' f'from {prior_ranges_path}') - seeds, seed_audit = classify_overlay_seeds(cap, data, load_addr, size, - crc32, toml) + seeds, seed_audit = classify_overlay_seeds( + cap, data, load_addr, size, crc32, toml, + root_enrichment=args.root_enrichment) this_ids = None # region func-ids once recompiled (None if skipped early) # Record this region's executed dispatch-proven PCs for the decoupled @@ -6492,6 +6863,7 @@ def main(): print(f' RECOMPILER ERROR:\n{r.stderr or r.stdout}') stats.add_fail(_label, 'recompiler', r.stderr or r.stdout) return + report_recompiler_guard_absorptions(r.stdout) # Find the generated _full.c stem = os.path.basename(psx_path) @@ -6526,7 +6898,7 @@ def main(): with open(debug_c, 'w') as f: f.write(src) if c_audit['unknown_bad'] or c_audit['unsupported_todo_addrs']: - fallback = optional_enrichment_fallback_capture(cap) + fallback = conservative_retry_capture(cap, seed_audit) if fallback is not None: print(' OPTIONAL ENRICHMENT AUDIT REJECTED; ' 'retrying conservative recipe\n') @@ -6688,7 +7060,7 @@ def main(): 'interpreter\n') stats.add_skip() return - fallback = optional_enrichment_fallback_capture(cap) + fallback = conservative_retry_capture(cap, seed_audit) if fallback is not None: print(' OPTIONAL ENRICHMENT COMPILE REJECTED; ' 'retrying conservative recipe\n') diff --git a/tools/enrich_overlay_captures.py b/tools/enrich_overlay_captures.py index b752c503..71fffe29 100644 --- a/tools/enrich_overlay_captures.py +++ b/tools/enrich_overlay_captures.py @@ -1,18 +1,23 @@ #!/usr/bin/env python3 -"""Add conservative static walk-root evidence to runtime overlay captures. +"""Inspect (and optionally materialize) static root enrichment for captures. -Runtime capture knows that a dirty-RAM PC was dispatched, but that is not the -same thing as knowing it is a function boundary: CPS continuations and jump -table cases are dispatched too. This tool therefore leaves the runtime-owned -``function_entry_pcs`` untouched and derives the optional, tool-owned -``static_discovery_entry_pcs`` from captured bytes. compile_overlays.py -re-validates every supplied root before compiling it. +compile_overlays.py derives these roots itself (enrichment is on by default +and part of the overlay cache key), so this tool is no longer a pipeline +step. It shows what the compiler's derivation finds in each captured image, +using exactly the same code (compile_overlays.derive_static_roots), and can +still write the roots into ``static_discovery_entry_pcs`` for offline A/B. + +Every derived root needs an image-local boundary proof (a non-delay-slot +stack-frame prologue, or a preceding ``jr $ra`` plus the bounded CFG probe). +A jal or pointer elsewhere in the image is not proof on its own: overlay +regions are swapped, so shared code can call an address that is a function +start only in a sibling image. compile_overlays.py additionally applies the +no-split guard to every root, derived or not. """ import argparse import base64 import json -import struct import sys from pathlib import Path @@ -22,64 +27,10 @@ sys.path.insert(0, str(ROOT / 'tools')) import compile_overlays as CO # noqa: E402 -def word_at(data, base, addr): - offset = addr - base - if offset < 0 or offset + 4 > len(data): - return None - return struct.unpack_from('> 26) == 0x03: # jal - target = 0x80000000 | ((word & 0x03ffffff) << 2) - if is_candidate(data, base, target): - roots.add(target) - pointers.append(word if word is not None and not (word & 3) and - base <= word < hi else None) - - # A dense run is a table only when it has three or more in-image pointers. - # Each member must still pass the same root proof; no bare pointer becomes - # authority, and a pointer to the table itself fails the CFG proof. - start = 0 - while start < len(pointers): - if pointers[start] is None: - start += 1 - continue - end = start + 1 - while end < len(pointers) and pointers[end] is not None: - end += 1 - if end - start >= 3: - for target in pointers[start:end]: - if is_candidate(data, base, target): - roots.add(target) - start = end - return sorted(roots) +def derived_roots(data, base, sources=None): + """The compiler's own enrichment roots for one captured image.""" + return sorted(CO.derive_static_roots(data, base, len(data), + sources=sources)) def enrich_record(record): @@ -102,7 +53,9 @@ def enrich_record(record): def main(argv=None): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--captures', required=True, help='runtime capture JSON') - parser.add_argument('--out', required=True, help='enriched JSON output path') + parser.add_argument('--out', help='write an enriched JSON copy here') + parser.add_argument('--list', action='store_true', + help='print every derived root and its evidence sources') args = parser.parse_args(argv) with open(args.captures, encoding='utf-8') as src: records = json.load(src) @@ -116,12 +69,22 @@ def main(argv=None): item, count = enrich_record(record) except ValueError as exc: raise SystemExit(f'capture {index}: {exc}') from exc + if args.list: + base = CO._parse_addr(record['load_addr']) + data = base64.b64decode(record['bytes_b64']) + sources = {} + roots = derived_roots(data, base, sources) + print(f'capture {index} load=0x{base:08X}: {len(roots)} derived root(s)') + for addr in roots: + print(f' 0x{addr:08X} {"+".join(sorted(sources[addr]))}') enriched.append(item) added += count - with open(args.out, 'w', encoding='utf-8', newline='\n') as dst: - json.dump(enriched, dst, indent=2) - dst.write('\n') - print(f'enriched {len(enriched)} capture(s); added {added} static discovery root(s)') + if args.out: + with open(args.out, 'w', encoding='utf-8', newline='\n') as dst: + json.dump(enriched, dst, indent=2) + dst.write('\n') + print(f'{len(enriched)} capture(s); {added} static discovery root(s) not ' + f'already declared') if __name__ == '__main__': diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py new file mode 100644 index 00000000..54cd52bd --- /dev/null +++ b/tools/tests/test_overlay_no_split_guard.py @@ -0,0 +1,374 @@ +#!/usr/bin/env python3 +"""The overlay no-split guard, image-local root enrichment, and the cache key. + +A walk root is a hard cap: the walk of the root below it stops there, so a +root inside another function's body splits that function. The generated C +still passes every audit, which is why this class never shows up as a shard +failure. Measured on Tomba (bead beads-eio.3.177): shared engine code, byte +identical in 12 swapped area images, does `jal 0x8011B1CC`; that address is a +function start in one image and a `subu` mid-way through host 0x8011AC10 in +X00. Rooting it there split the host and moved guest timing by one cycle +against the interpreter. + +Usage: test_overlay_no_split_guard.py --cmake [--recompiler ] +""" +import argparse +import os +import re +import shutil +import struct +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / 'tools')) +import compile_overlays as CO # noqa: E402 + +LOAD = 0x80100000 +NOP = 0x00000000 +JR_RA = 0x03E00008 +FRAME = 0x27BDFFE0 # addiu sp,sp,-32 +UNFRAME = 0x27BD0020 # addiu sp,sp,32 +ADDIU = 0x24420001 # addiu v0,v0,1 +SUBU = 0x00431023 # subu v0,v0,v1 +CMAKE = None +RECOMPILER = None + + +def jal(target): + return 0x0C000000 | ((target >> 2) & 0x03FFFFFF) + + +def bne_v0(pc, target): + return 0x14400000 | (((target - (pc + 4)) >> 2) & 0xFFFF) + + +def image(words: dict, size: int) -> bytes: + data = bytearray(size) + for addr, word in words.items(): + struct.pack_into('=3 pointer run, so + # enrichment derives them all. The table resolves only when every + # target is inside the host's walk, so they must be absorbed as a run. + host = LOAD + table = LOAD + 0x200 + cases = [LOAD + 0x68, LOAD + 0x78, LOAD + 0x88] + words = {host: FRAME, + LOAD + 0x04: 0x3C088010, # lui t0,0x8010 + LOAD + 0x08: 0x25100200, # addiu s0,t0,0x200 + LOAD + 0x0C: 0x2C620003, # sltiu v0,v1,3 + LOAD + 0x10: 0x10400000 | ((0x98 - 0x14) >> 2), # beq v0,zero + LOAD + 0x14: NOP, + LOAD + 0x18: 0x00031080, # sll v0,v1,2 + LOAD + 0x1C: 0x00501021, # addu v0,v0,s0 + LOAD + 0x20: 0x8C420000, # lw v0,0(v0) + LOAD + 0x24: NOP, + LOAD + 0x28: 0x00400008, # jr v0 + LOAD + 0x2C: NOP, + LOAD + 0x98: JR_RA, LOAD + 0x9C: UNFRAME} + for index, case in enumerate(cases): + words[case] = ADDIU + words[case + 4] = JR_RA + words[case + 8] = UNFRAME + words[table + index * 4] = case + data = image(words, 0x300) + derived = CO.derive_static_roots(data, LOAD, len(data)) + self.assertTrue(set(cases[1:]) <= derived) + # The guard alone (the case labels supplied as explicit roots, as a + # #386-enriched capture would) absorbs the whole run. + seeds, audit = classify(data, False, function_entry_pcs=[host], + static_discovery_entry_pcs=cases[1:]) + for case in cases[1:]: + self.assertEqual(audit['included_reasons'][case], + 'DISPATCH_INTERIOR', hex(case)) + self.assertNotIn(case, root_seeds(seeds)) + # Default enrichment: the guarded prepass resolves the table, so the + # case labels are never offered as roots at all. + seeds, audit = classify(data, True, function_entry_pcs=[host]) + for case in cases: + self.assertIn(audit['included_reasons'].get(case), + (None, 'DISPATCH_INTERIOR'), hex(case)) + self.assertNotIn(case, root_seeds(seeds)) + self.assertEqual(root_seeds(seeds), {host}) + + # (c) ------------------------------------------------------------------ + def test_jal_proven_function_start_stays_a_root(self): + caller, framed, frameless = LOAD, LOAD + 0x40, LOAD + 0x80 + data = image({caller: FRAME, caller + 4: jal(framed), + caller + 8: NOP, caller + 12: jal(frameless), + caller + 16: NOP, caller + 20: JR_RA, + caller + 24: UNFRAME, + framed: FRAME, framed + 4: ADDIU, framed + 8: JR_RA, + framed + 12: UNFRAME, + frameless - 8: JR_RA, frameless - 4: NOP, + frameless: ADDIU, frameless + 4: JR_RA, + frameless + 8: NOP}, 0x100) + derived = CO.derive_static_roots(data, LOAD, len(data)) + self.assertTrue({caller, framed, frameless} <= derived) + seeds, audit = classify(data, True) + roots = root_seeds(seeds) + self.assertTrue({caller, framed, frameless} <= roots) + self.assertEqual(audit['guard_demoted'], {}) + + # (d) ------------------------------------------------------------------ + def _cross_image(self, target_is_function): + engine, host = LOAD, LOAD + 0x100 + target = LOAD + 0x110 + words = {engine: FRAME, engine + 4: jal(target), engine + 8: NOP, + engine + 12: JR_RA, engine + 16: UNFRAME} + if target_is_function: + # Sibling image: the called address is a real function here. + words.update({host: ADDIU, host + 4: JR_RA, host + 8: NOP, + target: FRAME, target + 4: ADDIU, + target + 8: JR_RA, target + 12: UNFRAME}) + else: + # This image: the same address is mid-way through a host + # reached by pure fallthrough (the X00 shape). + words.update({host: FRAME, host + 4: ADDIU, host + 8: ADDIU, + host + 12: ADDIU, target: SUBU, + target + 4: ADDIU, target + 8: JR_RA, + target + 12: UNFRAME}) + return image(words, 0x200), engine, host, target + + def test_shared_jal_into_mid_body_is_not_rooted(self): + data, engine, host, target = self._cross_image(False) + derived = CO.derive_static_roots(data, LOAD, len(data)) + self.assertNotIn(target, derived) + self.assertTrue({engine, host} <= derived) + # Enrichment on (default): the rooted engine walk still derives the + # jal target, and the guard absorbs it into its host. + seeds, audit = classify(data, True) + self.assertNotIn(target, root_seeds(seeds)) + self.assertEqual(audit['included_reasons'][target], + 'DISPATCH_INTERIOR') + self.assertIn(host, root_seeds(seeds)) + # A #386-enriched capture that already names it is demoted too. + seeds, audit = classify(data, True, + static_discovery_entry_pcs=[target]) + self.assertNotIn(target, root_seeds(seeds)) + self.assertEqual(audit['guard_demoted'][target], + ('STATIC_DISCOVERY_ROOT', host)) + + def test_shared_jal_into_real_function_in_sibling_image_is_rooted(self): + data, _engine, _host, target = self._cross_image(True) + self.assertIn(target, CO.derive_static_roots(data, LOAD, len(data))) + seeds, _audit = classify(data, True) + self.assertIn(target, root_seeds(seeds)) + + # enrichment policy ---------------------------------------------------- + def test_enrichment_is_default_and_opt_out_is_diagnostic(self): + data, engine, host, _target = self._cross_image(False) + old = os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None) + try: + self.assertTrue(CO.root_enrichment_default()) + _seeds, audit = classify(data, None) + self.assertTrue(audit['root_enrichment']) + self.assertTrue({engine, host} <= audit['derived_static_roots']) + os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = '0' + self.assertFalse(CO.root_enrichment_default()) + _seeds, audit = classify(data, None) + self.assertFalse(audit['root_enrichment']) + self.assertEqual(audit['derived_static_roots'], set()) + finally: + os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None) + if old is not None: + os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = old + # A conservative retry recipe is never enriched again. + cap = {'schema': 'psxrecomp overlay capture v2', + CO.ROOT_ENRICHMENT_OFF_KEY: True} + _seeds, audit = CO.classify_overlay_seeds( + cap, data, LOAD, len(data), 0, {}, root_enrichment=True) + self.assertEqual(audit['derived_static_roots'], set()) + retry = CO.conservative_retry_capture( + {'schema': 'x'}, {'derived_static_roots': {engine}}) + self.assertTrue(retry[CO.ROOT_ENRICHMENT_OFF_KEY]) + self.assertIsNone(CO.conservative_retry_capture( + retry, {'derived_static_roots': {engine}})) + + +# (e) ---------------------------------------------------------------------- +class CacheKeyTests(unittest.TestCase): + def _cmake_list(self): + with tempfile.TemporaryDirectory() as tmp: + script = Path(tmp) / 'list.cmake' + script.write_text( + f'set(PSXRECOMP_CODEGEN_HASH_ROOT "{ROOT.as_posix()}")\n' + f'include("{(ROOT / "runtime/codegen_hash_sources.cmake").as_posix()}")\n' + 'message(STATUS "SRCS=${PSXRECOMP_CODEGEN_HASH_SRCS}")\n', + encoding='utf-8') + proc = subprocess.run([CMAKE, '-P', str(script)], + capture_output=True, text=True) + self.assertEqual(proc.returncode, 0, proc.stderr) + match = re.search(r'SRCS=(.*)', proc.stdout + proc.stderr) + self.assertIsNotNone(match) + return match.group(1).strip().split(';') + + def _hash_header(self, sources, out_dir): + header = Path(out_dir) / 'overlay_codegen_hash.h' + proc = subprocess.run( + [CMAKE, f'-DOUT={header.as_posix()}', + '-DSRCS=' + ';'.join(sources), + '-P', str(ROOT / 'runtime' / 'hash_codegen.cmake')], + capture_output=True, text=True) + self.assertEqual(proc.returncode, 0, proc.stderr) + shutil.copy2(ROOT / 'runtime' / 'include' / 'overlay_api.h', + Path(out_dir) / 'overlay_api.h') + return CO.codegen_hash(str(out_dir)) + + def test_root_policy_is_part_of_the_cache_namespace(self): + sources = self._cmake_list() + classifier = (ROOT / 'tools' / 'compile_overlays.py').as_posix() + self.assertIn(classifier, sources) + with tempfile.TemporaryDirectory() as tmp: + tmp = Path(tmp) + changed = tmp / 'compile_overlays.py' + changed.write_text( + Path(classifier).read_text(encoding='utf-8') + + '\n# a root-policy change\n', encoding='utf-8') + (tmp / 'a').mkdir() + (tmp / 'b').mkdir() + before = self._hash_header(sources, tmp / 'a') + after = self._hash_header( + [changed.as_posix() if s == classifier else s + for s in sources], tmp / 'b') + self.assertNotEqual(before, after) + saved = CO.overlay_config_hash + CO.overlay_config_hash = lambda recompiler, toml: 0x1234ABCD + try: + tag_a = CO.cache_tag(str(tmp / 'a'), 'r', 'g', 0) + tag_b = CO.cache_tag(str(tmp / 'b'), 'r', 'g', 0) + finally: + CO.overlay_config_hash = saved + self.assertNotEqual(tag_a, tag_b) + + +# recompiler enforcement --------------------------------------------------- +class RecompilerGuardTests(unittest.TestCase): + def test_recompiler_absorbs_explicit_fallthrough_root(self): + if not RECOMPILER: + self.skipTest('pass --recompiler (ctest always does)') + host, mid = LOAD + 0x20, LOAD + 0x30 + data = image({host: FRAME, host + 4: ADDIU, host + 8: ADDIU, + host + 12: ADDIU, mid: SUBU, mid + 4: ADDIU, + mid + 8: JR_RA, mid + 12: UNFRAME}, 0x1000) + with tempfile.TemporaryDirectory() as tmp: + psx = Path(tmp) / 'split.psx' + psx.write_bytes(CO.make_psxexe(LOAD, host, data, guard_bytes=0)) + seeds = Path(tmp) / 'seeds.txt' + # Seeds the classifier would never write: both as call roots. + seeds.write_text(f'call_root 0x{host:08X}\ncall_root 0x{mid:08X}\n') + out = Path(tmp) / 'out' + proc = subprocess.run( + [RECOMPILER, str(psx), '--seeds', str(seeds), + '--out-dir', str(out), '--overlay'], + capture_output=True, text=True, cwd=str(ROOT)) + self.assertEqual(proc.returncode, 0, proc.stderr or proc.stdout) + self.assertEqual( + CO.report_recompiler_guard_absorptions(proc.stdout), 1, + proc.stdout) + ranges = next(out.glob('*_full.ranges')).read_text() + # Both PCs stay callable; the mid-body one only as an alias whose + # identity covers the host range (it starts at the host). + self.assertIn(f'F {host:08X}', ranges) + self.assertIn(f'F {mid:08X}', ranges) + mid_rows = ranges.split(f'F {mid:08X}', 1)[1].splitlines()[1:] + self.assertTrue(any(row.startswith(f'R {host:08X}') + for row in mid_rows[:2]), ranges) + + +if __name__ == '__main__': + ap = argparse.ArgumentParser() + ap.add_argument('--cmake', default=shutil.which('cmake')) + ap.add_argument('--recompiler') + args, rest = ap.parse_known_args() + if not args.cmake: + sys.exit('cmake not found: pass --cmake') + CMAKE = args.cmake + RECOMPILER = args.recompiler + unittest.main(argv=[sys.argv[0]] + rest) From da59cee92622a8e6a605799ead2ed8e5a170f57f Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 19:21:58 -0700 Subject: [PATCH 03/13] overlays: weak enrichment targets kept only when no possible function start reaches them Co-Authored-By: Claude Opus 5.5 (1M context) --- recompiler/src/function_analysis.cpp | 9 +- tools/compile_overlays.py | 153 +++++++++++++++++---- tools/enrich_overlay_captures.py | 40 ++++-- tools/tests/test_overlay_no_split_guard.py | 31 ++++- 4 files changed, 194 insertions(+), 39 deletions(-) diff --git a/recompiler/src/function_analysis.cpp b/recompiler/src/function_analysis.cpp index 0705d6ed..eea0be96 100644 --- a/recompiler/src/function_analysis.cpp +++ b/recompiler/src/function_analysis.cpp @@ -1650,8 +1650,15 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries( const uint32_t host = roots[index]; kept.push_back(host); const ExactWalkResult reach = walk(host, analysis_end); + // A wider range can reject a jump table, so the walk capped + // at the successor is not always a subset: consult both. + const ExactWalkResult succ = index + 1 < roots.size() + ? walk(host, roots[index + 1]) : ExactWalkResult{}; + auto reached = [&](uint32_t a) { + return reach.visited.count(a) || succ.visited.count(a); + }; size_t end = index + 1; - while (end < roots.size() && reach.visited.count(roots[end]) && + while (end < roots.size() && reached(roots[end]) && !trusted_entries.count(roots[end])) end++; while (end > index + 1) { diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index edf7a124..c6c0e7c6 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -1335,28 +1335,38 @@ def _frameless_dispatch_root_proven(data: bytes, load_addr: int, size: int, # classify_overlay_seeds). A candidate that a root below it reaches by # fallthrough or branch (never by jal) -- walked without the caps of the # candidates in between -- is inside that function. It is demoted to -# DISPATCH_INTERIOR, an alias entry that never caps. Applied to dispatch entries, captured function -# entries, static discovery roots, TOML entries, derived call targets and -# enrichment alike; only promoted kernel orphans (DISPATCH_ROOT) are -# exempt, because they are promoted precisely when no walk covers them. -# A candidate that sits in a delay slot is never a root and never an -# alias (entering a host at a delay slot would make the slot a block -# leader); it is left to the interpreter. +# DISPATCH_INTERIOR, an alias entry that never caps. Applied to dispatch +# entries, captured function entries, static discovery roots, TOML +# entries, derived call targets and enrichment alike; only promoted +# kernel orphans (DISPATCH_ROOT) are exempt, because they are promoted +# precisely when no walk covers them. A candidate that sits in a delay +# slot is never a root and never an alias (entering a host at a delay +# slot would make the slot a block leader); it is left to the +# interpreter. # -# 2. Image-local entry proof for enrichment (image_local_entry_proven). A +# 2. Enrichment evidence is judged per image (derive_enrichment_roots). A # jal or a pointer table is call evidence for the image that is resident # when it executes, and overlay regions are swapped: shared engine code # at 0x80100984 is byte-identical in 12 Tomba area images and calls -# 0x8011B1CC, which is a function start in exactly one of them. So a jal -# or table target becomes a root only when THIS image's bytes also prove -# a boundary there: a non-delay-slot stack-frame prologue, or a preceding -# `jr $ra` (plus alignment padding) together with the bounded CFG probe. -# A frameless target that only the CFG probe accepts is not rooted from -# enrichment; if a rooted walk calls it, the classifier's own derived -# DIRECT_JAL_TARGET path still finds it, under the guard. +# 0x8011B1CC, which is a function start in exactly one of them. So: +# * STRONG: a non-delay-slot stack-frame prologue, or a jal/table +# target that THIS image's bytes also bound (preceding `jr $ra` plus +# alignment padding, and the bounded CFG probe). A strong root that +# turns out to be inside a function is demoted by rule 1. +# * WEAK: a jal/table target with only the CFG probe. It is added only +# when no possible function start below it reaches it -- every +# capture root, every strong or weak candidate, every prologue and +# every word after an unconditional transfer's delay slot, walked +# with no caps at all -- and a weak candidate that fails is DROPPED, +# never aliased. X00's 0x8011B1CC is reached from the prologue at +# host 0x8011AC10, so it is dropped there and kept in X10. +# Measured on Tomba (25 AOT images): strong-only enrichment adds zero net +# roots after rule 1; the weak class is where the coverage is. # # With both in place a wrong guess can only cost speed (an alias entry or an -# interpreted PC), never correctness. +# interpreted PC), never correctness: a surviving weak root inside a function +# whose start none of the above recognises can only begin a partial walk of +# an unrooted (interpreted) host; it caps no native walk. # Tool-owned capture key: set on a conservative retry recipe so enrichment is # not derived again after the enriched recipe was rejected. @@ -1424,13 +1434,17 @@ def image_local_entry_proven(data: bytes, load_addr: int, size: int, def derive_static_roots(data: bytes, load_addr: int, size: int, producer_ranges=(), analysis_hi: int | None = None, - sources: dict | None = None) -> set[int]: - """Enrichment: independently proven function starts in one image. + sources: dict | None = None, + weak_out: set | None = None) -> set[int]: + """Enrichment candidates in one image: the STRONG set is returned. Candidates come from three generic sources -- stack-frame prologues, - direct jal targets, and runs of three or more in-image pointers -- and - every candidate must pass image_local_entry_proven. ``sources`` (optional) - receives {addr: set of source names} for inspection. + direct jal targets, and runs of three or more in-image pointers. Those + passing image_local_entry_proven are STRONG. A jal/table target that only + passes the bounded CFG probe (plausible_callable_target, not in a delay + slot) is WEAK and goes to ``weak_out`` when given; derive_enrichment_roots + decides which weak ones survive. ``sources`` (optional) receives + {addr: set of source names} for every returned or weak address. """ hi = load_addr + size if analysis_hi is None else analysis_hi ranges = sorted(producer_ranges or ()) @@ -1480,9 +1494,86 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, roots.add(addr) if sources is not None: sources[addr] = set(why) + elif (weak_out is not None and why & {'jal', 'pointer_table'} and + not addr & 3 and load_addr <= addr < hi and + not _in_delay_slot(data, load_addr, addr) and + plausible_callable_target(data, load_addr, size, addr, + min(producer_hi, hi))): + weak_out.add(addr) + if sources is not None: + sources[addr] = set(why) | {'weak'} return roots +def possible_function_starts(data: bytes, load_addr: int, + analysis_hi: int, producer_ranges=()) -> set[int]: + """Every address a function could plausibly start at, for the WEAK + enrichment test: image start, producer starts, stack-frame prologues, and + the first non-padding word after an unconditional transfer's delay slot + (`jr`, `j`, always-taken branch). Generous on purpose -- a wider host set + can only drop more weak candidates.""" + starts = {load_addr} + starts.update(lo for lo, _hi in producer_ranges or ()) + for addr in range(load_addr, analysis_hi, 4): + word = _word_at(data, load_addr, addr) + if word is None or not _is_valid_mips_word(word): + continue + if _is_addiu_sp_neg(word) and not _in_delay_slot(data, load_addr, addr): + starts.add(addr) + if _is_control_flow(word): + kind, _target = _classify_cf(addr, word) + if kind in ('j', 'jr', 'jr_ra'): + nxt = addr + 8 + for _ in range(8): + if nxt >= analysis_hi or _word_at(data, load_addr, nxt) != 0: + break + nxt += 4 + if nxt < analysis_hi: + starts.add(nxt) + return starts + + +def derive_enrichment_roots(data: bytes, load_addr: int, size: int, + producer_ranges=(), analysis_hi: int | None = None, + explicit=(), walk=None) -> tuple[set, dict]: + """The enrichment roots compile_overlays adds for one image. + + STRONG candidates are returned as-is (the no-split guard demotes any that + lie inside a function). A WEAK candidate is returned only when no other + possible function start -- explicit capture roots, strong and weak + candidates, possible_function_starts() -- reaches it by an uncapped walk. + ``walk(entry, cap)`` is the caller's memoized _walk_overlay_function. + Returns (roots, stats). + """ + hi = load_addr + size if analysis_hi is None else analysis_hi + if walk is None: + cache = {} + + def walk(entry, cap): + key = (entry, cap) + if key not in cache: + cache[key] = _walk_overlay_function( + data, load_addr, size, entry, cap, producer_ranges) + return cache[key] + weak = set() + strong = derive_static_roots(data, load_addr, size, producer_ranges, hi, + weak_out=weak) + hosts = ({a for a in explicit if load_addr <= a < hi and not a & 3} | + strong | weak | + possible_function_starts(data, load_addr, hi, producer_ranges)) + reached = set() + if weak: + for host in sorted(hosts): + hits = walk(host, load_addr + size)['visited'] & weak + hits.discard(host) + reached |= hits + accepted = weak - reached + return strong | accepted, { + 'strong': len(strong), 'weak': len(weak), + 'weak_accepted': len(accepted), 'weak_dropped': len(reached), + 'hosts': len(hosts)} + + def no_split_partition(candidates, exempt, walk, hi: int): """Fixed point of the no-split guard over one root partition. @@ -1514,7 +1605,12 @@ def no_split_partition(candidates, exempt, walk, hi: int): index = 0 while index < len(roots): host = roots[index] + # Uncapped reach, plus the walk capped at the successor: a wider + # range can reject a jump table (more sources may enter its + # protected suffix), so the capped walk is not always a subset. reach = walk(host, hi)['visited'] + if index + 1 < len(roots): + reach = reach | walk(host, roots[index + 1])['visited'] end = index + 1 while (end < len(roots) and roots[end] in reach and roots[end] not in exempt): @@ -1891,9 +1987,13 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, # demotes jump-table case labels and unproven frameless heads). Every # derived root still passes include() and the no-split guard below. derived_static_roots = set() + enrichment_stats = {} if root_enrichment: - derived_static_roots = derive_static_roots( - data, load_addr, size, producer_ranges, fragment_hi) + derived_static_roots, enrichment_stats = derive_enrichment_roots( + data, load_addr, size, producer_ranges, fragment_hi, + explicit=(static_discovery_entries | captured_function_entries | + dispatch_entry_pcs | toml_entries | legacy_seeds), + walk=walk) derived_static_roots -= (static_discovery_entries | captured_function_entries | dispatch_entry_pcs | toml_entries | @@ -2323,6 +2423,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, # No-split guard / enrichment provenance, for audits and retries. 'root_enrichment': bool(root_enrichment), 'derived_static_roots': derived_static_roots, + 'enrichment_stats': enrichment_stats, 'guard_demoted': guard_demoted, 'delay_slot_rejected': delay_slot_rejected, } @@ -2375,8 +2476,12 @@ def print_seed_audit(audit: dict) -> None: print(f'toml_entries_included: {audit["counts"].get("TOML_DECLARED_ENTRY", 0)}') print(f'dispatch_interior_included: {audit["counts"].get("DISPATCH_INTERIOR", 0)}') print(f'dispatch_roots_promoted: {audit["counts"].get("DISPATCH_ROOT", 0)}') + stats = audit.get('enrichment_stats') or {} print(f'root_enrichment: {"on" if audit.get("root_enrichment") else "off"}' - f' (derived {len(audit.get("derived_static_roots", ()))})') + f' (derived {len(audit.get("derived_static_roots", ()))}' + + (f'; strong {stats["strong"]}, weak {stats["weak"]} accepted ' + f'{stats["weak_accepted"]} dropped {stats["weak_dropped"]}' + if stats else '') + ')') guard_demoted = audit.get('guard_demoted', {}) print(f'no_split_guard_demoted: {len(guard_demoted)}') print(f'delay_slot_roots_rejected: ' diff --git a/tools/enrich_overlay_captures.py b/tools/enrich_overlay_captures.py index 71fffe29..b4d3cf34 100644 --- a/tools/enrich_overlay_captures.py +++ b/tools/enrich_overlay_captures.py @@ -7,12 +7,14 @@ step. It shows what the compiler's derivation finds in each captured image, using exactly the same code (compile_overlays.derive_static_roots), and can still write the roots into ``static_discovery_entry_pcs`` for offline A/B. -Every derived root needs an image-local boundary proof (a non-delay-slot -stack-frame prologue, or a preceding ``jr $ra`` plus the bounded CFG probe). A jal or pointer elsewhere in the image is not proof on its own: overlay regions are swapped, so shared code can call an address that is a function -start only in a sibling image. compile_overlays.py additionally applies the -no-split guard to every root, derived or not. +start only in a sibling image. A candidate is STRONG when this image's bytes +bound it (a non-delay-slot stack-frame prologue, or a preceding ``jr $ra`` +plus the bounded CFG probe) and WEAK when only the CFG probe accepts it; a +weak candidate is kept only if no possible function start below it reaches +it (compile_overlays.derive_enrichment_roots). compile_overlays.py +additionally applies the no-split guard to every root, derived or not. """ import argparse @@ -27,10 +29,23 @@ sys.path.insert(0, str(ROOT / 'tools')) import compile_overlays as CO # noqa: E402 -def derived_roots(data, base, sources=None): +def derived_roots(data, base, explicit=(), sources=None, stats=None): """The compiler's own enrichment roots for one captured image.""" - return sorted(CO.derive_static_roots(data, base, len(data), - sources=sources)) + roots, info = CO.derive_enrichment_roots(data, base, len(data), + explicit=explicit) + if sources is not None: + weak = set() + CO.derive_static_roots(data, base, len(data), sources=sources, + weak_out=weak) + if stats is not None: + stats.update(info) + return sorted(roots) + + +def explicit_roots(record): + return set().union(*(CO._parse_addr_list(record.get(key, [])) for key in ( + 'function_entry_pcs', 'dispatch_entry_pcs', + 'static_discovery_entry_pcs', 'seeds'))) def enrich_record(record): @@ -45,7 +60,8 @@ def enrich_record(record): if size != len(data) or base & 3 or len(data) & 3: raise ValueError('capture bytes must be a word-aligned image matching size') prior = CO._parse_addr_list(record.get('static_discovery_entry_pcs', [])) - roots = sorted(prior | set(derived_roots(data, base))) + roots = sorted(prior | set(derived_roots(data, base, + explicit_roots(record)))) out['static_discovery_entry_pcs'] = [f'0x{addr:08X}' for addr in roots] return out, len(roots) - len(prior) @@ -72,9 +88,11 @@ def main(argv=None): if args.list: base = CO._parse_addr(record['load_addr']) data = base64.b64decode(record['bytes_b64']) - sources = {} - roots = derived_roots(data, base, sources) - print(f'capture {index} load=0x{base:08X}: {len(roots)} derived root(s)') + sources, stats = {}, {} + roots = derived_roots(data, base, explicit_roots(record), + sources, stats) + print(f'capture {index} load=0x{base:08X}: {len(roots)} derived ' + f'root(s) {stats}') for addr in roots: print(f' 0x{addr:08X} {"+".join(sorted(sources[addr]))}') enriched.append(item) diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py index 54cd52bd..29ff5dc9 100644 --- a/tools/tests/test_overlay_no_split_guard.py +++ b/tools/tests/test_overlay_no_split_guard.py @@ -218,9 +218,14 @@ class NoSplitGuardTests(unittest.TestCase): def test_shared_jal_into_mid_body_is_not_rooted(self): data, engine, host, target = self._cross_image(False) - derived = CO.derive_static_roots(data, LOAD, len(data)) - self.assertNotIn(target, derived) - self.assertTrue({engine, host} <= derived) + weak = set() + strong = CO.derive_static_roots(data, LOAD, len(data), weak_out=weak) + self.assertNotIn(target, strong) + self.assertIn(target, weak) # the CFG probe alone accepts it + self.assertTrue({engine, host} <= strong) + derived, stats = CO.derive_enrichment_roots(data, LOAD, len(data)) + self.assertNotIn(target, derived) # host's prologue reaches it + self.assertEqual(stats['weak_dropped'], 1) # Enrichment on (default): the rooted engine walk still derives the # jal target, and the guard absorbs it into its host. seeds, audit = classify(data, True) @@ -235,6 +240,26 @@ class NoSplitGuardTests(unittest.TestCase): self.assertEqual(audit['guard_demoted'][target], ('STATIC_DISCOVERY_ROOT', host)) + def test_weak_target_nothing_reaches_is_rooted(self): + # A frameless function right after another function's tail jump, + # called by jal: no `jr $ra` before it, so no boundary proof, but no + # possible function start reaches it either. + caller, tail, leaf, far = LOAD, LOAD + 0x40, LOAD + 0x4C, LOAD + 0x80 + data = image({caller: FRAME, caller + 4: jal(leaf), caller + 8: NOP, + caller + 12: JR_RA, caller + 16: UNFRAME, + tail: ADDIU, + tail + 4: 0x08000000 | ((far >> 2) & 0x03FFFFFF), + tail + 8: NOP, + leaf: ADDIU, leaf + 4: JR_RA, leaf + 8: NOP, + far: JR_RA, far + 4: NOP}, 0x100) + weak = set() + CO.derive_static_roots(data, LOAD, len(data), weak_out=weak) + self.assertIn(leaf, weak) + derived, _stats = CO.derive_enrichment_roots(data, LOAD, len(data)) + self.assertIn(leaf, derived) + seeds, _audit = classify(data, True) + self.assertIn(leaf, root_seeds(seeds)) + def test_shared_jal_into_real_function_in_sibling_image_is_rooted(self): data, _engine, _host, target = self._cross_image(True) self.assertIn(target, CO.derive_static_roots(data, LOAD, len(data))) From 98903fca502bda466efe6839de1f619bd1f03f74 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 19:25:58 -0700 Subject: [PATCH 04/13] overlays: cross-producer call evidence never counts as weak enrichment evidence Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/compile_overlays.py | 51 +++++++++++++++++----- tools/tests/test_overlay_no_split_guard.py | 20 +++++++++ 2 files changed, 60 insertions(+), 11 deletions(-) diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index c6c0e7c6..4874a809 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -1435,7 +1435,8 @@ def image_local_entry_proven(data: bytes, load_addr: int, size: int, def derive_static_roots(data: bytes, load_addr: int, size: int, producer_ranges=(), analysis_hi: int | None = None, sources: dict | None = None, - weak_out: set | None = None) -> set[int]: + weak_out: set | None = None, + strict_producer_ranges: bool = False) -> set[int]: """Enrichment candidates in one image: the STRONG set is returned. Candidates come from three generic sources -- stack-frame prologues, @@ -1443,7 +1444,10 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, passing image_local_entry_proven are STRONG. A jal/table target that only passes the bounded CFG probe (plausible_callable_target, not in a delay slot) is WEAK and goes to ``weak_out`` when given; derive_enrichment_roots - decides which weak ones survive. ``sources`` (optional) receives + decides which weak ones survive. With producer ranges, a jal or table in + one producer is only weak evidence about an address in the SAME producer; + across producers it can support a STRONG root (never under + ``strict_producer_ranges``). ``sources`` (optional) receives {addr: set of source names} for every returned or weak address. """ hi = load_addr + size if analysis_hi is None else analysis_hi @@ -1459,7 +1463,24 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, candidates: dict[int, set] = {} - def note(addr: int, source: str) -> None: + def producer_index(addr: int): + if not ranges: + return 0 + for index, (range_lo, range_hi) in enumerate(ranges): + if range_lo <= addr < range_hi: + return index + return None + + def note(addr: int, source: str, source_pc: int | None = None) -> None: + # Evidence from one producer about an address in another is the + # cross-producer case compile_overlays already treats as weaker: + # strict captures reject it outright, others need a local boundary. + if source_pc is not None: + src = producer_index(source_pc) + if src is None: + return + if src != producer_index(addr): + source += '_cross' candidates.setdefault(addr, set()).add(source) pointers = [] @@ -1468,7 +1489,7 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, if _is_addiu_sp_neg(word): note(addr, 'prologue') if word is not None and (word >> 26) == 0x03: - note(_jump_target(addr, word), 'jal') + note(_jump_target(addr, word), 'jal', addr) pointers.append(word if word is not None and not (word & 3) and load_addr <= word < hi else None) start = 0 @@ -1480,17 +1501,21 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, while end < len(pointers) and pointers[end] is not None: end += 1 if end - start >= 3: - for target in pointers[start:end]: - note(target, 'pointer_table') + for offset in range(start, end): + note(pointers[offset], 'pointer_table', load_addr + offset * 4) start = end + local = {'prologue', 'jal', 'pointer_table'} + cross = {'jal_cross', 'pointer_table_cross'} roots = set() for addr, why in candidates.items(): producer_hi = producer_hi_for(addr) if producer_hi is None: continue - if image_local_entry_proven(data, load_addr, size, addr, - producer_hi, hi): + strong_evidence = bool(why & local) or ( + bool(why & cross) and not strict_producer_ranges) + if strong_evidence and image_local_entry_proven( + data, load_addr, size, addr, producer_hi, hi): roots.add(addr) if sources is not None: sources[addr] = set(why) @@ -1535,7 +1560,9 @@ def possible_function_starts(data: bytes, load_addr: int, def derive_enrichment_roots(data: bytes, load_addr: int, size: int, producer_ranges=(), analysis_hi: int | None = None, - explicit=(), walk=None) -> tuple[set, dict]: + explicit=(), walk=None, + strict_producer_ranges: bool = False + ) -> tuple[set, dict]: """The enrichment roots compile_overlays adds for one image. STRONG candidates are returned as-is (the no-split guard demotes any that @@ -1557,7 +1584,8 @@ def derive_enrichment_roots(data: bytes, load_addr: int, size: int, return cache[key] weak = set() strong = derive_static_roots(data, load_addr, size, producer_ranges, hi, - weak_out=weak) + weak_out=weak, + strict_producer_ranges=strict_producer_ranges) hosts = ({a for a in explicit if load_addr <= a < hi and not a & 3} | strong | weak | possible_function_starts(data, load_addr, hi, producer_ranges)) @@ -1993,7 +2021,8 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, data, load_addr, size, producer_ranges, fragment_hi, explicit=(static_discovery_entries | captured_function_entries | dispatch_entry_pcs | toml_entries | legacy_seeds), - walk=walk) + walk=walk, + strict_producer_ranges=not allow_cross_producer_calls) derived_static_roots -= (static_discovery_entries | captured_function_entries | dispatch_entry_pcs | toml_entries | diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py index 29ff5dc9..d1f97568 100644 --- a/tools/tests/test_overlay_no_split_guard.py +++ b/tools/tests/test_overlay_no_split_guard.py @@ -266,6 +266,26 @@ class NoSplitGuardTests(unittest.TestCase): seeds, _audit = classify(data, True) self.assertIn(target, root_seeds(seeds)) + def test_cross_producer_call_evidence(self): + # Producer A calls into producer B. A cross-producer call is only + # ever weaker evidence: never WEAK evidence, STRONG only when B's own + # bytes bound the target and the capture is not strict. + a_fn, bounded, frameless = LOAD, LOAD + 0x110, LOAD + 0x130 + data = image({a_fn: FRAME, a_fn + 4: jal(bounded), a_fn + 8: NOP, + a_fn + 12: jal(frameless), a_fn + 16: NOP, + a_fn + 20: JR_RA, a_fn + 24: UNFRAME, + bounded - 8: JR_RA, bounded - 4: NOP, + bounded: ADDIU, bounded + 4: JR_RA, bounded + 8: NOP, + frameless - 4: ADDIU, + frameless: ADDIU, frameless + 4: JR_RA, + frameless + 8: NOP}, 0x200) + ranges = [(LOAD, LOAD + 0x100), (LOAD + 0x100, LOAD + 0x200)] + for strict in (True, False): + roots, _ = CO.derive_enrichment_roots( + data, LOAD, len(data), ranges, strict_producer_ranges=strict) + self.assertNotIn(frameless, roots) + self.assertEqual(bounded in roots, not strict) + # enrichment policy ---------------------------------------------------- def test_enrichment_is_default_and_opt_out_is_diagnostic(self): data, engine, host, _target = self._cross_image(False) From 97a8841a5d6d84c842dccc172a0044eb18eda71a Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 19:44:10 -0700 Subject: [PATCH 05/13] overlays: drop enrichment roots that a host's local branch jumps over Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/compile_overlays.py | 84 ++++++++++++++++++---- tools/tests/test_overlay_no_split_guard.py | 30 ++++++++ 2 files changed, 99 insertions(+), 15 deletions(-) diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index 4874a809..5a507fd7 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -1602,7 +1602,8 @@ def derive_enrichment_roots(data: bytes, load_addr: int, size: int, 'hosts': len(hosts)} -def no_split_partition(candidates, exempt, walk, hi: int): +def no_split_partition(candidates, exempt, walk, hi: int, droppable=(), + dropped_out: set | None = None): """Fixed point of the no-split guard over one root partition. For each root h, ascending: @@ -1623,6 +1624,14 @@ def no_split_partition(candidates, exempt, walk, hi: int): that intervening root would leave hostless. Absorption only removes caps, so whole passes repeat until nothing more is absorbed. + ``droppable`` candidates (weak evidence: enrichment roots) are also + removed when h's walk does NOT reach them but one of h's local branches + jumps over them -- an unreached word inside h's body (dead code, + embedded data) would cap h and turn that branch into a cross-function + exit. They are dropped, never aliased (not reachable from h), and + reported in ``dropped_out``. Capture-evidence roots keep their existing + behaviour in that position. + Returns (roots, {absorbed: owner}). """ roots = sorted(candidates) @@ -1640,20 +1649,41 @@ def no_split_partition(candidates, exempt, walk, hi: int): if index + 1 < len(roots): reach = reach | walk(host, roots[index + 1])['visited'] end = index + 1 - while (end < len(roots) and roots[end] in reach and - roots[end] not in exempt): + while (end < len(roots) and roots[end] not in exempt and + (roots[end] in reach or roots[end] in droppable)): end += 1 while end > index + 1: cap = roots[end] if end < len(roots) else hi - inner = walk(host, cap)['visited'] - blocked = next((k for k in range(index + 1, end) - if roots[k] not in inner), None) + entry_walk = walk(host, cap) + inner = entry_walk['visited'] + spans = entry_walk['local_branch_spans'] if droppable else () + blocked = None + for k in range(index + 1, end): + addr = roots[k] + if addr in inner: + continue + # Dropped only when one of h's own local branches (any + # conditional branch, or an always-taken b; never an + # absolute j, which is also how tail calls are encoded) + # jumps over it: compiled code never branches across a + # function boundary, so the word is an unreached part of + # h's body. + if addr in droppable and any( + src < addr < dst for src, dst in spans): + continue + blocked = k + break if blocked is None: break end = blocked if end > index + 1: + inner = walk(host, roots[end] if end < len(roots) + else hi)['visited'] for addr in roots[index + 1:end]: - absorbed[addr] = host + if addr in inner: + absorbed[addr] = host + elif dropped_out is not None: + dropped_out.add(addr) del roots[index + 1:end] changed = True index += 1 @@ -1678,6 +1708,11 @@ def _walk_overlay_function(data: bytes, load_addr: int, size: int, forward_branch_targets = set() rejected_cross_producer_calls = set() accepted_cross_producer_calls = set() + # (low, high) of every reached LOCAL branch -- conditional, or an + # always-taken b encoded as a branch; never an absolute j (also how tail + # calls are encoded). The no-split guard uses these to see an unreached + # word inside a function body. + local_branch_spans = [] def producer_for(addr: int): for range_lo, range_hi in producer_ranges: @@ -1737,6 +1772,8 @@ def _walk_overlay_function(data: bytes, load_addr: int, size: int, kind, target = _classify_cf(pc, word) delay = pc + 4 + if kind in ('branch', 'j') and (word >> 26) != 0x02: + local_branch_spans.append((min(pc, target), max(pc, target))) if kind == 'normal': if in_function(pc + 4): work.append(pc + 4) @@ -1832,6 +1869,7 @@ def _walk_overlay_function(data: bytes, load_addr: int, size: int, 'forward_branch_targets': forward_branch_targets, 'rejected_cross_producer_calls': rejected_cross_producer_calls, 'accepted_cross_producer_calls': accepted_cross_producer_calls, + 'local_branch_spans': local_branch_spans, } @@ -2042,7 +2080,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, pre_roots_guarded, _ = no_split_partition( {a for a in pre_roots if region(a) and not _in_delay_slot(data, load_addr, a)}, - set(), walk, hi) + set(), walk, hi, derived_static_roots) pre_roots_sorted = sorted(pre_roots_guarded) for i, entry in enumerate(pre_roots_sorted): hard_cap = pre_roots_sorted[i + 1] if i + 1 < len(pre_roots_sorted) else hi @@ -2176,7 +2214,10 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, # The explicit universe is fixed; the no-split guard decides, each round # and from scratch, which of it (plus derived targets) actually roots. initial_known = explicit_candidates - known, _ = no_split_partition(initial_known, set(), walk, hi) + # Enrichment roots are weak evidence: dropped (not aliased) when a host + # walks past them without reaching them. + droppable = initial_known & derived_static_roots + known, _ = no_split_partition(initial_known, set(), walk, hi, droppable) derived_reasons = {} promoted_roots = set() suppressed_derived = set() @@ -2218,7 +2259,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, + (f' ({unstable_text})' if unstable_text else '')) known, _ = no_split_partition( initial_known | promoted_roots, promoted_roots, - walk, hi) + walk, hi, droppable) derived_reasons = {} discovery_round = 0 seen_states = {} @@ -2230,7 +2271,8 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, f'unstable={unstable_text})' if cycle_start is not None else ' (round limit)')) known, _ = no_split_partition( - initial_known | promoted_roots, promoted_roots, walk, hi) + initial_known | promoted_roots, promoted_roots, walk, hi, + droppable) derived_reasons = {} break seen_states[state] = len(state_history) @@ -2240,7 +2282,11 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, sorted_known = sorted(known) def derivable(target: int) -> bool: - if target in initial_known or target in suppressed_derived: + # An enrichment root is weak evidence; a call from rooted code + # is stronger, so it may still derive (and stop being + # droppable). + if ((target in initial_known and target not in droppable) or + target in suppressed_derived): return False if _in_delay_slot(data, load_addr, target): delay_slot_rejected.setdefault(target, 'DERIVED') @@ -2292,7 +2338,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, # precisely because no walk covered them). normalized, _ = no_split_partition( initial_known | promoted_roots | set(next_reasons), - promoted_roots, walk, hi) + promoted_roots, walk, hi, droppable - set(next_reasons)) if normalized == known and next_reasons == derived_reasons: all_branch_targets = round_branch_targets @@ -2352,6 +2398,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, # the guard's fixed point that host exists by construction; the fallback # branch only keeps a broken invariant from minting a hostless alias. guard_demoted = {} + enrichment_shadowed = set() for addr in sorted(initial_known - known): prior = included.get(addr) index = bisect_left(sorted_known, addr) @@ -2361,8 +2408,12 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, guard_demoted[addr] = (prior, host) else: included.pop(addr, None) - excluded[addr] = ('OBSERVED_PC_ONLY' - if addr in executed_pcs else 'UNKNOWN') + if addr in droppable: + enrichment_shadowed.add(addr) + excluded[addr] = 'ENRICHMENT_SHADOWED' + else: + excluded[addr] = ('OBSERVED_PC_ONLY' + if addr in executed_pcs else 'UNKNOWN') # A DISPATCH_INTERIOR is emitted only when the final partition has a real # CFG host for that exact PC. Reconsider every derived target after all @@ -2454,6 +2505,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, 'derived_static_roots': derived_static_roots, 'enrichment_stats': enrichment_stats, 'guard_demoted': guard_demoted, + 'enrichment_shadowed': enrichment_shadowed, 'delay_slot_rejected': delay_slot_rejected, } # Interior entries carry the 'interior' marker so the recompiler emits @@ -2513,6 +2565,8 @@ def print_seed_audit(audit: dict) -> None: if stats else '') + ')') guard_demoted = audit.get('guard_demoted', {}) print(f'no_split_guard_demoted: {len(guard_demoted)}') + print(f'enrichment_roots_shadowed: ' + f'{len(audit.get("enrichment_shadowed", ()))}') print(f'delay_slot_roots_rejected: ' f'{len(audit.get("delay_slot_rejected", {}))}') for addr, (prior, host) in sorted(guard_demoted.items()): diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py index d1f97568..b3c6ea09 100644 --- a/tools/tests/test_overlay_no_split_guard.py +++ b/tools/tests/test_overlay_no_split_guard.py @@ -115,6 +115,36 @@ class NoSplitGuardTests(unittest.TestCase): self.assertNotIn(target, root_seeds(seeds)) self.assertEqual(audit['guard_demoted'][target][1], host) + def test_enrichment_root_in_unreached_body_word_is_dropped(self): + # The host branches (b) over a dead block whose first word looks like + # a prologue. As a root it would cap the host and turn the branch + # into a cross-function exit, although nothing reaches it. Enrichment + # evidence is weak: it is dropped, not rooted and not aliased. + host = LOAD + 0x20 + dead, label = host + 0x10, host + 0x30 + data = image({host: FRAME, + host + 4: 0x10000000 | (((label - (host + 8)) >> 2) + & 0xFFFF), + host + 8: NOP, + dead: FRAME, dead + 4: ADDIU, dead + 8: JR_RA, + dead + 12: UNFRAME, + label: ADDIU, label + 4: JR_RA, + label + 8: UNFRAME}, 0x100) + self.assertIn(dead, CO.derive_static_roots(data, LOAD, len(data))) + seeds, audit = classify(data, True, function_entry_pcs=[host]) + self.assertNotIn(dead, root_seeds(seeds)) + self.assertNotIn(dead, audit['included_reasons']) + self.assertEqual(audit['excluded_reasons'][dead], + 'ENRICHMENT_SHADOWED') + self.assertEqual(root_seeds(seeds), {host}) + # An absolute j over it is how tail calls are encoded, so that alone + # does not make the candidate part of the host. + tail = bytearray(data) + struct.pack_into('> 2) & 0x03FFFFFF)) + seeds, audit = classify(bytes(tail), True, function_entry_pcs=[host]) + self.assertIn(dead, root_seeds(seeds)) + def test_delay_slot_candidate_is_never_root_nor_alias(self): host, callee = LOAD + 0x20, LOAD + 0x80 slot = host + 8 From 61610b6e3b7524665e7e9b6ddb7b4c32f92381ce Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 20:17:38 -0700 Subject: [PATCH 06/13] overlays: surface recompiler guard absorptions from fragment batches; plan status Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/internal/OVERLAY_ROOT_ENRICHMENT.md | 31 ++++++++++++++++++++++++ tools/compile_overlays.py | 1 + 2 files changed, 32 insertions(+) diff --git a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md index 77310569..f5318e8f 100644 --- a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md +++ b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md @@ -74,6 +74,37 @@ mis-splits**, then as much coverage as possible. - Then play and explore areas that load new overlays: levels, menus, cutscenes, save and load. +## Status: steps 1 and 2 (beads-eio.3.177) + +- **Guard** (`no_split_partition` in `compile_overlays.py`, same rule in + `FunctionAnalyzer::analyze_exact_entries` for overlay mode). For each root, + the following candidates its uncapped walk reaches are absorbed together + (a switch resolves only when all its case labels are inside the walk); the + run stops at the first unreached root, so a tail call over another function + keeps its target. Applies to every root source; promoted kernel orphans are + exempt. Delay-slot candidates are never roots or aliases. +- **Evidence.** A jal or pointer only proves a function start in the image + that is resident when it runs. STRONG = a prologue, or a jal/table target + that this image bounds (`jr $ra` before it plus the CFG probe). WEAK = CFG + probe only; kept only if no possible function start reaches it, and dropped + if a host's local branch jumps over it. Cross-producer calls are never weak + evidence. +- **Default on, one code path.** `classify_overlay_seeds` derives the roots; + `tools/enrich_overlay_captures.py` is an inspection tool built on the same + functions. `--no-root-enrichment` / `PSX_OVERLAY_ROOT_ENRICHMENT=0` is + diagnostic only. +- **Cache key.** `tools/compile_overlays.py` is in + `runtime/codegen_hash_sources.cmake`, so any root-policy change moves the + cache namespace. +- **Tomba (25 AOT images).** Emitted fallthrough splits: master 57, #386 + enrichment 2,072, guard 0. Fingerprints match master on every value and + timing field over the 30.7k-frame route and the 32k attract run. Sound + enrichment adds almost nothing on Tomba: 12.3k of #386's extra roots were + jump-table case labels inside functions. The way to that coverage is + resolving those tables, not rooting their labels. +- **Not caused by splits:** master's own 3-4 cycle drift from the interpreter + at frame 2111. It is unchanged with the guard. + ## Done when - The no-split guard is in, with a test that a mid-function candidate is diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index 5a507fd7..60010118 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -4929,6 +4929,7 @@ def compile_fragment_batch(requested_entries, data: bytes, load_addr: int, env=sub_env) if r.returncode != 0: return None, f'recompiler-error: {(r.stderr or r.stdout or "").strip()}' + report_recompiler_guard_absorptions(r.stdout) full_c = ranges_src = None for fn in os.listdir(out_dir_tmp): if fn.endswith('_full.c'): From a7e6520753077bbed5194cbc1b16c72c60dbfe4a Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 20:29:56 -0700 Subject: [PATCH 07/13] tests: no-split guard cache-key test finds cmake on PATH when run directly Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/tests/test_overlay_no_split_guard.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py index b3c6ea09..792fe139 100644 --- a/tools/tests/test_overlay_no_split_guard.py +++ b/tools/tests/test_overlay_no_split_guard.py @@ -34,7 +34,7 @@ FRAME = 0x27BDFFE0 # addiu sp,sp,-32 UNFRAME = 0x27BD0020 # addiu sp,sp,32 ADDIU = 0x24420001 # addiu v0,v0,1 SUBU = 0x00431023 # subu v0,v0,v1 -CMAKE = None +CMAKE = shutil.which('cmake') # --cmake overrides; ctest always passes it RECOMPILER = None @@ -350,6 +350,8 @@ class NoSplitGuardTests(unittest.TestCase): # (e) ---------------------------------------------------------------------- class CacheKeyTests(unittest.TestCase): def _cmake_list(self): + self.assertIsNotNone(CMAKE, 'cmake not found: pass --cmake or put ' + 'cmake on PATH') with tempfile.TemporaryDirectory() as tmp: script = Path(tmp) / 'list.cmake' script.write_text( From 39490b23b3fd36e562126b8955b27326482a8bf0 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 22:09:26 -0700 Subject: [PATCH 08/13] overlays: strong-root supplement fragments carry their hosts' aliases Without them a candidate the no-split guard absorbed into a host compiled outside the region shard lost its native entry (Tomba 2: 127 captured dispatch entries). Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/compile_overlays.py | 79 +- tools/tests/test_overlay_no_split_guard.py | 1016 +++++++++++--------- 2 files changed, 638 insertions(+), 457 deletions(-) diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index 60010118..14eb4973 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -2432,6 +2432,18 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, del included[addr] excluded[addr] = ('OBSERVED_PC_ONLY' if addr in executed_pcs else 'UNKNOWN') + # Every surviving alias with the final root(s) whose walk reaches it. The + # region shard carries all aliases because it compiles every root; a host + # compiled instead in a supplemental root fragment (candidate capacity, + # region audit fallout) must carry its aliases too, or an address the + # no-split guard absorbed loses its callable identity and runs + # interpreted where the unguarded build had a (split) native entry. + interior_hosts = { + addr: {host for host, entry_walk in final_walks.items() + if addr in entry_walk['visited']} + for addr, reason in included.items() + if reason == 'DISPATCH_INTERIOR' + } for addr in delay_slot_rejected: if addr not in included: @@ -2505,6 +2517,7 @@ def classify_overlay_seeds(cap: dict, data: bytes, load_addr: int, size: int, 'derived_static_roots': derived_static_roots, 'enrichment_stats': enrichment_stats, 'guard_demoted': guard_demoted, + 'interior_hosts': interior_hosts, 'enrichment_shadowed': enrichment_shadowed, 'delay_slot_rejected': delay_slot_rejected, } @@ -4305,6 +4318,12 @@ def make_interior_fragment_job(phys_addr: int, load_addr: int, size: int, addr: {reason} for addr, reason in seed_audit['included_reasons'].items() }, + # DISPATCH_INTERIOR alias -> final host root(s) that reach it; a root + # fragment carries the aliases of the hosts it compiles. + 'interior_hosts': { + _canonical_guest_addr(addr): {_canonical_guest_addr(h) for h in hosts} + for addr, hosts in seed_audit.get('interior_hosts', {}).items() + }, 'static_interval_demands': static_interval_demands, 'forced': forced, 'producer_ranges': tuple(seed_audit['producer_ranges']), @@ -4352,6 +4371,10 @@ def merge_fragment_jobs_by_recipe(jobs: list[dict], addr: set(reasons) for addr, reasons in job.get('included_reasons', {}).items() } + merged['interior_hosts'] = { + addr: set(hosts) + for addr, hosts in job.get('interior_hosts', {}).items() + } grouped[key] = merged continue for field in set_fields: @@ -4359,6 +4382,9 @@ def merge_fragment_jobs_by_recipe(jobs: list[dict], for addr, reasons in job.get('included_reasons', {}).items(): merged.setdefault('included_reasons', {}).setdefault( addr, set()).update(reasons) + for addr, hosts in job.get('interior_hosts', {}).items(): + merged.setdefault('interior_hosts', {}).setdefault( + addr, set()).update(hosts) if merged.get('resident_cap') is None and job.get('resident_cap'): merged['resident_cap'] = job['resident_cap'] return [grouped[key] for key in sorted(grouped)] @@ -4622,6 +4648,35 @@ def partition_strong_root_demands(static_exact: set[int], executed: set[int], return (sorted(missing - isolated), sorted(missing & isolated)) +def fragment_interior_aliases(job: dict, entries) -> list[int]: + """DISPATCH_INTERIOR aliases whose final host is one of ``entries``. + + A strong-root supplement fragment compiles hosts the region shard left + out. Their aliases (dispatch interiors and every candidate the no-split + guard absorbed) must ride along, or the absorbed address is served by + neither shard and dispatches to it fall back to the interpreter. + """ + roots = {_canonical_guest_addr(entry) for entry in entries} + return sorted( + alias for alias, hosts in job.get('interior_hosts', {}).items() + if alias not in roots and hosts & roots) + + +def compile_root_fragment_with_aliases(entries, aliases, build): + """Build one root fragment carrying its hosts' aliases. + + ``build(alias_list)`` returns the usual ``(func_ids, status)``. A failing + multi-root batch is returned as-is so the caller bisects it; a single root + whose aliases break its fragment is rebuilt without them, so an alias can + never cost its host its native entry. + """ + frag_ids, status = build(list(aliases)) + if (not frag_ids and aliases and len(set(entries)) == 1 and + not str(status).startswith('candidate-capacity:')): + frag_ids, status = build([]) + return frag_ids, status + + def compile_batched_fragment_roots(entries, compile_one, on_success, on_singleton_failure, still_needed=lambda _entry: True, @@ -4849,7 +4904,7 @@ def compile_fragment_batch(requested_entries, data: bytes, load_addr: int, producer_ranges=(), cross_call_allow=(), hosted_owners: dict | None = None, manifest_provenance: str | None = None, - *, guard_bytes: int): + *, guard_bytes: int, interior_aliases=()): """Compile an ISOLATED interior-entry 'island' fragment that ENTERS at an executed orphan DISPATCH_INTERIOR PC (a host that static analysis never discovered, e.g. an FMV driver reached via a computed jump) and covers the @@ -4903,6 +4958,14 @@ def compile_fragment_batch(requested_entries, data: bytes, load_addr: int, if hosted_specs is None: for entry in requested: f.write(f'dispatch_root 0x{entry:08X}\n') + # Alias candidates of the roots compiled here, exactly as the + # region shard passes them: never walk roots, emitted only + # where the recompiler's own walk of a requested root reaches + # the PC (orphans are dropped by the recompiler). + for alias in sorted({_canonical_guest_addr(a) + for a in interior_aliases} - + set(requested)): + f.write(f'interior 0x{alias:08X}\n') else: host_markers = {} for entry, spec in sorted(hosted_specs.items()): @@ -7433,11 +7496,15 @@ def main(): continue def compile_strong_batch(entries): - return compile_fragment_batch( - entries, data, load_addr, size, phys_addr, cache_dir, - args, frag_env, toml, job['producer_ranges'], - job['cross_call_allow'], - guard_bytes=job['guard_bytes']) + def build(alias_list): + return compile_fragment_batch( + entries, data, load_addr, size, phys_addr, cache_dir, + args, frag_env, toml, job['producer_ranges'], + job['cross_call_allow'], + guard_bytes=job['guard_bytes'], + interior_aliases=alias_list) + return compile_root_fragment_with_aliases( + entries, fragment_interior_aliases(job, entries), build) def strong_batch_success(entries, frag_ids, status): strong_handled.update(entries) diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py index 792fe139..7d474767 100644 --- a/tools/tests/test_overlay_no_split_guard.py +++ b/tools/tests/test_overlay_no_split_guard.py @@ -1,451 +1,565 @@ -#!/usr/bin/env python3 -"""The overlay no-split guard, image-local root enrichment, and the cache key. - -A walk root is a hard cap: the walk of the root below it stops there, so a -root inside another function's body splits that function. The generated C -still passes every audit, which is why this class never shows up as a shard -failure. Measured on Tomba (bead beads-eio.3.177): shared engine code, byte -identical in 12 swapped area images, does `jal 0x8011B1CC`; that address is a -function start in one image and a `subu` mid-way through host 0x8011AC10 in -X00. Rooting it there split the host and moved guest timing by one cycle -against the interpreter. - -Usage: test_overlay_no_split_guard.py --cmake [--recompiler ] -""" -import argparse -import os -import re -import shutil -import struct -import subprocess -import sys -import tempfile -import unittest -from pathlib import Path - -ROOT = Path(__file__).resolve().parents[2] -sys.path.insert(0, str(ROOT / 'tools')) -import compile_overlays as CO # noqa: E402 - -LOAD = 0x80100000 -NOP = 0x00000000 -JR_RA = 0x03E00008 -FRAME = 0x27BDFFE0 # addiu sp,sp,-32 -UNFRAME = 0x27BD0020 # addiu sp,sp,32 -ADDIU = 0x24420001 # addiu v0,v0,1 -SUBU = 0x00431023 # subu v0,v0,v1 -CMAKE = shutil.which('cmake') # --cmake overrides; ctest always passes it -RECOMPILER = None - - -def jal(target): - return 0x0C000000 | ((target >> 2) & 0x03FFFFFF) - - -def bne_v0(pc, target): - return 0x14400000 | (((target - (pc + 4)) >> 2) & 0xFFFF) - - -def image(words: dict, size: int) -> bytes: - data = bytearray(size) - for addr, word in words.items(): - struct.pack_into('> 2) - & 0xFFFF), - host + 8: NOP, - dead: FRAME, dead + 4: ADDIU, dead + 8: JR_RA, - dead + 12: UNFRAME, - label: ADDIU, label + 4: JR_RA, - label + 8: UNFRAME}, 0x100) - self.assertIn(dead, CO.derive_static_roots(data, LOAD, len(data))) - seeds, audit = classify(data, True, function_entry_pcs=[host]) - self.assertNotIn(dead, root_seeds(seeds)) - self.assertNotIn(dead, audit['included_reasons']) - self.assertEqual(audit['excluded_reasons'][dead], - 'ENRICHMENT_SHADOWED') - self.assertEqual(root_seeds(seeds), {host}) - # An absolute j over it is how tail calls are encoded, so that alone - # does not make the candidate part of the host. - tail = bytearray(data) - struct.pack_into('> 2) & 0x03FFFFFF)) - seeds, audit = classify(bytes(tail), True, function_entry_pcs=[host]) - self.assertIn(dead, root_seeds(seeds)) - - def test_delay_slot_candidate_is_never_root_nor_alias(self): - host, callee = LOAD + 0x20, LOAD + 0x80 - slot = host + 8 - data = image({host: FRAME, host + 4: jal(callee), slot: ADDIU, - host + 12: JR_RA, host + 16: UNFRAME, - callee: JR_RA, callee + 4: NOP}, 0x100) - seeds, audit = classify(data, True, function_entry_pcs=[host], - static_discovery_entry_pcs=[slot], - dispatch_entry_pcs=[slot]) - self.assertNotIn(slot, audit['included_reasons']) - self.assertEqual(audit['excluded_reasons'][slot], 'DELAY_SLOT') - self.assertFalse(any(seed.endswith(f'0x{slot:08X}') for seed in seeds)) - self.assertIn(slot, audit['delay_slot_rejected']) - self.assertFalse(CO.image_local_entry_proven( - data, LOAD, len(data), slot, LOAD + len(data))) - - def test_switch_case_roots_absorbed_together(self): - # Every case label passes the image-local proof (each case before it - # ends in `jr ra`), and the jump table is a >=3 pointer run, so - # enrichment derives them all. The table resolves only when every - # target is inside the host's walk, so they must be absorbed as a run. - host = LOAD - table = LOAD + 0x200 - cases = [LOAD + 0x68, LOAD + 0x78, LOAD + 0x88] - words = {host: FRAME, - LOAD + 0x04: 0x3C088010, # lui t0,0x8010 - LOAD + 0x08: 0x25100200, # addiu s0,t0,0x200 - LOAD + 0x0C: 0x2C620003, # sltiu v0,v1,3 - LOAD + 0x10: 0x10400000 | ((0x98 - 0x14) >> 2), # beq v0,zero - LOAD + 0x14: NOP, - LOAD + 0x18: 0x00031080, # sll v0,v1,2 - LOAD + 0x1C: 0x00501021, # addu v0,v0,s0 - LOAD + 0x20: 0x8C420000, # lw v0,0(v0) - LOAD + 0x24: NOP, - LOAD + 0x28: 0x00400008, # jr v0 - LOAD + 0x2C: NOP, - LOAD + 0x98: JR_RA, LOAD + 0x9C: UNFRAME} - for index, case in enumerate(cases): - words[case] = ADDIU - words[case + 4] = JR_RA - words[case + 8] = UNFRAME - words[table + index * 4] = case - data = image(words, 0x300) - derived = CO.derive_static_roots(data, LOAD, len(data)) - self.assertTrue(set(cases[1:]) <= derived) - # The guard alone (the case labels supplied as explicit roots, as a - # #386-enriched capture would) absorbs the whole run. - seeds, audit = classify(data, False, function_entry_pcs=[host], - static_discovery_entry_pcs=cases[1:]) - for case in cases[1:]: - self.assertEqual(audit['included_reasons'][case], - 'DISPATCH_INTERIOR', hex(case)) - self.assertNotIn(case, root_seeds(seeds)) - # Default enrichment: the guarded prepass resolves the table, so the - # case labels are never offered as roots at all. - seeds, audit = classify(data, True, function_entry_pcs=[host]) - for case in cases: - self.assertIn(audit['included_reasons'].get(case), - (None, 'DISPATCH_INTERIOR'), hex(case)) - self.assertNotIn(case, root_seeds(seeds)) - self.assertEqual(root_seeds(seeds), {host}) - - # (c) ------------------------------------------------------------------ - def test_jal_proven_function_start_stays_a_root(self): - caller, framed, frameless = LOAD, LOAD + 0x40, LOAD + 0x80 - data = image({caller: FRAME, caller + 4: jal(framed), - caller + 8: NOP, caller + 12: jal(frameless), - caller + 16: NOP, caller + 20: JR_RA, - caller + 24: UNFRAME, - framed: FRAME, framed + 4: ADDIU, framed + 8: JR_RA, - framed + 12: UNFRAME, - frameless - 8: JR_RA, frameless - 4: NOP, - frameless: ADDIU, frameless + 4: JR_RA, - frameless + 8: NOP}, 0x100) - derived = CO.derive_static_roots(data, LOAD, len(data)) - self.assertTrue({caller, framed, frameless} <= derived) - seeds, audit = classify(data, True) - roots = root_seeds(seeds) - self.assertTrue({caller, framed, frameless} <= roots) - self.assertEqual(audit['guard_demoted'], {}) - - # (d) ------------------------------------------------------------------ - def _cross_image(self, target_is_function): - engine, host = LOAD, LOAD + 0x100 - target = LOAD + 0x110 - words = {engine: FRAME, engine + 4: jal(target), engine + 8: NOP, - engine + 12: JR_RA, engine + 16: UNFRAME} - if target_is_function: - # Sibling image: the called address is a real function here. - words.update({host: ADDIU, host + 4: JR_RA, host + 8: NOP, - target: FRAME, target + 4: ADDIU, - target + 8: JR_RA, target + 12: UNFRAME}) - else: - # This image: the same address is mid-way through a host - # reached by pure fallthrough (the X00 shape). - words.update({host: FRAME, host + 4: ADDIU, host + 8: ADDIU, - host + 12: ADDIU, target: SUBU, - target + 4: ADDIU, target + 8: JR_RA, - target + 12: UNFRAME}) - return image(words, 0x200), engine, host, target - - def test_shared_jal_into_mid_body_is_not_rooted(self): - data, engine, host, target = self._cross_image(False) - weak = set() - strong = CO.derive_static_roots(data, LOAD, len(data), weak_out=weak) - self.assertNotIn(target, strong) - self.assertIn(target, weak) # the CFG probe alone accepts it - self.assertTrue({engine, host} <= strong) - derived, stats = CO.derive_enrichment_roots(data, LOAD, len(data)) - self.assertNotIn(target, derived) # host's prologue reaches it - self.assertEqual(stats['weak_dropped'], 1) - # Enrichment on (default): the rooted engine walk still derives the - # jal target, and the guard absorbs it into its host. - seeds, audit = classify(data, True) - self.assertNotIn(target, root_seeds(seeds)) - self.assertEqual(audit['included_reasons'][target], - 'DISPATCH_INTERIOR') - self.assertIn(host, root_seeds(seeds)) - # A #386-enriched capture that already names it is demoted too. - seeds, audit = classify(data, True, - static_discovery_entry_pcs=[target]) - self.assertNotIn(target, root_seeds(seeds)) - self.assertEqual(audit['guard_demoted'][target], - ('STATIC_DISCOVERY_ROOT', host)) - - def test_weak_target_nothing_reaches_is_rooted(self): - # A frameless function right after another function's tail jump, - # called by jal: no `jr $ra` before it, so no boundary proof, but no - # possible function start reaches it either. - caller, tail, leaf, far = LOAD, LOAD + 0x40, LOAD + 0x4C, LOAD + 0x80 - data = image({caller: FRAME, caller + 4: jal(leaf), caller + 8: NOP, - caller + 12: JR_RA, caller + 16: UNFRAME, - tail: ADDIU, - tail + 4: 0x08000000 | ((far >> 2) & 0x03FFFFFF), - tail + 8: NOP, - leaf: ADDIU, leaf + 4: JR_RA, leaf + 8: NOP, - far: JR_RA, far + 4: NOP}, 0x100) - weak = set() - CO.derive_static_roots(data, LOAD, len(data), weak_out=weak) - self.assertIn(leaf, weak) - derived, _stats = CO.derive_enrichment_roots(data, LOAD, len(data)) - self.assertIn(leaf, derived) - seeds, _audit = classify(data, True) - self.assertIn(leaf, root_seeds(seeds)) - - def test_shared_jal_into_real_function_in_sibling_image_is_rooted(self): - data, _engine, _host, target = self._cross_image(True) - self.assertIn(target, CO.derive_static_roots(data, LOAD, len(data))) - seeds, _audit = classify(data, True) - self.assertIn(target, root_seeds(seeds)) - - def test_cross_producer_call_evidence(self): - # Producer A calls into producer B. A cross-producer call is only - # ever weaker evidence: never WEAK evidence, STRONG only when B's own - # bytes bound the target and the capture is not strict. - a_fn, bounded, frameless = LOAD, LOAD + 0x110, LOAD + 0x130 - data = image({a_fn: FRAME, a_fn + 4: jal(bounded), a_fn + 8: NOP, - a_fn + 12: jal(frameless), a_fn + 16: NOP, - a_fn + 20: JR_RA, a_fn + 24: UNFRAME, - bounded - 8: JR_RA, bounded - 4: NOP, - bounded: ADDIU, bounded + 4: JR_RA, bounded + 8: NOP, - frameless - 4: ADDIU, - frameless: ADDIU, frameless + 4: JR_RA, - frameless + 8: NOP}, 0x200) - ranges = [(LOAD, LOAD + 0x100), (LOAD + 0x100, LOAD + 0x200)] - for strict in (True, False): - roots, _ = CO.derive_enrichment_roots( - data, LOAD, len(data), ranges, strict_producer_ranges=strict) - self.assertNotIn(frameless, roots) - self.assertEqual(bounded in roots, not strict) - - # enrichment policy ---------------------------------------------------- - def test_enrichment_is_default_and_opt_out_is_diagnostic(self): - data, engine, host, _target = self._cross_image(False) - old = os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None) - try: - self.assertTrue(CO.root_enrichment_default()) - _seeds, audit = classify(data, None) - self.assertTrue(audit['root_enrichment']) - self.assertTrue({engine, host} <= audit['derived_static_roots']) - os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = '0' - self.assertFalse(CO.root_enrichment_default()) - _seeds, audit = classify(data, None) - self.assertFalse(audit['root_enrichment']) - self.assertEqual(audit['derived_static_roots'], set()) - finally: - os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None) - if old is not None: - os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = old - # A conservative retry recipe is never enriched again. - cap = {'schema': 'psxrecomp overlay capture v2', - CO.ROOT_ENRICHMENT_OFF_KEY: True} - _seeds, audit = CO.classify_overlay_seeds( - cap, data, LOAD, len(data), 0, {}, root_enrichment=True) - self.assertEqual(audit['derived_static_roots'], set()) - retry = CO.conservative_retry_capture( - {'schema': 'x'}, {'derived_static_roots': {engine}}) - self.assertTrue(retry[CO.ROOT_ENRICHMENT_OFF_KEY]) - self.assertIsNone(CO.conservative_retry_capture( - retry, {'derived_static_roots': {engine}})) - - -# (e) ---------------------------------------------------------------------- -class CacheKeyTests(unittest.TestCase): - def _cmake_list(self): - self.assertIsNotNone(CMAKE, 'cmake not found: pass --cmake or put ' - 'cmake on PATH') - with tempfile.TemporaryDirectory() as tmp: - script = Path(tmp) / 'list.cmake' - script.write_text( - f'set(PSXRECOMP_CODEGEN_HASH_ROOT "{ROOT.as_posix()}")\n' - f'include("{(ROOT / "runtime/codegen_hash_sources.cmake").as_posix()}")\n' - 'message(STATUS "SRCS=${PSXRECOMP_CODEGEN_HASH_SRCS}")\n', - encoding='utf-8') - proc = subprocess.run([CMAKE, '-P', str(script)], - capture_output=True, text=True) - self.assertEqual(proc.returncode, 0, proc.stderr) - match = re.search(r'SRCS=(.*)', proc.stdout + proc.stderr) - self.assertIsNotNone(match) - return match.group(1).strip().split(';') - - def _hash_header(self, sources, out_dir): - header = Path(out_dir) / 'overlay_codegen_hash.h' - proc = subprocess.run( - [CMAKE, f'-DOUT={header.as_posix()}', - '-DSRCS=' + ';'.join(sources), - '-P', str(ROOT / 'runtime' / 'hash_codegen.cmake')], - capture_output=True, text=True) - self.assertEqual(proc.returncode, 0, proc.stderr) - shutil.copy2(ROOT / 'runtime' / 'include' / 'overlay_api.h', - Path(out_dir) / 'overlay_api.h') - return CO.codegen_hash(str(out_dir)) - - def test_root_policy_is_part_of_the_cache_namespace(self): - sources = self._cmake_list() - classifier = (ROOT / 'tools' / 'compile_overlays.py').as_posix() - self.assertIn(classifier, sources) - with tempfile.TemporaryDirectory() as tmp: - tmp = Path(tmp) - changed = tmp / 'compile_overlays.py' - changed.write_text( - Path(classifier).read_text(encoding='utf-8') + - '\n# a root-policy change\n', encoding='utf-8') - (tmp / 'a').mkdir() - (tmp / 'b').mkdir() - before = self._hash_header(sources, tmp / 'a') - after = self._hash_header( - [changed.as_posix() if s == classifier else s - for s in sources], tmp / 'b') - self.assertNotEqual(before, after) - saved = CO.overlay_config_hash - CO.overlay_config_hash = lambda recompiler, toml: 0x1234ABCD - try: - tag_a = CO.cache_tag(str(tmp / 'a'), 'r', 'g', 0) - tag_b = CO.cache_tag(str(tmp / 'b'), 'r', 'g', 0) - finally: - CO.overlay_config_hash = saved - self.assertNotEqual(tag_a, tag_b) - - -# recompiler enforcement --------------------------------------------------- -class RecompilerGuardTests(unittest.TestCase): - def test_recompiler_absorbs_explicit_fallthrough_root(self): - if not RECOMPILER: - self.skipTest('pass --recompiler (ctest always does)') - host, mid = LOAD + 0x20, LOAD + 0x30 - data = image({host: FRAME, host + 4: ADDIU, host + 8: ADDIU, - host + 12: ADDIU, mid: SUBU, mid + 4: ADDIU, - mid + 8: JR_RA, mid + 12: UNFRAME}, 0x1000) - with tempfile.TemporaryDirectory() as tmp: - psx = Path(tmp) / 'split.psx' - psx.write_bytes(CO.make_psxexe(LOAD, host, data, guard_bytes=0)) - seeds = Path(tmp) / 'seeds.txt' - # Seeds the classifier would never write: both as call roots. - seeds.write_text(f'call_root 0x{host:08X}\ncall_root 0x{mid:08X}\n') - out = Path(tmp) / 'out' - proc = subprocess.run( - [RECOMPILER, str(psx), '--seeds', str(seeds), - '--out-dir', str(out), '--overlay'], - capture_output=True, text=True, cwd=str(ROOT)) - self.assertEqual(proc.returncode, 0, proc.stderr or proc.stdout) - self.assertEqual( - CO.report_recompiler_guard_absorptions(proc.stdout), 1, - proc.stdout) - ranges = next(out.glob('*_full.ranges')).read_text() - # Both PCs stay callable; the mid-body one only as an alias whose - # identity covers the host range (it starts at the host). - self.assertIn(f'F {host:08X}', ranges) - self.assertIn(f'F {mid:08X}', ranges) - mid_rows = ranges.split(f'F {mid:08X}', 1)[1].splitlines()[1:] - self.assertTrue(any(row.startswith(f'R {host:08X}') - for row in mid_rows[:2]), ranges) - - -if __name__ == '__main__': - ap = argparse.ArgumentParser() - ap.add_argument('--cmake', default=shutil.which('cmake')) - ap.add_argument('--recompiler') - args, rest = ap.parse_known_args() - if not args.cmake: - sys.exit('cmake not found: pass --cmake') - CMAKE = args.cmake - RECOMPILER = args.recompiler - unittest.main(argv=[sys.argv[0]] + rest) +#!/usr/bin/env python3 +"""The overlay no-split guard, image-local root enrichment, and the cache key. + +A walk root is a hard cap: the walk of the root below it stops there, so a +root inside another function's body splits that function. The generated C +still passes every audit, which is why this class never shows up as a shard +failure. Measured on Tomba (bead beads-eio.3.177): shared engine code, byte +identical in 12 swapped area images, does `jal 0x8011B1CC`; that address is a +function start in one image and a `subu` mid-way through host 0x8011AC10 in +X00. Rooting it there split the host and moved guest timing by one cycle +against the interpreter. + +Usage: test_overlay_no_split_guard.py --cmake [--recompiler ] +""" +import argparse +import os +import re +import shutil +import struct +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / 'tools')) +import compile_overlays as CO # noqa: E402 + +LOAD = 0x80100000 +NOP = 0x00000000 +JR_RA = 0x03E00008 +FRAME = 0x27BDFFE0 # addiu sp,sp,-32 +UNFRAME = 0x27BD0020 # addiu sp,sp,32 +ADDIU = 0x24420001 # addiu v0,v0,1 +SUBU = 0x00431023 # subu v0,v0,v1 +CMAKE = shutil.which('cmake') # --cmake overrides; ctest always passes it +RECOMPILER = None + + +def jal(target): + return 0x0C000000 | ((target >> 2) & 0x03FFFFFF) + + +def bne_v0(pc, target): + return 0x14400000 | (((target - (pc + 4)) >> 2) & 0xFFFF) + + +def image(words: dict, size: int) -> bytes: + data = bytearray(size) + for addr, word in words.items(): + struct.pack_into('> 2) + & 0xFFFF), + host + 8: NOP, + dead: FRAME, dead + 4: ADDIU, dead + 8: JR_RA, + dead + 12: UNFRAME, + label: ADDIU, label + 4: JR_RA, + label + 8: UNFRAME}, 0x100) + self.assertIn(dead, CO.derive_static_roots(data, LOAD, len(data))) + seeds, audit = classify(data, True, function_entry_pcs=[host]) + self.assertNotIn(dead, root_seeds(seeds)) + self.assertNotIn(dead, audit['included_reasons']) + self.assertEqual(audit['excluded_reasons'][dead], + 'ENRICHMENT_SHADOWED') + self.assertEqual(root_seeds(seeds), {host}) + # An absolute j over it is how tail calls are encoded, so that alone + # does not make the candidate part of the host. + tail = bytearray(data) + struct.pack_into('> 2) & 0x03FFFFFF)) + seeds, audit = classify(bytes(tail), True, function_entry_pcs=[host]) + self.assertIn(dead, root_seeds(seeds)) + + def test_delay_slot_candidate_is_never_root_nor_alias(self): + host, callee = LOAD + 0x20, LOAD + 0x80 + slot = host + 8 + data = image({host: FRAME, host + 4: jal(callee), slot: ADDIU, + host + 12: JR_RA, host + 16: UNFRAME, + callee: JR_RA, callee + 4: NOP}, 0x100) + seeds, audit = classify(data, True, function_entry_pcs=[host], + static_discovery_entry_pcs=[slot], + dispatch_entry_pcs=[slot]) + self.assertNotIn(slot, audit['included_reasons']) + self.assertEqual(audit['excluded_reasons'][slot], 'DELAY_SLOT') + self.assertFalse(any(seed.endswith(f'0x{slot:08X}') for seed in seeds)) + self.assertIn(slot, audit['delay_slot_rejected']) + self.assertFalse(CO.image_local_entry_proven( + data, LOAD, len(data), slot, LOAD + len(data))) + + def test_switch_case_roots_absorbed_together(self): + # Every case label passes the image-local proof (each case before it + # ends in `jr ra`), and the jump table is a >=3 pointer run, so + # enrichment derives them all. The table resolves only when every + # target is inside the host's walk, so they must be absorbed as a run. + host = LOAD + table = LOAD + 0x200 + cases = [LOAD + 0x68, LOAD + 0x78, LOAD + 0x88] + words = {host: FRAME, + LOAD + 0x04: 0x3C088010, # lui t0,0x8010 + LOAD + 0x08: 0x25100200, # addiu s0,t0,0x200 + LOAD + 0x0C: 0x2C620003, # sltiu v0,v1,3 + LOAD + 0x10: 0x10400000 | ((0x98 - 0x14) >> 2), # beq v0,zero + LOAD + 0x14: NOP, + LOAD + 0x18: 0x00031080, # sll v0,v1,2 + LOAD + 0x1C: 0x00501021, # addu v0,v0,s0 + LOAD + 0x20: 0x8C420000, # lw v0,0(v0) + LOAD + 0x24: NOP, + LOAD + 0x28: 0x00400008, # jr v0 + LOAD + 0x2C: NOP, + LOAD + 0x98: JR_RA, LOAD + 0x9C: UNFRAME} + for index, case in enumerate(cases): + words[case] = ADDIU + words[case + 4] = JR_RA + words[case + 8] = UNFRAME + words[table + index * 4] = case + data = image(words, 0x300) + derived = CO.derive_static_roots(data, LOAD, len(data)) + self.assertTrue(set(cases[1:]) <= derived) + # The guard alone (the case labels supplied as explicit roots, as a + # #386-enriched capture would) absorbs the whole run. + seeds, audit = classify(data, False, function_entry_pcs=[host], + static_discovery_entry_pcs=cases[1:]) + for case in cases[1:]: + self.assertEqual(audit['included_reasons'][case], + 'DISPATCH_INTERIOR', hex(case)) + self.assertNotIn(case, root_seeds(seeds)) + # Default enrichment: the guarded prepass resolves the table, so the + # case labels are never offered as roots at all. + seeds, audit = classify(data, True, function_entry_pcs=[host]) + for case in cases: + self.assertIn(audit['included_reasons'].get(case), + (None, 'DISPATCH_INTERIOR'), hex(case)) + self.assertNotIn(case, root_seeds(seeds)) + self.assertEqual(root_seeds(seeds), {host}) + + # (c) ------------------------------------------------------------------ + def test_jal_proven_function_start_stays_a_root(self): + caller, framed, frameless = LOAD, LOAD + 0x40, LOAD + 0x80 + data = image({caller: FRAME, caller + 4: jal(framed), + caller + 8: NOP, caller + 12: jal(frameless), + caller + 16: NOP, caller + 20: JR_RA, + caller + 24: UNFRAME, + framed: FRAME, framed + 4: ADDIU, framed + 8: JR_RA, + framed + 12: UNFRAME, + frameless - 8: JR_RA, frameless - 4: NOP, + frameless: ADDIU, frameless + 4: JR_RA, + frameless + 8: NOP}, 0x100) + derived = CO.derive_static_roots(data, LOAD, len(data)) + self.assertTrue({caller, framed, frameless} <= derived) + seeds, audit = classify(data, True) + roots = root_seeds(seeds) + self.assertTrue({caller, framed, frameless} <= roots) + self.assertEqual(audit['guard_demoted'], {}) + + # (d) ------------------------------------------------------------------ + def _cross_image(self, target_is_function): + engine, host = LOAD, LOAD + 0x100 + target = LOAD + 0x110 + words = {engine: FRAME, engine + 4: jal(target), engine + 8: NOP, + engine + 12: JR_RA, engine + 16: UNFRAME} + if target_is_function: + # Sibling image: the called address is a real function here. + words.update({host: ADDIU, host + 4: JR_RA, host + 8: NOP, + target: FRAME, target + 4: ADDIU, + target + 8: JR_RA, target + 12: UNFRAME}) + else: + # This image: the same address is mid-way through a host + # reached by pure fallthrough (the X00 shape). + words.update({host: FRAME, host + 4: ADDIU, host + 8: ADDIU, + host + 12: ADDIU, target: SUBU, + target + 4: ADDIU, target + 8: JR_RA, + target + 12: UNFRAME}) + return image(words, 0x200), engine, host, target + + def test_shared_jal_into_mid_body_is_not_rooted(self): + data, engine, host, target = self._cross_image(False) + weak = set() + strong = CO.derive_static_roots(data, LOAD, len(data), weak_out=weak) + self.assertNotIn(target, strong) + self.assertIn(target, weak) # the CFG probe alone accepts it + self.assertTrue({engine, host} <= strong) + derived, stats = CO.derive_enrichment_roots(data, LOAD, len(data)) + self.assertNotIn(target, derived) # host's prologue reaches it + self.assertEqual(stats['weak_dropped'], 1) + # Enrichment on (default): the rooted engine walk still derives the + # jal target, and the guard absorbs it into its host. + seeds, audit = classify(data, True) + self.assertNotIn(target, root_seeds(seeds)) + self.assertEqual(audit['included_reasons'][target], + 'DISPATCH_INTERIOR') + self.assertIn(host, root_seeds(seeds)) + # A #386-enriched capture that already names it is demoted too. + seeds, audit = classify(data, True, + static_discovery_entry_pcs=[target]) + self.assertNotIn(target, root_seeds(seeds)) + self.assertEqual(audit['guard_demoted'][target], + ('STATIC_DISCOVERY_ROOT', host)) + + def test_weak_target_nothing_reaches_is_rooted(self): + # A frameless function right after another function's tail jump, + # called by jal: no `jr $ra` before it, so no boundary proof, but no + # possible function start reaches it either. + caller, tail, leaf, far = LOAD, LOAD + 0x40, LOAD + 0x4C, LOAD + 0x80 + data = image({caller: FRAME, caller + 4: jal(leaf), caller + 8: NOP, + caller + 12: JR_RA, caller + 16: UNFRAME, + tail: ADDIU, + tail + 4: 0x08000000 | ((far >> 2) & 0x03FFFFFF), + tail + 8: NOP, + leaf: ADDIU, leaf + 4: JR_RA, leaf + 8: NOP, + far: JR_RA, far + 4: NOP}, 0x100) + weak = set() + CO.derive_static_roots(data, LOAD, len(data), weak_out=weak) + self.assertIn(leaf, weak) + derived, _stats = CO.derive_enrichment_roots(data, LOAD, len(data)) + self.assertIn(leaf, derived) + seeds, _audit = classify(data, True) + self.assertIn(leaf, root_seeds(seeds)) + + def test_shared_jal_into_real_function_in_sibling_image_is_rooted(self): + data, _engine, _host, target = self._cross_image(True) + self.assertIn(target, CO.derive_static_roots(data, LOAD, len(data))) + seeds, _audit = classify(data, True) + self.assertIn(target, root_seeds(seeds)) + + def test_cross_producer_call_evidence(self): + # Producer A calls into producer B. A cross-producer call is only + # ever weaker evidence: never WEAK evidence, STRONG only when B's own + # bytes bound the target and the capture is not strict. + a_fn, bounded, frameless = LOAD, LOAD + 0x110, LOAD + 0x130 + data = image({a_fn: FRAME, a_fn + 4: jal(bounded), a_fn + 8: NOP, + a_fn + 12: jal(frameless), a_fn + 16: NOP, + a_fn + 20: JR_RA, a_fn + 24: UNFRAME, + bounded - 8: JR_RA, bounded - 4: NOP, + bounded: ADDIU, bounded + 4: JR_RA, bounded + 8: NOP, + frameless - 4: ADDIU, + frameless: ADDIU, frameless + 4: JR_RA, + frameless + 8: NOP}, 0x200) + ranges = [(LOAD, LOAD + 0x100), (LOAD + 0x100, LOAD + 0x200)] + for strict in (True, False): + roots, _ = CO.derive_enrichment_roots( + data, LOAD, len(data), ranges, strict_producer_ranges=strict) + self.assertNotIn(frameless, roots) + self.assertEqual(bounded in roots, not strict) + + # enrichment policy ---------------------------------------------------- + def test_enrichment_is_default_and_opt_out_is_diagnostic(self): + data, engine, host, _target = self._cross_image(False) + old = os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None) + try: + self.assertTrue(CO.root_enrichment_default()) + _seeds, audit = classify(data, None) + self.assertTrue(audit['root_enrichment']) + self.assertTrue({engine, host} <= audit['derived_static_roots']) + os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = '0' + self.assertFalse(CO.root_enrichment_default()) + _seeds, audit = classify(data, None) + self.assertFalse(audit['root_enrichment']) + self.assertEqual(audit['derived_static_roots'], set()) + finally: + os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None) + if old is not None: + os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = old + # A conservative retry recipe is never enriched again. + cap = {'schema': 'psxrecomp overlay capture v2', + CO.ROOT_ENRICHMENT_OFF_KEY: True} + _seeds, audit = CO.classify_overlay_seeds( + cap, data, LOAD, len(data), 0, {}, root_enrichment=True) + self.assertEqual(audit['derived_static_roots'], set()) + retry = CO.conservative_retry_capture( + {'schema': 'x'}, {'derived_static_roots': {engine}}) + self.assertTrue(retry[CO.ROOT_ENRICHMENT_OFF_KEY]) + self.assertIsNone(CO.conservative_retry_capture( + retry, {'derived_static_roots': {engine}})) + + +# (e) ---------------------------------------------------------------------- +class CacheKeyTests(unittest.TestCase): + def _cmake_list(self): + self.assertIsNotNone(CMAKE, 'cmake not found: pass --cmake or put ' + 'cmake on PATH') + with tempfile.TemporaryDirectory() as tmp: + script = Path(tmp) / 'list.cmake' + script.write_text( + f'set(PSXRECOMP_CODEGEN_HASH_ROOT "{ROOT.as_posix()}")\n' + f'include("{(ROOT / "runtime/codegen_hash_sources.cmake").as_posix()}")\n' + 'message(STATUS "SRCS=${PSXRECOMP_CODEGEN_HASH_SRCS}")\n', + encoding='utf-8') + proc = subprocess.run([CMAKE, '-P', str(script)], + capture_output=True, text=True) + self.assertEqual(proc.returncode, 0, proc.stderr) + match = re.search(r'SRCS=(.*)', proc.stdout + proc.stderr) + self.assertIsNotNone(match) + return match.group(1).strip().split(';') + + def _hash_header(self, sources, out_dir): + header = Path(out_dir) / 'overlay_codegen_hash.h' + proc = subprocess.run( + [CMAKE, f'-DOUT={header.as_posix()}', + '-DSRCS=' + ';'.join(sources), + '-P', str(ROOT / 'runtime' / 'hash_codegen.cmake')], + capture_output=True, text=True) + self.assertEqual(proc.returncode, 0, proc.stderr) + shutil.copy2(ROOT / 'runtime' / 'include' / 'overlay_api.h', + Path(out_dir) / 'overlay_api.h') + return CO.codegen_hash(str(out_dir)) + + def test_root_policy_is_part_of_the_cache_namespace(self): + sources = self._cmake_list() + classifier = (ROOT / 'tools' / 'compile_overlays.py').as_posix() + self.assertIn(classifier, sources) + with tempfile.TemporaryDirectory() as tmp: + tmp = Path(tmp) + changed = tmp / 'compile_overlays.py' + changed.write_text( + Path(classifier).read_text(encoding='utf-8') + + '\n# a root-policy change\n', encoding='utf-8') + (tmp / 'a').mkdir() + (tmp / 'b').mkdir() + before = self._hash_header(sources, tmp / 'a') + after = self._hash_header( + [changed.as_posix() if s == classifier else s + for s in sources], tmp / 'b') + self.assertNotEqual(before, after) + saved = CO.overlay_config_hash + CO.overlay_config_hash = lambda recompiler, toml: 0x1234ABCD + try: + tag_a = CO.cache_tag(str(tmp / 'a'), 'r', 'g', 0) + tag_b = CO.cache_tag(str(tmp / 'b'), 'r', 'g', 0) + finally: + CO.overlay_config_hash = saved + self.assertNotEqual(tag_a, tag_b) + + +# recompiler enforcement --------------------------------------------------- +class RecompilerGuardTests(unittest.TestCase): + def test_recompiler_absorbs_explicit_fallthrough_root(self): + if not RECOMPILER: + self.skipTest('pass --recompiler (ctest always does)') + host, mid = LOAD + 0x20, LOAD + 0x30 + data = image({host: FRAME, host + 4: ADDIU, host + 8: ADDIU, + host + 12: ADDIU, mid: SUBU, mid + 4: ADDIU, + mid + 8: JR_RA, mid + 12: UNFRAME}, 0x1000) + with tempfile.TemporaryDirectory() as tmp: + psx = Path(tmp) / 'split.psx' + psx.write_bytes(CO.make_psxexe(LOAD, host, data, guard_bytes=0)) + seeds = Path(tmp) / 'seeds.txt' + # Seeds the classifier would never write: both as call roots. + seeds.write_text(f'call_root 0x{host:08X}\ncall_root 0x{mid:08X}\n') + out = Path(tmp) / 'out' + proc = subprocess.run( + [RECOMPILER, str(psx), '--seeds', str(seeds), + '--out-dir', str(out), '--overlay'], + capture_output=True, text=True, cwd=str(ROOT)) + self.assertEqual(proc.returncode, 0, proc.stderr or proc.stdout) + self.assertEqual( + CO.report_recompiler_guard_absorptions(proc.stdout), 1, + proc.stdout) + ranges = next(out.glob('*_full.ranges')).read_text() + # Both PCs stay callable; the mid-body one only as an alias whose + # identity covers the host range (it starts at the host). + self.assertIn(f'F {host:08X}', ranges) + self.assertIn(f'F {mid:08X}', ranges) + mid_rows = ranges.split(f'F {mid:08X}', 1)[1].splitlines()[1:] + self.assertTrue(any(row.startswith(f'R {host:08X}') + for row in mid_rows[:2]), ranges) + + +# supplemental root fragments carry the aliases of their hosts -------------- +# +# Measured on Tomba 2 (bead beads-eio.3.177): the 0x80038000 capture has more +# roots than the region shard admits, so ~960 roots are compiled in strong-root +# supplement fragments. Those fragments were built from `dispatch_root` seeds +# only, so every candidate the guard absorbed into such a host (0x8003A678 in +# host 0x8003A614, captured dispatch entries) was served by neither shard: 127 +# captured dispatch entries lost the native identity the unguarded build had. +class SupplementFragmentAliasTests(unittest.TestCase): + HOST, MID = LOAD + 0x20, LOAD + 0x30 + + def fallthrough_image(self, size=0x100): + host, mid = self.HOST, self.MID + return image({host: FRAME, host + 4: ADDIU, host + 8: ADDIU, + host + 12: ADDIU, mid: SUBU, mid + 4: ADDIU, + mid + 8: JR_RA, mid + 12: UNFRAME}, size) + + def job(self, data): + cap = {'schema': 'psxrecomp overlay capture v2', + 'function_entry_pcs': [f'0x{self.HOST:08X}'], + 'dispatch_entry_pcs': [f'0x{self.HOST:08X}'], + 'static_discovery_entry_pcs': [f'0x{self.MID:08X}']} + _seeds, audit = CO.classify_overlay_seeds( + cap, data, LOAD, len(data), 0, {}, root_enrichment=True) + return audit, CO.make_interior_fragment_job( + LOAD & 0x1FFFFFFF, LOAD, len(data), data, audit, set(), cap) + + def test_classifier_records_final_host_of_every_alias(self): + audit, _job = self.job(self.fallthrough_image()) + self.assertEqual(audit['included_reasons'][self.MID], + 'DISPATCH_INTERIOR') + self.assertEqual(audit['interior_hosts'][self.MID], {self.HOST}) + self.assertEqual(audit['guard_demoted'][self.MID], + ('STATIC_DISCOVERY_ROOT', self.HOST)) + + def test_root_fragment_selects_its_hosts_aliases(self): + _audit, job = self.job(self.fallthrough_image()) + self.assertIn(self.HOST, job['static_exact_demands']) + self.assertEqual(CO.fragment_interior_aliases(job, [self.HOST]), + [self.MID]) + self.assertEqual(CO.fragment_interior_aliases(job, [LOAD + 0x80]), []) + # merged byte-variant recipes union the hosts + merged = CO.merge_fragment_jobs_by_recipe([job, job]) + self.assertEqual(merged[0]['interior_hosts'][self.MID], {self.HOST}) + + def test_fragment_seed_file_carries_aliases_as_interior(self): + data = self.fallthrough_image() + seen = {} + real_run = CO.subprocess.run + + def fake_run(cmd, **kw): + seen['seeds'] = Path(cmd[cmd.index('--seeds') + 1]).read_text() + return subprocess.CompletedProcess(cmd, 1, '', 'stop') + args = argparse.Namespace(recompiler='recomp.exe', game_toml=__file__, + project_root=None, runtime_include='.') + CO.subprocess.run = fake_run + try: + CO.compile_fragment_batch( + [self.HOST], data, LOAD, len(data), LOAD & 0x1FFFFFFF, '.', + args, {}, {}, guard_bytes=0, interior_aliases=[self.MID]) + finally: + CO.subprocess.run = real_run + lines = seen['seeds'].splitlines() + self.assertIn(f'dispatch_root 0x{self.HOST:08X}', lines) + self.assertIn(f'interior 0x{self.MID:08X}', lines) + + def test_alias_failure_never_costs_the_host(self): + calls = [] + + def build(aliases): + calls.append(list(aliases)) + return ((None, 'generated-c-audit: 1 unknown_bad') if aliases + else ([('ok',)], 'built')) + ids, status = CO.compile_root_fragment_with_aliases( + [self.HOST], [self.MID], build) + self.assertEqual((ids, status, calls), ([('ok',)], 'built', + [[self.MID], []])) + # a multi-root batch is left to the caller's bisection + calls.clear() + ids, _status = CO.compile_root_fragment_with_aliases( + [self.HOST, LOAD + 0x80], [self.MID], build) + self.assertIsNone(ids) + self.assertEqual(calls, [[self.MID]]) + # capacity verdicts are never retried + calls.clear() + CO.compile_root_fragment_with_aliases( + [self.HOST], [self.MID], + lambda a: calls.append(a) or (None, 'candidate-capacity: full; x')) + self.assertEqual(len(calls), 1) + + def test_recompiler_emits_alias_in_root_fragment(self): + if not RECOMPILER: + self.skipTest('pass --recompiler (ctest always does)') + data = self.fallthrough_image(0x1000) + with tempfile.TemporaryDirectory() as tmp: + psx = Path(tmp) / 'frag.psx' + psx.write_bytes(CO.make_psxexe(LOAD, self.HOST, data, + guard_bytes=0)) + seeds = Path(tmp) / 'seeds.txt' + seeds.write_text(f'dispatch_root 0x{self.HOST:08X}\n' + f'interior 0x{self.MID:08X}\n') + out = Path(tmp) / 'out' + proc = subprocess.run( + [RECOMPILER, str(psx), '--seeds', str(seeds), + '--out-dir', str(out), '--overlay'], + capture_output=True, text=True, cwd=str(ROOT)) + self.assertEqual(proc.returncode, 0, proc.stderr or proc.stdout) + ranges = next(out.glob('*_full.ranges')).read_text() + mid_rows = ranges.split(f'F {self.MID:08X}', 1) + self.assertEqual(len(mid_rows), 2, ranges) + self.assertTrue(mid_rows[1].splitlines()[1].startswith( + f'R {self.HOST:08X}'), ranges) + + +if __name__ == '__main__': + ap = argparse.ArgumentParser() + ap.add_argument('--cmake', default=shutil.which('cmake')) + ap.add_argument('--recompiler') + args, rest = ap.parse_known_args() + if not args.cmake: + sys.exit('cmake not found: pass --cmake') + CMAKE = args.cmake + RECOMPILER = args.recompiler + unittest.main(argv=[sys.argv[0]] + rest) From 5a094db697bd0be503412703c8eb3034429b94a9 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 23:32:41 -0700 Subject: [PATCH 09/13] overlays: guard-absorbed candidates of interpreted hosts compile as roots Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/compile_overlays.py | 61 ++++++++++++++++++++-- tools/tests/test_overlay_no_split_guard.py | 14 +++++ 2 files changed, 72 insertions(+), 3 deletions(-) diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index 14eb4973..4f75d73d 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -4324,6 +4324,13 @@ def make_interior_fragment_job(phys_addr: int, load_addr: int, size: int, _canonical_guest_addr(addr): {_canonical_guest_addr(h) for h in hosts} for addr, hosts in seed_audit.get('interior_hosts', {}).items() }, + # Candidates the no-split guard demoted from a root reason. If none of + # their hosts ends up native, they are compiled as the roots they + # would have been without the guard (see orphaned_guard_aliases). + 'guard_demoted_aliases': { + _canonical_guest_addr(addr) + for addr in seed_audit.get('guard_demoted', {}) + }, 'static_interval_demands': static_interval_demands, 'forced': forced, 'producer_ranges': tuple(seed_audit['producer_ranges']), @@ -4351,7 +4358,7 @@ def merge_fragment_jobs_by_recipe(jobs: list[dict], set_fields = ( 'candidates', 'executed', 'static_demands', 'static_exact_demands', 'hosted_donor_demands', - 'static_interval_demands', 'forced', + 'static_interval_demands', 'forced', 'guard_demoted_aliases', ) for job in jobs: key = ( @@ -4366,7 +4373,7 @@ def merge_fragment_jobs_by_recipe(jobs: list[dict], if merged is None: merged = dict(job) for field in set_fields: - merged[field] = set(job[field]) + merged[field] = set(job.get(field, ())) merged['included_reasons'] = { addr: set(reasons) for addr, reasons in job.get('included_reasons', {}).items() @@ -4378,7 +4385,7 @@ def merge_fragment_jobs_by_recipe(jobs: list[dict], grouped[key] = merged continue for field in set_fields: - merged[field].update(job[field]) + merged[field].update(job.get(field, ())) for addr, reasons in job.get('included_reasons', {}).items(): merged.setdefault('included_reasons', {}).setdefault( addr, set()).update(reasons) @@ -4662,6 +4669,30 @@ def fragment_interior_aliases(job: dict, entries) -> list[int]: if alias not in roots and hosts & roots) +def orphaned_guard_aliases(job: dict, served_phys) -> list[int]: + """Guard-absorbed candidates none of whose hosts is native. + + The no-split guard demotes a candidate to an alias only because a host + walk reaches it. When every such host fails to compile (a host walk that + runs through data audits out as unsupported code) or is otherwise left + to the interpreter, the alias has nothing to attach to. Those candidates + are compiled as the roots they were before the guard: splitting a host + that is interpreted anyway costs nothing, and dropping them would lose + native coverage the unguarded build had. + """ + served = {int(entry) & 0x1FFFFFFF for entry in served_phys} + hosts_of = job.get('interior_hosts', {}) + out = [] + for alias in sorted(job.get('guard_demoted_aliases', ())): + if (alias & 0x1FFFFFFF) in served: + continue + hosts = hosts_of.get(alias, set()) + if any((host & 0x1FFFFFFF) in served for host in hosts): + continue + out.append(alias) + return out + + def compile_root_fragment_with_aliases(entries, aliases, build): """Build one root fragment carrying its hosts' aliases. @@ -7600,6 +7631,30 @@ def main(): strong_shards += 1 else: strong_singleton_failure(entry, status) + # Guard-absorbed candidates whose every host stayed interpreted + # (failed, memoized-doomed, or never compiled) get their root back. + orphaned = [ + entry for entry in orphaned_guard_aliases( + job, current_variant_entries) + if fragment_capacity_allows_entry( + candidate_capacity_exhausted, args.force, + job['forced'], entry) + ] + if orphaned: + # Same failure policy as the static root each one was before + # the guard: a deterministic audit rejection is a safe skip. + job['static_demands'] = set(job['static_demands']) | set(orphaned) + orphan_shards = compile_batched_fragment_roots( + orphaned, compile_strong_batch, strong_batch_success, + strong_singleton_failure, + lambda entry: ((entry & 0x1FFFFFFF) not in + current_variant_entries), + fragment_batch_failure_is_partitionable, + strong_batch_failure) + strong_shards += orphan_shards + print(f' guard-alias fallback @0x{phys_addr:08X}: ' + f'{len(orphaned)} alias(es) of interpreted hosts -> ' + f'{orphan_shards} root shard(s)') if strong_candidates: print(f' strong-root supplement @0x{phys_addr:08X}: ' f'{len(strong_candidates)} demand(s) -> ' diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py index 7d474767..96ce2825 100644 --- a/tools/tests/test_overlay_no_split_guard.py +++ b/tools/tests/test_overlay_no_split_guard.py @@ -505,6 +505,20 @@ class SupplementFragmentAliasTests(unittest.TestCase): self.assertIn(f'dispatch_root 0x{self.HOST:08X}', lines) self.assertIn(f'interior 0x{self.MID:08X}', lines) + def test_alias_of_interpreted_host_gets_its_root_back(self): + # Tomba 2: host 0x800BDF10 (a walk through data, 48 unsupported + # instructions) fails its fragment; the 8-byte stubs the guard had + # absorbed into it were native roots in the unguarded build. + _audit, job = self.job(self.fallthrough_image()) + self.assertEqual(job['guard_demoted_aliases'], {self.MID}) + self.assertEqual(CO.orphaned_guard_aliases(job, set()), [self.MID]) + self.assertEqual(CO.orphaned_guard_aliases( + job, {self.HOST & 0x1FFFFFFF}), []) + self.assertEqual(CO.orphaned_guard_aliases( + job, {self.MID & 0x1FFFFFFF}), []) + merged = CO.merge_fragment_jobs_by_recipe([job, dict(job)]) + self.assertEqual(merged[0]['guard_demoted_aliases'], {self.MID}) + def test_alias_failure_never_costs_the_host(self): calls = [] From 29417854dde5ceee7ab44c959448a81f30daef6c Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Fri, 25 Sep 2026 23:56:01 -0700 Subject: [PATCH 10/13] docs: Tomba 2 guard result and the two supplement-fragment fixes Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/internal/OVERLAY_ROOT_ENRICHMENT.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md index f5318e8f..662d69cc 100644 --- a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md +++ b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md @@ -104,6 +104,25 @@ mis-splits**, then as much coverage as possible. resolving those tables, not rooting their labels. - **Not caused by splits:** master's own 3-4 cycle drift from the interpreter at frame 2111. It is unchanged with the guard. +- **Tomba 2 (23 captures, autocompile path).** Two guard gaps found and + fixed. (1) Roots the region shard leaves out are built in strong-root + supplement fragments from `dispatch_root` seeds only, so aliases of those + hosts were served by no shard (127 captured dispatch entries lost). The + supplement now passes each host's aliases as `interior` seeds, and a single + root whose aliases break its fragment is rebuilt without them. (2) When + every host of an absorbed candidate stays interpreted (0x800BDF10 walks + into data and fails the audit), the candidate is compiled as the root it + was before the guard. Left: 14 captured dispatch entries at delay slots, + which the guard keeps interpreted by design (master aliased them). +- **Tomba 2 fingerprints do not match master exactly,** and the cause is + master's own splits. Frame 6797: master splits 0x8009B0C0 from its + fallthrough 0x8009B0CC; the split leaves the nested call unit, so master + takes a CD IRQ inside the loop, while the unsplit function runs as one + unit and the runtime's nested-unit rule defers the IRQ to its return + (with `PSX_OVERLAY_UNIT_DEFER=0` in both arms this fork goes away). + Frame 7169/7363: master splits switch 0x8006D654 at its case label + 0x8006D690; the jump-table exit costs master 2 extra cycles. Values stay + identical, and every route and attract screenshot is identical. ## Done when From 7111271bae36bd7a4c77fd84e6eb5fe64969afdc Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Sat, 26 Sep 2026 06:18:18 -0700 Subject: [PATCH 11/13] docs: MMX6 and Ape guard results, the gate used, and what is left Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/internal/OVERLAY_ROOT_ENRICHMENT.md | 44 ++++++++++++++++++++---- 1 file changed, 37 insertions(+), 7 deletions(-) diff --git a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md index 662d69cc..1e0db5c9 100644 --- a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md +++ b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md @@ -124,11 +124,41 @@ mis-splits**, then as much coverage as possible. 0x8006D690; the jump-table exit costs master 2 extra cycles. Values stay identical, and every route and attract screenshot is identical. -## Done when +- **Mega Man X6 and Ape Escape (guard at 5a094db6).** Same gate as below, + arms N (master) and G (guard), own sandboxes and caches, both proven to run + native. MMX6: interpreter fallback 14.04M vs 14.11M on the route, 0/0 + shard failures, 74/74 route and 13/13 attract screenshots identical; the + only fingerprint fork is the store-PC hash at 8303, where overlay code + master interprets runs native in G (same address, value and cycle). + Ape: fallback 3.28M vs 3.31M, memcard LOAD GAME passes, 67/67 and 39/39 + screenshots identical; fork at 3564 is the same interpreted-to-native class + plus a 3-cycle difference in how native and interpreter cost one + straight-line multiply block (framework behaviour, beads-eio.3.185). + Tomba re-checked on 5a094db6: 0 diverging frames vs master. -- The no-split guard is in, with a test that a mid-function candidate is - demoted, not rooted. -- Enrichment is on by default in `compile_overlays.py`, and part of the - cache key. -- The four titles match interpreted-only fingerprints and pass a play check. -- The `no-func-ids` cause is known and fixed at the source. +## Gate used, and what is left + +The gate the four titles passed is **no regression against master**: +interpreter fallback not higher, no new shard failures, no freeze or wedge, +all checkpoint screenshots identical. Small timing forks that come from +removing master's own splits are accepted. Matching the interpreted-only +arm exactly is not the gate: master itself drifts a few cycles from it. + +Done: + +- The no-split guard, with tests that mid-function, branch-reached and + delay-slot candidates are demoted, not rooted. +- Enrichment on by default in `compile_overlays.py`, and part of the cache + key. +- Four-title regression gate (above). + +Not done: + +- Step 3 as a widening. The evidence rules were made stricter (image-local + proof, cross-producer calls excluded), not extended with new categories. +- Step 4, the `no-func-ids` root cause. +- Re-measuring Ace Combat 3. #386's AC3 numbers (rootless images 250 to 1, + capacity skips 3,028 to 0) were taken without the guard; the stricter + evidence rule may root fewer of those images. +- Delay-slot dispatch PCs that master aliased stay interpreted under the + guard by design (Tomba 96, Tomba 2 14, Ape 10); no fallback cost measured. From ae4e57295079f88ca71b8c45de26a89384a2fa7e Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Sat, 26 Sep 2026 08:26:48 -0700 Subject: [PATCH 12/13] overlays: a stack-frame prologue roots its function's true start, not the stack adjust Co-Authored-By: Claude Opus 5.5 (1M context) --- tools/compile_overlays.py | 244 +++++++++++++++++++-- tools/tests/test_overlay_no_split_guard.py | 188 ++++++++++++++++ 2 files changed, 416 insertions(+), 16 deletions(-) diff --git a/tools/compile_overlays.py b/tools/compile_overlays.py index 4f75d73d..0ae29d86 100644 --- a/tools/compile_overlays.py +++ b/tools/compile_overlays.py @@ -24,6 +24,7 @@ import argparse import itertools import io import contextlib +import functools import base64 import binascii from bisect import bisect_left @@ -1349,11 +1350,20 @@ def _frameless_dispatch_root_proven(data: bytes, load_addr: int, size: int, # when it executes, and overlay regions are swapped: shared engine code # at 0x80100984 is byte-identical in 12 Tomba area images and calls # 0x8011B1CC, which is a function start in exactly one of them. So: -# * STRONG: a non-delay-slot stack-frame prologue, or a jal/table -# target that THIS image's bytes also bound (preceding `jr $ra` plus -# alignment padding, and the bounded CFG probe). A strong root that -# turns out to be inside a function is demoted by rule 1. -# * WEAK: a jal/table target with only the CFG probe. It is added only +# * STRONG: a stack-frame prologue AT a boundary (image/producer +# start, after an unconditional transfer's delay slot or a `break`, +# or after data), the true start in front of a prologue that loads +# come before (walked back over the straight-line preamble to such +# a boundary; prologue_function_start), or a jal/table target that +# THIS image's bytes also bound (preceding `jr $ra` plus alignment +# padding, and the bounded CFG probe). A strong root that turns out +# to be inside a function is demoted by rule 1. Ace Combat 3 showed +# why the prologue needs the boundary: rooting the `addiu sp` itself +# capped 359 functions a few words in (beads-eio.3.191). Captured +# entry evidence at a stack adjust (a dispatch PC) keeps its own +# gate; the true start's walk absorbs it by rule 1. +# * WEAK: a jal/table target with only the CFG probe, or a prologue +# whose start the preamble walk cannot prove. It is added only # when no possible function start below it reaches it -- every # capture root, every strong or weak candidate, every prologue and # every word after an unconditional transfer's delay slot, walked @@ -1405,14 +1415,173 @@ def _return_precedes(data: bytes, load_addr: int, addr: int) -> bool: return False +# Longest straight-line preamble walked back from a stack-frame prologue to +# its function's start. Scheduled loads before the frame are 1-5 words on Ace +# Combat 3 (beads-eio.3.191); a longer run proves nothing more. Zero words +# (alignment padding, zero fill) are walked but not counted. +PROLOGUE_PREAMBLE_WORDS = 32 + + +@functools.lru_cache(maxsize=8) +def _local_branch_sources(data: bytes, load_addr: int) -> dict[int, int]: + """{target: lowest source pc} of every local branch in the image: a + conditional branch or an always-taken `b`, never an absolute j/jal (also + how tail calls and calls are encoded) and never a register jump. Compiled + code never branches across a function boundary, so a word a branch from + BELOW reaches is inside the function that branch belongs to.""" + hi = load_addr + len(data) + sources = {} + for pc in range(load_addr, hi, 4): + word = _word_at(data, load_addr, pc) + if word is None or not _is_control_flow(word) or \ + (word >> 26) in (0x00, 0x02, 0x03): + continue + kind, target = _classify_cf(pc, word) + if kind in ('branch', 'j') and pc < sources.get(target, hi): + sources[target] = pc + return sources + + +def _pointer_shaped(word, load_addr: int, hi: int) -> bool: + """An aligned RAM (KSEG0) or in-image address: a table entry. As an + instruction it is `lb rt,imm(zero|at|v0|v1)`, which real code also uses + (`lui v0,0x8010; lb v0,0x5CEC(v0)`), so one such word alone is not data.""" + return (word is not None and not word & 3 and + (load_addr <= word < hi or 0x80000000 <= word < 0x80800000)) + + +def _setup_shaped_preamble(data: bytes, load_addr: int, start: int, + end: int) -> bool: + """Every word in [start, end) is zero or global-data setup: `lui rt`, + `li rt` (addiu/ori from $zero), or a load whose base a preceding `lui` + set -- the recompiler's pre-prologue rule (preprologue_window_is_valid). + Destination never $zero, $sp or $ra.""" + bases = set() + for pc in range(start, end, 4): + word = _word_at(data, load_addr, pc) + if word is None: + return False + if word == 0: + continue + op, rs, rt = word >> 26, (word >> 21) & 31, (word >> 16) & 31 + if rt in (0, 29, 31): + return False + if op == 0x0F and rs == 0: + bases.add(rt) + elif op in (0x09, 0x0D) and rs == 0: + bases.discard(rt) + elif 0x20 <= op <= 0x26 and rs in bases: + bases.discard(rt) + else: + return False + return True + + +def prologue_function_start(data: bytes, load_addr: int, addr: int, + lo: int | None = None): + """The start of the function whose stack-frame prologue is at ``addr``. + + A stack adjust is not a function boundary by itself: compilers schedule + loads in front of it (`lui v0; lw v1,..(v0); addiu sp,sp,-24`), and + rooting the adjust caps the real function a few words in (Ace Combat 3: + 359 such roots, 175 with the true start executed). Walk back over the + straight-line preamble to the nearest boundary, then step over zero + padding. A boundary is ``lo`` (image or producer start), the word after + the delay slot of an unconditional transfer (`jr ra`, `jr`, `j`, `b`) or + after a `break` trap, or the word after data: an invalid instruction, or + the last of two pointer-shaped words (a table). After data the preamble + must be global-data setup (_setup_shaped_preamble), since data can decode + as valid instructions. + + Returns ``addr`` when the prologue itself is at a boundary, an earlier + address when a preamble leads to it, and None when no start is provable: + ``addr`` is not a prologue or sits in a delay slot; the walk meets a + conditional branch or a call (the prologue is reached by fallthrough from + code whose start this walk does not bound) or runs past the window; the + preamble after data is not setup; or a local branch from below the start + lands in [start, addr] (the start is then a label of the function before + it, past an early return or a divide check). + """ + lo = load_addr if lo is None else lo + hi = load_addr + len(data) + if addr & 3 or not (lo <= addr < hi) or \ + not _is_addiu_sp_neg(_word_at(data, load_addr, addr)): + return None + pc = addr + preamble = 0 + after_data = False + while pc > lo: + prev = _word_at(data, load_addr, pc - 4) + if prev is None: + return None + # A control-transfer shape counts only as a valid instruction: data + # like 0x06040200 (REGIMM with no such rt) decodes as a "branch". + if _is_valid_mips_word(prev) and _is_control_flow(prev): + return None # pc is a delay slot + prev2 = _word_at(data, load_addr, pc - 8) if pc - 8 >= lo else None + if (_is_valid_mips_word(prev2) and _is_control_flow(prev2) and + _is_valid_mips_word(prev)): + kind, _target = _classify_cf(pc - 8, prev2) + if kind in ('j', 'jr', 'jr_ra'): + break # after an unconditional transfer + return None # conditional fallthrough / call return + if (prev >> 26) == 0 and (prev & 0x3F) == 0x0D: + break # after a `break` trap + if not _is_valid_mips_word(prev) or ( + _pointer_shaped(prev, load_addr, hi) and prev2 is not None and + (not _is_valid_mips_word(prev2) or + _pointer_shaped(prev2, load_addr, hi))): + after_data = True + break + if prev: # zero fill is padding, not preamble + preamble += 1 + if preamble > PROLOGUE_PREAMBLE_WORDS: + return None + pc -= 4 + while pc < addr and _word_at(data, load_addr, pc) == 0: + pc += 4 + if after_data and pc < addr and \ + not _setup_shaped_preamble(data, load_addr, pc, addr): + return None + sources = _local_branch_sources(bytes(data), load_addr) + for label in range(pc, addr + 4, 4): + if sources.get(label, pc) < pc: + return None + return pc + + +def _late_prologue_start_proven(data: bytes, load_addr: int, start: int, + producer_hi: int, hi: int) -> bool: + """Proof for the start prologue_function_start found BEFORE a prologue. + + The evidence is the same as for a prologue at a boundary -- a stack frame + opened by straight-line code from a boundary -- only relocated to where + that code begins, so it needs no more than a prologue does: the checks + every root gets (in range, valid non-zero word, not a delay slot or a + pointer table). Not the frameless-boundary CFG probe: that stands in for + the code evidence a prologue already gives, and it rejects functions that + run past a page-sized capture (13 of them on Ace Combat 3) which the + prologue itself roots today.""" + if start & 3 or not (load_addr <= start < min(producer_hi, hi)): + return False + word = _word_at(data, load_addr, start) + if not word or not _is_valid_mips_word(word): + return False + return not (_dense_local_pointer_table(data, load_addr, start) or + _in_delay_slot(data, load_addr, start)) + + def image_local_entry_proven(data: bytes, load_addr: int, size: int, addr: int, producer_hi: int, - analysis_hi: int | None = None) -> bool: + analysis_hi: int | None = None, + producer_lo: int | None = None) -> bool: """Does THIS image's own byte content prove a function boundary at addr? Used for every enrichment source (prologue scan, jal targets, pointer tables). A call or pointer elsewhere in the image is not accepted as proof - by itself; see the section comment above for why. + by itself; see the section comment above for why. A stack-frame prologue + proves a boundary only when it is at one (prologue_function_start returns + it unchanged); a prologue after a preamble proves its true start instead. """ hi = load_addr + size if analysis_hi is None else analysis_hi if addr & 3 or not (load_addr <= addr < hi): @@ -1424,7 +1593,9 @@ def image_local_entry_proven(data: bytes, load_addr: int, size: int, return False if _in_delay_slot(data, load_addr, addr): return False - if _is_addiu_sp_neg(word): + if _is_addiu_sp_neg(word) and prologue_function_start( + data, load_addr, addr, + load_addr if producer_lo is None else producer_lo) == addr: return True if _return_precedes(data, load_addr, addr): return plausible_callable_target(data, load_addr, size, addr, @@ -1483,11 +1654,32 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, source += '_cross' candidates.setdefault(addr, set()).add(source) + def producer_lo_for(addr: int): + if not ranges: + return load_addr + for range_lo, range_hi in ranges: + if range_lo <= addr < range_hi: + return range_lo + return None + + # A prologue roots its function's true start (prologue_function_start): + # at a boundary it is its own start; after a straight-line preamble the + # start is the candidate, and it must pass _late_prologue_start_proven. + # A prologue with no provable start is only weak evidence. + late_prologues: dict[int, set] = {} pointers = [] for addr in range(load_addr, hi, 4): word = _word_at(data, load_addr, addr) - if _is_addiu_sp_neg(word): - note(addr, 'prologue') + if _is_addiu_sp_neg(word) and not _in_delay_slot(data, load_addr, addr): + plo = producer_lo_for(addr) + start = (None if plo is None else prologue_function_start( + data, load_addr, addr, plo)) + if start == addr: + note(addr, 'prologue') + elif start is not None: + late_prologues.setdefault(start, set()).add(addr) + else: + note(addr, 'prologue_unbounded') if word is not None and (word >> 26) == 0x03: note(_jump_target(addr, word), 'jal', addr) pointers.append(word if word is not None and not (word & 3) and @@ -1505,8 +1697,18 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, note(pointers[offset], 'pointer_table', load_addr + offset * 4) start = end + for start, prologue_addrs in late_prologues.items(): + producer_hi = producer_hi_for(start) + if producer_hi is not None and _late_prologue_start_proven( + data, load_addr, start, producer_hi, hi): + note(start, 'prologue_start') + else: + for addr in prologue_addrs: + note(addr, 'prologue_unbounded') + local = {'prologue', 'jal', 'pointer_table'} cross = {'jal_cross', 'pointer_table_cross'} + weak_sources = {'jal', 'pointer_table', 'prologue_unbounded'} roots = set() for addr, why in candidates.items(): producer_hi = producer_hi_for(addr) @@ -1514,12 +1716,14 @@ def derive_static_roots(data: bytes, load_addr: int, size: int, continue strong_evidence = bool(why & local) or ( bool(why & cross) and not strict_producer_ranges) - if strong_evidence and image_local_entry_proven( - data, load_addr, size, addr, producer_hi, hi): + if 'prologue_start' in why or ( + strong_evidence and image_local_entry_proven( + data, load_addr, size, addr, producer_hi, hi, + producer_lo_for(addr))): roots.add(addr) if sources is not None: sources[addr] = set(why) - elif (weak_out is not None and why & {'jal', 'pointer_table'} and + elif (weak_out is not None and why & weak_sources and not addr & 3 and load_addr <= addr < hi and not _in_delay_slot(data, load_addr, addr) and plausible_callable_target(data, load_addr, size, addr, @@ -1583,8 +1787,9 @@ def derive_enrichment_roots(data: bytes, load_addr: int, size: int, data, load_addr, size, entry, cap, producer_ranges) return cache[key] weak = set() + why = {} strong = derive_static_roots(data, load_addr, size, producer_ranges, hi, - weak_out=weak, + sources=why, weak_out=weak, strict_producer_ranges=strict_producer_ranges) hosts = ({a for a in explicit if load_addr <= a < hi and not a & 3} | strong | weak | @@ -1599,7 +1804,11 @@ def derive_enrichment_roots(data: bytes, load_addr: int, size: int, return strong | accepted, { 'strong': len(strong), 'weak': len(weak), 'weak_accepted': len(accepted), 'weak_dropped': len(reached), - 'hosts': len(hosts)} + 'hosts': len(hosts), + 'late_prologue_starts': sum('prologue_start' in why.get(a, ()) + for a in strong), + 'unbounded_prologues_accepted': sum( + 'prologue_unbounded' in why.get(a, ()) for a in accepted)} def no_split_partition(candidates, exempt, walk, hi: int, droppable=(), @@ -2574,7 +2783,10 @@ def print_seed_audit(audit: dict) -> None: print(f'root_enrichment: {"on" if audit.get("root_enrichment") else "off"}' f' (derived {len(audit.get("derived_static_roots", ()))}' + (f'; strong {stats["strong"]}, weak {stats["weak"]} accepted ' - f'{stats["weak_accepted"]} dropped {stats["weak_dropped"]}' + f'{stats["weak_accepted"]} dropped {stats["weak_dropped"]}; ' + f'late-prologue starts {stats.get("late_prologue_starts", 0)}, ' + f'unbounded prologues kept ' + f'{stats.get("unbounded_prologues_accepted", 0)}' if stats else '') + ')') guard_demoted = audit.get('guard_demoted', {}) print(f'no_split_guard_demoted: {len(guard_demoted)}') diff --git a/tools/tests/test_overlay_no_split_guard.py b/tools/tests/test_overlay_no_split_guard.py index 96ce2825..5135bed6 100644 --- a/tools/tests/test_overlay_no_split_guard.py +++ b/tools/tests/test_overlay_no_split_guard.py @@ -316,6 +316,194 @@ class NoSplitGuardTests(unittest.TestCase): self.assertNotIn(frameless, roots) self.assertEqual(bounded in roots, not strict) + # late prologues (beads-eio.3.191) ------------------------------------- + # A stack adjust is a boundary only when it is at one. Compilers schedule + # loads in front of it, so rooting every `addiu sp,sp,-N` capped 359 Ace + # Combat 3 functions two to five words in, 175 of them with the real + # start executed. The prologue must root the function's true start. + AC3 = { # Ace Combat 3 USA image 0 (load 0x800C8000), 0x800D1250.. + 0x00: JR_RA, # jr ra + 0x04: 0x27BD0018, # addiu sp,sp,24 (delay slot) + 0x08: 0x3C02800F, # lui v0,0x800F <- real start + 0x0C: 0x8C435D4C, # lw v1,0x5D4C(v0) + 0x10: 0x27BDFFE8, # addiu sp,sp,-24 <- was rooted + 0x14: 0xAFBF0014, # sw ra,20(sp) + 0x18: 0xAFB00010, # sw s0,16(sp) + 0x1C: 0x8C620000, # lw v0,0(v1) + 0x20: NOP, + 0x24: 0x8FBF0014, # lw ra,20(sp) + 0x28: 0x8FB00010, # lw s0,16(sp) + 0x2C: JR_RA, + 0x30: 0x27BD0018, # addiu sp,sp,24 + } + + def ac3_image(self, base=LOAD + 0x40, extra=None): + words = {LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME} + words.update({base + off: word for off, word in self.AC3.items()}) + words.update(extra or {}) + return image(words, 0x100), base + 0x08, base + 0x10 + + def test_late_prologue_roots_its_true_start(self): + data, start, prologue = self.ac3_image() + self.assertEqual(CO.prologue_function_start(data, LOAD, prologue), + start) + self.assertFalse(CO.image_local_entry_proven( + data, LOAD, len(data), prologue, LOAD + len(data))) + sources = {} + strong = CO.derive_static_roots(data, LOAD, len(data), + sources=sources) + self.assertIn(start, strong) + self.assertNotIn(prologue, strong) + self.assertIn('prologue_start', sources[start]) + _roots, stats = CO.derive_enrichment_roots(data, LOAD, len(data)) + self.assertEqual(stats['late_prologue_starts'], 1) + seeds, audit = classify(data, True) + self.assertIn(start, root_seeds(seeds)) + self.assertNotIn(prologue, root_seeds(seeds)) + # Captured entry evidence at the stack adjust is absorbed by the true + # start: an alias, never a cap. + seeds, audit = classify(data, True, dispatch_entry_pcs=[prologue]) + self.assertIn(start, root_seeds(seeds)) + self.assertEqual(audit['included_reasons'][prologue], + 'DISPATCH_INTERIOR') + self.assertEqual(audit['guard_demoted'][prologue][1], start) + + def test_prologue_at_a_boundary_is_its_own_start(self): + # image start, after jr ra + padding, after j, after a break trap + far = LOAD + 0xF0 + data = image({LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME, + LOAD + 0x20: FRAME, LOAD + 0x24: JR_RA, + LOAD + 0x28: UNFRAME, + LOAD + 0x40: 0x08000000 | ((far >> 2) & 0x03FFFFFF), + LOAD + 0x44: NOP, LOAD + 0x48: FRAME, + LOAD + 0x4C: JR_RA, LOAD + 0x50: UNFRAME, + LOAD + 0x60: 0x0000000D, # break + LOAD + 0x64: 0x3C02800F, # lui v0 <- start + LOAD + 0x68: FRAME, LOAD + 0x6C: JR_RA, + LOAD + 0x70: UNFRAME, + far: JR_RA, far + 4: NOP}, 0x100) + for addr in (LOAD, LOAD + 0x20, LOAD + 0x48): + self.assertEqual(CO.prologue_function_start(data, LOAD, addr), + addr, hex(addr)) + self.assertTrue(CO.image_local_entry_proven( + data, LOAD, len(data), addr, LOAD + len(data)), hex(addr)) + self.assertEqual(CO.prologue_function_start(data, LOAD, LOAD + 0x68), + LOAD + 0x64) + self.assertTrue({LOAD, LOAD + 0x20, LOAD + 0x48, LOAD + 0x64} <= + CO.derive_static_roots(data, LOAD, len(data))) + + def test_preamble_with_branch_or_call_proves_no_start(self): + # `jr ra; nop; lw v0,0(a0); beqz v0,out; nop; addiu sp` -- the stack + # adjust is reached by fallthrough from code this walk does not + # bound. It is weak evidence, and the start after `jr ra` (a possible + # function start) reaches it, so it is dropped: never a root. + base = LOAD + 0x40 + out = base + 0x30 + for shape, word in (('branch', bne_v0(base + 0x0C, out)), + ('call', jal(LOAD))): + data = image({LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME, + base: JR_RA, base + 4: NOP, + base + 8: 0x8C820000, # lw v0,0(a0) + base + 0x0C: word, base + 0x10: NOP, + base + 0x14: FRAME, base + 0x18: ADDIU, + base + 0x1C: JR_RA, base + 0x20: UNFRAME, + out: JR_RA, out + 4: NOP}, 0x100) + prologue = base + 0x14 + self.assertIsNone(CO.prologue_function_start(data, LOAD, + prologue), shape) + self.assertFalse(CO.image_local_entry_proven( + data, LOAD, len(data), prologue, LOAD + len(data)), shape) + weak = set() + strong = CO.derive_static_roots(data, LOAD, len(data), + weak_out=weak) + self.assertNotIn(prologue, strong, shape) + self.assertNotIn(base + 8, strong, shape) + roots, _stats = CO.derive_enrichment_roots(data, LOAD, len(data)) + self.assertNotIn(prologue, roots, shape) + seeds, _audit = classify(data, True) + self.assertNotIn(prologue, root_seeds(seeds), shape) + + def test_branch_from_below_into_preamble_proves_no_start(self): + # The function before returns early (`jr ra`) and branches past it to + # a label inside what looks like the next function's preamble: the + # words after that `jr ra` are still its body. Also the same with the + # label exactly at the would-be start. + host = LOAD + 0x20 + for label_off in (0x0C, 0x08): + label = host + 0x10 + label_off + data = image({host: FRAME, + host + 4: bne_v0(host + 4, label), + host + 8: NOP, + host + 0x0C: JR_RA, host + 0x10: UNFRAME, + host + 0x18: 0x3C02800F, # lui v0,0x800F + host + 0x1C: 0x8C435D4C, # lw v1,..(v0) + host + 0x20: FRAME, host + 0x24: ADDIU, + host + 0x28: JR_RA, host + 0x2C: UNFRAME}, 0x100) + prologue = host + 0x20 + self.assertIsNone(CO.prologue_function_start(data, LOAD, + prologue), + hex(label)) + strong = CO.derive_static_roots(data, LOAD, len(data)) + self.assertNotIn(host + 0x18, strong, hex(label)) + self.assertNotIn(prologue, strong, hex(label)) + seeds, _audit = classify(data, True, function_entry_pcs=[host]) + self.assertEqual(root_seeds(seeds), {host}, hex(label)) + + def test_prologue_after_data(self): + # A pointer table (two pointer-shaped words) or an invalid word ends + # the walk: the prologue right after it is its own start; a preamble + # after data must be global-data setup, since data decodes as + # instructions too. + table = LOAD + 0x40 + words = {LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME, + table: LOAD + 0x80, table + 4: LOAD + 0x90, + table + 8: FRAME, table + 12: JR_RA, table + 16: UNFRAME} + data = image(words, 0x100) + self.assertEqual(CO.prologue_function_start(data, LOAD, table + 8), + table + 8) + setup = dict(words) + setup.update({table + 8: 0x3C02800F, table + 12: 0x8C435D4C, + table + 16: FRAME, table + 20: JR_RA, + table + 24: UNFRAME}) + data = image(setup, 0x100) + self.assertEqual(CO.prologue_function_start(data, LOAD, table + 16), + table + 8) + other = dict(setup) + other[table + 12] = 0xA6020020 # sh v0,32(s0) + data = image(other, 0x100) + self.assertIsNone(CO.prologue_function_start(data, LOAD, table + 16)) + invalid = {LOAD + 0x40: 0x0000003D, LOAD + 0x44: FRAME, + LOAD + 0x48: JR_RA, LOAD + 0x4C: UNFRAME, + # REGIMM with no such rt: data shaped like a branch + LOAD + 0x60: 0x06040200, LOAD + 0x64: FRAME, + LOAD + 0x68: JR_RA, LOAD + 0x6C: UNFRAME} + data = image(invalid, 0x100) + for addr in (LOAD + 0x44, LOAD + 0x64): + self.assertEqual(CO.prologue_function_start(data, LOAD, addr), + addr, hex(addr)) + + def test_single_pointer_shaped_word_is_code(self): + # `lui v0,0x8010; lb v0,0x5CEC(v0)` is 0x80425CEC: one aligned RAM + # address in a preamble is an instruction, not a table. + base = LOAD + 0x40 + data = image({base: JR_RA, base + 4: UNFRAME, + base + 8: 0x3C028010, base + 12: 0x80425CEC, + base + 16: FRAME, base + 20: JR_RA, + base + 24: UNFRAME}, 0x100) + self.assertEqual(CO.prologue_function_start(data, LOAD, base + 16), + base + 8) + + def test_late_prologue_start_respects_producer_start(self): + # The walk never leaves the prologue's producer. + data, start, prologue = self.ac3_image(base=LOAD + 0xC0) + ranges = [(LOAD, LOAD + 0xC8), (LOAD + 0xC8, LOAD + 0x100)] + self.assertEqual(CO.prologue_function_start( + data, LOAD, prologue, LOAD + 0xC8), start) + self.assertEqual(CO.prologue_function_start( + data, LOAD, prologue, LOAD + 0xCC), LOAD + 0xCC) + self.assertIn(start, CO.derive_static_roots(data, LOAD, len(data), + ranges)) + # enrichment policy ---------------------------------------------------- def test_enrichment_is_default_and_opt_out_is_diagnostic(self): data, engine, host, _target = self._cross_image(False) From 6e7ea9791f4dc29c4149867fff5b3afe1f0594b2 Mon Sep 17 00:00:00 2001 From: Matthew Stanley <1379tech@gmail.com> Date: Sat, 26 Sep 2026 09:24:14 -0700 Subject: [PATCH 13/13] docs: late-prologue rule, AC3 and four-title results Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/internal/OVERLAY_ROOT_ENRICHMENT.md | 37 ++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md index 1e0db5c9..61f5c267 100644 --- a/docs/internal/OVERLAY_ROOT_ENRICHMENT.md +++ b/docs/internal/OVERLAY_ROOT_ENRICHMENT.md @@ -136,6 +136,43 @@ mis-splits**, then as much coverage as possible. straight-line multiply block (framework behaviour, beads-eio.3.185). Tomba re-checked on 5a094db6: 0 diverging frames vs master. +## Late prologues (beads-eio.3.191) + +The STRONG prologue rule rooted every `addiu sp,sp,-N` that was not in a +delay slot. Compilers schedule loads in front of the stack adjust +(`lui v0; lw v1,..(v0); addiu sp,sp,-24`), so the root capped the function +two to five words in. Ace Combat 3 had 359 such roots under the guard, 175 +with the real start executed. The guard could not absorb them because the +real start was not a root. + +Now a prologue roots its function's true start +(`prologue_function_start`): walk back over the straight-line preamble to +the nearest boundary (image or producer start, after an unconditional +transfer's delay slot or a `break`, or after data) and root that word. After +data the preamble must be global-data setup (`lui`, `li`, loads from a +`lui` base). No provable start (a conditional branch or call in the +preamble, a local branch from below into it) makes the prologue weak +evidence, which the host above always reaches, so it is not rooted. +Captured entries at a stack adjust keep their own gate and are absorbed by +the true start. + +Measured: + +- **Ace Combat 3** (73 captures): late-prologue walk roots 359 to 1 (the one + is a jal target right after a pointer table, a real start). 357 roots + moved to their true start, 5 mid-function prologues (after a conditional + branch, or entered by a branch from above) are no longer roots. Rootless + 0, capacity skips 0, shards 890/2/61 (the 2 are the guard-word class). + Served identities 6219 vs 6337: the stack adjusts are no longer emitted + as entries, since nothing enters there; captured dispatch entries served + 3671 in both, covered words +278. +- **Four-title gate** (same method as below, master 19b5a65f, own + sandboxes and caches): interpreter fallback G <= N on every run, no shard + failures, every screenshot identical. Forks from N are the same frames + and fields as the previous guard gate. Against the previous guard runs, + only the store/MMIO attribution hashes move (code that changed between + native and interpreted); writes and cycles are identical. + ## Gate used, and what is left The gate the four titles passed is **no regression against master**: