mirror of
https://github.com/RetroPortingToolKit/psxrecomp
synced 2026-10-25 18:05:12 -04:00
Integrate PR392 preserving current import and parser context
This commit is contained in:
Matt Stanley
@@ -0,0 +1,201 @@
|
||||
# Overlay root enrichment: next steps after #386
|
||||
|
||||
#386 added `tools/enrich_overlay_captures.py`. It scans each captured overlay
|
||||
image for likely function starts and writes them to
|
||||
`static_discovery_entry_pcs`. Nothing runs it automatically yet. This page is
|
||||
the plan for making it the default and getting more coverage from it, without
|
||||
any risk of softlocks.
|
||||
|
||||
## Where we are
|
||||
|
||||
- **Measured on Ace Combat 3** (x6568tank's captures, compile tooling fixed):
|
||||
rootless images went from 250 to 1, and capacity skips from 3,028 to 0.
|
||||
There were 8 shard failures, all `fragment: no-func-ids`, and all of them
|
||||
also appear in the comparison tool. So no new failures.
|
||||
- **A richer tool finds 47% more starts** (15,008 vs 10,191). That gave +31%
|
||||
entries served and +19% covered bytes. It also gave 19 failures instead of
|
||||
8, a 38% larger cache and 39% more compile time.
|
||||
- **Our titles' captures** have no `function_entry_pcs`, only
|
||||
`dispatch_entry_pcs`. Enrichment would add:
|
||||
|
||||
| Title | Dispatch entries | Starts added |
|
||||
|---|---|---|
|
||||
| Tomba | 38 | 764 |
|
||||
| Tomba 2 | 178 | 3,316 |
|
||||
| Mega Man X6 | 325 | 901 |
|
||||
| Ape Escape | 149 | 459 |
|
||||
|
||||
## The rule that matters
|
||||
|
||||
A shard **failure** is safe. The compiler refuses the piece and it runs
|
||||
interpreted. The real risk is a wrong start point that **passes** every check.
|
||||
A walk root is a hard cap, so a false root in the middle of a function splits
|
||||
that function. The output is valid C that passes every audit, but it behaves
|
||||
wrong. That is the mid-function-seed softlock class, and it never shows up in
|
||||
the failure count.
|
||||
|
||||
So the failure count is not the safety metric. The goal is **zero
|
||||
mis-splits**, then as much coverage as possible.
|
||||
|
||||
## Plan
|
||||
|
||||
1. **No-split guard** (in `compile_overlays.py`).
|
||||
- Walk each candidate root without hard caps.
|
||||
- A candidate that another root reaches by fallthrough or branch (not by
|
||||
`jal`) is inside that function. Demote it to `DISPATCH_INTERIOR`, which
|
||||
makes it an alias entry and never a cap.
|
||||
- This extends the protection dispatch entries already get to every
|
||||
static root.
|
||||
- With this guard, a wrong guess can only cost speed, never correctness.
|
||||
2. **Move enrichment into `compile_overlays.py`.**
|
||||
- Call the same root derivation when building `pre_roots`, instead of
|
||||
running a separate pipeline step.
|
||||
- Add it to the overlay cache/recipe key, so shards built without it are
|
||||
never reused.
|
||||
- Keep the script for inspecting captures.
|
||||
3. **Add more root categories, behind the guard.** These are the sources the
|
||||
richer tool uses and #386 does not:
|
||||
- `jal` edges from the main EXE and other images into the overlay. These
|
||||
are real call evidence, and likely the safest win.
|
||||
- Frameless starts right after `jr ra` + delay slot. These are riskiest;
|
||||
`compile_overlays` itself says this spot can be data.
|
||||
- Pointer tables with fewer than 3 entries, and `lui`/`addiu`-built
|
||||
pointers.
|
||||
|
||||
Measure each category on its own: roots added, entries served, failures.
|
||||
4. **Explain the `no-func-ids` failures.**
|
||||
- Find which requested entries produce an empty ranges manifest, and why
|
||||
they were requested.
|
||||
- Fix the request side so nothing unprovable is asked for. Do not
|
||||
reclassify these as skips just to reach zero.
|
||||
5. **Validate on Tomba, Tomba 2, Mega Man X6 and Ape Escape.**
|
||||
- Run interpreted-only and enriched-native to the same frame, and compare
|
||||
`frame_fingerprint` rings. They must match.
|
||||
- Then play and explore areas that load new overlays: levels, menus,
|
||||
cutscenes, save and load.
|
||||
|
||||
## Status: steps 1 and 2 (beads-eio.3.177)
|
||||
|
||||
- **Guard** (`no_split_partition` in `compile_overlays.py`, same rule in
|
||||
`FunctionAnalyzer::analyze_exact_entries` for overlay mode). For each root,
|
||||
the following candidates its uncapped walk reaches are absorbed together
|
||||
(a switch resolves only when all its case labels are inside the walk); the
|
||||
run stops at the first unreached root, so a tail call over another function
|
||||
keeps its target. Applies to every root source; promoted kernel orphans are
|
||||
exempt. Delay-slot candidates are never roots or aliases.
|
||||
- **Evidence.** A jal or pointer only proves a function start in the image
|
||||
that is resident when it runs. STRONG = a prologue, or a jal/table target
|
||||
that this image bounds (`jr $ra` before it plus the CFG probe). WEAK = CFG
|
||||
probe only; kept only if no possible function start reaches it, and dropped
|
||||
if a host's local branch jumps over it. Cross-producer calls are never weak
|
||||
evidence.
|
||||
- **Default on, one code path.** `classify_overlay_seeds` derives the roots;
|
||||
`tools/enrich_overlay_captures.py` is an inspection tool built on the same
|
||||
functions. `--no-root-enrichment` / `PSX_OVERLAY_ROOT_ENRICHMENT=0` is
|
||||
diagnostic only.
|
||||
- **Cache key.** `tools/compile_overlays.py` is in
|
||||
`runtime/codegen_hash_sources.cmake`, so any root-policy change moves the
|
||||
cache namespace.
|
||||
- **Tomba (25 AOT images).** Emitted fallthrough splits: master 57, #386
|
||||
enrichment 2,072, guard 0. Fingerprints match master on every value and
|
||||
timing field over the 30.7k-frame route and the 32k attract run. Sound
|
||||
enrichment adds almost nothing on Tomba: 12.3k of #386's extra roots were
|
||||
jump-table case labels inside functions. The way to that coverage is
|
||||
resolving those tables, not rooting their labels.
|
||||
- **Not caused by splits:** master's own 3-4 cycle drift from the interpreter
|
||||
at frame 2111. It is unchanged with the guard.
|
||||
- **Tomba 2 (23 captures, autocompile path).** Two guard gaps found and
|
||||
fixed. (1) Roots the region shard leaves out are built in strong-root
|
||||
supplement fragments from `dispatch_root` seeds only, so aliases of those
|
||||
hosts were served by no shard (127 captured dispatch entries lost). The
|
||||
supplement now passes each host's aliases as `interior` seeds, and a single
|
||||
root whose aliases break its fragment is rebuilt without them. (2) When
|
||||
every host of an absorbed candidate stays interpreted (0x800BDF10 walks
|
||||
into data and fails the audit), the candidate is compiled as the root it
|
||||
was before the guard. Left: 14 captured dispatch entries at delay slots,
|
||||
which the guard keeps interpreted by design (master aliased them).
|
||||
- **Tomba 2 fingerprints do not match master exactly,** and the cause is
|
||||
master's own splits. Frame 6797: master splits 0x8009B0C0 from its
|
||||
fallthrough 0x8009B0CC; the split leaves the nested call unit, so master
|
||||
takes a CD IRQ inside the loop, while the unsplit function runs as one
|
||||
unit and the runtime's nested-unit rule defers the IRQ to its return
|
||||
(with `PSX_OVERLAY_UNIT_DEFER=0` in both arms this fork goes away).
|
||||
Frame 7169/7363: master splits switch 0x8006D654 at its case label
|
||||
0x8006D690; the jump-table exit costs master 2 extra cycles. Values stay
|
||||
identical, and every route and attract screenshot is identical.
|
||||
|
||||
- **Mega Man X6 and Ape Escape (guard at 5a094db6).** Same gate as below,
|
||||
arms N (master) and G (guard), own sandboxes and caches, both proven to run
|
||||
native. MMX6: interpreter fallback 14.04M vs 14.11M on the route, 0/0
|
||||
shard failures, 74/74 route and 13/13 attract screenshots identical; the
|
||||
only fingerprint fork is the store-PC hash at 8303, where overlay code
|
||||
master interprets runs native in G (same address, value and cycle).
|
||||
Ape: fallback 3.28M vs 3.31M, memcard LOAD GAME passes, 67/67 and 39/39
|
||||
screenshots identical; fork at 3564 is the same interpreted-to-native class
|
||||
plus a 3-cycle difference in how native and interpreter cost one
|
||||
straight-line multiply block (framework behaviour, beads-eio.3.185).
|
||||
Tomba re-checked on 5a094db6: 0 diverging frames vs master.
|
||||
|
||||
## Late prologues (beads-eio.3.191)
|
||||
|
||||
The STRONG prologue rule rooted every `addiu sp,sp,-N` that was not in a
|
||||
delay slot. Compilers schedule loads in front of the stack adjust
|
||||
(`lui v0; lw v1,..(v0); addiu sp,sp,-24`), so the root capped the function
|
||||
two to five words in. Ace Combat 3 had 359 such roots under the guard, 175
|
||||
with the real start executed. The guard could not absorb them because the
|
||||
real start was not a root.
|
||||
|
||||
Now a prologue roots its function's true start
|
||||
(`prologue_function_start`): walk back over the straight-line preamble to
|
||||
the nearest boundary (image or producer start, after an unconditional
|
||||
transfer's delay slot or a `break`, or after data) and root that word. After
|
||||
data the preamble must be global-data setup (`lui`, `li`, loads from a
|
||||
`lui` base). No provable start (a conditional branch or call in the
|
||||
preamble, a local branch from below into it) makes the prologue weak
|
||||
evidence, which the host above always reaches, so it is not rooted.
|
||||
Captured entries at a stack adjust keep their own gate and are absorbed by
|
||||
the true start.
|
||||
|
||||
Measured:
|
||||
|
||||
- **Ace Combat 3** (73 captures): late-prologue walk roots 359 to 1 (the one
|
||||
is a jal target right after a pointer table, a real start). 357 roots
|
||||
moved to their true start, 5 mid-function prologues (after a conditional
|
||||
branch, or entered by a branch from above) are no longer roots. Rootless
|
||||
0, capacity skips 0, shards 890/2/61 (the 2 are the guard-word class).
|
||||
Served identities 6219 vs 6337: the stack adjusts are no longer emitted
|
||||
as entries, since nothing enters there; captured dispatch entries served
|
||||
3671 in both, covered words +278.
|
||||
- **Four-title gate** (same method as below, master 19b5a65f, own
|
||||
sandboxes and caches): interpreter fallback G <= N on every run, no shard
|
||||
failures, every screenshot identical. Forks from N are the same frames
|
||||
and fields as the previous guard gate. Against the previous guard runs,
|
||||
only the store/MMIO attribution hashes move (code that changed between
|
||||
native and interpreted); writes and cycles are identical.
|
||||
|
||||
## Gate used, and what is left
|
||||
|
||||
The gate the four titles passed is **no regression against master**:
|
||||
interpreter fallback not higher, no new shard failures, no freeze or wedge,
|
||||
all checkpoint screenshots identical. Small timing forks that come from
|
||||
removing master's own splits are accepted. Matching the interpreted-only
|
||||
arm exactly is not the gate: master itself drifts a few cycles from it.
|
||||
|
||||
Done:
|
||||
|
||||
- The no-split guard, with tests that mid-function, branch-reached and
|
||||
delay-slot candidates are demoted, not rooted.
|
||||
- Enrichment on by default in `compile_overlays.py`, and part of the cache
|
||||
key.
|
||||
- Four-title regression gate (above).
|
||||
|
||||
Not done:
|
||||
|
||||
- Step 3 as a widening. The evidence rules were made stricter (image-local
|
||||
proof, cross-producer calls excluded), not extended with new categories.
|
||||
- Step 4, the `no-func-ids` root cause.
|
||||
- Re-measuring Ace Combat 3. #386's AC3 numbers (rootless images 250 to 1,
|
||||
capacity skips 3,028 to 0) were taken without the guard; the stricter
|
||||
evidence rule may root fewer of those images.
|
||||
- Delay-slot dispatch PCs that master aliased stay interpreted under the
|
||||
guard by design (Tomba 96, Tomba 2 14, Ape 10); no fallback cost measured.
|
||||
@@ -920,6 +920,18 @@ if(BUILD_TESTING)
|
||||
COMMAND ${Python3_EXECUTABLE}
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../tools/tests/test_enrich_overlay_captures.py
|
||||
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..)
|
||||
# A walk root is a hard cap: a root inside another function splits it
|
||||
# and still passes every audit. The classifier's no-split guard, the
|
||||
# image-local enrichment proof, the recompiler's own guard on explicit
|
||||
# roots, and the root policy's place in the cache namespace
|
||||
# (beads-eio.3.177).
|
||||
add_test(
|
||||
NAME overlay_no_split_guard
|
||||
COMMAND ${Python3_EXECUTABLE}
|
||||
${CMAKE_CURRENT_SOURCE_DIR}/../tools/tests/test_overlay_no_split_guard.py
|
||||
--cmake ${CMAKE_COMMAND}
|
||||
--recompiler $<TARGET_FILE:psxrecomp-game>
|
||||
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/..)
|
||||
|
||||
# The runtime exe name must come from CMake, not be re-derived.
|
||||
add_test(
|
||||
|
||||
@@ -172,10 +172,22 @@ public:
|
||||
// reachable from them. Used by runtime-loaded overlays and opt-in main-EXE
|
||||
// reachable discovery. Unresolved jalr/indirect targets do not mint
|
||||
// functions; evidence-backed entries must be supplied explicitly.
|
||||
//
|
||||
// no_split_guard (overlay mode): an EXPLICIT entry that the entry below
|
||||
// it reaches by fallthrough or branch -- walked with the entry's own cap
|
||||
// removed -- is inside that function. It is absorbed like a derived
|
||||
// call target and exported in absorbed_entries (an alias, never a cap).
|
||||
// `trusted_entries` (classifier-promoted dispatch roots) are exempt.
|
||||
// compile_overlays.py applies the same rule before writing seeds; this is
|
||||
// the enforcement point for walk proofs only this analyzer can make
|
||||
// (computed-stride and self-limited jump tables). Off by default so the
|
||||
// main-EXE reachable-discovery partition is unchanged.
|
||||
FunctionAnalysisResult analyze_exact_entries(
|
||||
const std::vector<uint32_t>& entries,
|
||||
const std::vector<std::pair<uint32_t, uint32_t>>& producer_ranges = {},
|
||||
const std::set<uint32_t>& cross_call_allow = {});
|
||||
const std::set<uint32_t>& cross_call_allow = {},
|
||||
bool no_split_guard = false,
|
||||
const std::set<uint32_t>& trusted_entries = {});
|
||||
|
||||
// Add a forced entry point address that is treated as a function start
|
||||
// even if it has no standard ADDIU $sp prologue. The function will be
|
||||
|
||||
@@ -1418,7 +1418,9 @@ bool resolve_self_limited_jump_table(
|
||||
FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(
|
||||
const std::vector<uint32_t>& entries,
|
||||
const std::vector<std::pair<uint32_t, uint32_t>>& producer_ranges,
|
||||
const std::set<uint32_t>& cross_call_allow) {
|
||||
const std::set<uint32_t>& cross_call_allow,
|
||||
bool no_split_guard,
|
||||
const std::set<uint32_t>& trusted_entries) {
|
||||
FunctionAnalysisResult result;
|
||||
result.total_instructions = 0;
|
||||
result.jr_ra_count = 0;
|
||||
@@ -1628,6 +1630,61 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(
|
||||
std::map<uint32_t, std::set<std::pair<uint32_t, bool>>> derived_evidence;
|
||||
fmt::print("Explicit entries: {}\n", explicit_count);
|
||||
|
||||
// No-split guard: fixed point over EVERY non-trusted entry (explicit and
|
||||
// derived); same rule as compile_overlays.no_split_partition. For each
|
||||
// host, the run of following entries its UNCAPPED walk reaches (stopping
|
||||
// at the first it does not reach, or a trusted entry) may be inside it;
|
||||
// walk the host capped at the end of the run, shrink the run to the first
|
||||
// entry that walk misses, and absorb the rest together. Absorbing the run
|
||||
// at once lets a switch whose table targets several entries resolve
|
||||
// (every target must lie inside the walk); stopping at the first
|
||||
// unreached entry keeps a forward tail call over an unrelated function
|
||||
// from orphaning its target. Absorbing only removes caps, so repeat.
|
||||
auto absorb_split_entries = [&](std::set<uint32_t>& entries) {
|
||||
const uint32_t analysis_end = exe_.analysis_end_address();
|
||||
bool changed = true;
|
||||
while (changed) {
|
||||
changed = false;
|
||||
std::vector<uint32_t> roots(entries.begin(), entries.end());
|
||||
std::vector<uint32_t> kept;
|
||||
size_t index = 0;
|
||||
while (index < roots.size()) {
|
||||
const uint32_t host = roots[index];
|
||||
kept.push_back(host);
|
||||
const ExactWalkResult reach = walk(host, analysis_end);
|
||||
// A wider range can reject a jump table, so the walk capped
|
||||
// at the successor is not always a subset: consult both.
|
||||
const ExactWalkResult succ = index + 1 < roots.size()
|
||||
? walk(host, roots[index + 1]) : ExactWalkResult{};
|
||||
auto reached = [&](uint32_t a) {
|
||||
return reach.visited.count(a) || succ.visited.count(a);
|
||||
};
|
||||
size_t end = index + 1;
|
||||
while (end < roots.size() && reached(roots[end]) &&
|
||||
!trusted_entries.count(roots[end]))
|
||||
end++;
|
||||
while (end > index + 1) {
|
||||
const uint32_t cap =
|
||||
end < roots.size() ? roots[end] : analysis_end;
|
||||
const ExactWalkResult inner = walk(host, cap);
|
||||
size_t blocked = end;
|
||||
for (size_t k = index + 1; k < end; k++) {
|
||||
if (!inner.visited.count(roots[k])) {
|
||||
blocked = k;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (blocked == end) break;
|
||||
end = blocked;
|
||||
}
|
||||
if (end > index + 1) changed = true;
|
||||
index = end;
|
||||
}
|
||||
entries = std::set<uint32_t>(kept.begin(), kept.end());
|
||||
}
|
||||
};
|
||||
if (no_split_guard) absorb_split_entries(known_entries);
|
||||
|
||||
// Discover callees in rounds against one stable entry partition. Processing
|
||||
// roots sequentially let an early caller insert a JAL target that actually
|
||||
// lay inside a later explicit root. That new entry hard-capped the later
|
||||
@@ -1643,6 +1700,7 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(
|
||||
fmt::print("WARNING: exact-entry ownership did not converge; "
|
||||
"falling back to explicit roots only\n");
|
||||
known_entries = explicit_entries;
|
||||
if (no_split_guard) absorb_split_entries(known_entries);
|
||||
derived_entries.clear();
|
||||
derived_evidence.clear();
|
||||
break;
|
||||
@@ -1680,7 +1738,9 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(
|
||||
// "absorbed forever" set therefore loses reachable code.
|
||||
known_entries = explicit_entries;
|
||||
known_entries.insert(derived_entries.begin(), derived_entries.end());
|
||||
for (uint32_t target : derived_entries) {
|
||||
if (no_split_guard) {
|
||||
absorb_split_entries(known_entries);
|
||||
} else for (uint32_t target : derived_entries) {
|
||||
auto target_it = known_entries.find(target);
|
||||
if (target_it == known_entries.end() ||
|
||||
explicit_entries.count(target) ||
|
||||
@@ -1701,8 +1761,15 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(
|
||||
derived_entries == previous_derived) break;
|
||||
}
|
||||
|
||||
result.call_discovered_count = static_cast<int>(known_entries.size() - explicit_count);
|
||||
size_t explicit_kept = 0;
|
||||
for (uint32_t entry : known_entries)
|
||||
explicit_kept += explicit_entries.count(entry);
|
||||
result.call_discovered_count =
|
||||
static_cast<int>(known_entries.size() - explicit_kept);
|
||||
fmt::print("Direct-JAL entries: {}\n", result.call_discovered_count);
|
||||
if (no_split_guard)
|
||||
fmt::print("No-split guard absorbed explicit entries: {}\n",
|
||||
explicit_count - explicit_kept);
|
||||
fmt::print("Total exact entries: {}\n\n", known_entries.size());
|
||||
|
||||
std::vector<uint32_t> starts_vec(known_entries.begin(), known_entries.end());
|
||||
@@ -1784,6 +1851,26 @@ FunctionAnalysisResult FunctionAnalyzer::analyze_exact_entries(
|
||||
break;
|
||||
}
|
||||
}
|
||||
// Explicit entries the no-split guard absorbed: the proof is the host's
|
||||
// own walk (fallthrough/branch), so the host start is the source. A
|
||||
// delay-slot target is not exported: an alias there would make the slot
|
||||
// a block leader. It stays with the interpreter.
|
||||
if (no_split_guard) {
|
||||
for (uint32_t target : explicit_entries) {
|
||||
if (final_starts.count(target)) continue;
|
||||
if (target >= exe_.header.load_address + 4u) {
|
||||
auto prev = exe_.read_word(target - 4u);
|
||||
if (prev.has_value() && is_branch_or_jump(*prev)) continue;
|
||||
}
|
||||
for (const auto& [host, wr] : final_walks) {
|
||||
if (!wr.visited.count(target)) continue;
|
||||
result.absorbed_entries.push_back(
|
||||
{target, host.start_addr, host.end_addr,
|
||||
host.start_addr, false});
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
@@ -1161,8 +1161,15 @@ static int psxrecomp_game_main(int argc, char** argv) {
|
||||
{
|
||||
PSXRecomp::FunctionAnalyzer analyzer(*exe);
|
||||
std::vector<uint32_t> roots_vec(roots.begin(), roots.end());
|
||||
// Overlay mode enforces the no-split guard on explicit roots too
|
||||
// (compile_overlays.py applies the same rule first; this catches
|
||||
// walk proofs only this analyzer makes). Classifier-promoted
|
||||
// dispatch roots are exempt. Main-EXE reachable discovery keeps
|
||||
// its existing partition.
|
||||
analysis_result = analyzer.analyze_exact_entries(
|
||||
roots_vec, producer_ranges, cross_call_allow);
|
||||
roots_vec, producer_ranges, cross_call_allow,
|
||||
/*no_split_guard=*/overlay_mode,
|
||||
overlay_mode ? trusted_root_seeds : std::set<uint32_t>{});
|
||||
}
|
||||
|
||||
std::vector<AliasEntry> alias_entries;
|
||||
|
||||
@@ -760,6 +760,83 @@ int main() {
|
||||
thunk_starts.count(kLoad + 0x610),
|
||||
"packed BIOS thunk bounds the following frameless leaf");
|
||||
|
||||
// ---- No-split guard on EXPLICIT entries (overlay mode) ----------------
|
||||
// A root is a hard cap, so an explicit entry that the entry below it
|
||||
// reaches by fallthrough splits that function (beads-eio.3.177: Tomba
|
||||
// X00 root 0x8011B1CC, a subu mid-way through host 0x8011AC10).
|
||||
{
|
||||
auto split_image = make_exe_buffer(0x1000);
|
||||
put32(split_image, text + 0x100, 0x27BDFFE0u); // addiu sp,sp,-32
|
||||
put32(split_image, text + 0x104, 0x24420001u);
|
||||
put32(split_image, text + 0x108, 0x00431023u); // subu v0,v0,v1
|
||||
put32(split_image, text + 0x10C, 0x03E00008u);
|
||||
put32(split_image, text + 0x110, 0x27BD0020u);
|
||||
auto split_exe = parse(split_image);
|
||||
PSXRecomp::FunctionAnalyzer unguarded(split_exe);
|
||||
CHECK(starts(unguarded.analyze_exact_entries(
|
||||
{kLoad + 0x100, kLoad + 0x108})) ==
|
||||
std::set<uint32_t>({kLoad + 0x100, kLoad + 0x108}),
|
||||
"without the guard an explicit mid-function entry still caps");
|
||||
PSXRecomp::FunctionAnalyzer guarded(split_exe);
|
||||
const auto guarded_result = guarded.analyze_exact_entries(
|
||||
{kLoad + 0x100, kLoad + 0x108}, {}, {}, true);
|
||||
CHECK(starts(guarded_result) == std::set<uint32_t>({kLoad + 0x100}),
|
||||
"no-split guard absorbs an explicit entry reached by fallthrough");
|
||||
bool aliased = false;
|
||||
for (const auto& absorbed : guarded_result.absorbed_entries)
|
||||
aliased |= absorbed.addr == kLoad + 0x108 &&
|
||||
absorbed.host_start == kLoad + 0x100 &&
|
||||
absorbed.addr < absorbed.host_end;
|
||||
CHECK(aliased, "absorbed explicit entry is exported as a hosted alias");
|
||||
|
||||
PSXRecomp::FunctionAnalyzer trusted(split_exe);
|
||||
CHECK(starts(trusted.analyze_exact_entries(
|
||||
{kLoad + 0x100, kLoad + 0x108}, {}, {}, true,
|
||||
{kLoad + 0x108})) ==
|
||||
std::set<uint32_t>({kLoad + 0x100, kLoad + 0x108}),
|
||||
"trusted (promoted dispatch) entries are exempt from the guard");
|
||||
}
|
||||
{
|
||||
// Every case label of a switch is an explicit entry. Each one caps
|
||||
// the host below its siblings, so the table (which resolves only when
|
||||
// all of its targets are inside the walk) is only visible when the
|
||||
// whole run of case labels is absorbed together.
|
||||
auto sw = make_ape_switch();
|
||||
auto sw_exe = parse(sw.image);
|
||||
std::vector<uint32_t> entries = {sw.entry};
|
||||
entries.insert(entries.end(), sw.cases.begin(), sw.cases.end());
|
||||
PSXRecomp::FunctionAnalyzer guarded(sw_exe);
|
||||
const auto result = guarded.analyze_exact_entries(entries, {}, {}, true);
|
||||
CHECK(starts(result) == std::set<uint32_t>({sw.entry}),
|
||||
"no-split guard absorbs every case-label entry of a switch host");
|
||||
size_t hosted = 0;
|
||||
for (const auto& absorbed : result.absorbed_entries)
|
||||
hosted += absorbed.host_start == sw.entry;
|
||||
CHECK(hosted == sw.cases.size(),
|
||||
"every absorbed case label is hosted by the switch function");
|
||||
}
|
||||
{
|
||||
// A forward tail call over an unrelated function must not demote the
|
||||
// tail-call target: the intervening root would leave it hostless.
|
||||
auto tail_image = make_exe_buffer(0x1000);
|
||||
put32(tail_image, text + 0x700, 0x08000000u |
|
||||
(((kLoad + 0x740) >> 2) & 0x03FFFFFFu));
|
||||
put32(tail_image, text + 0x704, 0x00000000u);
|
||||
put32(tail_image, text + 0x720, 0x03E00008u);
|
||||
put32(tail_image, text + 0x724, 0x00000000u);
|
||||
put32(tail_image, text + 0x740, 0x24020001u);
|
||||
put32(tail_image, text + 0x744, 0x03E00008u);
|
||||
put32(tail_image, text + 0x748, 0x00000000u);
|
||||
auto tail_exe = parse(tail_image);
|
||||
PSXRecomp::FunctionAnalyzer guarded(tail_exe);
|
||||
CHECK(starts(guarded.analyze_exact_entries(
|
||||
{kLoad + 0x700, kLoad + 0x720, kLoad + 0x740},
|
||||
{}, {}, true)) ==
|
||||
std::set<uint32_t>(
|
||||
{kLoad + 0x700, kLoad + 0x720, kLoad + 0x740}),
|
||||
"tail call over an unrelated root keeps its target a root");
|
||||
}
|
||||
|
||||
PSXRecomp::FunctionAnalyzer seeded_analyzer(exe);
|
||||
const auto seeded = starts(
|
||||
seeded_analyzer.analyze_exact_entries({kLoad, kLoad + 0x200}));
|
||||
|
||||
@@ -58,6 +58,16 @@ set(PSXRECOMP_CODEGEN_HASH_SRCS
|
||||
${PSXRECOMP_CODEGEN_HASH_ROOT}/recompiler/include/basic_block.h
|
||||
${PSXRECOMP_CODEGEN_HASH_ROOT}/recompiler/include/code_generator.h
|
||||
|
||||
# --- Root policy ------------------------------------------------------------
|
||||
# compile_overlays.py decides which captured PCs become walk roots, alias
|
||||
# entries or interpreter-only PCs (the seed classifier, the no-split guard
|
||||
# and default-on root enrichment all live there). Roots are hard caps, so
|
||||
# the same bytes and the same emitter produce DIFFERENT shards under a
|
||||
# different root policy: a shard built before the no-split guard can carry
|
||||
# a mis-split host (beads-eio.3.177). Hashing the classifier moves the cache
|
||||
# namespace whenever the policy changes, so such shards are never reused.
|
||||
${PSXRECOMP_CODEGEN_HASH_ROOT}/tools/compile_overlays.py
|
||||
|
||||
# --- Shard COMPILE surface -------------------------------------------------
|
||||
# The cg tag must invalidate on anything a shard compiles/links against, not
|
||||
# just what the emitter WRITES. A change to the overlay ABI header, the
|
||||
|
||||
+967
-72
File diff suppressed because it is too large
Load Diff
@@ -1,18 +1,25 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Add conservative static walk-root evidence to runtime overlay captures.
|
||||
"""Inspect (and optionally materialize) static root enrichment for captures.
|
||||
|
||||
Runtime capture knows that a dirty-RAM PC was dispatched, but that is not the
|
||||
same thing as knowing it is a function boundary: CPS continuations and jump
|
||||
table cases are dispatched too. This tool therefore leaves the runtime-owned
|
||||
``function_entry_pcs`` untouched and derives the optional, tool-owned
|
||||
``static_discovery_entry_pcs`` from captured bytes. compile_overlays.py
|
||||
re-validates every supplied root before compiling it.
|
||||
compile_overlays.py derives these roots itself (enrichment is on by default
|
||||
and part of the overlay cache key), so this tool is no longer a pipeline
|
||||
step. It shows what the compiler's derivation finds in each captured image,
|
||||
using exactly the same code (compile_overlays.derive_static_roots), and can
|
||||
still write the roots into ``static_discovery_entry_pcs`` for offline A/B.
|
||||
|
||||
A jal or pointer elsewhere in the image is not proof on its own: overlay
|
||||
regions are swapped, so shared code can call an address that is a function
|
||||
start only in a sibling image. A candidate is STRONG when this image's bytes
|
||||
bound it (a non-delay-slot stack-frame prologue, or a preceding ``jr $ra``
|
||||
plus the bounded CFG probe) and WEAK when only the CFG probe accepts it; a
|
||||
weak candidate is kept only if no possible function start below it reaches
|
||||
it (compile_overlays.derive_enrichment_roots). compile_overlays.py
|
||||
additionally applies the no-split guard to every root, derived or not.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import base64
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
@@ -22,66 +29,25 @@ sys.path.insert(0, str(ROOT / 'tools'))
|
||||
import compile_overlays as CO # noqa: E402
|
||||
|
||||
|
||||
def word_at(data, base, addr):
|
||||
offset = addr - base
|
||||
if offset < 0 or offset + 4 > len(data):
|
||||
return None
|
||||
return struct.unpack_from('<I', data, offset)[0]
|
||||
|
||||
|
||||
def is_candidate(data, base, addr):
|
||||
"""Use only the compiler's existing root proofs.
|
||||
|
||||
A classic stack-frame prologue is already accepted by overlay discovery.
|
||||
Frameless entries additionally need its bounded CFG proof, which rejects
|
||||
tables and sequential data that happen to decode as MIPS.
|
||||
"""
|
||||
if addr & 3 or not base <= addr < base + len(data):
|
||||
return False
|
||||
word = word_at(data, base, addr)
|
||||
previous = word_at(data, base, addr - 4)
|
||||
if CO._is_addiu_sp_neg(word) and not CO._is_control_flow(previous):
|
||||
return True
|
||||
return CO.plausible_callable_target(data, base, len(data), addr,
|
||||
base + len(data))
|
||||
|
||||
|
||||
def derived_roots(data, base):
|
||||
"""Find generic, independently evidenced roots in one captured image."""
|
||||
hi = base + len(data)
|
||||
roots = set()
|
||||
pointers = []
|
||||
for addr in range(base, hi - 3, 4):
|
||||
word = word_at(data, base, addr)
|
||||
if CO._is_addiu_sp_neg(word) and not CO._is_control_flow(
|
||||
word_at(data, base, addr - 4)):
|
||||
roots.add(addr)
|
||||
if word is not None and (word >> 26) == 0x03: # jal
|
||||
target = 0x80000000 | ((word & 0x03ffffff) << 2)
|
||||
if is_candidate(data, base, target):
|
||||
roots.add(target)
|
||||
pointers.append(word if word is not None and not (word & 3) and
|
||||
base <= word < hi else None)
|
||||
|
||||
# A dense run is a table only when it has three or more in-image pointers.
|
||||
# Each member must still pass the same root proof; no bare pointer becomes
|
||||
# authority, and a pointer to the table itself fails the CFG proof.
|
||||
start = 0
|
||||
while start < len(pointers):
|
||||
if pointers[start] is None:
|
||||
start += 1
|
||||
continue
|
||||
end = start + 1
|
||||
while end < len(pointers) and pointers[end] is not None:
|
||||
end += 1
|
||||
if end - start >= 3:
|
||||
for target in pointers[start:end]:
|
||||
if is_candidate(data, base, target):
|
||||
roots.add(target)
|
||||
start = end
|
||||
def derived_roots(data, base, explicit=(), sources=None, stats=None):
|
||||
"""The compiler's own enrichment roots for one captured image."""
|
||||
roots, info = CO.derive_enrichment_roots(data, base, len(data),
|
||||
explicit=explicit)
|
||||
if sources is not None:
|
||||
weak = set()
|
||||
CO.derive_static_roots(data, base, len(data), sources=sources,
|
||||
weak_out=weak)
|
||||
if stats is not None:
|
||||
stats.update(info)
|
||||
return sorted(roots)
|
||||
|
||||
|
||||
def explicit_roots(record):
|
||||
return set().union(*(CO._parse_addr_list(record.get(key, [])) for key in (
|
||||
'function_entry_pcs', 'dispatch_entry_pcs',
|
||||
'static_discovery_entry_pcs', 'seeds')))
|
||||
|
||||
|
||||
def enrich_record(record):
|
||||
"""Return a copy of *record* with discovered roots unioned into it."""
|
||||
out = dict(record)
|
||||
@@ -94,7 +60,8 @@ def enrich_record(record):
|
||||
if size != len(data) or base & 3 or len(data) & 3:
|
||||
raise ValueError('capture bytes must be a word-aligned image matching size')
|
||||
prior = CO._parse_addr_list(record.get('static_discovery_entry_pcs', []))
|
||||
roots = sorted(prior | set(derived_roots(data, base)))
|
||||
roots = sorted(prior | set(derived_roots(data, base,
|
||||
explicit_roots(record))))
|
||||
out['static_discovery_entry_pcs'] = [f'0x{addr:08X}' for addr in roots]
|
||||
return out, len(roots) - len(prior)
|
||||
|
||||
@@ -102,7 +69,9 @@ def enrich_record(record):
|
||||
def main(argv=None):
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--captures', required=True, help='runtime capture JSON')
|
||||
parser.add_argument('--out', required=True, help='enriched JSON output path')
|
||||
parser.add_argument('--out', help='write an enriched JSON copy here')
|
||||
parser.add_argument('--list', action='store_true',
|
||||
help='print every derived root and its evidence sources')
|
||||
args = parser.parse_args(argv)
|
||||
with open(args.captures, encoding='utf-8') as src:
|
||||
records = json.load(src)
|
||||
@@ -116,12 +85,24 @@ def main(argv=None):
|
||||
item, count = enrich_record(record)
|
||||
except ValueError as exc:
|
||||
raise SystemExit(f'capture {index}: {exc}') from exc
|
||||
if args.list:
|
||||
base = CO._parse_addr(record['load_addr'])
|
||||
data = base64.b64decode(record['bytes_b64'])
|
||||
sources, stats = {}, {}
|
||||
roots = derived_roots(data, base, explicit_roots(record),
|
||||
sources, stats)
|
||||
print(f'capture {index} load=0x{base:08X}: {len(roots)} derived '
|
||||
f'root(s) {stats}')
|
||||
for addr in roots:
|
||||
print(f' 0x{addr:08X} {"+".join(sorted(sources[addr]))}')
|
||||
enriched.append(item)
|
||||
added += count
|
||||
with open(args.out, 'w', encoding='utf-8', newline='\n') as dst:
|
||||
json.dump(enriched, dst, indent=2)
|
||||
dst.write('\n')
|
||||
print(f'enriched {len(enriched)} capture(s); added {added} static discovery root(s)')
|
||||
if args.out:
|
||||
with open(args.out, 'w', encoding='utf-8', newline='\n') as dst:
|
||||
json.dump(enriched, dst, indent=2)
|
||||
dst.write('\n')
|
||||
print(f'{len(enriched)} capture(s); {added} static discovery root(s) not '
|
||||
f'already declared')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -0,0 +1,767 @@
|
||||
#!/usr/bin/env python3
|
||||
"""The overlay no-split guard, image-local root enrichment, and the cache key.
|
||||
|
||||
A walk root is a hard cap: the walk of the root below it stops there, so a
|
||||
root inside another function's body splits that function. The generated C
|
||||
still passes every audit, which is why this class never shows up as a shard
|
||||
failure. Measured on Tomba (bead beads-eio.3.177): shared engine code, byte
|
||||
identical in 12 swapped area images, does `jal 0x8011B1CC`; that address is a
|
||||
function start in one image and a `subu` mid-way through host 0x8011AC10 in
|
||||
X00. Rooting it there split the host and moved guest timing by one cycle
|
||||
against the interpreter.
|
||||
|
||||
Usage: test_overlay_no_split_guard.py --cmake <cmake> [--recompiler <exe>]
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / 'tools'))
|
||||
import compile_overlays as CO # noqa: E402
|
||||
|
||||
LOAD = 0x80100000
|
||||
NOP = 0x00000000
|
||||
JR_RA = 0x03E00008
|
||||
FRAME = 0x27BDFFE0 # addiu sp,sp,-32
|
||||
UNFRAME = 0x27BD0020 # addiu sp,sp,32
|
||||
ADDIU = 0x24420001 # addiu v0,v0,1
|
||||
SUBU = 0x00431023 # subu v0,v0,v1
|
||||
CMAKE = shutil.which('cmake') # --cmake overrides; ctest always passes it
|
||||
RECOMPILER = None
|
||||
|
||||
|
||||
def jal(target):
|
||||
return 0x0C000000 | ((target >> 2) & 0x03FFFFFF)
|
||||
|
||||
|
||||
def bne_v0(pc, target):
|
||||
return 0x14400000 | (((target - (pc + 4)) >> 2) & 0xFFFF)
|
||||
|
||||
|
||||
def image(words: dict, size: int) -> bytes:
|
||||
data = bytearray(size)
|
||||
for addr, word in words.items():
|
||||
struct.pack_into('<I', data, addr - LOAD, word)
|
||||
return bytes(data)
|
||||
|
||||
|
||||
def classify(data, enrichment=True, **cap_fields):
|
||||
cap = {'schema': 'psxrecomp overlay capture v2',
|
||||
'function_entry_pcs': [], 'dispatch_entry_pcs': []}
|
||||
for key, values in cap_fields.items():
|
||||
cap[key] = [f'0x{addr:08X}' for addr in values]
|
||||
return CO.classify_overlay_seeds(cap, data, LOAD, len(data), 0, {},
|
||||
root_enrichment=enrichment)
|
||||
|
||||
|
||||
def root_seeds(seeds):
|
||||
return {int(seed.split()[-1], 16) for seed in seeds
|
||||
if seed.split()[0] in ('call_root', 'dispatch_root') or
|
||||
seed.startswith('0x')}
|
||||
|
||||
|
||||
class NoSplitGuardTests(unittest.TestCase):
|
||||
# (a) ------------------------------------------------------------------
|
||||
def test_fallthrough_candidate_is_demoted_not_rooted(self):
|
||||
host, mid = LOAD + 0x20, LOAD + 0x30
|
||||
data = image({host: FRAME, host + 4: ADDIU, host + 8: ADDIU,
|
||||
host + 12: ADDIU, mid: SUBU, mid + 4: ADDIU,
|
||||
mid + 8: JR_RA, mid + 12: UNFRAME}, 0x100)
|
||||
# A #386-style enriched capture declared the mid-body word as a
|
||||
# static discovery root; it passes the old CFG-only probe.
|
||||
self.assertTrue(CO.plausible_callable_target(
|
||||
data, LOAD, len(data), mid, LOAD + len(data)))
|
||||
for enrichment in (True, False):
|
||||
seeds, audit = classify(data, enrichment,
|
||||
function_entry_pcs=[host],
|
||||
static_discovery_entry_pcs=[mid])
|
||||
self.assertEqual(audit['included_reasons'][mid],
|
||||
'DISPATCH_INTERIOR')
|
||||
self.assertIn(f'interior 0x{mid:08X}', seeds)
|
||||
self.assertNotIn(mid, root_seeds(seeds))
|
||||
self.assertIn(host, root_seeds(seeds))
|
||||
self.assertEqual(audit['guard_demoted'][mid],
|
||||
('STATIC_DISCOVERY_ROOT', host))
|
||||
|
||||
# (b) ------------------------------------------------------------------
|
||||
def test_branch_reached_candidate_is_demoted(self):
|
||||
host = LOAD + 0x40
|
||||
target = host + 0x14
|
||||
caller = LOAD + 0x80
|
||||
data = image({host: FRAME,
|
||||
host + 4: bne_v0(host + 4, target), host + 8: NOP,
|
||||
host + 12: JR_RA, host + 16: UNFRAME,
|
||||
# The branch target sits right after `jr ra; delay`:
|
||||
# it looks like a frameless function and passes the
|
||||
# image-local proof, and something jals it, so
|
||||
# enrichment derives it.
|
||||
target: ADDIU, target + 4: JR_RA,
|
||||
target + 8: UNFRAME,
|
||||
caller: FRAME, caller + 4: jal(target),
|
||||
caller + 8: NOP, caller + 12: JR_RA,
|
||||
caller + 16: UNFRAME}, 0x100)
|
||||
self.assertIn(target, CO.derive_static_roots(data, LOAD, len(data)))
|
||||
seeds, audit = classify(data, True, function_entry_pcs=[host])
|
||||
self.assertEqual(audit['included_reasons'][target],
|
||||
'DISPATCH_INTERIOR')
|
||||
self.assertNotIn(target, root_seeds(seeds))
|
||||
self.assertEqual(audit['guard_demoted'][target][1], host)
|
||||
|
||||
def test_enrichment_root_in_unreached_body_word_is_dropped(self):
|
||||
# The host branches (b) over a dead block whose first word looks like
|
||||
# a prologue. As a root it would cap the host and turn the branch
|
||||
# into a cross-function exit, although nothing reaches it. Enrichment
|
||||
# evidence is weak: it is dropped, not rooted and not aliased.
|
||||
host = LOAD + 0x20
|
||||
dead, label = host + 0x10, host + 0x30
|
||||
data = image({host: FRAME,
|
||||
host + 4: 0x10000000 | (((label - (host + 8)) >> 2)
|
||||
& 0xFFFF),
|
||||
host + 8: NOP,
|
||||
dead: FRAME, dead + 4: ADDIU, dead + 8: JR_RA,
|
||||
dead + 12: UNFRAME,
|
||||
label: ADDIU, label + 4: JR_RA,
|
||||
label + 8: UNFRAME}, 0x100)
|
||||
self.assertIn(dead, CO.derive_static_roots(data, LOAD, len(data)))
|
||||
seeds, audit = classify(data, True, function_entry_pcs=[host])
|
||||
self.assertNotIn(dead, root_seeds(seeds))
|
||||
self.assertNotIn(dead, audit['included_reasons'])
|
||||
self.assertEqual(audit['excluded_reasons'][dead],
|
||||
'ENRICHMENT_SHADOWED')
|
||||
self.assertEqual(root_seeds(seeds), {host})
|
||||
# An absolute j over it is how tail calls are encoded, so that alone
|
||||
# does not make the candidate part of the host.
|
||||
tail = bytearray(data)
|
||||
struct.pack_into('<I', tail, host + 4 - LOAD,
|
||||
0x08000000 | ((label >> 2) & 0x03FFFFFF))
|
||||
seeds, audit = classify(bytes(tail), True, function_entry_pcs=[host])
|
||||
self.assertIn(dead, root_seeds(seeds))
|
||||
|
||||
def test_delay_slot_candidate_is_never_root_nor_alias(self):
|
||||
host, callee = LOAD + 0x20, LOAD + 0x80
|
||||
slot = host + 8
|
||||
data = image({host: FRAME, host + 4: jal(callee), slot: ADDIU,
|
||||
host + 12: JR_RA, host + 16: UNFRAME,
|
||||
callee: JR_RA, callee + 4: NOP}, 0x100)
|
||||
seeds, audit = classify(data, True, function_entry_pcs=[host],
|
||||
static_discovery_entry_pcs=[slot],
|
||||
dispatch_entry_pcs=[slot])
|
||||
self.assertNotIn(slot, audit['included_reasons'])
|
||||
self.assertEqual(audit['excluded_reasons'][slot], 'DELAY_SLOT')
|
||||
self.assertFalse(any(seed.endswith(f'0x{slot:08X}') for seed in seeds))
|
||||
self.assertIn(slot, audit['delay_slot_rejected'])
|
||||
self.assertFalse(CO.image_local_entry_proven(
|
||||
data, LOAD, len(data), slot, LOAD + len(data)))
|
||||
|
||||
def test_switch_case_roots_absorbed_together(self):
|
||||
# Every case label passes the image-local proof (each case before it
|
||||
# ends in `jr ra`), and the jump table is a >=3 pointer run, so
|
||||
# enrichment derives them all. The table resolves only when every
|
||||
# target is inside the host's walk, so they must be absorbed as a run.
|
||||
host = LOAD
|
||||
table = LOAD + 0x200
|
||||
cases = [LOAD + 0x68, LOAD + 0x78, LOAD + 0x88]
|
||||
words = {host: FRAME,
|
||||
LOAD + 0x04: 0x3C088010, # lui t0,0x8010
|
||||
LOAD + 0x08: 0x25100200, # addiu s0,t0,0x200
|
||||
LOAD + 0x0C: 0x2C620003, # sltiu v0,v1,3
|
||||
LOAD + 0x10: 0x10400000 | ((0x98 - 0x14) >> 2), # beq v0,zero
|
||||
LOAD + 0x14: NOP,
|
||||
LOAD + 0x18: 0x00031080, # sll v0,v1,2
|
||||
LOAD + 0x1C: 0x00501021, # addu v0,v0,s0
|
||||
LOAD + 0x20: 0x8C420000, # lw v0,0(v0)
|
||||
LOAD + 0x24: NOP,
|
||||
LOAD + 0x28: 0x00400008, # jr v0
|
||||
LOAD + 0x2C: NOP,
|
||||
LOAD + 0x98: JR_RA, LOAD + 0x9C: UNFRAME}
|
||||
for index, case in enumerate(cases):
|
||||
words[case] = ADDIU
|
||||
words[case + 4] = JR_RA
|
||||
words[case + 8] = UNFRAME
|
||||
words[table + index * 4] = case
|
||||
data = image(words, 0x300)
|
||||
derived = CO.derive_static_roots(data, LOAD, len(data))
|
||||
self.assertTrue(set(cases[1:]) <= derived)
|
||||
# The guard alone (the case labels supplied as explicit roots, as a
|
||||
# #386-enriched capture would) absorbs the whole run.
|
||||
seeds, audit = classify(data, False, function_entry_pcs=[host],
|
||||
static_discovery_entry_pcs=cases[1:])
|
||||
for case in cases[1:]:
|
||||
self.assertEqual(audit['included_reasons'][case],
|
||||
'DISPATCH_INTERIOR', hex(case))
|
||||
self.assertNotIn(case, root_seeds(seeds))
|
||||
# Default enrichment: the guarded prepass resolves the table, so the
|
||||
# case labels are never offered as roots at all.
|
||||
seeds, audit = classify(data, True, function_entry_pcs=[host])
|
||||
for case in cases:
|
||||
self.assertIn(audit['included_reasons'].get(case),
|
||||
(None, 'DISPATCH_INTERIOR'), hex(case))
|
||||
self.assertNotIn(case, root_seeds(seeds))
|
||||
self.assertEqual(root_seeds(seeds), {host})
|
||||
|
||||
# (c) ------------------------------------------------------------------
|
||||
def test_jal_proven_function_start_stays_a_root(self):
|
||||
caller, framed, frameless = LOAD, LOAD + 0x40, LOAD + 0x80
|
||||
data = image({caller: FRAME, caller + 4: jal(framed),
|
||||
caller + 8: NOP, caller + 12: jal(frameless),
|
||||
caller + 16: NOP, caller + 20: JR_RA,
|
||||
caller + 24: UNFRAME,
|
||||
framed: FRAME, framed + 4: ADDIU, framed + 8: JR_RA,
|
||||
framed + 12: UNFRAME,
|
||||
frameless - 8: JR_RA, frameless - 4: NOP,
|
||||
frameless: ADDIU, frameless + 4: JR_RA,
|
||||
frameless + 8: NOP}, 0x100)
|
||||
derived = CO.derive_static_roots(data, LOAD, len(data))
|
||||
self.assertTrue({caller, framed, frameless} <= derived)
|
||||
seeds, audit = classify(data, True)
|
||||
roots = root_seeds(seeds)
|
||||
self.assertTrue({caller, framed, frameless} <= roots)
|
||||
self.assertEqual(audit['guard_demoted'], {})
|
||||
|
||||
# (d) ------------------------------------------------------------------
|
||||
def _cross_image(self, target_is_function):
|
||||
engine, host = LOAD, LOAD + 0x100
|
||||
target = LOAD + 0x110
|
||||
words = {engine: FRAME, engine + 4: jal(target), engine + 8: NOP,
|
||||
engine + 12: JR_RA, engine + 16: UNFRAME}
|
||||
if target_is_function:
|
||||
# Sibling image: the called address is a real function here.
|
||||
words.update({host: ADDIU, host + 4: JR_RA, host + 8: NOP,
|
||||
target: FRAME, target + 4: ADDIU,
|
||||
target + 8: JR_RA, target + 12: UNFRAME})
|
||||
else:
|
||||
# This image: the same address is mid-way through a host
|
||||
# reached by pure fallthrough (the X00 shape).
|
||||
words.update({host: FRAME, host + 4: ADDIU, host + 8: ADDIU,
|
||||
host + 12: ADDIU, target: SUBU,
|
||||
target + 4: ADDIU, target + 8: JR_RA,
|
||||
target + 12: UNFRAME})
|
||||
return image(words, 0x200), engine, host, target
|
||||
|
||||
def test_shared_jal_into_mid_body_is_not_rooted(self):
|
||||
data, engine, host, target = self._cross_image(False)
|
||||
weak = set()
|
||||
strong = CO.derive_static_roots(data, LOAD, len(data), weak_out=weak)
|
||||
self.assertNotIn(target, strong)
|
||||
self.assertIn(target, weak) # the CFG probe alone accepts it
|
||||
self.assertTrue({engine, host} <= strong)
|
||||
derived, stats = CO.derive_enrichment_roots(data, LOAD, len(data))
|
||||
self.assertNotIn(target, derived) # host's prologue reaches it
|
||||
self.assertEqual(stats['weak_dropped'], 1)
|
||||
# Enrichment on (default): the rooted engine walk still derives the
|
||||
# jal target, and the guard absorbs it into its host.
|
||||
seeds, audit = classify(data, True)
|
||||
self.assertNotIn(target, root_seeds(seeds))
|
||||
self.assertEqual(audit['included_reasons'][target],
|
||||
'DISPATCH_INTERIOR')
|
||||
self.assertIn(host, root_seeds(seeds))
|
||||
# A #386-enriched capture that already names it is demoted too.
|
||||
seeds, audit = classify(data, True,
|
||||
static_discovery_entry_pcs=[target])
|
||||
self.assertNotIn(target, root_seeds(seeds))
|
||||
self.assertEqual(audit['guard_demoted'][target],
|
||||
('STATIC_DISCOVERY_ROOT', host))
|
||||
|
||||
def test_weak_target_nothing_reaches_is_rooted(self):
|
||||
# A frameless function right after another function's tail jump,
|
||||
# called by jal: no `jr $ra` before it, so no boundary proof, but no
|
||||
# possible function start reaches it either.
|
||||
caller, tail, leaf, far = LOAD, LOAD + 0x40, LOAD + 0x4C, LOAD + 0x80
|
||||
data = image({caller: FRAME, caller + 4: jal(leaf), caller + 8: NOP,
|
||||
caller + 12: JR_RA, caller + 16: UNFRAME,
|
||||
tail: ADDIU,
|
||||
tail + 4: 0x08000000 | ((far >> 2) & 0x03FFFFFF),
|
||||
tail + 8: NOP,
|
||||
leaf: ADDIU, leaf + 4: JR_RA, leaf + 8: NOP,
|
||||
far: JR_RA, far + 4: NOP}, 0x100)
|
||||
weak = set()
|
||||
CO.derive_static_roots(data, LOAD, len(data), weak_out=weak)
|
||||
self.assertIn(leaf, weak)
|
||||
derived, _stats = CO.derive_enrichment_roots(data, LOAD, len(data))
|
||||
self.assertIn(leaf, derived)
|
||||
seeds, _audit = classify(data, True)
|
||||
self.assertIn(leaf, root_seeds(seeds))
|
||||
|
||||
def test_shared_jal_into_real_function_in_sibling_image_is_rooted(self):
|
||||
data, _engine, _host, target = self._cross_image(True)
|
||||
self.assertIn(target, CO.derive_static_roots(data, LOAD, len(data)))
|
||||
seeds, _audit = classify(data, True)
|
||||
self.assertIn(target, root_seeds(seeds))
|
||||
|
||||
def test_cross_producer_call_evidence(self):
|
||||
# Producer A calls into producer B. A cross-producer call is only
|
||||
# ever weaker evidence: never WEAK evidence, STRONG only when B's own
|
||||
# bytes bound the target and the capture is not strict.
|
||||
a_fn, bounded, frameless = LOAD, LOAD + 0x110, LOAD + 0x130
|
||||
data = image({a_fn: FRAME, a_fn + 4: jal(bounded), a_fn + 8: NOP,
|
||||
a_fn + 12: jal(frameless), a_fn + 16: NOP,
|
||||
a_fn + 20: JR_RA, a_fn + 24: UNFRAME,
|
||||
bounded - 8: JR_RA, bounded - 4: NOP,
|
||||
bounded: ADDIU, bounded + 4: JR_RA, bounded + 8: NOP,
|
||||
frameless - 4: ADDIU,
|
||||
frameless: ADDIU, frameless + 4: JR_RA,
|
||||
frameless + 8: NOP}, 0x200)
|
||||
ranges = [(LOAD, LOAD + 0x100), (LOAD + 0x100, LOAD + 0x200)]
|
||||
for strict in (True, False):
|
||||
roots, _ = CO.derive_enrichment_roots(
|
||||
data, LOAD, len(data), ranges, strict_producer_ranges=strict)
|
||||
self.assertNotIn(frameless, roots)
|
||||
self.assertEqual(bounded in roots, not strict)
|
||||
|
||||
# late prologues (beads-eio.3.191) -------------------------------------
|
||||
# A stack adjust is a boundary only when it is at one. Compilers schedule
|
||||
# loads in front of it, so rooting every `addiu sp,sp,-N` capped 359 Ace
|
||||
# Combat 3 functions two to five words in, 175 of them with the real
|
||||
# start executed. The prologue must root the function's true start.
|
||||
AC3 = { # Ace Combat 3 USA image 0 (load 0x800C8000), 0x800D1250..
|
||||
0x00: JR_RA, # jr ra
|
||||
0x04: 0x27BD0018, # addiu sp,sp,24 (delay slot)
|
||||
0x08: 0x3C02800F, # lui v0,0x800F <- real start
|
||||
0x0C: 0x8C435D4C, # lw v1,0x5D4C(v0)
|
||||
0x10: 0x27BDFFE8, # addiu sp,sp,-24 <- was rooted
|
||||
0x14: 0xAFBF0014, # sw ra,20(sp)
|
||||
0x18: 0xAFB00010, # sw s0,16(sp)
|
||||
0x1C: 0x8C620000, # lw v0,0(v1)
|
||||
0x20: NOP,
|
||||
0x24: 0x8FBF0014, # lw ra,20(sp)
|
||||
0x28: 0x8FB00010, # lw s0,16(sp)
|
||||
0x2C: JR_RA,
|
||||
0x30: 0x27BD0018, # addiu sp,sp,24
|
||||
}
|
||||
|
||||
def ac3_image(self, base=LOAD + 0x40, extra=None):
|
||||
words = {LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME}
|
||||
words.update({base + off: word for off, word in self.AC3.items()})
|
||||
words.update(extra or {})
|
||||
return image(words, 0x100), base + 0x08, base + 0x10
|
||||
|
||||
def test_late_prologue_roots_its_true_start(self):
|
||||
data, start, prologue = self.ac3_image()
|
||||
self.assertEqual(CO.prologue_function_start(data, LOAD, prologue),
|
||||
start)
|
||||
self.assertFalse(CO.image_local_entry_proven(
|
||||
data, LOAD, len(data), prologue, LOAD + len(data)))
|
||||
sources = {}
|
||||
strong = CO.derive_static_roots(data, LOAD, len(data),
|
||||
sources=sources)
|
||||
self.assertIn(start, strong)
|
||||
self.assertNotIn(prologue, strong)
|
||||
self.assertIn('prologue_start', sources[start])
|
||||
_roots, stats = CO.derive_enrichment_roots(data, LOAD, len(data))
|
||||
self.assertEqual(stats['late_prologue_starts'], 1)
|
||||
seeds, audit = classify(data, True)
|
||||
self.assertIn(start, root_seeds(seeds))
|
||||
self.assertNotIn(prologue, root_seeds(seeds))
|
||||
# Captured entry evidence at the stack adjust is absorbed by the true
|
||||
# start: an alias, never a cap.
|
||||
seeds, audit = classify(data, True, dispatch_entry_pcs=[prologue])
|
||||
self.assertIn(start, root_seeds(seeds))
|
||||
self.assertEqual(audit['included_reasons'][prologue],
|
||||
'DISPATCH_INTERIOR')
|
||||
self.assertEqual(audit['guard_demoted'][prologue][1], start)
|
||||
|
||||
def test_prologue_at_a_boundary_is_its_own_start(self):
|
||||
# image start, after jr ra + padding, after j, after a break trap
|
||||
far = LOAD + 0xF0
|
||||
data = image({LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME,
|
||||
LOAD + 0x20: FRAME, LOAD + 0x24: JR_RA,
|
||||
LOAD + 0x28: UNFRAME,
|
||||
LOAD + 0x40: 0x08000000 | ((far >> 2) & 0x03FFFFFF),
|
||||
LOAD + 0x44: NOP, LOAD + 0x48: FRAME,
|
||||
LOAD + 0x4C: JR_RA, LOAD + 0x50: UNFRAME,
|
||||
LOAD + 0x60: 0x0000000D, # break
|
||||
LOAD + 0x64: 0x3C02800F, # lui v0 <- start
|
||||
LOAD + 0x68: FRAME, LOAD + 0x6C: JR_RA,
|
||||
LOAD + 0x70: UNFRAME,
|
||||
far: JR_RA, far + 4: NOP}, 0x100)
|
||||
for addr in (LOAD, LOAD + 0x20, LOAD + 0x48):
|
||||
self.assertEqual(CO.prologue_function_start(data, LOAD, addr),
|
||||
addr, hex(addr))
|
||||
self.assertTrue(CO.image_local_entry_proven(
|
||||
data, LOAD, len(data), addr, LOAD + len(data)), hex(addr))
|
||||
self.assertEqual(CO.prologue_function_start(data, LOAD, LOAD + 0x68),
|
||||
LOAD + 0x64)
|
||||
self.assertTrue({LOAD, LOAD + 0x20, LOAD + 0x48, LOAD + 0x64} <=
|
||||
CO.derive_static_roots(data, LOAD, len(data)))
|
||||
|
||||
def test_preamble_with_branch_or_call_proves_no_start(self):
|
||||
# `jr ra; nop; lw v0,0(a0); beqz v0,out; nop; addiu sp` -- the stack
|
||||
# adjust is reached by fallthrough from code this walk does not
|
||||
# bound. It is weak evidence, and the start after `jr ra` (a possible
|
||||
# function start) reaches it, so it is dropped: never a root.
|
||||
base = LOAD + 0x40
|
||||
out = base + 0x30
|
||||
for shape, word in (('branch', bne_v0(base + 0x0C, out)),
|
||||
('call', jal(LOAD))):
|
||||
data = image({LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME,
|
||||
base: JR_RA, base + 4: NOP,
|
||||
base + 8: 0x8C820000, # lw v0,0(a0)
|
||||
base + 0x0C: word, base + 0x10: NOP,
|
||||
base + 0x14: FRAME, base + 0x18: ADDIU,
|
||||
base + 0x1C: JR_RA, base + 0x20: UNFRAME,
|
||||
out: JR_RA, out + 4: NOP}, 0x100)
|
||||
prologue = base + 0x14
|
||||
self.assertIsNone(CO.prologue_function_start(data, LOAD,
|
||||
prologue), shape)
|
||||
self.assertFalse(CO.image_local_entry_proven(
|
||||
data, LOAD, len(data), prologue, LOAD + len(data)), shape)
|
||||
weak = set()
|
||||
strong = CO.derive_static_roots(data, LOAD, len(data),
|
||||
weak_out=weak)
|
||||
self.assertNotIn(prologue, strong, shape)
|
||||
self.assertNotIn(base + 8, strong, shape)
|
||||
roots, _stats = CO.derive_enrichment_roots(data, LOAD, len(data))
|
||||
self.assertNotIn(prologue, roots, shape)
|
||||
seeds, _audit = classify(data, True)
|
||||
self.assertNotIn(prologue, root_seeds(seeds), shape)
|
||||
|
||||
def test_branch_from_below_into_preamble_proves_no_start(self):
|
||||
# The function before returns early (`jr ra`) and branches past it to
|
||||
# a label inside what looks like the next function's preamble: the
|
||||
# words after that `jr ra` are still its body. Also the same with the
|
||||
# label exactly at the would-be start.
|
||||
host = LOAD + 0x20
|
||||
for label_off in (0x0C, 0x08):
|
||||
label = host + 0x10 + label_off
|
||||
data = image({host: FRAME,
|
||||
host + 4: bne_v0(host + 4, label),
|
||||
host + 8: NOP,
|
||||
host + 0x0C: JR_RA, host + 0x10: UNFRAME,
|
||||
host + 0x18: 0x3C02800F, # lui v0,0x800F
|
||||
host + 0x1C: 0x8C435D4C, # lw v1,..(v0)
|
||||
host + 0x20: FRAME, host + 0x24: ADDIU,
|
||||
host + 0x28: JR_RA, host + 0x2C: UNFRAME}, 0x100)
|
||||
prologue = host + 0x20
|
||||
self.assertIsNone(CO.prologue_function_start(data, LOAD,
|
||||
prologue),
|
||||
hex(label))
|
||||
strong = CO.derive_static_roots(data, LOAD, len(data))
|
||||
self.assertNotIn(host + 0x18, strong, hex(label))
|
||||
self.assertNotIn(prologue, strong, hex(label))
|
||||
seeds, _audit = classify(data, True, function_entry_pcs=[host])
|
||||
self.assertEqual(root_seeds(seeds), {host}, hex(label))
|
||||
|
||||
def test_prologue_after_data(self):
|
||||
# A pointer table (two pointer-shaped words) or an invalid word ends
|
||||
# the walk: the prologue right after it is its own start; a preamble
|
||||
# after data must be global-data setup, since data decodes as
|
||||
# instructions too.
|
||||
table = LOAD + 0x40
|
||||
words = {LOAD: FRAME, LOAD + 4: JR_RA, LOAD + 8: UNFRAME,
|
||||
table: LOAD + 0x80, table + 4: LOAD + 0x90,
|
||||
table + 8: FRAME, table + 12: JR_RA, table + 16: UNFRAME}
|
||||
data = image(words, 0x100)
|
||||
self.assertEqual(CO.prologue_function_start(data, LOAD, table + 8),
|
||||
table + 8)
|
||||
setup = dict(words)
|
||||
setup.update({table + 8: 0x3C02800F, table + 12: 0x8C435D4C,
|
||||
table + 16: FRAME, table + 20: JR_RA,
|
||||
table + 24: UNFRAME})
|
||||
data = image(setup, 0x100)
|
||||
self.assertEqual(CO.prologue_function_start(data, LOAD, table + 16),
|
||||
table + 8)
|
||||
other = dict(setup)
|
||||
other[table + 12] = 0xA6020020 # sh v0,32(s0)
|
||||
data = image(other, 0x100)
|
||||
self.assertIsNone(CO.prologue_function_start(data, LOAD, table + 16))
|
||||
invalid = {LOAD + 0x40: 0x0000003D, LOAD + 0x44: FRAME,
|
||||
LOAD + 0x48: JR_RA, LOAD + 0x4C: UNFRAME,
|
||||
# REGIMM with no such rt: data shaped like a branch
|
||||
LOAD + 0x60: 0x06040200, LOAD + 0x64: FRAME,
|
||||
LOAD + 0x68: JR_RA, LOAD + 0x6C: UNFRAME}
|
||||
data = image(invalid, 0x100)
|
||||
for addr in (LOAD + 0x44, LOAD + 0x64):
|
||||
self.assertEqual(CO.prologue_function_start(data, LOAD, addr),
|
||||
addr, hex(addr))
|
||||
|
||||
def test_single_pointer_shaped_word_is_code(self):
|
||||
# `lui v0,0x8010; lb v0,0x5CEC(v0)` is 0x80425CEC: one aligned RAM
|
||||
# address in a preamble is an instruction, not a table.
|
||||
base = LOAD + 0x40
|
||||
data = image({base: JR_RA, base + 4: UNFRAME,
|
||||
base + 8: 0x3C028010, base + 12: 0x80425CEC,
|
||||
base + 16: FRAME, base + 20: JR_RA,
|
||||
base + 24: UNFRAME}, 0x100)
|
||||
self.assertEqual(CO.prologue_function_start(data, LOAD, base + 16),
|
||||
base + 8)
|
||||
|
||||
def test_late_prologue_start_respects_producer_start(self):
|
||||
# The walk never leaves the prologue's producer.
|
||||
data, start, prologue = self.ac3_image(base=LOAD + 0xC0)
|
||||
ranges = [(LOAD, LOAD + 0xC8), (LOAD + 0xC8, LOAD + 0x100)]
|
||||
self.assertEqual(CO.prologue_function_start(
|
||||
data, LOAD, prologue, LOAD + 0xC8), start)
|
||||
self.assertEqual(CO.prologue_function_start(
|
||||
data, LOAD, prologue, LOAD + 0xCC), LOAD + 0xCC)
|
||||
self.assertIn(start, CO.derive_static_roots(data, LOAD, len(data),
|
||||
ranges))
|
||||
|
||||
# enrichment policy ----------------------------------------------------
|
||||
def test_enrichment_is_default_and_opt_out_is_diagnostic(self):
|
||||
data, engine, host, _target = self._cross_image(False)
|
||||
old = os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None)
|
||||
try:
|
||||
self.assertTrue(CO.root_enrichment_default())
|
||||
_seeds, audit = classify(data, None)
|
||||
self.assertTrue(audit['root_enrichment'])
|
||||
self.assertTrue({engine, host} <= audit['derived_static_roots'])
|
||||
os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = '0'
|
||||
self.assertFalse(CO.root_enrichment_default())
|
||||
_seeds, audit = classify(data, None)
|
||||
self.assertFalse(audit['root_enrichment'])
|
||||
self.assertEqual(audit['derived_static_roots'], set())
|
||||
finally:
|
||||
os.environ.pop('PSX_OVERLAY_ROOT_ENRICHMENT', None)
|
||||
if old is not None:
|
||||
os.environ['PSX_OVERLAY_ROOT_ENRICHMENT'] = old
|
||||
# A conservative retry recipe is never enriched again.
|
||||
cap = {'schema': 'psxrecomp overlay capture v2',
|
||||
CO.ROOT_ENRICHMENT_OFF_KEY: True}
|
||||
_seeds, audit = CO.classify_overlay_seeds(
|
||||
cap, data, LOAD, len(data), 0, {}, root_enrichment=True)
|
||||
self.assertEqual(audit['derived_static_roots'], set())
|
||||
retry = CO.conservative_retry_capture(
|
||||
{'schema': 'x'}, {'derived_static_roots': {engine}})
|
||||
self.assertTrue(retry[CO.ROOT_ENRICHMENT_OFF_KEY])
|
||||
self.assertIsNone(CO.conservative_retry_capture(
|
||||
retry, {'derived_static_roots': {engine}}))
|
||||
|
||||
|
||||
# (e) ----------------------------------------------------------------------
|
||||
class CacheKeyTests(unittest.TestCase):
|
||||
def _cmake_list(self):
|
||||
self.assertIsNotNone(CMAKE, 'cmake not found: pass --cmake or put '
|
||||
'cmake on PATH')
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
script = Path(tmp) / 'list.cmake'
|
||||
script.write_text(
|
||||
f'set(PSXRECOMP_CODEGEN_HASH_ROOT "{ROOT.as_posix()}")\n'
|
||||
f'include("{(ROOT / "runtime/codegen_hash_sources.cmake").as_posix()}")\n'
|
||||
'message(STATUS "SRCS=${PSXRECOMP_CODEGEN_HASH_SRCS}")\n',
|
||||
encoding='utf-8')
|
||||
proc = subprocess.run([CMAKE, '-P', str(script)],
|
||||
capture_output=True, text=True)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
match = re.search(r'SRCS=(.*)', proc.stdout + proc.stderr)
|
||||
self.assertIsNotNone(match)
|
||||
return match.group(1).strip().split(';')
|
||||
|
||||
def _hash_header(self, sources, out_dir):
|
||||
header = Path(out_dir) / 'overlay_codegen_hash.h'
|
||||
proc = subprocess.run(
|
||||
[CMAKE, f'-DOUT={header.as_posix()}',
|
||||
'-DSRCS=' + ';'.join(sources),
|
||||
'-P', str(ROOT / 'runtime' / 'hash_codegen.cmake')],
|
||||
capture_output=True, text=True)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
shutil.copy2(ROOT / 'runtime' / 'include' / 'overlay_api.h',
|
||||
Path(out_dir) / 'overlay_api.h')
|
||||
return CO.codegen_hash(str(out_dir))
|
||||
|
||||
def test_root_policy_is_part_of_the_cache_namespace(self):
|
||||
sources = self._cmake_list()
|
||||
classifier = (ROOT / 'tools' / 'compile_overlays.py').as_posix()
|
||||
self.assertIn(classifier, sources)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
tmp = Path(tmp)
|
||||
changed = tmp / 'compile_overlays.py'
|
||||
changed.write_text(
|
||||
Path(classifier).read_text(encoding='utf-8') +
|
||||
'\n# a root-policy change\n', encoding='utf-8')
|
||||
(tmp / 'a').mkdir()
|
||||
(tmp / 'b').mkdir()
|
||||
before = self._hash_header(sources, tmp / 'a')
|
||||
after = self._hash_header(
|
||||
[changed.as_posix() if s == classifier else s
|
||||
for s in sources], tmp / 'b')
|
||||
self.assertNotEqual(before, after)
|
||||
saved = CO.overlay_config_hash
|
||||
CO.overlay_config_hash = lambda recompiler, toml: 0x1234ABCD
|
||||
try:
|
||||
tag_a = CO.cache_tag(str(tmp / 'a'), 'r', 'g', 0)
|
||||
tag_b = CO.cache_tag(str(tmp / 'b'), 'r', 'g', 0)
|
||||
finally:
|
||||
CO.overlay_config_hash = saved
|
||||
self.assertNotEqual(tag_a, tag_b)
|
||||
|
||||
|
||||
# recompiler enforcement ---------------------------------------------------
|
||||
class RecompilerGuardTests(unittest.TestCase):
|
||||
def test_recompiler_absorbs_explicit_fallthrough_root(self):
|
||||
if not RECOMPILER:
|
||||
self.skipTest('pass --recompiler (ctest always does)')
|
||||
host, mid = LOAD + 0x20, LOAD + 0x30
|
||||
data = image({host: FRAME, host + 4: ADDIU, host + 8: ADDIU,
|
||||
host + 12: ADDIU, mid: SUBU, mid + 4: ADDIU,
|
||||
mid + 8: JR_RA, mid + 12: UNFRAME}, 0x1000)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
psx = Path(tmp) / 'split.psx'
|
||||
psx.write_bytes(CO.make_psxexe(LOAD, host, data, guard_bytes=0))
|
||||
seeds = Path(tmp) / 'seeds.txt'
|
||||
# Seeds the classifier would never write: both as call roots.
|
||||
seeds.write_text(f'call_root 0x{host:08X}\ncall_root 0x{mid:08X}\n')
|
||||
out = Path(tmp) / 'out'
|
||||
proc = subprocess.run(
|
||||
[RECOMPILER, str(psx), '--seeds', str(seeds),
|
||||
'--out-dir', str(out), '--overlay'],
|
||||
capture_output=True, text=True, cwd=str(ROOT))
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr or proc.stdout)
|
||||
self.assertEqual(
|
||||
CO.report_recompiler_guard_absorptions(proc.stdout), 1,
|
||||
proc.stdout)
|
||||
ranges = next(out.glob('*_full.ranges')).read_text()
|
||||
# Both PCs stay callable; the mid-body one only as an alias whose
|
||||
# identity covers the host range (it starts at the host).
|
||||
self.assertIn(f'F {host:08X}', ranges)
|
||||
self.assertIn(f'F {mid:08X}', ranges)
|
||||
mid_rows = ranges.split(f'F {mid:08X}', 1)[1].splitlines()[1:]
|
||||
self.assertTrue(any(row.startswith(f'R {host:08X}')
|
||||
for row in mid_rows[:2]), ranges)
|
||||
|
||||
|
||||
# supplemental root fragments carry the aliases of their hosts --------------
|
||||
#
|
||||
# Measured on Tomba 2 (bead beads-eio.3.177): the 0x80038000 capture has more
|
||||
# roots than the region shard admits, so ~960 roots are compiled in strong-root
|
||||
# supplement fragments. Those fragments were built from `dispatch_root` seeds
|
||||
# only, so every candidate the guard absorbed into such a host (0x8003A678 in
|
||||
# host 0x8003A614, captured dispatch entries) was served by neither shard: 127
|
||||
# captured dispatch entries lost the native identity the unguarded build had.
|
||||
class SupplementFragmentAliasTests(unittest.TestCase):
|
||||
HOST, MID = LOAD + 0x20, LOAD + 0x30
|
||||
|
||||
def fallthrough_image(self, size=0x100):
|
||||
host, mid = self.HOST, self.MID
|
||||
return image({host: FRAME, host + 4: ADDIU, host + 8: ADDIU,
|
||||
host + 12: ADDIU, mid: SUBU, mid + 4: ADDIU,
|
||||
mid + 8: JR_RA, mid + 12: UNFRAME}, size)
|
||||
|
||||
def job(self, data):
|
||||
cap = {'schema': 'psxrecomp overlay capture v2',
|
||||
'function_entry_pcs': [f'0x{self.HOST:08X}'],
|
||||
'dispatch_entry_pcs': [f'0x{self.HOST:08X}'],
|
||||
'static_discovery_entry_pcs': [f'0x{self.MID:08X}']}
|
||||
_seeds, audit = CO.classify_overlay_seeds(
|
||||
cap, data, LOAD, len(data), 0, {}, root_enrichment=True)
|
||||
return audit, CO.make_interior_fragment_job(
|
||||
LOAD & 0x1FFFFFFF, LOAD, len(data), data, audit, set(), cap)
|
||||
|
||||
def test_classifier_records_final_host_of_every_alias(self):
|
||||
audit, _job = self.job(self.fallthrough_image())
|
||||
self.assertEqual(audit['included_reasons'][self.MID],
|
||||
'DISPATCH_INTERIOR')
|
||||
self.assertEqual(audit['interior_hosts'][self.MID], {self.HOST})
|
||||
self.assertEqual(audit['guard_demoted'][self.MID],
|
||||
('STATIC_DISCOVERY_ROOT', self.HOST))
|
||||
|
||||
def test_root_fragment_selects_its_hosts_aliases(self):
|
||||
_audit, job = self.job(self.fallthrough_image())
|
||||
self.assertIn(self.HOST, job['static_exact_demands'])
|
||||
self.assertEqual(CO.fragment_interior_aliases(job, [self.HOST]),
|
||||
[self.MID])
|
||||
self.assertEqual(CO.fragment_interior_aliases(job, [LOAD + 0x80]), [])
|
||||
# merged byte-variant recipes union the hosts
|
||||
merged = CO.merge_fragment_jobs_by_recipe([job, job])
|
||||
self.assertEqual(merged[0]['interior_hosts'][self.MID], {self.HOST})
|
||||
|
||||
def test_fragment_seed_file_carries_aliases_as_interior(self):
|
||||
data = self.fallthrough_image()
|
||||
seen = {}
|
||||
real_run = CO.subprocess.run
|
||||
|
||||
def fake_run(cmd, **kw):
|
||||
seen['seeds'] = Path(cmd[cmd.index('--seeds') + 1]).read_text()
|
||||
return subprocess.CompletedProcess(cmd, 1, '', 'stop')
|
||||
args = argparse.Namespace(recompiler='recomp.exe', game_toml=__file__,
|
||||
project_root=None, runtime_include='.')
|
||||
CO.subprocess.run = fake_run
|
||||
try:
|
||||
CO.compile_fragment_batch(
|
||||
[self.HOST], data, LOAD, len(data), LOAD & 0x1FFFFFFF, '.',
|
||||
args, {}, {}, guard_bytes=0, interior_aliases=[self.MID])
|
||||
finally:
|
||||
CO.subprocess.run = real_run
|
||||
lines = seen['seeds'].splitlines()
|
||||
self.assertIn(f'dispatch_root 0x{self.HOST:08X}', lines)
|
||||
self.assertIn(f'interior 0x{self.MID:08X}', lines)
|
||||
|
||||
def test_alias_of_interpreted_host_gets_its_root_back(self):
|
||||
# Tomba 2: host 0x800BDF10 (a walk through data, 48 unsupported
|
||||
# instructions) fails its fragment; the 8-byte stubs the guard had
|
||||
# absorbed into it were native roots in the unguarded build.
|
||||
_audit, job = self.job(self.fallthrough_image())
|
||||
self.assertEqual(job['guard_demoted_aliases'], {self.MID})
|
||||
self.assertEqual(CO.orphaned_guard_aliases(job, set()), [self.MID])
|
||||
self.assertEqual(CO.orphaned_guard_aliases(
|
||||
job, {self.HOST & 0x1FFFFFFF}), [])
|
||||
self.assertEqual(CO.orphaned_guard_aliases(
|
||||
job, {self.MID & 0x1FFFFFFF}), [])
|
||||
merged = CO.merge_fragment_jobs_by_recipe([job, dict(job)])
|
||||
self.assertEqual(merged[0]['guard_demoted_aliases'], {self.MID})
|
||||
|
||||
def test_alias_failure_never_costs_the_host(self):
|
||||
calls = []
|
||||
|
||||
def build(aliases):
|
||||
calls.append(list(aliases))
|
||||
return ((None, 'generated-c-audit: 1 unknown_bad') if aliases
|
||||
else ([('ok',)], 'built'))
|
||||
ids, status = CO.compile_root_fragment_with_aliases(
|
||||
[self.HOST], [self.MID], build)
|
||||
self.assertEqual((ids, status, calls), ([('ok',)], 'built',
|
||||
[[self.MID], []]))
|
||||
# a multi-root batch is left to the caller's bisection
|
||||
calls.clear()
|
||||
ids, _status = CO.compile_root_fragment_with_aliases(
|
||||
[self.HOST, LOAD + 0x80], [self.MID], build)
|
||||
self.assertIsNone(ids)
|
||||
self.assertEqual(calls, [[self.MID]])
|
||||
# capacity verdicts are never retried
|
||||
calls.clear()
|
||||
CO.compile_root_fragment_with_aliases(
|
||||
[self.HOST], [self.MID],
|
||||
lambda a: calls.append(a) or (None, 'candidate-capacity: full; x'))
|
||||
self.assertEqual(len(calls), 1)
|
||||
|
||||
def test_recompiler_emits_alias_in_root_fragment(self):
|
||||
if not RECOMPILER:
|
||||
self.skipTest('pass --recompiler (ctest always does)')
|
||||
data = self.fallthrough_image(0x1000)
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
psx = Path(tmp) / 'frag.psx'
|
||||
psx.write_bytes(CO.make_psxexe(LOAD, self.HOST, data,
|
||||
guard_bytes=0))
|
||||
seeds = Path(tmp) / 'seeds.txt'
|
||||
seeds.write_text(f'dispatch_root 0x{self.HOST:08X}\n'
|
||||
f'interior 0x{self.MID:08X}\n')
|
||||
out = Path(tmp) / 'out'
|
||||
proc = subprocess.run(
|
||||
[RECOMPILER, str(psx), '--seeds', str(seeds),
|
||||
'--out-dir', str(out), '--overlay'],
|
||||
capture_output=True, text=True, cwd=str(ROOT))
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr or proc.stdout)
|
||||
ranges = next(out.glob('*_full.ranges')).read_text()
|
||||
mid_rows = ranges.split(f'F {self.MID:08X}', 1)
|
||||
self.assertEqual(len(mid_rows), 2, ranges)
|
||||
self.assertTrue(mid_rows[1].splitlines()[1].startswith(
|
||||
f'R {self.HOST:08X}'), ranges)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--cmake', default=shutil.which('cmake'))
|
||||
ap.add_argument('--recompiler')
|
||||
args, rest = ap.parse_known_args()
|
||||
if not args.cmake:
|
||||
sys.exit('cmake not found: pass --cmake')
|
||||
CMAKE = args.cmake
|
||||
RECOMPILER = args.recompiler
|
||||
unittest.main(argv=[sys.argv[0]] + rest)
|
||||
Reference in New Issue
Block a user