mirror/psxrecomp
1
0
Fork 0
mirror of https://github.com/RetroPortingToolKit/psxrecomp synced 2026-10-25 18:05:12 -04:00
Code Issues Packages Projects Releases Wiki Activity

runtime: run each overlay shard only for its own segment's PCs

Browse Source 
docs/SEGMENT_AWARE_CODE.md §5.7 (decision 4). #417 made every KUSEG and
KSEG1 overlay dispatch fall to the interpreter, because every shard was
compiled for KSEG0. The loader now keys shards by segment, so a shard
compiled for KUSEG or KSEG1 runs for PCs of that segment.

Cache layout. KSEG0 shards keep their directory and {phys}_{crc} names, so
no existing cache is invalidated. KUSEG and KSEG1 shards live in the cache
tag's seg-kuseg/ and seg-kseg1/ subdirectories, under the same names; both
tiers scan them, and a shard's segment is its directory's
(cache_seg_from_path).

Manifest. A KUSEG or KSEG1 manifest starts with `S <segment>` (00000000 or
A0000000) before its first F. F entries are full VAs in that segment; the
parser no longer forces them to KSEG0. R records stay KSEG0-spelled byte
extents. Without S a manifest is KSEG0, as before. Refused, when indexing,
when loading and through live publication alike:
- a manifest whose segment differs from its directory's;
- an F entry outside its segment;
- S after an F, S twice, or S that is not a RAM segment base.

Dispatch. Each candidate records its segment. The physical indexes stay (the
bytes are one identity), and every lookup matches the PC's segment:
- the entry chain (idx_head_seg);
- CPS continuation ownership;
- the lazy manifest index and try_load_region;
- the interpreter's native-call contract.

The lazy-miss and range-owner memos are keyed by the full PC. A PC in a
segment that maps no RAM never runs a shard. Exports are resolved by their
full VA.

Counters. segment_alias_interp (from #417) now counts KUSEG/KSEG1 dispatches
the loader leaves to the interpreter because no shard of their segment is
cached or valid yet. The new segment_native counts those a shard of their
own segment runs. Both are in the TCP overlay_loader_status reply. A warm
cache takes the first to 0.

Full-VA compares:
- the overlay idle note's return test compares the resume PC with $ra
  exactly, and counts it as internal only for a running shard of its segment;
- the shadow diff's two "returned to the caller" tests compare the exact PC;
- the diff mode's kernel-window exclusion tests the physical address, so
  KSEG0 and KUSEG kernel candidates are treated alike. Before, it tested the
  full PC, so only a KUSEG PC ever matched, and #417's gate never let one reach
  it. This is opt-in (PSX_OVERLAY_DIFF).

Tests (loader harness, overlay_segment_gate):
- a KUSEG shard runs natively for its KUSEG PCs, entry and continuation,
  while KSEG0/KSEG1 aliases are interpreted;
- one shard per segment for the same word, preloaded and lazy;
- a stale own-segment shard is interpreted rather than handed to another
  segment's;
- a KUSEG continuation where KSEG0 has an entry at that word, and a lazy one
  whose competing KSEG0 bundle is newer;
- refused manifests, including through live publication;
- the native-call contract;
- a source check that the interpreter records the segment from the full PC,
  and that no site clears dispatch evidence without its segment bits.

The harness gains a segment-run scenario that drives real compiled shards
through a CPS dispatch loop. overlay_init_guard and overlay_retry_c11 follow
the per-segment lookup, and so do the disabled interpreter_perf_guards'
negative-miss and try_load_region checks (it still stops where it did on D).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Shokunin
2026-09-30 01:48:42 -07:00
parent 752f8ff2e7
commit 67bdf98eaf
9 changed files with 698 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
runtime/include/overlay_loader.h
+19 -10
View File
@@ -19,14 +19,19 @@
extern "C" {
#endif
/* The segment every overlay shard is compiled for. Captures record their
* bytes at the KSEG0 address (overlay_capture.c) and the manifest parser keys
* entries there; the emitter bakes that segment into every PC a shard makes:
* the $ra a jal/jalr writes, its jump and exception-resume PCs, and its I-cache
* fetch tags. The same bytes reached through KUSEG or KSEG1 run at a different
* architectural PC (a different link value; KSEG1 is uncached), so a shard runs
* only for PCs in this segment and any other alias is interpreted at its own
* PC. (OpenBIOS's exception path enters its RAM patch slots at KUSEG.) */
/* The segment an overlay shard is compiled for unless its manifest says
* otherwise (docs/SEGMENT_AWARE_CODE.md §5.7). Captures record their bytes and
* dispatch_entry_pcs at the KSEG0 address (overlay_capture.c), and a manifest
* without an S record keys its entries there. The emitter bakes a shard's
* segment into every PC it makes: the $ra a jal/jalr writes, its jump and
* exception-resume PCs, its I-cache fetch tags and its store-PC stamps. The
* same bytes reached through KUSEG or KSEG1 run at a different architectural
* PC (a different link value; KSEG1 is uncached), so capture records each
* dispatch's segment, compile_overlays.py builds a shard per segment with
* entries (KUSEG and KSEG1 ones in the seg-kuseg/ and seg-kseg1/ cache
* subdirectories), and a shard runs only for PCs in its own segment; a PC with
* no shard of its segment is interpreted at its own PC. (OpenBIOS's exception
* path enters its RAM patch slots at KUSEG.) */
#define PSX_OVERLAY_CODE_SEGMENT 0x80000000u
static inline int psx_overlay_code_segment_pc(uint32_t pc) {
return (pc & 0xE0000000u) == PSX_OVERLAY_CODE_SEGMENT;
@@ -151,9 +156,13 @@ int overlay_loader_lazy_manifest_count(void);
int overlay_loader_lazy_manifest_overflow(void);
uint64_t overlay_loader_candidate_overflow(void);
uint64_t overlay_loader_pair_aliases(void);
/* Dispatches sent to the interpreter because the PC is a KUSEG/KSEG1 alias of
* code compiled for PSX_OVERLAY_CODE_SEGMENT. */
/* KUSEG/KSEG1 (and unmapped-segment) dispatches the loader left to the
* interpreter: no shard compiled for their segment is cached or valid yet
* (docs/SEGMENT_AWARE_CODE.md §5.7). Capture records their segment, so a
* warm cache takes this to 0. */
uint64_t overlay_loader_segment_alias_interp(void);
/* KUSEG/KSEG1 dispatches run natively by a shard compiled for their segment. */
uint64_t overlay_loader_segment_native(void);
int overlay_loader_dump_lazy_at(uint32_t addr, char *out, int cap);
/* Overlay CI wrapper early-return attribution (PSX_POST_LOAD_PROBE). */
runtime/src/debug_server.c
+3 -2
View File
@@ -12513,7 +12513,7 @@ static void handle_overlay_loader_status(int id, const char *json)
"\"gen_fastpath\":%llu,\"range_links\":%d,\"range_index_overflow\":%d,"
"\"lazy_manifests\":%d,\"lazy_manifest_overflow\":%d,"
"\"candidate_overflow\":%llu,\"pair_aliases\":%llu,"
"\"segment_alias_interp\":%llu",
"\"segment_alias_interp\":%llu,\"segment_native\":%llu",
r0v, r0w, r0lo, r0hi, r0crc, ratt, rmiss, rlast,
(unsigned long long)overlay_loader_gen_fastpath(),
overlay_loader_range_link_count(),
@@ -12522,7 +12522,8 @@ static void handle_overlay_loader_status(int id, const char *json)
overlay_loader_lazy_manifest_overflow(),
(unsigned long long)overlay_loader_candidate_overflow(),
(unsigned long long)overlay_loader_pair_aliases(),
(unsigned long long)overlay_loader_segment_alias_interp());
(unsigned long long)overlay_loader_segment_alias_interp(),
(unsigned long long)overlay_loader_segment_native());
uint64_t nd=0, ni=0, sn=0, ss=0, sc=0, sx=0;
psx_interrupt_delivery_diag(&nd, &ni, &sn, &ss, &sc, &sx);
n += snprintf(buf + n, sizeof(buf) - n,
runtime/src/overlay_loader.c
+215 -72
View File
@@ -70,6 +70,9 @@ enum { ENTRY_VALID = 0, ENTRY_INVALID = 1, ENTRY_BLACKLIST = 2 };
typedef struct {
uint32_t addr; /* phys entry address */
uint32_t seg; /* segment the shard was compiled for
* (docs/SEGMENT_AWARE_CODE.md §5.7):
* it runs only for PCs in it */
OverlayFn fn;
uint32_t range_lo[MAX_CODE_RANGES]; /* phys code-range starts */
uint32_t range_len[MAX_CODE_RANGES];
@@ -131,7 +134,7 @@ static int s_range_index_overflow = 0;
* hundreds of historical, currently-invalid owners in a reused region. */
#define RANGE_PC_CACHE_CAP 16384u
#define RANGE_PC_CACHE_MASK (RANGE_PC_CACHE_CAP - 1u)
typedef struct { uint32_t phys, generation; int cand; } RangePcCache;
typedef struct { uint32_t phys, generation; int cand; } RangePcCache; /* phys = full PC */
static RangePcCache s_range_pc_cache[RANGE_PC_CACHE_CAP];
static uint32_t s_range_candidate_generation = 1u;
@@ -216,6 +219,10 @@ static void idx_set_head(uint32_t phys, int head) {
}
}
}
/* The chain keeps every segment's candidates for one physical entry (the
* bytes are one identity); a PC may run only its own segment's (§5.7). First
* candidate of the chain compiled for `seg`, or -1. */
static int idx_head_seg(uint32_t phys, uint32_t seg);
/* Active native-entry stack (self-modification detection, §8.5). */
static int s_active_stack[64];
@@ -501,7 +508,8 @@ static uint32_t s_rehash_miss = 0; /* hashes that didn't match crc_code */
static uint64_t s_gen_fastpath = 0; /* dispatches that skipped the crc32 */
/* via the unchanged page-generation */
/* fast path (overlay-cache v2 P2) */
static uint64_t s_segment_alias_interp = 0; /* KUSEG/KSEG1 PCs kept off shards */
static uint64_t s_segment_alias_interp = 0; /* KUSEG/KSEG1 PCs interpreted */
static uint64_t s_segment_native = 0; /* KUSEG/KSEG1 PCs run by own shard */
static uint64_t s_diffgate_interp = 0; /* CPS interior re-entries sent to the */
/* interp because their candidate is */
/* still inside the diff verify budget */
@@ -798,7 +806,13 @@ enum {
static int man_structurally_valid(const ManFn *m) {
if (!m || !m->has_crc || m->n < 1 || m->n > MAX_CODE_RANGES)
return 0;
if ((m->entry & OVERLAY_KSEG0_WINDOW_MASK) != 0x80000000u) return 0;
/* F entries carry the full VA of the segment the shard was compiled for
* (docs/SEGMENT_AWARE_CODE.md §5.7): KUSEG, KSEG0 or KSEG1, inside that
* segment's RAM decode window. R ranges stay byte extents, written at
* KSEG0 whatever the shard's segment. */
if (psx_code_segment_index(m->entry) < 0 ||
(m->entry & OVERLAY_KSEG0_WINDOW_MASK) != (m->entry & 0xE0000000u))
return 0;
uint32_t entry = m->entry & 0x1FFFFFFFu;
if ((entry & 3u) != 0u || entry >= OVERLAY_RAM_SIZE) return 0;
int entry_covered = 0;
@@ -853,18 +867,25 @@ static int manifest_record_end(const char *cursor) {
return *manifest_skip_space(cursor) == '\0';
}
/* `S <seg>` (§5.7) names the segment the shard was compiled for: 00000000
* (KUSEG), 80000000 (KSEG0) or A0000000 (KSEG1). It is optional and absent
* means KSEG0, so every manifest written before it stays valid. Every F entry
* must lie in that segment; *out_seg receives it. */
static ManFn *parse_manifest(const char *path, int *out_n,
uint64_t *out_pair_id,
int *out_has_pair_id,
int *out_provenance) {
int *out_provenance,
uint32_t *out_seg) {
*out_n = 0;
if (out_seg) *out_seg = PSX_OVERLAY_CODE_SEGMENT;
if (out_pair_id) *out_pair_id = 0;
if (out_has_pair_id) *out_has_pair_id = 0;
if (out_provenance) *out_provenance = MANIFEST_PROVENANCE_AUTHORITY;
FILE *f = fopen(path, "rb");
if (!f) return NULL;
int cap = 1024, n = 0;
int invalid = 0, pair_seen = 0, provenance_seen = 0;
int invalid = 0, pair_seen = 0, provenance_seen = 0, seg_seen = 0;
uint32_t seg = PSX_OVERLAY_CODE_SEGMENT;
int provenance = MANIFEST_PROVENANCE_AUTHORITY;
ManFn *arr = (ManFn *)malloc(sizeof(ManFn) * cap);
if (!arr) { fclose(f); return NULL; }
@@ -900,7 +921,8 @@ static ManFn *parse_manifest(const char *path, int *out_n,
? parsed_provenance : MANIFEST_PROVENANCE_AMBIGUOUS;
}
const char *record = manifest_skip_space(line);
int recognized = ((*record == 'P' || *record == 'F' || *record == 'R') &&
int recognized = ((*record == 'P' || *record == 'F' || *record == 'R' ||
*record == 'S') &&
(record[1] == '\0' ||
manifest_is_space((unsigned char)record[1])));
int embedded_nul = memchr(line, '\0', line_len) != NULL;
@@ -916,7 +938,18 @@ static ManFn *parse_manifest(const char *path, int *out_n,
continue;
}
if (!recognized) continue;
if (*record == 'P') {
if (*record == 'S') {
const char *cursor = record + 1;
uint64_t parsed = 0;
if (!seg_seen && n == 0 &&
manifest_hex_field(&cursor, 8, &parsed) &&
manifest_record_end(cursor) &&
psx_code_segment_index((uint32_t)parsed) >= 0 &&
((uint32_t)parsed & 0x1FFFFFFFu) == 0u) {
seg = (uint32_t)parsed;
seg_seen = 1;
} else invalid = 1; /* twice, after an F, or not a RAM segment */
} else if (*record == 'P') {
const char *cursor = record + 1;
uint64_t parsed = 0;
if (manifest_hex_field(&cursor, 16, &parsed) &&
@@ -939,7 +972,8 @@ static ManFn *parse_manifest(const char *path, int *out_n,
arr = na;
}
cur = &arr[n++];
cur->entry = ((uint32_t)parsed_e & 0x1FFFFFFFu) | PSX_OVERLAY_CODE_SEGMENT;
cur->entry = (uint32_t)parsed_e;
if ((cur->entry & 0xE0000000u) != seg) invalid = 1;
cur->crc = (uint32_t)parsed_crc;
cur->has_crc = 1;
cur->n = 0;
@@ -981,6 +1015,7 @@ static ManFn *parse_manifest(const char *path, int *out_n,
return NULL;
}
if (out_provenance) *out_provenance = provenance;
if (out_seg) *out_seg = seg;
*out_n = n;
return arr;
}
@@ -1169,6 +1204,7 @@ static int cand_register(uint32_t phys, OverlayFn fn, const ManFn *m, int dll,
int idx = s_cand_n++;
Candidate *c = &s_cand[idx];
c->addr = phys;
c->seg = m->entry & 0xE0000000u;
c->fn = fn;
c->dll = dll;
c->tier = (uint8_t)tier;
@@ -1222,6 +1258,12 @@ static int cand_register(uint32_t phys, OverlayFn fn, const ManFn *m, int dll,
return 1;
}
static int idx_head_seg(uint32_t phys, uint32_t seg) {
for (int i = idx_head(phys); i >= 0; i = s_cand[i].next)
if (s_cand[i].seg == seg) return i;
return -1;
}
/* (sljit removed 2026-07-15: register_sljit_candidate, the JIT-on-miss memo,
* the async compile-worker glue, the master tier gate, and try_sljit_region
* lived here. Overlay misses now fall to the interpreter.) */
@@ -1265,6 +1307,7 @@ enum { CACHE_TIER_UNKNOWN = 0, CACHE_TIER_TCC = 1, CACHE_TIER_GCC = 2 };
typedef struct {
uint32_t region_start;
uint32_t logical_crc;
uint32_t seg; /* shard segment, from its directory (§5.7) */
uint64_t mtime;
int func_count;
int indexed_func_count;
@@ -1349,6 +1392,33 @@ static int cache_tier_from_path(const char *path) {
return CACHE_TIER_UNKNOWN;
}
/* Per-segment cache layout (docs/SEGMENT_AWARE_CODE.md §5.7, decision 4):
* KSEG0 shards stay directly in the cache tag directory, under their old
* names, so no existing cache is invalidated; KUSEG and KSEG1 shards live in
* the seg-kuseg/ and seg-kseg1/ subdirectories of the same tag, under the
* same {phys}_{crc} grammar. A shard's segment is its directory's, and its
* manifest's S record must agree. */
#define OVERLAY_SEG_DIR_KUSEG "seg-kuseg"
#define OVERLAY_SEG_DIR_KSEG1 "seg-kseg1"
static int cache_dir_name_eq(const char *start, size_t len, const char *want) {
if (len != strlen(want)) return 0;
for (size_t i = 0; i < len; i++)
if (tolower((unsigned char)start[i]) != want[i]) return 0;
return 1;
}
static uint32_t cache_seg_from_path(const char *path) {
const char *end = NULL;
for (const char *p = path; *p; p++)
if (*p == '/' || *p == '\\') end = p;
if (!end) return PSX_OVERLAY_CODE_SEGMENT;
const char *start = end;
while (start > path && start[-1] != '/' && start[-1] != '\\') start--;
size_t len = (size_t)(end - start);
if (cache_dir_name_eq(start, len, OVERLAY_SEG_DIR_KUSEG)) return 0x00000000u;
if (cache_dir_name_eq(start, len, OVERLAY_SEG_DIR_KSEG1)) return 0xA0000000u;
return PSX_OVERLAY_CODE_SEGMENT;
}
static int cache_name_is_immutable(const char *name) {
if (!name) return 0;
size_t len = strlen(name);
@@ -1564,7 +1634,7 @@ static int s_lazy_range_link_n = 0;
#define LAZY_MISS_CACHE_CAP 16384u
#define LAZY_MISS_CACHE_MASK (LAZY_MISS_CACHE_CAP - 1u)
typedef struct {
uint32_t phys;
uint32_t phys; /* the full PC (segment | phys) */
uint32_t code_gen;
uint32_t watched_code_gen;
uint32_t loader_gen;
@@ -1581,16 +1651,18 @@ static void lazy_miss_invalidate_loader(void) {
}
}
static int lazy_miss_cached(uint32_t phys) {
LazyMissEntry *e = &s_lazy_miss_cache[(phys >> 2) & LAZY_MISS_CACHE_MASK];
return e->phys == phys && e->code_gen == g_dirty_ram_code_gen &&
/* Keyed by the full PC: a miss for one segment says nothing about another
* segment's shard of the same bytes (§5.7). */
static int lazy_miss_cached(uint32_t pc) {
LazyMissEntry *e = &s_lazy_miss_cache[(pc >> 2) & LAZY_MISS_CACHE_MASK];
return e->phys == pc && e->code_gen == g_dirty_ram_code_gen &&
e->watched_code_gen == s_lazy_watched_code_gen &&
e->loader_gen == s_lazy_loader_gen;
}
static void lazy_miss_record(uint32_t phys) {
LazyMissEntry *e = &s_lazy_miss_cache[(phys >> 2) & LAZY_MISS_CACHE_MASK];
e->phys = phys;
static void lazy_miss_record(uint32_t pc) {
LazyMissEntry *e = &s_lazy_miss_cache[(pc >> 2) & LAZY_MISS_CACHE_MASK];
e->phys = pc;
e->code_gen = g_dirty_ram_code_gen;
e->watched_code_gen = s_lazy_watched_code_gen;
e->loader_gen = s_lazy_loader_gen;
@@ -1630,8 +1702,18 @@ static void rebuild_lazy_manifest_index(void) {
snprintf(path + n - OVERLAY_SHARED_EXT_LEN,
sizeof(path) - (n - OVERLAY_SHARED_EXT_LEN), ".ranges");
int man_n = 0;
ManFn *man = parse_manifest(path, &man_n, NULL, NULL, NULL);
uint32_t man_seg = 0;
ManFn *man = parse_manifest(path, &man_n, NULL, NULL, NULL, &man_seg);
if (!man) continue;
if (man_seg != s_cache_idx[ci].seg) {
/* A shard runs only in the segment it was compiled for, and its
* directory says which one; a disagreeing manifest is not used. */
loader_log("manifest %s names segment 0x%08X, its directory 0x%08X "
"-- left to the interpreter", path, man_seg,
s_cache_idx[ci].seg);
free(man);
continue;
}
s_cache_idx[ci].manifest_ok = 1;
s_cache_idx[ci].func_count = man_n;
for (int mi = 0; mi < man_n; mi++) {
@@ -1774,6 +1856,7 @@ const char *overlay_loader_arch_abi(void) { return PSX_OVERLAY_ARCH_ABI; }
#ifndef _WIN32
static int add_posix_cache_file(const PsxOverlayCacheFile *file, void *opaque) {
int tier = opaque ? *(const int *)opaque : CACHE_TIER_UNKNOWN;
const uint32_t seg = cache_seg_from_path(file->path);
/* Legacy 8_8 pairs were published by a replace protocol that could leave a
* new manifest beside an old DLL. Keep them on disk for Python seed
* migration, but never let them suppress or execute instead of a bound
@@ -1789,6 +1872,7 @@ static int add_posix_cache_file(const PsxOverlayCacheFile *file, void *opaque) {
CacheEntry *e = &s_cache_idx[s_cache_idx_count++];
e->region_start = file->region_start;
e->logical_crc = file->content_crc;
e->seg = seg;
e->mtime = file->mtime;
e->func_count = 0;
e->indexed_func_count = 0;
@@ -1832,6 +1916,7 @@ static void scan_one_cache_dir(const char *dir, int tier) {
CacheEntry *e = &s_cache_idx[s_cache_idx_count++];
e->region_start = addr;
e->logical_crc = crc;
e->seg = cache_seg_from_path(full);
e->mtime = ((uint64_t)fd.ftLastWriteTime.dwHighDateTime << 32) |
(uint64_t)fd.ftLastWriteTime.dwLowDateTime;
e->func_count = 0;
@@ -2042,23 +2127,31 @@ static void abi_preflight_sweep(const char *dir) {
#endif
}
static void scan_cache_dir(void) {
static void scan_tier_cache_dirs(const char *tier_name, int tier) {
char dir[768];
snprintf(dir, sizeof(dir), "%s/%s/%s/%s/cg%d_%08x_gc%08x_f%u",
s_cache_dir, s_game_id, tier_name, PSX_OVERLAY_ARCH_ABI,
PSX_OVERLAY_CODEGEN_VER, (unsigned)PSX_OVERLAY_CODEGEN_HASH,
(unsigned)s_config_hash, (unsigned)PSX_OVERLAY_FLAVOR);
scan_one_cache_dir(dir, tier);
abi_preflight_sweep(dir);
/* KUSEG and KSEG1 shards: per-segment subdirectories of the same tag. */
static const char *const seg_dirs[] = {
OVERLAY_SEG_DIR_KUSEG, OVERLAY_SEG_DIR_KSEG1 };
for (size_t i = 0; i < sizeof(seg_dirs) / sizeof(seg_dirs[0]); i++) {
char seg_dir[800];
snprintf(seg_dir, sizeof(seg_dir), "%s/%s", dir, seg_dirs[i]);
scan_one_cache_dir(seg_dir, tier);
abi_preflight_sweep(seg_dir);
}
}
static void scan_cache_dir(void) {
/* Index both tiers and every immutable artifact. Runtime selection prefers
* usable GCC over TCC; an invalid GCC artifact cannot suppress a valid TCC
* fallback merely because its filename was enumerated first. */
snprintf(dir, sizeof(dir), "%s/%s/gcc/%s/cg%d_%08x_gc%08x_f%u",
s_cache_dir, s_game_id, PSX_OVERLAY_ARCH_ABI, PSX_OVERLAY_CODEGEN_VER,
(unsigned)PSX_OVERLAY_CODEGEN_HASH, (unsigned)s_config_hash,
(unsigned)PSX_OVERLAY_FLAVOR);
scan_one_cache_dir(dir, CACHE_TIER_GCC);
abi_preflight_sweep(dir);
snprintf(dir, sizeof(dir), "%s/%s/tcc/%s/cg%d_%08x_gc%08x_f%u",
s_cache_dir, s_game_id, PSX_OVERLAY_ARCH_ABI, PSX_OVERLAY_CODEGEN_VER,
(unsigned)PSX_OVERLAY_CODEGEN_HASH, (unsigned)s_config_hash,
(unsigned)PSX_OVERLAY_FLAVOR);
scan_one_cache_dir(dir, CACHE_TIER_TCC);
abi_preflight_sweep(dir);
scan_tier_cache_dirs("gcc", CACHE_TIER_GCC);
scan_tier_cache_dirs("tcc", CACHE_TIER_TCC);
refresh_bios_resident_flags();
rebuild_lazy_manifest_index();
@@ -2302,12 +2395,16 @@ static void overlay_ci_wrapper(CPUState *cpu) {
psx_check_interrupts(cpu);
}
/* Full-VA compares (§5.7): the resume PC a shard bakes is in its own segment,
* so it is the caller's return point only when it equals $ra exactly, and it
* is internal only to a running shard of the same segment. */
static int overlay_idle_note_is_internal_or_return(const CPUState *cpu,
uint32_t resume_pc) {
uint32_t phys = resume_pc & 0x1FFFFFFFu;
if ((resume_pc & 0x1FFFFFFFu) == (cpu->gpr[31] & 0x1FFFFFFFu)) return 1;
if (resume_pc == cpu->gpr[31]) return 1;
if (s_active_depth <= 0) return 0;
Candidate *c = &s_cand[s_active_stack[s_active_depth - 1]];
if ((resume_pc & 0xE0000000u) != c->seg) return 0;
for (int r = 0; r < c->nranges; r++) {
uint32_t lo = c->range_lo[r];
if (phys >= lo && phys < lo + c->range_len[r]) return 1;
@@ -3139,16 +3236,26 @@ static int load_one_dll(const char *dll_path,
uint64_t manifest_pair_id = 0;
int manifest_has_pair_id = 0;
int manifest_provenance = MANIFEST_PROVENANCE_AUTHORITY;
uint32_t manifest_seg = 0;
ManFn *man = parse_manifest(ranges_path, &man_n,
&manifest_pair_id,
&manifest_has_pair_id,
&manifest_provenance);
&manifest_provenance,
&manifest_seg);
if (!man || man_n == 0) {
loader_log("no/empty manifest %s — DLL left to interpreter", ranges_path);
free(man);
overlay_library_close(prepared);
return 0;
}
if (manifest_seg != cache_seg_from_path(dll_path)) {
loader_log("manifest %s names segment 0x%08X, not its directory's "
"0x%08X -- DLL left to interpreter", ranges_path,
manifest_seg, cache_seg_from_path(dll_path));
free(man);
overlay_library_close(prepared);
return 0;
}
/* A complete known pair remains a zero-slot operation even when no new
* bundle of this size can fit. Parse and compare before durably suppressing
* the cache entry; otherwise safe aliases become unreachable at the cap. */
@@ -3308,7 +3415,8 @@ void overlay_loader_discard_prepared(OverlayPreparedImage *image) {
free(image);
}
static int lazy_man_contains(const ManFn *m, uint32_t phys) {
static int lazy_man_contains(const ManFn *m, uint32_t seg, uint32_t phys) {
if ((m->entry & 0xE0000000u) != seg) return 0; /* another segment's shard */
if ((m->entry & 0x1FFFFFFFu) == phys) return 1;
for (int r = 0; r < m->n; r++) {
uint32_t lo = m->lo[r] & 0x1FFFFFFFu;
@@ -3402,18 +3510,21 @@ static void overlay_image_warm_seed_boot_text(void) {
overlay_image_warm_queue(indices, count);
}
static int lazy_is_loadable(int li, uint32_t region_start, uint32_t phys,
int require_region_start) {
static int lazy_is_loadable(int li, uint32_t region_start, uint32_t seg,
uint32_t phys, int require_region_start) {
if (li < 0 || li >= s_lazy_man_n) return 0;
LazyMan *lm = &s_lazy_man[li];
int ci = lm->cache_idx;
if (ci < 0 || ci >= s_cache_idx_count) return 0;
/* A manifest's entries are in its directory's segment (checked when the
* index is built), so lazy_man_contains()'s segment test selects the
* shards of `seg`. */
return (!require_region_start ||
s_cache_idx[ci].region_start == region_start) &&
!s_cache_idx[ci].load_failed &&
!s_cache_idx[ci].capacity_suppressed &&
!dll_already_loaded(s_cache_idx[ci].path) &&
lazy_man_contains(&lm->fn, phys) && lazy_man_matches(lm);
lazy_man_contains(&lm->fn, seg, phys) && lazy_man_matches(lm);
}
static int lazy_better_bundle(int li, int best) {
@@ -3445,7 +3556,10 @@ static int lazy_load_selected(int li) {
return loaded;
}
static int try_load_region(uint32_t phys) {
/* Load the cached shard of `seg` (§5.7) that serves phys. The region walkback
* and the manifest indexes are physical: the bytes, and so the region, are the
* same in every segment. lazy_is_loadable() admits only `seg`'s manifests. */
static int try_load_region(uint32_t seg, uint32_t phys) {
extern uint32_t dirty_ram_get_bitmap_word(uint32_t word_index);
if (!overlay_loads_allowed())
@@ -3499,7 +3613,7 @@ retry_artifact:
* variants, so walkback can recover an older adjacent run's base
* (e.g. 0x106000 instead of an exact live 0x108000 shard). Rejecting
* the exact entry on that heuristic strands valid DLLs forever. */
if (!lazy_is_loadable(li, region_start, phys, 0)) continue;
if (!lazy_is_loadable(li, region_start, seg, phys, 0)) continue;
int ci = s_lazy_man[li].cache_idx;
/* Cross-region recovery is safe only for a fully coherent CPS bundle.
* A partial exact-function match can have snapshot-specific internal
@@ -3525,7 +3639,7 @@ retry_artifact:
/* Non-exact range ownership remains region-narrowed: unlike an
* exact manifested entry, an interior PC alone is not a unique
* identity for a reused-address CPS bundle. */
if (!lazy_is_loadable(li, region_start, phys, 1)) continue;
if (!lazy_is_loadable(li, region_start, seg, phys, 1)) continue;
if (lazy_candidate_preferred(li, fallback)) fallback = li;
int ci = s_lazy_man[li].cache_idx;
if (lazy_bundle_matches(ci) && lazy_candidate_preferred(li, best))
@@ -3546,7 +3660,7 @@ retry_artifact:
* interior CPS continuation. The latter must use its already-loaded range owner
* first (the Whoopee 0x107624 fix); the former must be allowed to publish its
* exact DLL even when a broader conservative owner contains the same PC. */
static int lazy_has_exact_entry(uint32_t phys) {
static int lazy_has_exact_entry(uint32_t seg, uint32_t phys) {
/* Same overlay-off hazard as overlay_find_by_range: the lazy entry index
* (s_lazy_entry_head) is -1-initialized only when overlay_cache is enabled;
* with overlay off it is zero-initialized and the link traversal below
@@ -3555,7 +3669,7 @@ static int lazy_has_exact_entry(uint32_t phys) {
uint32_t bucket = (phys * 2654435761u) & LAZY_ENTRY_MASK;
for (int li = s_lazy_entry_head[bucket]; li >= 0;
li = s_lazy_man[li].next_entry) {
if ((s_lazy_man[li].fn.entry & 0x1FFFFFFFu) == phys)
if (s_lazy_man[li].fn.entry == (seg | phys))
return 1;
}
return 0;
@@ -3570,9 +3684,10 @@ static int lazy_has_exact_entry(uint32_t phys) {
* idx_head() (entry-keyed) misses; we re-enter the owning function with
* cpu->pc = that address so its entry-switch routes to the right block. Returns
* the candidate index, or -1. */
static int range_candidate_matches(int i, uint32_t phys) {
static int range_candidate_matches(int i, uint32_t seg, uint32_t phys) {
Candidate *c = &s_cand[i];
if (c->state == ENTRY_BLACKLIST) return 0;
if (c->seg != seg) return 0; /* another segment's shard (§5.7) */
int contains = 0;
uint32_t first_range_lo = UINT32_MAX;
for (int r = 0; r < c->nranges; r++) {
@@ -3629,7 +3744,7 @@ static int range_candidate_preferred(int candidate, int current) {
return candidate > current; /* newest same-tier repair */
}
static int overlay_find_by_range(uint32_t phys) {
static int overlay_find_by_range(uint32_t seg, uint32_t phys) {
/* Definitive guard for the overlay-off case: the range page index
* (s_range_page_head) is only initialized to -1 by overlay_loader_init,
* which runs solely when overlay_cache is enabled. With overlay off it is
@@ -3640,36 +3755,39 @@ static int overlay_find_by_range(uint32_t phys) {
uint32_t page = phys >> 12;
if (page >= RANGE_PAGE_COUNT) return -1;
/* Cached by the full PC: each segment has its own owner (§5.7). */
const uint32_t key = seg | phys;
RangePcCache *pc = &s_range_pc_cache[
(phys * 2654435761u) & RANGE_PC_CACHE_MASK];
(key * 2654435761u) & RANGE_PC_CACHE_MASK];
if (pc->generation == s_range_candidate_generation &&
pc->cand >= 0 && pc->phys == phys &&
range_candidate_matches(pc->cand, phys))
pc->cand >= 0 && pc->phys == key &&
range_candidate_matches(pc->cand, seg, phys))
return pc->cand;
int best = -1;
if (s_range_index_overflow) {
for (int i = 0; i < s_cand_n; i++)
if (range_candidate_matches(i, phys) &&
if (range_candidate_matches(i, seg, phys) &&
range_candidate_preferred(i, best))
best = i;
} else {
for (int li = s_range_page_head[page]; li >= 0;
li = s_range_links[li].next) {
int i = s_range_links[li].cand;
if (range_candidate_matches(i, phys) &&
if (range_candidate_matches(i, seg, phys) &&
range_candidate_preferred(i, best))
best = i;
}
}
pc->phys = phys;
pc->phys = key;
pc->generation = s_range_candidate_generation;
pc->cand = best;
return best;
}
static int overlay_loader_dispatch_seg(CPUState *cpu, uint32_t addr);
int overlay_loader_dispatch(CPUState *cpu, uint32_t addr) {
uint32_t phys = addr & 0x1FFFFFFFu;
/* Overlay dispatch is a no-op when the overlay loader is inactive
* (overlay_cache=false): there are no candidates to match, so this must
* return 0 (dispatch to interp). This guard is also a HARD SAFETY:
@@ -3683,28 +3801,46 @@ int overlay_loader_dispatch(CPUState *cpu, uint32_t addr) {
* overlay-off + CPS game hit this and wedged at boot before any game code ran
* (found via Ape Escape, the only overlay-off title). Fail closed here. */
if (!s_active) return 0;
/* A shard's body bakes the KSEG0 PCs it was compiled for; a KUSEG or KSEG1
* alias of its bytes must run at its own PC ($ra, EPC, I-cache tags). The
* CPS continuation switch already rejects such a PC (bad entry); this
* keeps function entries to the same rule. */
int ran = overlay_loader_dispatch_seg(cpu, addr);
/* KUSEG/KSEG1 dispatches, split by outcome (docs/SEGMENT_AWARE_CODE.md
* §5.7): run by a shard compiled for their segment, or left to the
* interpreter (none is cached or valid yet). The interpreted ones are
* what capture records for the next compile, so on a warm cache the
* second count falls to 0. */
if (!psx_overlay_code_segment_pc(addr)) {
s_segment_alias_interp++;
if (ran) s_segment_native++;
else s_segment_alias_interp++;
}
return ran;
}
/* A shard's body bakes the PCs of the segment it was compiled for: the $ra a
* jal/jalr writes, its jump and exception-resume PCs, its I-cache fetch tags
* and store-PC stamps (docs/SEGMENT_AWARE_CODE.md §5.7). A PC therefore runs
* only a shard of its own segment; another segment's shard of the same bytes
* is not a candidate for it. Candidates, manifests and the negative caches
* are indexed physically (the bytes are one identity) and matched by segment;
* a PC outside the three segments that map RAM never runs a shard. */
static int overlay_loader_dispatch_seg(CPUState *cpu, uint32_t addr) {
const uint32_t phys = addr & 0x1FFFFFFFu;
const uint32_t seg = addr & 0xE0000000u;
if (psx_code_segment_index(addr) < 0) {
s_disp_interp++;
return 0;
}
if (overlay_cache_window_contains(phys) && lazy_miss_cached(phys)) {
if (overlay_cache_window_contains(phys) && lazy_miss_cached(addr)) {
s_disp_interp++;
return 0;
}
int lazy_loaded = 0;
retry_candidates:
; /* C11 labels must precede a statement. */
int head = idx_head(phys);
int head = idx_head_seg(phys, seg);
int loaded_range_ci = -1;
int lazy_exact = 0;
int exact_needs_load = 0;
if (head < 0 && s_active && overlay_cache_window_contains(phys)) {
lazy_exact = head < 0 && lazy_has_exact_entry(phys);
lazy_exact = head < 0 && lazy_has_exact_entry(seg, phys);
/* A CPS continuation is normally not a registered function ENTRY. Its
* already-loaded range owner must be checked before lazy discovery:
* try_load_region otherwise scans the 12K+ manifest index and may load
@@ -3713,11 +3849,11 @@ retry_candidates:
* manifested entries are different: a broader range owner must not mask
* their valid cached DLL (Tomba FMV's 0x80106424/0x80106688 case). */
if (head < 0 && g_psx_cps_mode)
loaded_range_ci = overlay_find_by_range(phys);
loaded_range_ci = overlay_find_by_range(seg, phys);
exact_needs_load = lazy_exact &&
(loaded_range_ci < 0 || s_cand[loaded_range_ci].device_touch);
if (head < 0 && (loaded_range_ci < 0 || exact_needs_load) &&
!lazy_loaded && try_load_region(phys)) {
!lazy_loaded && try_load_region(seg, phys)) {
lazy_loaded = 1;
goto retry_candidates;
}
@@ -3735,7 +3871,7 @@ retry_candidates:
* so they run native directly here; device-touch funcs still go to interp. */
if (head < 0 && g_psx_cps_mode) {
int ci = loaded_range_ci >= 0 ? loaded_range_ci
: overlay_find_by_range(phys);
: overlay_find_by_range(seg, phys);
int _probe = (s_cps_probe_pc && phys == s_cps_probe_pc);
if (_probe) {
s_cps_probe_count++;
@@ -3785,7 +3921,7 @@ retry_candidates:
int in_own_shadow = (s_in_shadow && (const void *)c == s_shadow_cand);
if (!in_own_shadow) {
int want_diff = s_diff_mode && cand_selected_for_diff(c);
if (want_diff && addr < 0x10000u) want_diff = 0;
if (want_diff && phys < 0x10000u) want_diff = 0;
if (want_diff && (s_diff_addr || c->diff_passes < OVERLAY_DIFF_BUDGET)) {
if (_probe) s_cps_probe_outcome = 5;
s_diffgate_interp++;
@@ -3844,6 +3980,7 @@ retry_candidates:
for (int i = head; i >= 0; i = s_cand[i].next) {
Candidate *c = &s_cand[i];
if (c->seg != seg) continue; /* another segment's shard */
if (c->state == ENTRY_BLACKLIST) continue;
/* Generation-gated validation (overlay-cache v2 P2). The ONLY way this
@@ -3905,7 +4042,7 @@ retry_candidates:
* "diverges" (observed: 2016/2017 calls, all 0xB0, flooding the
* ring). The diff harness validates OVERLAY shard codegen against
* the interp oracle; kernel gates stay on their normal route. */
if (want_diff && addr < 0x10000u) want_diff = 0;
if (want_diff && phys < 0x10000u) want_diff = 0;
if (want_diff && !s_in_shadow &&
(s_diff_addr || c->diff_passes < OVERLAY_DIFF_BUDGET)) {
/* NEVER start a shadow inside an exception dispatch: the guest's
@@ -4004,12 +4141,12 @@ retry_candidates:
* preserves additive variant coverage without turning one guest transition
* into an unbounded synchronous LoadLibrary loop. */
if (!lazy_loaded && s_active && overlay_cache_window_contains(phys) &&
try_load_region(phys)) {
try_load_region(seg, phys)) {
lazy_loaded = 1;
goto retry_candidates;
}
if (overlay_cache_window_contains(phys)) lazy_miss_record(phys);
if (overlay_cache_window_contains(phys)) lazy_miss_record(addr);
s_disp_interp++;
return 0;
}
@@ -4129,6 +4266,7 @@ int overlay_loader_lazy_manifest_overflow(void) { return s_lazy_man_overflow; }
uint64_t overlay_loader_candidate_overflow(void) { return s_cand_overflow; }
uint64_t overlay_loader_pair_aliases(void) { return s_pair_aliases; }
uint64_t overlay_loader_segment_alias_interp(void) { return s_segment_alias_interp; }
uint64_t overlay_loader_segment_native(void) { return s_segment_native; }
/* (sljit removed 2026-07-15: overlay_loader_sljit_obsoleted and the
* overlay_loader_sljit_probe one-shot JIT helper lived here.) */
@@ -4321,7 +4459,7 @@ static void run_shadow_diff_legacy(CPUState *cpu, Candidate *c, uint32_t addr) {
int guard = 0;
while (cpu->pc != 0 && !g_psx_call_bail && guard++ < 8192) {
uint32_t tv = cpu->pc;
if ((tv & 0x1FFFFFFFu) == (stop_ra & 0x1FFFFFFFu)) break; /* returned */
if (tv == stop_ra) break; /* returned (the exact PC, §5.7) */
cpu->pc = 0;
int prev_phase = g_exec_phase;
g_exec_phase = 3; /* compiled route; dirty/native callees re-tag inside */
@@ -4540,7 +4678,7 @@ static void run_shadow_diff(CPUState *cpu, Candidate *c, uint32_t addr) {
int guard = 0;
while (cpuN.pc != 0 && !g_psx_call_bail && guard++ < 8192) {
uint32_t tv = cpuN.pc;
if ((tv & 0x1FFFFFFFu) == (stop_ra & 0x1FFFFFFFu)) break;
if (tv == stop_ra) break;
cpuN.pc = 0;
int prev_phase = g_exec_phase;
g_exec_phase = 3;
@@ -4811,7 +4949,9 @@ int overlay_loader_call_native(CPUState *cpu, uint32_t addr) {
if (!s_active || !s_native_exec)
return 0; /* inactive/interp mode: keep the legacy inline path */
uint32_t phys = addr & 0x1FFFFFFFu;
if (idx_head(phys) < 0 && !lazy_has_exact_entry(phys))
uint32_t seg = addr & 0xE0000000u;
if (psx_code_segment_index(addr) < 0 ||
(idx_head_seg(phys, seg) < 0 && !lazy_has_exact_entry(seg, phys)))
return 0; /* neither a registered nor an exact cached entry */
uint32_t in_regs[34];
int fp = overlay_fp_enabled();
@@ -4929,11 +5069,12 @@ int overlay_loader_dump_candidates_at(uint32_t addr, char *out, int cap) {
const int live_ok = cand_live_crc(c, &live);
uint32_t sum = cand_gensum(c);
n += snprintf(out + n, cap - n,
"%s{\"index\":%d,\"addr\":\"0x%08X\",\"state\":%d,\"nranges\":%d,"
"%s{\"index\":%d,\"addr\":\"0x%08X\",\"seg\":\"0x%08X\","
"\"state\":%d,\"nranges\":%d,"
"\"crc\":\"0x%08X\",\"live\":\"0x%08X\",\"match\":%d,"
"\"val_gen\":%u,\"gen\":%u,\"dll\":%d,\"diff_passes\":%u,"
"\"device_touch\":%d}",
first ? "" : ",", i, c->addr, c->state, c->nranges,
first ? "" : ",", i, c->addr, c->seg, c->state, c->nranges,
c->crc_code, live, (live_ok && live == c->crc_code) ? 1 : 0,
c->val_gen, sum, c->dll, c->diff_passes, c->device_touch);
first = 0;
@@ -4971,13 +5112,15 @@ int overlay_loader_dump_lazy_at(uint32_t addr, char *out, int cap) {
const int live_ok = lazy_man_crc(&lm->fn, &live);
n += snprintf(out + n, cap - n,
"%s{\"li\":%d,\"ci\":%d,\"region\":\"0x%08X\","
"\"file\":\"%s\",\"funcs\":%d,\"indexed\":%d,"
"\"entry\":\"0x%08X\",\"file\":\"%s\",\"funcs\":%d,\"indexed\":%d,"
"\"loaded\":%d,\"crc\":\"0x%08X\",\"live\":\"0x%08X\","
"\"match\":%d,\"contains\":%d}",
first ? "" : ",", li, ci, s_cache_idx[ci].region_start,
base, s_cache_idx[ci].func_count, s_cache_idx[ci].indexed_func_count,
lm->fn.entry, base, s_cache_idx[ci].func_count,
s_cache_idx[ci].indexed_func_count,
dll_already_loaded(s_cache_idx[ci].path), lm->fn.crc, live,
live_ok && live == lm->fn.crc, lazy_man_contains(&lm->fn, phys));
live_ok && live == lm->fn.crc,
lazy_man_contains(&lm->fn, lm->fn.entry & 0xE0000000u, phys));
first = 0;
}
n += snprintf(out + n, cap - n, "]}");
runtime/tests/overlay_pair_dedup_fixture.c
+17 -4
View File
@@ -53,14 +53,27 @@ TEST_EXPORT int test_init_count(void) { return s_init_count; }
TEST_EXPORT int test_flush_count(void) { return s_flush_count; }
TEST_EXPORT int test_call_count(void) { return s_call_count; }
TEST_EXPORT void func_80010000(CPUState *cpu) {
/* A shard's exports are named by the full VA of the segment it was compiled
* for (docs/SEGMENT_AWARE_CODE.md §5.1, §5.7): KSEG0 by default, KUSEG or
* KSEG1 when the segment tests build it that way. */
#if defined(TEST_SEGMENT_KUSEG)
#define ENTRY(lo) func_0001##lo
#elif defined(TEST_SEGMENT_KSEG1)
#define ENTRY(lo) func_A001##lo
#else
#define ENTRY(lo) func_8001##lo
#endif
TEST_EXPORT void ENTRY(0000)(CPUState *cpu) {
s_call_count++;
cpu->gpr[2] = TEST_MARKER;
cpu->gpr[6] = TEST_INSTANCE; /* which shard ran */
cpu->gpr[7] = cpu->pc; /* the PC it was entered for (CPS routing) */
trace("call");
}
#ifndef TEST_PARTIAL_EXPORTS
TEST_EXPORT void func_80010004(CPUState *cpu) { cpu->gpr[3] = TEST_MARKER; }
TEST_EXPORT void func_80010008(CPUState *cpu) { cpu->gpr[4] = TEST_MARKER; }
TEST_EXPORT void func_8001000C(CPUState *cpu) { cpu->gpr[5] = TEST_MARKER; }
TEST_EXPORT void ENTRY(0004)(CPUState *cpu) { cpu->gpr[3] = TEST_MARKER; }
TEST_EXPORT void ENTRY(0008)(CPUState *cpu) { cpu->gpr[4] = TEST_MARKER; }
TEST_EXPORT void ENTRY(000C)(CPUState *cpu) { cpu->gpr[5] = TEST_MARKER; }
#endif
runtime/tests/overlay_pair_dedup_harness.c
+225 -2
View File
@@ -86,7 +86,8 @@ int psx_cycle_replay_begin(uint64_t cycle) { (void)cycle; return 1; }
uint64_t psx_cycle_replay_end(void) { return 0; }
uint32_t psx_cyc_load_word(CPUState *cpu, uint32_t addr, uint32_t rt,
uint32_t mask) {
(void)cpu; (void)addr; (void)rt; (void)mask; return 0;
(void)rt; (void)mask;
return cpu->read_word ? cpu->read_word(addr) : 0;
}
uint32_t psx_cyc_load_word_slow(CPUState *cpu, uint32_t addr, uint32_t rt,
uint32_t mask) {
@@ -111,7 +112,14 @@ int psx_icache_shadow_record_begin(void) { return 1; }
int psx_icache_shadow_replay_begin(void) { return 1; }
void psx_icache_shadow_replay_end(void) {}
void psx_icache_shadow_abort(void) {}
void psx_icache_fetch(CPUState *cpu, uint32_t addr) { (void)cpu; (void)addr; }
/* Fetch tags the shards hand the I-cache model, in order (segment-run). */
static uint32_t s_fetch_log[1024];
static unsigned s_fetch_n;
void psx_icache_fetch(CPUState *cpu, uint32_t addr) {
(void)cpu;
if (s_fetch_n < sizeof(s_fetch_log) / sizeof(s_fetch_log[0]))
s_fetch_log[s_fetch_n++] = addr;
}
void psx_icache_fetch_fn(CPUState *cpu, uint32_t addr) {
psx_icache_fetch(cpu, addr);
}
@@ -339,6 +347,209 @@ static int segment_alias(const char *first) {
return 0;
}
/* Per-segment shards (docs/SEGMENT_AWARE_CODE.md §5.7). The fixture is built
* once per segment (instance 1 KSEG0, 3 KUSEG, 4 KSEG1), each with one
* function at physical 0x10000 whose manifest range covers 0x10000-0x10007,
* so 0x10004 is a CPS continuation inside it. Each PC runs only the shard of
* its own segment, entry or continuation alike; a PC with no shard of its
* segment, or in a segment that maps no RAM, is interpreted. */
struct SegCase { uint32_t pc; int ran; int instance; };
static int segment_dispatch(const struct SegCase *cases, unsigned n) {
int ok = 1;
for (unsigned i = 0; i < n; i++) {
CPUState cpu;
char what[64];
memset(&cpu, 0, sizeof(cpu));
int ran = overlay_loader_dispatch(&cpu, cases[i].pc);
snprintf(what, sizeof(what), "dispatch 0x%08X ran", cases[i].pc);
ok &= expect_int(what, ran, cases[i].ran);
snprintf(what, sizeof(what), "dispatch 0x%08X shard", cases[i].pc);
ok &= expect_int(what, cpu.gpr[6], cases[i].ran ? cases[i].instance : 0);
if (cases[i].ran) {
/* A continuation re-enters its owner with cpu->pc = the PC. */
snprintf(what, sizeof(what), "dispatch 0x%08X entry pc", cases[i].pc);
ok &= expect_int(what, cpu.gpr[7],
(cases[i].pc & 0x1FFFFFFFu) == 0x10000u ? 0 : cases[i].pc);
}
}
return ok;
}
extern int overlay_loader_call_native(CPUState *cpu, uint32_t addr);
static int segment_shards(int kseg0, int kuseg, int kseg1, const char *publish) {
extern int g_psx_cps_mode;
g_psx_cps_mode = 1;
const struct SegCase cases[] = {
{ 0x00010000u, kuseg, 3 }, { 0x80010000u, kseg0, 1 },
{ 0xA0010000u, kseg1, 4 },
{ 0x00010004u, kuseg, 3 }, { 0x80010004u, kseg0, 1 },
{ 0xA0010004u, kseg1, 4 },
/* No RAM there in Beetle (addr_mask leaves them unmasked). */
{ 0x20010000u, 0, 0 }, { 0xC0010000u, 0, 0 },
};
int ok = segment_dispatch(cases, sizeof(cases) / sizeof(cases[0]));
/* Again, now that every shard is loaded and cached: same answers. */
ok &= segment_dispatch(cases, sizeof(cases) / sizeof(cases[0]));
long long native = 2 * (2 * kuseg + 2 * kseg1);
long long interp = 2 * (2 * !kuseg + 2 * !kseg1 + 2);
ok &= expect_int("segment_native", (long long)overlay_loader_segment_native(),
native);
ok &= expect_int("segment_alias_interp",
(long long)overlay_loader_segment_alias_interp(), interp);
ok &= expect_int("registered", overlay_loader_registered_count(),
kseg0 + kuseg + kseg1);
ok &= expect_int("pair aliases", (long long)overlay_loader_pair_aliases(), 0);
/* Only manifests whose segment agrees with their directory are indexed. */
ok &= expect_int("lazy manifests", overlay_loader_lazy_manifest_count(),
kseg0 + kuseg + kseg1);
/* The interpreter's native-call contract asks the loader only for a PC
* with a shard of its own segment: no dispatch, no count, for the rest. */
static const uint32_t segs[] = { 0x00000000u, 0xA0000000u };
const int have[] = { kuseg, kseg1 };
for (unsigned i = 0; i < 2; i++) {
CPUState cpu;
memset(&cpu, 0, sizeof(cpu));
uint64_t n0 = overlay_loader_segment_native();
uint64_t i0 = overlay_loader_segment_alias_interp();
char what[64];
snprintf(what, sizeof(what), "call_native 0x%08X", segs[i] | 0x10000u);
ok &= expect_int(what, overlay_loader_call_native(&cpu, segs[i] | 0x10000u),
have[i]);
ok &= expect_int("call_native segment_native",
(long long)(overlay_loader_segment_native() - n0), have[i]);
ok &= expect_int("call_native segment_alias_interp",
(long long)(overlay_loader_segment_alias_interp() - i0), 0);
}
/* A refused pair stays refused through live publication. */
if (publish && strcmp(publish, "-") != 0) {
OverlayPreparedImage *image = overlay_loader_prepare_published(publish);
int published = image ? overlay_loader_commit_published(image) : 0;
ok &= expect_int("published refused pair", published, 0);
ok &= expect_int("registered after publication",
overlay_loader_registered_count(), kseg0 + kuseg + kseg1);
}
if (!ok) {
fprintf(stderr, "loader: %s\n", overlay_loader_last_msg());
return 1;
}
printf("PASS segment-shards kseg0=%d kuseg=%d kseg1=%d\n", kseg0, kuseg, kseg1);
return 0;
}
/* segment-stale: the KUSEG shard's bytes no longer match live RAM, the KSEG0
* shard's do. The KUSEG PC is interpreted, never handed to the KSEG0 shard
* that follows it in the physical chain. */
static int segment_stale(void) {
const struct SegCase cases[] = {
{ 0x00010000u, 0, 0 }, { 0x80010000u, 1, 1 }, { 0xA0010000u, 0, 0 },
};
/* Both are loaded; only the KSEG0 candidate validates. */
int ok = expect_int("valid candidates", overlay_loader_registered_count(), 1);
ok &= segment_dispatch(cases, sizeof(cases) / sizeof(cases[0]));
if (!ok) { fprintf(stderr, "loader: %s\n", overlay_loader_last_msg()); return 1; }
printf("PASS segment-stale\n");
return 0;
}
/* segment-continuation: a KSEG0 shard has a function ENTRY at 0x10004, and
* the KUSEG shard owns 0x10004 as a CPS continuation of its 0x10000
* function. The KUSEG PC resumes the KUSEG owner; another segment's entry at
* the word is not an entry for it. With `lazy`, both shards are unloaded and
* the KSEG0 one is newer: the continuation still loads the KUSEG shard. */
static int segment_continuation(int lazy) {
extern int g_psx_cps_mode;
g_psx_cps_mode = 1;
CPUState cpu;
int ok = 1;
memset(&cpu, 0, sizeof(cpu));
ok &= expect_int("kuseg continuation ran", overlay_loader_dispatch(&cpu, 0x00010004u), 1);
ok &= expect_int("kuseg continuation shard", cpu.gpr[6], 3);
ok &= expect_int("kuseg continuation pc", cpu.gpr[7], 0x00010004u);
if (!lazy) {
memset(&cpu, 0, sizeof(cpu));
ok &= expect_int("kseg0 entry ran", overlay_loader_dispatch(&cpu, 0x80010004u), 1);
ok &= expect_int("kseg0 entry marker", cpu.gpr[3], TEST_MARKER);
ok &= expect_int("kseg0 entry is no kuseg shard", cpu.gpr[6], 0);
}
if (!ok) { fprintf(stderr, "loader: %s\n", overlay_loader_last_msg()); return 1; }
printf("PASS segment-continuation%s\n", lazy ? " (lazy)" : "");
return 0;
}
/* segment-run: real shards compiled by compile_overlays.py from one image,
* one per segment it was captured in (docs/SEGMENT_AWARE_CODE.md §5.7). The
* image is placed at its physical address and entered at `phys` in each
* segment, driving a minimal CPS dispatch loop (each tail transfer goes back
* through overlay_loader_dispatch) until the function returns to its caller.
* One line per segment reports what the shards did; the Python side judges
* it: the links, fetch tags and store-PC stamps each run produced, and
* whether every transfer ran natively. */
static uint32_t s_store_log[64];
static unsigned s_store_n;
/* The host store-PC stamp the shards write through (ABI v24). This harness
* builds the loader with PSX_OVERLAY_DLL_BUILD, where cpu_state.h maps
* g_debug_last_store_pc onto *g_psx_last_store_pc_p, so point that at a
* real word before the loader hands its address to the shards. */
static uint32_t s_host_store_pc;
static uint32_t ram_read_word(uint32_t addr) {
uint32_t v;
memcpy(&v, &s_ram[(addr & (RAM_SIZE - 1u)) & ~3u], 4);
return v;
}
static void ram_write_word(uint32_t addr, uint32_t value) {
if (s_store_n < sizeof(s_store_log) / sizeof(s_store_log[0]))
s_store_log[s_store_n++] = s_host_store_pc;
memcpy(&s_ram[(addr & (RAM_SIZE - 1u)) & ~3u], &value, 4);
}
static int segment_run(const char *image_path, const char *phys_text) {
extern int g_psx_cps_mode;
const uint32_t ra_sentinel = 0x0000BEE0u;
uint32_t phys = (uint32_t)strtoul(phys_text, NULL, 16);
FILE *f = fopen(image_path, "rb");
if (!f) { perror(image_path); return 3; }
size_t got = fread(&s_ram[phys], 1, RAM_SIZE - phys, f);
fclose(f);
if (got == 0) return 3;
g_psx_cps_mode = 1;
static const uint32_t segs[] = { 0x00000000u, 0x80000000u, 0xA0000000u };
for (unsigned si = 0; si < 3; si++) {
CPUState cpu;
memset(&cpu, 0, sizeof(cpu));
cpu.read_word = ram_read_word;
cpu.write_word = ram_write_word;
cpu.gpr[29] = 0x001FF000u;
cpu.gpr[31] = ra_sentinel;
s_fetch_n = 0;
s_store_n = 0;
uint32_t pc = segs[si] | phys, interp_at = 0;
unsigned steps = 0, native = 0;
while (pc != ra_sentinel && steps++ < 64) {
cpu.pc = 0;
if (!overlay_loader_dispatch(&cpu, pc)) { interp_at = pc; break; }
native++;
if (cpu.pc == 0) break; /* returned to the C caller */
pc = cpu.pc;
}
printf("seg=%08X native=%u interp_at=%08X end=%08X v1=%08X t3=%u sp=%08X "
"fetch=", segs[si], native, interp_at, pc, cpu.gpr[3],
cpu.gpr[11], cpu.gpr[29]);
for (unsigned i = 0; i < s_fetch_n; i++)
printf("%s%08X", i ? "," : "", s_fetch_log[i]);
printf(" store_pc=");
for (unsigned i = 0; i < s_store_n; i++)
printf("%s%08X", i ? "," : "", s_store_log[i]);
printf("\n");
}
printf("segment_native=%llu segment_alias_interp=%llu registered=%d\n",
(unsigned long long)overlay_loader_segment_native(),
(unsigned long long)overlay_loader_segment_alias_interp(),
overlay_loader_registered_count());
return 0;
}
int main(int argc, char **argv) {
if (argc != 5) {
fprintf(stderr, "usage: %s <cache-root> <scenario> <first> <second>\n",
@@ -349,8 +560,20 @@ int main(int argc, char **argv) {
const char *first = argv[3];
const char *second = argv[4];
memset(s_ram, 0, sizeof(s_ram));
g_psx_last_store_pc_p = &s_host_store_pc;
overlay_loader_init(argv[1], "PAIR-TEST", 0);
if (strcmp(scenario, "segment-alias") == 0) return segment_alias(first);
/* segment-shards: argv[3] names which segments have a shard ("kseg0",
* "kuseg", "kseg1", comma-separated, or "none"). */
if (strcmp(scenario, "segment-shards") == 0)
return segment_shards(strstr(first, "kseg0") != NULL,
strstr(first, "kuseg") != NULL,
strstr(first, "kseg1") != NULL, second);
if (strcmp(scenario, "segment-stale") == 0) return segment_stale();
if (strcmp(scenario, "segment-continuation") == 0)
return segment_continuation(strcmp(first, "lazy") == 0);
/* segment-run <image> <phys-hex> */
if (strcmp(scenario, "segment-run") == 0) return segment_run(first, second);
int alias = strcmp(scenario, "alias-at-cap") == 0;
int partial = strcmp(scenario, "partial-first") == 0;
runtime/tests/test_interpreter_perf_guards.py
+8 -6
View File
@@ -120,10 +120,12 @@ def main():
raise AssertionError(
"interpreter does not take a pending rfe escape before local dirty flow")
dispatch = body(loader, "overlay_loader_dispatch")
cached = dispatch.find("lazy_miss_cached(phys)")
lookup = dispatch.find("idx_head(phys)")
record_miss = dispatch.rfind("lazy_miss_record(phys)")
# Keyed by the full PC and matched by segment since §5.7
# (docs/SEGMENT_AWARE_CODE.md); the body lives in the per-segment half.
dispatch = body(loader, "overlay_loader_dispatch_seg")
cached = dispatch.find("lazy_miss_cached(addr)")
lookup = dispatch.find("idx_head_seg(phys, seg)")
record_miss = dispatch.rfind("lazy_miss_record(addr)")
if min(cached, lookup, record_miss) < 0 or not cached < lookup < record_miss:
raise AssertionError("negative miss cache is not guarding lookup/recorded at final miss")
if "lazy_miss_invalidate_loader();" not in body(loader, "overlay_loader_rescan"):
@@ -153,9 +155,9 @@ def main():
if guard not in has_crc:
raise AssertionError(f"cached-CRC query accepts unusable artifacts: {guard}")
try_load = body(loader, "try_load_region")
if "lazy_is_loadable(li, region_start, phys, 0)" not in try_load:
if "lazy_is_loadable(li, region_start, seg, phys, 0)" not in try_load:
raise AssertionError("exact manifest entries are still gated by dirty-run base")
if "lazy_is_loadable(li, region_start, phys, 1)" not in try_load:
if "lazy_is_loadable(li, region_start, seg, phys, 1)" not in try_load:
raise AssertionError("non-exact range fallback lost region narrowing")
if "lazy_candidate_preferred" not in try_load:
raise AssertionError("lazy artifact selection is still directory-order dependent")
runtime/tests/test_overlay_init_guard.py
+4 -2
View File
@@ -49,11 +49,13 @@ def require_before(body: str, guard_pattern: str, state_pattern: str, name: str)
def main() -> int:
source = SOURCE.read_text(encoding="utf-8")
# overlay_loader_dispatch guards, then hands the PC to the per-segment
# lookup (docs/SEGMENT_AWARE_CODE.md §5.7), which reads the index tables.
dispatch = function_body(source, "overlay_loader_dispatch")
require_before(
dispatch,
r"if\s*\(\s*!s_active\s*\)\s*return\s+0\s*;",
r"idx_head\s*\(",
r"(?:idx_head(?:_seg)?|overlay_loader_dispatch_seg)\s*\(",
"overlay_loader_dispatch",
)
@@ -69,7 +71,7 @@ def main() -> int:
require_before(
call_native,
r"if\s*\(\s*!s_active\s*\|\|\s*!s_native_exec\s*\)\s*return\s+0\s*;",
r"idx_head\s*\(",
r"idx_head(?:_seg)?\s*\(",
"overlay_loader_call_native",
)
runtime/tests/test_overlay_retry_c11.py
+2 -1
View File
@@ -15,6 +15,7 @@ end = text.index('int loaded_range_ci', start)
fragment = text[start:end]
with tempfile.TemporaryDirectory() as tmp:
source = Path(tmp)/'label.c'
source.write_text('static int idx_head(int p) { return p; }\nint main(void) { int phys=0; if (phys) goto retry_candidates;\n'+fragment+'return head; }\n')
# The retry label heads the per-segment lookup (idx_head_seg, §5.7).
source.write_text('static int idx_head_seg(int p, unsigned s) { return p + (int)s; }\nint main(void) { int phys=0; unsigned seg=0; if (phys) goto retry_candidates;\n'+fragment+'return head; }\n')
subprocess.run([shutil.which(args.compiler) or args.compiler,'-std=c11','-pedantic-errors','-fsyntax-only',str(source)],check=True)
print('production retry label: strict C11 passes')
runtime/tests/test_overlay_segment_gate.py
+205 -14
View File
@@ -1,17 +1,43 @@
#!/usr/bin/env python3
"""Build the real overlay loader and prove shards run only for their segment.
Overlay shards are compiled for the KSEG0 address of their bytes and bake that
segment into every PC they produce: jal/jalr link values, jump and exception
resume PCs, I-cache fetch tags. OpenBIOS's exception path enters its RAM patch
slots at KUSEG (0x0000281C), and a KSEG0 shard run there wrote $ra=0x8000282C
instead of 0x0000282C and missed I-cache lines the kernel had just filled with
KUSEG tags: R4 warm (native) and cold (interpreted) overlay-cache runs took the
next VBlank IRQ at different instructions and split at guest frame 194.
A shard bakes the segment it was compiled for into every PC it produces:
jal/jalr link values, jump and exception resume PCs, I-cache fetch tags and
store-PC stamps. OpenBIOS's exception path enters its RAM patch slots at KUSEG
(0x0000281C), and a KSEG0 shard run there wrote $ra=0x8000282C instead of
0x0000282C and missed I-cache lines the kernel had just filled with KUSEG tags:
R4 warm (native) and cold (interpreted) overlay-cache runs took the next VBlank
IRQ at different instructions and split at guest frame 194 (#417).
This drives the loader harness (overlay_pair_dedup_harness.c) with a fixture
shard at 0x80010000: KUSEG and KSEG1 aliases must fall to the interpreter
without calling the shard, and the KSEG0 PC must still run native.
docs/SEGMENT_AWARE_CODE.md §5.7 (rollout PR E) keys shards by segment: capture
records each dispatch's segment, compile_overlays.py builds one shard per
segment with entries, KUSEG and KSEG1 shards live in the seg-kuseg/ and
seg-kseg1/ subdirectories of the cache tag (KSEG0 keeps its old place and
names), and a manifest's `S <segment>` record names the segment its full-VA F
entries are in. This drives the loader harness (overlay_pair_dedup_harness.c)
through the real overlay_loader.c:
segment-alias a KSEG0 shard at 0x80010000 runs only for its KSEG0 PC; the
KUSEG and KSEG1 aliases fall to the interpreter (#417's case).
kuseg-only a KUSEG-compiled shard runs natively for its KUSEG PCs, entry
and CPS continuation, and the KSEG0 and KSEG1 aliases of the
same bytes are interpreted (the §7.1 acceptance case).
all-segments one shard per segment for the same bytes: each PC runs its
own segment's shard, loaded lazily from its directory, and
none pair-aliases another; PCs in segments that map no RAM
never run one.
rejected a manifest whose S record, or whose F entries, disagree with
its directory's segment is never indexed, loaded or
published.
stale a PC whose own segment's shard no longer matches live bytes
is interpreted, not handed to another segment's shard.
continuation a CPS continuation resumes its own segment's owner, even
where another segment has a function entry at that word,
and loads it lazily even when another segment's bundle is
newer.
capture wiring the interpreter records each dispatch's segment from its
full PC, and every clear of dispatch evidence clears the
segment bits too (a source check).
Usage: python runtime/tests/test_overlay_segment_gate.py
Exit 0 = PASS.
@@ -23,15 +49,91 @@ import argparse
import os
import pathlib
import platform
import re
import shutil
import sys
import tempfile
import zlib
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
import test_overlay_pair_dedup_runtime as pair # noqa: E402
KUSEG, KSEG0, KSEG1 = 0x00000000, 0x80000000, 0xA0000000
SEG_DIR = {KUSEG: "seg-kuseg", KSEG0: None, KSEG1: "seg-kseg1"}
INSTANCE = {KSEG0: 1, KUSEG: 3, KSEG1: 4}
SEG_DEFINE = {KSEG0: None, KUSEG: "TEST_SEGMENT_KUSEG", KSEG1: "TEST_SEGMENT_KSEG1"}
def segment_manifest(seg: int, *, s_record: int | None, f_seg: int | None = None,
length: int = 8, phys: int = 0x00010000,
stale: bool = False) -> str:
"""One function at `phys` covering `length` bytes. R stays a KSEG0-spelled
byte extent whatever the segment. `stale` records a CRC the live (zeroed)
RAM does not have."""
entry = (seg if f_seg is None else f_seg) | phys
crc = zlib.crc32(bytes(length) if not stale else b"\xFF" * length) & 0xFFFFFFFF
lines = []
if s_record is not None:
lines.append(f"S {s_record:08X}\n")
lines.append(f"P {pair.PAIR:016X}\n")
lines.append(f"F {entry:08X} {crc:08X}\n")
lines.append(f"R {KSEG0 | phys:08X} {length:08X}\n")
return "".join(lines)
def compile_segment_fixture(gcc: str, out: pathlib.Path, seg: int) -> None:
command = [gcc, "-std=c11", "-O0", "-Wall", "-Wextra", "-Werror", "-shared",
f"-I{pair.RUNTIME / 'include'}",
f"-DTEST_INSTANCE={INSTANCE[seg]}"]
if SEG_DEFINE[seg]:
command.append(f"-D{SEG_DEFINE[seg]}=1")
if platform.system() != "Windows":
command.append("-fPIC")
command += [str(pair.TESTS / "overlay_pair_dedup_fixture.c"), "-o", str(out)]
pair.run(command)
def publish(cache: pathlib.Path, subdir: str | None, library: pathlib.Path,
text: str, *, resident: bool) -> pathlib.Path:
ext = ".dll" if platform.system() == "Windows" else ".so"
leaf = cache / pair.GAME / "gcc" / pair.arch_abi() / pair.codegen_leaf()
if subdir:
leaf = leaf / subdir
leaf.mkdir(parents=True, exist_ok=True)
# The same {phys}_{crc} name in every segment's directory (decision 4).
target = leaf / f"00010000_11111111{ext}"
shutil.copy2(library, target)
target.with_suffix(".ranges").write_text(text, encoding="ascii", newline="")
if resident:
target.with_suffix(".resident").write_text(
pair.RESIDENT, encoding="ascii", newline="")
return target
def check_capture_wiring() -> None:
"""Capture records the segment of each interpreted dispatch from its full
PC, and no site drops dispatch evidence without dropping the segment bits
with it (a stale segment would name a segment the next epoch never ran)."""
src = (pair.RUNTIME / "src")
interp = (src / "dirty_ram_interp.c").read_text(encoding="utf-8")
start = interp.index("static int dirty_ram_dispatch_inner(CPUState* cpu, uint32_t addr, "
"uint32_t stop_addr) {")
body = interp[start:interp.index("\n}\n", start)]
any_bit = body.find("g_dirty_ram_dispatch_pc_bitmap[word >> 5] |=")
seg_bit = body.find("int seg = psx_code_segment_index(addr);")
seg_set = body.find("g_dirty_ram_dispatch_seg_bitmap[seg][word >> 5] |=")
if not (0 <= any_bit < seg_bit < seg_set):
raise AssertionError("dirty_ram_dispatch_inner must set the segment bit "
"from the full PC next to the dispatch bit")
for name in ("memory.c", "overlay_capture.c", "dirty_ram_interp.c"):
text = (src / name).read_text(encoding="utf-8")
if re.search(r"memset\(\s*&?g_dirty_ram_dispatch_pc_bitmap", text):
raise AssertionError(f"{name} clears dispatch evidence without its "
"segment bits (use dirty_ram_dispatch_evidence_clear)")
def main() -> int:
check_capture_wiring()
parser = argparse.ArgumentParser()
parser.add_argument("--gcc", default=shutil.which("gcc") or shutil.which("cc"))
args = parser.parse_args()
@@ -42,16 +144,105 @@ def main() -> int:
with tempfile.TemporaryDirectory(
prefix="psx-segment-gate-", ignore_cleanup_errors=True) as raw:
tmp = pathlib.Path(raw)
shard = tmp / f"shard{ext}"
harness = tmp / f"segment-harness{exe}"
pair.compile_fixture(args.gcc, shard, instance=1)
pair.compile_harness(args.gcc, harness)
cache = tmp / "cache"
os.environ.pop("PSX_PAIR_TEST_TRACE", None)
# #417: a KSEG0 shard, and no other segment's.
shard = tmp / f"shard{ext}"
pair.compile_fixture(args.gcc, shard, instance=1)
cache = tmp / "cache-alias"
first = pair.publish(cache, "gcc", f"00010000_11111111{ext}", shard,
pair.manifest([(0x80010000, 4), (0x80010004, 4)]))
os.environ.pop("PSX_PAIR_TEST_TRACE", None)
pair.run([str(harness), str(cache), "segment-alias", str(first),
str(tmp / f"unused{ext}")])
libs = {}
for seg in (KSEG0, KUSEG, KSEG1):
libs[seg] = tmp / f"seg-{seg:08X}{ext}"
compile_segment_fixture(args.gcc, libs[seg], seg)
# §7.1: a KUSEG shard alone, loaded lazily from seg-kuseg/.
cache = tmp / "cache-kuseg"
publish(cache, SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=KUSEG), resident=False)
pair.run([str(harness), str(cache), "segment-shards", "kuseg", "-"])
# All three segments for the same bytes: KSEG0 preloaded (resident),
# KUSEG and KSEG1 found lazily in their own directories. The KSEG0
# manifest has no S record, as every manifest before §5.7.
cache = tmp / "cache-all"
publish(cache, None, libs[KSEG0],
segment_manifest(KSEG0, s_record=None), resident=True)
publish(cache, SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=KUSEG), resident=False)
publish(cache, SEG_DIR[KSEG1], libs[KSEG1],
segment_manifest(KSEG1, s_record=KSEG1), resident=True)
pair.run([str(harness), str(cache), "segment-shards",
"kseg0,kuseg,kseg1", "-"])
# Disagreeing segments are never loaded, resident or lazy.
rejected = {
# A KUSEG shard whose manifest has no S record (so KSEG0).
"kuseg-no-s": (SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=None)),
# S names KSEG0 but the directory is seg-kuseg/.
"kuseg-s-kseg0": (SEG_DIR[KUSEG], libs[KSEG0],
segment_manifest(KSEG0, s_record=KSEG0)),
# A KUSEG manifest in the KSEG0 place.
"root-s-kuseg": (None, libs[KUSEG],
segment_manifest(KUSEG, s_record=KUSEG)),
# S says KSEG1, an F entry is KSEG0.
"kseg1-f-kseg0": (SEG_DIR[KSEG1], libs[KSEG0],
segment_manifest(KSEG1, s_record=KSEG1, f_seg=KSEG0)),
# S after an F record (the F is right for the S, not before it),
# and S twice.
"s-after-f": (None, libs[KSEG0],
segment_manifest(KSEG0, s_record=None) + "S 80000000\n"),
"s-twice": (SEG_DIR[KUSEG], libs[KUSEG],
"S 00000000\n" + segment_manifest(KUSEG, s_record=KUSEG)),
# S must name a segment base that maps RAM.
"s-kseg2": (SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=0xC0000000)),
"s-not-base": (SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=0x00010000)),
}
for name, (subdir, lib, text) in rejected.items():
for resident in (False, True):
cache = tmp / f"cache-{name}-{int(resident)}"
target = publish(cache, subdir, lib, text, resident=resident)
# Nothing indexes or loads it, and live publication refuses it.
pair.run([str(harness), str(cache), "segment-shards", "none",
str(target)])
# The KUSEG shard's bytes are stale, the KSEG0 shard's are live.
cache = tmp / "cache-stale"
publish(cache, None, libs[KSEG0],
segment_manifest(KSEG0, s_record=None), resident=True)
publish(cache, SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=KUSEG, stale=True), resident=True)
pair.run([str(harness), str(cache), "segment-stale", "-", "-"])
# 0x10004: a KSEG0 function entry, and a KUSEG CPS continuation.
cache = tmp / "cache-continuation"
publish(cache, None, libs[KSEG0],
segment_manifest(KSEG0, s_record=None, phys=0x00010004, length=4),
resident=True)
publish(cache, SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=KUSEG), resident=True)
pair.run([str(harness), str(cache), "segment-continuation", "-", "-"])
# Both unloaded, the KSEG0 bundle newer: the KUSEG continuation still
# loads the KUSEG shard.
cache = tmp / "cache-lazy-continuation"
k0 = publish(cache, None, libs[KSEG0],
segment_manifest(KSEG0, s_record=None), resident=False)
publish(cache, SEG_DIR[KUSEG], libs[KUSEG],
segment_manifest(KUSEG, s_record=KUSEG), resident=False)
newer = os.stat(k0).st_mtime + 60
for path in (k0, k0.with_suffix(".ranges")):
os.utime(path, (newer, newer))
pair.run([str(harness), str(cache), "segment-continuation", "lazy", "-"])
print("PASS: overlay shards run only for the segment they were compiled for")
return 0