docs/SEGMENT_AWARE_CODE.md §5.7. compile_overlays.py splits each capture
into one view per segment with entries (capture_segment_views). A v2
capture has no dispatch_entry_segments and comes back unchanged, KSEG0 only;
in v3, a dispatch entry that no segment list names is a KSEG0 entry. A view
spells load_addr and every PC in its segment, holds only that segment's
dispatch entries, and carries all the execution evidence, which has no
segment. A malformed segment record fails that capture loudly.
Two kinds of record also get a view where they saw no dispatch entry:
- a record with no dispatch entry in any segment (execution evidence only,
which the runtime writes) is one KSEG0 view, exactly as a v2 reader reads
it, so classification, forced interiors, declared entries and prior
manifests apply as before;
- declared_view_segments(): a --force-interior PC inside the record (KSEG0)
asks for the KSEG0 view, and a game.toml [[overlays]] table asks for the
segment its load address, or else each of its entries inside the record,
is spelled in.
Each view compiles like any capture, region and fragment passes alike:
- the image is compiled at `segment | phys`, so the recompiler names and
bakes it in that segment;
- helpers that spell PCs without an image address use the view's segment
(image_segment(); KSEG0 by default, so a KSEG0 compile is unchanged);
- the shard lands in segment_cache_dir(): the tag directory for KSEG0,
seg-kuseg/ and seg-kseg1/ otherwise.
The manifest writer adds `S <segment>` for KUSEG/KSEG1. The runtime-contract
parser keeps F entries' full VA, applies the S rules, and refuses a
manifest whose segment differs from its directory's. Exports are looked up
by their full VA.
Capacity accounting counts each leaf's segment directories in the leaf's
namespace, since the runtime's candidate table is shared, and tells shards
apart by segment and name. The static generator emits the exact PC per row,
so KUSEG, KSEG0 and KSEG1 rows of one word coexist, with a segment gate
over the segments it compiled. A KSEG0-only table emits the same text as
before. Non-KSEG0 static namespaces are ov_s0_ (KUSEG) and ov_s5_ (KSEG1).
The static isolated-fragment pass keys its images by segment too, so each
(entry, image) demand compiles against its own segment's view and one entry
dispatched in two segments gets a fragment in each. --force-interior PCs
stay KSEG0 and force KSEG0 views.
coverage_vault.py unions dispatch_entry_segments when it merges variants,
making a v2 record's entries explicit KSEG0 entries, and crops it when it
compacts regions.
Output for v2 captures is unchanged: the 52 R4 shard sources and manifests
compiled from D's race captures are byte-identical to D's.
Tests:
- new overlay_segment_keys (CTest). It runs capture views, manifests and the
cache layout, then the real compile_overlays.py driver, in process, on a v3
capture of an overlay entered at KUSEG and KSEG1, and checks:
- each shard lands in its directory with S and full-VA F records;
- every baked PC is in its segment, and KSEG1 charges a fetch per
instruction;
- a second run is served from each segment's cache;
- a KSEG0 view compiles byte-identically to the same bytes captured as v2;
- --static emits per-segment rows and namespaces, and an isolated
fragment per segment for an entry with no callable boundary;
- a dispatch-less v3 record with --force-interior compiles byte-identically
to its v2 reading, and a KUSEG-only record still gets its forced KSEG0
fragment;
- a KUSEG/KSEG1 view compiled into a cache holding a KSEG0 manifest of the
same bytes merges none of it, and the empty-primary reconciliation counts
only the view's own segment's fragments;
- the real loader runs the shards: links, store-PC stamps and fetch tags in
each segment, fetch cycles equal to Beetle's model, KSEG0 interpreted.
- coverage_vault: segment unions and crops.
- compile_overlays_static_split: a KUSEG row beside a KSEG0 row dispatches
by exact PC; unmapped-segment entries are refused. This replaces the
KSEG0-only rejection.
- aot_overlay_discovery: the capacity namespace lists the segment
directories.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Overlay shards are compiled at the KSEG0 address of their bytes: captures
record 0x80000000|offset and the manifest parser keys entries there. The
emitter bakes that segment into every PC a shard produces: the $ra a
jal/jalr writes, jump and exception-resume PCs, and I-cache fetch tags.
Dispatch matched candidates by the segment-stripped address, so a KUSEG or
KSEG1 alias of the same bytes ran the KSEG0 body at the wrong
architectural PC.
Found in R4 (SLUS-00797) with OpenBIOS: the exception handler enters its
RAM patch slots at KUSEG (0x0000281C, 0x0000357C). The kernel-page shard
linked $ra=0x8000282C where the interpreter (and hardware) gives
0x0000282C, and fetched 0x8000281C/0x8000357C, missing lines the kernel
had just filled with KUSEG tags: +4 cycles per slot, so the next VBlank
IRQ landed at a different instruction and warm (native) vs cold
(interpreted) overlay-cache runs saved different registers from frame
194 on. A KSEG1 entry would also be charged as cached.
PSX_OVERLAY_CODE_SEGMENT (overlay_loader.h) names the rule for capture,
the manifest parser, overlay_loader_dispatch and the generated static
overlay table: a PC outside it falls to the interpreter, which runs at
the PC it was handed. CPS continuations already failed closed this way
(their entry switch only knows KSEG0 PCs); function entries now match.
overlay_loader_status reports segment_alias_interp.
Static game code has the same shape for KUSEG-linked EXEs (the parser
normalizes their headers to KSEG0); that needs segment-relative emission
and is not changed here. Shard cache tags are unchanged.
Tests: test_overlay_segment_gate.py drives the real loader (pair-dedup
harness, which also gains the RAM-geometry globals it no longer linked
without) with a KSEG0 fixture shard: KUSEG and KSEG1 dispatches stay
interpreted and never call it, KSEG0 still runs native. It fails with the
gate removed. test_compile_overlays_static_split.py covers the generated
static table and rejects a non-KSEG0 entry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_compile_overlays_static_split.test_data_only_capture_reports_skip_with_captured_log
has been red on master since 4683e923 (2026-09-12). That commit reworded both
emit sites deliberately -- "no walk-root seeds (data-only region)" became
"no shared walk-root seeds; checking fragments separately", because the region
is no longer declared data-only: its executed dispatch demands are still
served by the isolated-fragment pass. The assertion still names the old text,
so the test fails on a clean checkout:
AssertionError: 'SKIP: no walk-root seeds' not found in '... SKIP: no
shared walk-root seeds; checking fragments separately ...'
The literal was duplicated at the two emit sites and copied a third time into
the test, so rewording two of the three left the third stale with nothing to
catch it. Give it a name -- NO_SHARED_WALK_ROOT_SEEDS_SKIP -- use it at both
emit sites, and have the test assert the constant rather than a copy of its
text. A future reword then updates all three at once or fails loudly.
The printed output is unchanged, verified byte for byte against the pre-change
text including both leading spaces and the trailing blank line.
tools/test_compile_overlays_static_split.py: 10 tests, was 9 passed 1 failed,
now 10 passed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Static mode (--static) was serial end to end and produced one translation
unit, so a whole title compiled on one core twice: once in Python, once in
gcc. Measured on BoF3 (339 captures, 79,688 identities) in a build-dbg loop:
phase "compile all overlay bands" ~12 min, runtime build ~4 min at -O0.
Three changes, each independently verified byte-identical:
1. static_capture_job(): the per-capture static work (recompiler subprocess,
C audit, range audit, namespacing) is now a module-level, picklable job
that returns its part + captured log. --jobs runs it in a
ProcessPoolExecutor; results merge in capture order, so static_parts and
entry sources are order-identical to the sequential loop. --jobs 1 and
--jobs 3 on three bands: diff -r of the outputs is empty.
2. add_cps_resume_cases(): the universal CPS resume-wrapper pass called
add_cps_resume_case() once per entry, and each call re-scanned and
rebuilt the WHOLE part source. cProfile on three bands: 7,386 entries ->
22,159 re.search calls, 366 s of 424 s total. The batch form locates
every signature once, edits each host's segment in isolation, rebuilds
once. Same three bands: 418 s -> 5 s, outputs byte-identical (diff -r
against the per-entry output). The per-entry function is kept; a unit
test asserts the two agree on fresh-switch, existing-switch, missing
label and missing host cases.
3. write_static_outputs(): overlays_static.c now holds the dispatcher plus an
extern prototype per compiled entry, and every part becomes its own unit
overlays_static_NNNN.c (each already carries its #includes and a private
namespace). runtime.cmake globs the siblings with CONFIGURE_DEPENDS, so a
changed part count needs no reconfigure. Unchanged units are not
rewritten (mtime preserved -> not recompiled); stale units are removed.
--static-single-file restores the monolithic file.
Result, same BoF3 loop on a 12-core box: phase 5a 12 min -> 20 s, runtime
build 4 min -> 69 s (358 units in parallel), whole loop ~16 min -> 90 s.
Shard summary unchanged (332 ok / 7 benign audit fails), identity count
unchanged. Headless boot of the rebuilt runtime: 485,650 static checks,
485,244 hits (99.92%), miss_total 0, 60 emu fps through the LOGO overlay.
Note: tools/test_compile_overlays_additive.py already fails on master (it
references removed functions: compile_and_publish_dll, load_additive_captures,
validated_cached_bundle_ids); untouched here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Shokunin