# Ahead-of-time overlay sharding Overlays can be recompiled before gameplay without first decompiling the game. The necessary knowledge is narrower: which disc bytes become executable, where they load, how they are transformed, and which entry points and dependencies can be established. Full source recovery is not a prerequisite. The goal is to prepare native coverage from original inputs so first visits do not depend on runtime capture and compilation. Interpreter and runtime-compiler fallbacks remain useful until individual titles demonstrate sufficient coverage. An inventory of every overlay file is not proof of every execution path. ## What exists today The framework already separates discovery from compilation: | Layer | Existing implementation | Responsibility | | --- | --- | --- | | Disc/container discovery | `tools/aot_overlay_spike/extract_generic.py`, `tools/extract_overlays.py` | Read disc files and recognized archives; recover candidate load layouts and static roots. | | Native production | `tools/compile_overlays.py`, recompiler function analysis | Walk code, recover supported indirect dispatch, compile and audit native candidates. | | Cache selection | Runtime overlay loader, `.dll`/`.ranges` pairs | Match the game, build/flavor ABI and current guarded bytes before dispatch. | | Release staging | `tools/release_overlay_stage.ps1` and shared packaging helpers | Stage matching cache namespaces and fallback tooling. | The extractor lives under `aot_overlay_spike`: it is useful tooling with partial format support, not a universal overlay detector. Per-title evidence and recipes are still required when an input format or loader layout is ambiguous. The production pipeline below adds declarative profiles for verified methods; arbitrary game loaders still require bounded investigation. The compiler's input option is named `--captures` for historical reasons. It also accepts records produced entirely from disc bytes. That filename/schema does not imply a playthrough or RAM capture is required. ## Production pipeline and declarative profiles `tools/aot_overlay_pipeline.py` owns extraction, inventory checks, native builds, original-byte auditing and staging. Games supply JSON facts with schema `psxrecomp AOT methods v1`; they do not carry copies of extractors or auditors. The pipeline contains no game-ID branches. ```sh python tools/aot_overlay_pipeline.py release \ --profile /game/aot/overlays.json --game-toml /game/game.toml \ --runtime-config /game/packaging/release/game.toml \ --recompiler /framework/recompiler/build/psxrecomp-game \ --work-dir /private/aot-build --stage /release/payload --gcc gcc --workers 3 ``` Use `extract` instead of `release` and omit `--stage` for bounded discovery. Every invocation creates a fresh work directory and reads the original disc; existing shards and gameplay captures are never inputs. `--runtime-config` selects the packaged code-generation settings/cache namespace while the source project supplies BIOS profiles and extraction inputs. Never rename cache tags to force compatibility. Python 3.11 or newer is required. Profiles declare these reusable methods: | Field / method | Contract | | --- | --- | | `disc_hashes` | Hash the cue's original data track before extracting. | | `disc_set` | Instead of `disc_hashes`: `[{name, disc_hashes}]`, every disc of a release whose executable images are identical. Any one matches; the images are pinned by their own content hashes, so which disc was read does not change the output. | | `file_aliases` | `{logical: [per-disc names]}` for one file that discs name differently (`SCUS_944.91`, `SCUS_945.84`, ...). Exactly one candidate must exist; the logical name reads as that file everywhere in the profile. | | `prepared_files[]` | Virtual files derived from disc bytes by a declared transform (see below). Each is verified, then usable as a `file` by every method and check. | | `cross_image_call_roots` | Opt in (`true`): a `jal` target that other images call becomes a static discovery root of the image holding it, when the evidence at the target is the same as for local calls and exactly one image occupies that address. Finds routines only an overlay (or boot code) calls. | | `generic_exclude_files[]` | `{files, reason}`: disc files the generic detector must not consider because the profile owns them (aliased program files are excluded automatically). | | `images[].method = fixed_address_files` | Whole original files placed at a verified `load_addr`. Optional duplicate-leaf checks require identical bytes. | | `images[].method = fixed_address_extents` | Declared `{file, file_offset, base, address, size, sha256}` byte extents placed at a verified `load_addr`. Offset rules match `words` checks; size and SHA-256 are required. | | `mod_packages[]` / `mod_package` | A named, verified selection of a trusted mod package. Images and checks carrying `mod_package` read the original disc with that selection applied; see below. | | `images[].transfer_entries` | `{from: "mod_package_writes", count}`: J/JAL targets inside the image, taken only from words the selected package wrote. The count pins the inventory. Optional `exclude_writes: [{start, end, reason}]` names intervals of package-written DATA (for example a pack linked into the EXE that the package replaced) whose words are not transfers. | | `images[].method = tagged_relocated_files` | Apply a trailing two-bit ABS32/HI16/LO16/J26 relocation stream to a verified `load_addr`, retaining only the original loader's image interval. Destinations are explicit evidence, never inferred from a historical RAM capture. | | `images[].method = counted_relocated_members` | Verify each original container's SHA-256, classify every counted member, and relocate its image at each separately established heap placement. Pin `image_size` and `relocation_count`; each member needs nonempty unique `placements` or an explained exclusion. New placements remain interpreted. See [counted containers](COUNTED_MIPS_RELOCATIONS.md). | | `images[].entry_pointer_offsets` | Aligned image-relative offsets of loader-established callable exports. Read the pointer after relocation, ignore null callbacks, and require every nonnull callback to target aligned code inside the image. Pointer-shaped script/data fields are not callable exports. | | `images[].method = psx_exe` | Read the load address and image from a PS-X EXE header; the generic extractor may split resident and overlay floors. | | `images[].method = packed_sector_members` | Decode u32 count/offset sector descriptors across ordered payload files. Verify every member classification and requested full-payload coverage; use loader-established addresses for executable members. | | `images[].method = sector_extent_members` | Decode `{sector offset, byte size}` pairs in an archive. Use archive-relative offsets, verify the exact count and terminator, reject gaps/overlaps, and account for every member after sector rounding. Each member needs a verified load address or an explicit exclusion. | | `images[].method = aligned_lzss_banks` | Read aligned stored-size pairs, decode the metadata and its tagged bank members using parameterized LZSS, verify every container and bank classification, and deduplicate exact decoded images at verified destinations. External-RAM references fail extraction. | | `images[].method = indexed_lzss_members` | Read a u32 count and `{offset, size, stored}` descriptors, decode every member (verbatim when `size == stored`, otherwise bit-stream LZSS with declared parameters), require exact sizes, zero padding and a fully accounted file, and classify every member as an executable image at a verified `load_addr` or an explained exclusion. | | `checks[].method = words` | Verify loader instructions or descriptors at explicit file offsets / virtual addresses. | | `checks[].method = tagged_relocations` | Validate a complete relocation stream, retained `image_size` and `relocation_count`, including modules whose heap placement remains unresolved and which are excluded from native production. | | `checks[].method = pointer_strings` | Verify a pointer-indexed filename table against exact expected strings. Optional `stride` (default 4, a multiple of 4) reads tables of `{id, name*}` records. | | `checks[].method = bcd_extent_table` | Verify an indexed BCD-MSF/size table against an ISO file's actual extent. | | `checks[].method = adjacent_files` | Require the named ISO extents to be sector aligned and physically adjacent, in order. | | `compositions` | Combine declared simultaneous producers in address order. `max_gap` bounds their separation; gap and alignment bytes remain unowned. | | `expected_records` | Stop on inventory drift, including unexpected additions requiring review. Every configured image must be represented. | | `strict_bounds` | Require emission to remain inside established producer intervals. | | `images[].excluded_ranges` | Explicit fallback intervals with aligned start/end, original-byte SHA-256 and a reason. Requires strict bounds; required loader entries cannot be excluded. | Image addresses must lie in the KSEG0 main-RAM decode window `0x80000000..0x80800000`. Extraction recipes describe bytes and KSEG0 entries, so their shards are KSEG0 shards. Code a title runs at KUSEG or KSEG1 gets shards of that segment from runtime captures, which record each entry's segment (docs/SEGMENT_AWARE_CODE.md §5.7). Retail 2 MiB hardware mirrors DRAM across all of it and expanded 8 MiB targets decode it uniquely, so an image a loader places in a mirror is valid; the runtime gates it on the folded bytes the CPU executes. On retail RAM an image must also fit inside one 2 MiB mirror: one that crosses a mirror boundary (or the 2 MiB end) has no contiguous backing, so the pipeline rejects it rather than emit a variant the runtime gate can never accept. A profile whose images need the opt-in 8 MB map declares `"main_ram_bytes": "0x800000"` (default `0x200000`). `allow_missing` opts a fixed-address source into static root discovery when the generic detector misses it. `entry_word` or `entries` can supply independently verified loader entry points. These facts do not turn every plausible code word into a proven function. There is no recipe option to ignore a failed compiler. Use `dispatch_entries` for independently verified dispatch PCs that may be inside an existing function, such as jump-table case labels. They are required by the coverage audit but do not force new function boundaries. The existing decoder and reachability checks decide whether each PC is a callable root or a hosted interior; an interior declaration cannot truncate the function that owns it. Each recipe compiles in an isolated cache. `tools/audit_aot_cache.py` checks every native pair's ABI/exports, every manifest guard against known original bytes, and nonzero native coverage for every recipe. Loader entry points supplied by the profile must also have native entries guarded by that image's original bytes. Staging copies precisely the audited platform pairs, verifies hashes again, rejects extras, and emits `AOT_CACHE_AUDIT.json` without game bytes. A missing disc, changed loader word, missing image, compiler failure, empty recipe coverage or failed audit stops release packaging. Pair counts are results, not inherited success thresholds. An inventory plus valid guards still does not prove complete static execution coverage or native semantics; keep fallback and perform gameplay spot checks. Jersey Devil uses `aligned_lzss_banks` for five renderer variants across 70 bank occurrences and `psx_exe` for five secondary executables. Its profile accounts for all 93 BZZ containers, including 23 without code banks. Exact-BIOS resident helper recipes also publish a `.resident` preload marker. The pipeline preserves it through consolidation and staging, verifies its BIOS hash against the recipe, and includes its hash in the audit receipt. Omitting that marker can leave an otherwise valid helper unused at runtime. Current consumers: - Tomba! USA: `TombaRecomp/aot/overlays.json`, 25 fixed-address images, including 17 area variants and eight support images. - Tomba! 2 USA / Tombi! 2 Italian: `Tomba2Recomp/aot/{usa,ita}.json`, each with MAIN.EXE and 28 raw images, including all 22 area overlays. Their loader tables establish different regional addresses; composition uses the same shared method (including the Italian gap). US START.BIN is explicitly accounted for. - Ape Escape USA: `ApeEscapeRecomp/aot/overlays.json`, 47 established packed sector-table overlays plus two minigame PS-X EXEs. Two additional archive members have unresolved load paths and are explicitly excluded. - The Legend of Dragoon USA: `aot/overlays.json`, `disc_set` over all four discs (identical executable images). The BPE launcher's embedded container is the engine (`prepared_files`, two code extents at 0x80010000) and the ten BPE overlays fill slot A (0x800C6688) or slot B (0x800FB778); all linked with `static`. - WipEout 3 SE PAL: `aot/overlays.json`, the front-end menu overlay (member 5 of `WIPEOUT3/NFE.PB`, `indexed_lzss_members`) from the stock disc and from the NTSC / PAL Mode package's patched disc, plus that package's engine image; all linked with `static` (see below). The other 88 packs on the disc (`TRACK*.PBP`) hold data only. ### Prepared images: packed executables and overlays Some engines run code that no disc file holds verbatim: a launcher EXE that unpacks the real executable, overlays stored compressed. The loader's transform is deterministic, so its output is an original input like any other, provided the transform and every byte it reads and writes are pinned. `prepared_files` declares such outputs; the pipeline rebuilds them from the user's own disc every time and nothing derived is stored or committed. The profile carries hashes, offsets, sizes and addresses only. ```json "prepared_files": [{ "name": "PREPARED/ENGINE", "method": "bpe_container", "source": {"file": "BOOT.EXE", "file_offset": "0xA00"}, "magic": "0x1A455042", "consumed": "0x36228", "source_sha256": "...", "size": "0xD7800", "sha256": "..." }] ``` The decoder must consume exactly `consumed` bytes, whose SHA-256 is `source_sha256`, and produce exactly `size` bytes with SHA-256 `sha256`; any difference stops the run. A prepared name may not be a real disc file or alias, and a later entry may read an earlier one. Prepared names then work in `fixed_address_files`, `fixed_address_extents`, `words` and `pointer_strings`, so the loader code that fixes the destination is verified against the very bytes that run. Methods: `bpe_container` (`tools/bpe_container.py`): block-wise byte-pair expansion, `u32 size, u32 magic, {u32 block size, pair table, symbols}*, 0`. The magic is a required parameter. A packed image is rarely all code (its BSS, tables and replaced overlay area are data). Use `fixed_address_extents` on the prepared file to name only the code runs, each with its `sha256`, so data is never a producer: a data run decoded as code fails the generated-C audit and costs the whole image. ### Executable images from a mod package A trusted package can add code that no disc file contains: patched boot-EXE text, or an engine the patched EXE copies elsewhere. Such bytes are still original inputs plus declared, hash-verified operations, so they get the same treatment as disc methods. `tools/mod_package_images.py` reproduces a package's selected operations; the pipeline never reads a package it was not told about. ```json "mod_packages": [{ "name": "engine-full", "manifest": "mods///manifest.toml", "manifest_sha256": "", "id": "", "version": "", "features": {"": {"