diff --git a/Makefile b/Makefile index 81be2974f..7c9ff8866 100644 --- a/Makefile +++ b/Makefile @@ -661,7 +661,7 @@ build/src/ov_SC01_077/ov_SC01_077_o0.o: CC1FLAGS := -quiet -O0 -G0 -mips1 -mcpu= # Phase-24: the whale func_80144B9C is a 2nd -O0 region (0x80144B9C..0x801457A4) present in EVERY # overlay (reach-134), carved into its own object _o0b by each overlay's splat config; the # struct-assign memcpy matches only at -O0. One wildcard rule -O0-compiles all overlays' _o0b.o -# (the ×134 rollout; tools/rollout_whale_o0.py). All share src/shared/func_80144B9C.h. +# (the ×134 rollout; tools/rollout_whale_o0.py — one-shot, retired S45). All share src/shared/func_80144B9C.h. # # P30 T2: the glob is `_o0?` (was `_o0b`) so ANY lettered -O0 sub-split is covered by this one rule. # A 4th -O0 region was found inside an -O2 jr split (0x80183CF0..0x80184920, 15 contiguous fns in @@ -674,7 +674,7 @@ WHALE_O0B_OBJS := $(patsubst src/%.c,build/src/%.o,$(wildcard src/ov_*/ov_*_o0?. $(WHALE_O0B_OBJS): CC1FLAGS := -quiet -O0 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker # Phase-29 T2 Arm A: the -O0 cluster (0x8013B568..0x8013C98C) carved per single-file overlay into -# _o0.o (tools/rollout_o0_cluster.py) — same per-file -O0 mechanism so its h_seq family members +# _o0.o (tools/rollout_o0_cluster.py — one-shot, retired S45; the generic driver is tools/rollout_o0.py) — same per-file -O0 mechanism so its h_seq family members # bank whole-binary (the Task-1 swing verdict: they masked-MATCH only at -O0). One wildcard rule # -O0-compiles every overlay's _o0.o; ov_SC01_077_o0.o already has its explicit rule above (filtered # out to avoid a duplicate target-specific assignment). `*_o0.c` never matches the whale's `*_o0b.c`. diff --git a/docs/SETUP.md b/docs/SETUP.md index 7a1b4e5db..9dbd378f8 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -571,6 +571,27 @@ The reusable **flat-blob recipe** (every Gen2 overlay follows it): - Per-binary `_GHIDRA_PROG` → `make sig-refresh BINARY=`; `diff_settings.py` + the three report scripts gain a `` entry; `make expected` is per-binary-safe (merge-copy, no sibling clobber). +**Module-class binaries — `md_*` (P30 S44/S45; the §S44 loader table, `docs/memory-map.md`):** +the small type-1 payloads (per-actor modules, the SC07 endgame pair) load at their OWN statically +derived slots (A `0x800CAE08` · B `0x800CCB1C` · boot/resident `0x800CEDF8` · SC07 `0x801A00D8`), +not the shared overlay slot. Onboard with **`tools/new_binary.sh +[TEXT_LO]`** (the generalized `new_overlay.sh`; registry `config/modules.mk` / `MODULE_BINARIES`). +Module-specific facts the recipe encodes: +- **TEXT_LO ≠ 0** (the §154 module-id law: payload word0 is a global module id, sometimes followed + by a fn-ptr table and/or data): derive per payload from the first-prologue scan (`27BDxxxx`) and + the min fn-ptr-table target — NOT min-table alone (functions can precede the lowest table entry: + the SC07 pair's real code start is 0xFC/0x158, their min table targets 0x930/0x370). +- **The header carve is a dot-typed `.rodata` PAIRED with the c segment** (same name), never a + standalone `rodata, hdr` object and never `bin`: a module header can hold a function's JUMP + TABLE, whose `.L` labels only resolve when jtbl and function assemble in the SAME object (the + EXE `[0x63238,.rodata,800]` precedent); `bin` assets link in the data block (wrong placement). +- **A4 symbol-window law:** a module whose window lies INSIDE another binary's symbol region must + NOT stack that binary's symbol file — the boot trio (`0x800CEDF8`) omits `symbols.resident.txt` + (DsMix @0x800D1BD8 minted a phantom fn boundary in md_MAIN_011 before this). +- **`make sig-modules`** signs every module at its own vram/TEXT_LO, seeding from the built ELF's + `func_*` symbols when a build exists (bootstrap's linear partition glues adjacent functions + around jtbl dispatch); fresh-clone fallback is `--bootstrap`, self-healing on the next run. + ### §6.8 Cross-binary dedup & code-sharing (Phase 11) — "one match unlocks many" Full how-to in `docs/matching-cookbook.md` §11. Command crib: - **`make sig-overlays`** — Ghidra-FREE sign all 134 location overlays (`SCxx 0.4.dec`) at the shared overlay @@ -642,8 +663,8 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `ExportSymbols.java` | Dump curated symbols (feeds `config/symbols.us.txt`, R15). | | | `DumpProgramInfo.java` | Dump program metadata (loader, language, ImageBase, function count). | | | `DumpFunctionSignatures.java` | Dump function signatures (feeds `make sig-refresh`). | -| | `ImportOverlay.java` | Import an overlay segment into the project. | -| | `VerifyOverlay.java` | Verify an imported overlay against expected bytes. | +| | `ImportOverlay.java` | **RETIRED (S45, R33)** — 1-overlay-era hardcoded import; `tools/ghidra_import_raw.sh` is the live path. | +| | `VerifyOverlay.java` | **RETIRED (S45, R33)** — companion of ImportOverlay.java; retired with it. | | | `GetSymbolAt.java` | Read the symbol at a given address (scripted lookup). | | | `DecompileAt.java` | Decompile the function at a given address (scripted scaffold). | | | `DefineFunctions.java` | Disassemble + create functions at splat's validated entry points (`.run/_funcs.txt`) — completes a raw-blob program's function set (Phase 10). | @@ -664,7 +685,8 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/permuter/` | decomp-permuter harness (PERM_ recipes/weights) for stubborn near-misses. | | | `diff_settings.py` *(repo root)* | asm-differ config (arch `mipsel`, object mode vs `expected/`). | | | `tools/new_overlay.sh` | One-command location-overlay onboarding: ` [ENTRY]` — instantiate `config/splat..yaml` from the template (+ non-4-aligned `bin` carve), register the binary in `config/overlays.mk` + the report/diff dicts, `make extract && build` byte-check. Idempotent (Phase 13, cookbook §13). **Phase-27 T7:** the optional `ENTRY` arg (default `0.4`) reaches a non-`0.4.dec` payload — the 4 SC07 overlays put code at PAC entry 1 (`1.4`). difficulty.py dropped from the insertion set (it derives now, T6). | -| | `tools/disc_code_sweep.py` | **(Phase-27 T7)** Disc-completeness audit: decode every extracted PAC payload (reusing `sig_image.make_insn`) and flag code-bearing ones by BOTH `valid ≥ 0.90` AND `jr $ra` density `≥ 0.01` (the `jr $ra` gate is decisive — type-0/2 data decodes ~100% "valid" but has 0 returns). Reconciles the onboarded set against the disc — the R34 answer to "what code did nobody onboard". Findings → `docs/disc-completeness.md` (138/138 type-4 complete; **39 un-onboarded type-1 resident-class modules** pending load-address RE). | +| | `tools/disc_code_sweep.py` | **RETIRED (S45, R33)** — superseded by `tools/disc_audit.py` / `make audit-disc` (whole-payload, BOTH raw+LZSS layers, residue-0 partition, claimed-by derived from `config/check..sha`). The sweep read only the RAW layer through a 4,096-word window and had no notion of a claim; its historical findings are preserved in `docs/disc-completeness.md`. | +| | `tools/new_binary.sh` | **(P30 S44/S45)** One-command onboarding for ANY flat-blob binary class: ` [TEXT_LO]` — ov_* (overlay slot, registry `overlays.mk`) or md_* (own §S44 slot, registry `modules.mk`). Signs (TEXT_LO-aware §154), instantiates the shared template (non-zero TEXT_LO ⇒ paired-`.rodata` header carve), writes check.sha + symbols, registers in the registry + the 3 report/diff dicts (sentinel-anchored, ast-checked), extract+build byte-check. Idempotent. `new_overlay.sh` is now a thin wrapper over it. §6.7 module recipe. | | **PsyQ library linking** (cookbook §8/§9) | `tools/psyq_lib_split.py` | Split a PsyQ `.LIB` into per-object members. | | | `tools/psyq_build_libs.sh` | Build the PsyQ libs from split members. | | | `tools/psyq_identify.py` | Identify which SDK objects a region's functions belong to. | diff --git a/docs/disc-completeness.md b/docs/disc-completeness.md index e9ca1a3e4..d4496a2fc 100644 --- a/docs/disc-completeness.md +++ b/docs/disc-completeness.md @@ -82,7 +82,11 @@ mostly-unmatched code to the denominator; every "×134" family propagation is no honest direction — the prior number was measured over an incomplete disc. ## Reproduce +> **⚠️ S45: `tools/disc_code_sweep.py` is RETIRED (R33)** — superseded by `make audit-disc` +> (`tools/disc_audit.py`): whole-payload classification at BOTH the raw and LZSS layers, a +> residue-0 partition over the disc, and `claimed-by` derived from `config/check..sha`. +> The commands below are kept for provenance of the 2026-07-15 findings; they no longer run. ``` -tools/disc_code_sweep.py # the table above + the ranked hidden list -tools/disc_code_sweep.py --types 1 # just the type-1 code modules +tools/disc_code_sweep.py # (retired) the table above + the ranked hidden list +tools/disc_code_sweep.py --types 1 # (retired) just the type-1 code modules ``` diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index f4b664b43..fc06b92f5 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -224,7 +224,17 @@ stub on a named wall/behemoth/queue ledger** — 140/140 byte-identical througho (12,682,962/13,487,439) · 95.96% fn-count · 87.6% distinct (78,596/90,143 uniq).** The drop from S44's 94.4% is the denominator growing +52k module ins (honest direction, P27/S39/S44 precedent). -- **II.2** (retirements + SETUP module recipe) — next. +- **II.2 ✅ — retirements (R33) + SETUP module recipe.** Deleted: `disc_code_sweep.py` (superseded + by `disc_audit.py`/`make audit-disc`), `reconcile_decls.py` (superseded by `reconcile_tu`; + its incumbent row removed from cdecl's differential — the audit's purpose was to enable exactly + this deletion), the 3 one-shot rollout drivers (`rollout_801457a4_o0` / `rollout_whale_o0` / + `rollout_o0_cluster` — `rollout_o0.py` is the live generic), `ImportOverlay.java` + + `VerifyOverlay.java` (`ghidra_import_raw.sh` is the live path). Reference check first: the + plan's "zero build refs" was wrong for 3 of them (comments + one live cdecl import — handled, + gate re-proven green). SETUP §6.7 gained the **module-class recipe** (TEXT_LO derivation law, + paired-.rodata header carve, A4 symbol-window law, ELF-seeded sig-modules) + inventory rows + (R21); disc-completeness Reproduce marked retired; Makefile comments annotated. +- **II.3** (metrics re-baseline + roadmap delta + decision-log) — next. --- diff --git a/tools/cdecl.py b/tools/cdecl.py index 8899ea54a..36857ef93 100644 --- a/tools/cdecl.py +++ b/tools/cdecl.py @@ -1239,7 +1239,8 @@ def audit_differential(): tool. The reverse (symbols only this one sees) is the measured size of the hole.""" sys.path.insert(0, os.path.join(REPO, 'tools')) import gen_harvest_targets as ght - import reconcile_decls as rd + # reconcile_decls RETIRED (S45, R33): its DATA_DECL_LINE_RE row proved the superset for + # 26-A..S44; the incumbent is deleted, so the differential now compares the survivors only. import sig_unify as su ec = os.path.join(REPO, 'src/shared/engine_core.h') @@ -1268,7 +1269,6 @@ def audit_differential(): mine_d = {n for n in mine if n.startswith('D_')} rows = [] for name, rx in (('gen_harvest_targets.DATA_DECL_RE', ght.DATA_DECL_RE), - ('reconcile_decls.DATA_DECL_LINE_RE', rd.DATA_DECL_LINE_RE), ('sig_unify.DATA_DECL_RE', su.DATA_DECL_RE)): theirs = set() for m in rx.finditer(text): diff --git a/tools/disc_code_sweep.py b/tools/disc_code_sweep.py deleted file mode 100644 index ee9e868c7..000000000 --- a/tools/disc_code_sweep.py +++ /dev/null @@ -1,166 +0,0 @@ -#!/usr/bin/env python3 -"""disc_code_sweep.py — find code-bearing PAC payloads the onboarding may have missed (Phase-27 T7). - -The disc-completeness question the byte-gate is structurally blind to (R34): the build only touches -binaries someone ONBOARDED, so a code payload no one onboarded is invisible to `make check-all` no -matter how green it is. The Phase-27 audit found four such overlays (SC07 FILE_006/007/010/011, code -at PAC entry 1 not 0). This sweeps EVERY extracted PAC payload and reports which decode as MIPS code, -so the onboarded set can be reconciled against the disc rather than trusted. - -Method (reuses sig_image's exact rabbitizer decode): decode the first `--window` words of each raw -payload as MIPS-LE, report the fraction rabbitizer calls valid. Overlay/EXE code runs ~0.97-1.00 -valid; data/graphics/audio sit far lower. A payload above `--threshold` that is NOT already onboarded -is a candidate the audit must explain (onboard it, or record why it is not a build binary — e.g. a -type-1 blob that loads at its own address like the resident, needing load-address analysis first). - -COVERAGE-ASSERTED (R32): every extracted `{index}.{type}` payload is classified; the onboarded set is -cross-checked so a hit that is already a binary is labelled, not re-flagged. - - tools/disc_code_sweep.py # sweep all types, print the report - tools/disc_code_sweep.py --types 1,6,7 # only the non-type-4 unknowns -""" -import argparse -import glob -import os -import re -import struct -import sys - -sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) -sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "bfm_extract")) -import sig_image # make_insn — the shared rabbitizer decode (GTE-aware) -import lzss # decompress — the SAME game-semantics decoder the extractor uses - -REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) -# a raw payload is /. with NO .dec suffix; type-4 also has a .dec sibling -_PAYLOAD = re.compile(r"/(\d+)\.(\d+)$") - - -def onboarded_payloads(): - """The set of payload paths already wired as a build binary (main, resident, every overlay). - Read from the Makefile/overlays.mk *_EXE assignments — the single source of truth (R33). - - KEYED BY THE RAW PAYLOAD PATH (Phase-28 T7). A type-4 overlay's `_EXE` is the DECOMPRESSED path - (`…/0.4.dec`), but this sweep iterates the RAW payloads (`…/0.4`) so it can decode the compressed - layer itself. Store the `.dec`-stripped form too, or all 138 type-4 overlays — now correctly seen - as code via decompression — would fail the onboarded match and flag as false HIDDEN hits.""" - paths = set() - for mk in ("Makefile", "config/overlays.mk"): - p = os.path.join(REPO, mk) - if not os.path.exists(p): - continue - for m in re.finditer(r"^\w+_EXE\s*:=\s*(\S+)", open(p).read(), re.M): - norm = os.path.normpath(m.group(1)) - paths.add(norm) - if norm.endswith(".dec"): - paths.add(norm[:-len(".dec")]) # the raw payload this .dec came from - return paths - - -_JR_RA = 0x03E00008 # `jr $ra` — a function return; the discriminator rabbitizer.isValid() lacks - - -def _signals(data, window): - n = min(window, len(data) // 4) - if n == 0: - return 0.0, 0.0, 0 - ok = jr = 0 - for k in range(n): - word = struct.unpack_from("