mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-29 15:18:24 -04:00
chore(phase-30 S45 II.2): retirements (R33) + SETUP module recipe
- DELETED: disc_code_sweep.py (superseded by disc_audit/make audit-disc), reconcile_decls.py (superseded by reconcile_tu; incumbent row removed from cdecl audit_differential — the differential existed to prove this deletion safe), rollout_801457a4_o0/rollout_whale_o0/ rollout_o0_cluster one-shots (rollout_o0.py is the live generic), ImportOverlay.java + VerifyOverlay.java (ghidra_import_raw.sh is the live path) - reference check first (R14): the plan's 'zero build refs' was wrong for 3 — comment refs annotated, the one LIVE import (cdecl) reworked; audit-cdecl + tools-health re-proven green - SETUP §6.7: module-class recipe (TEXT_LO derivation, paired-.rodata hdr carve, A4 symbol- window law, ELF-seeded sig-modules) + new_binary.sh inventory row + 3 RETIRED rows (R21); disc-completeness Reproduce marked retired
This commit is contained in:
Drew T
@@ -661,7 +661,7 @@ build/src/ov_SC01_077/ov_SC01_077_o0.o: CC1FLAGS := -quiet -O0 -G0 -mips1 -mcpu=
|
||||
# Phase-24: the whale func_80144B9C is a 2nd -O0 region (0x80144B9C..0x801457A4) present in EVERY
|
||||
# overlay (reach-134), carved into its own object <ov>_o0b by each overlay's splat config; the
|
||||
# struct-assign memcpy matches only at -O0. One wildcard rule -O0-compiles all overlays' _o0b.o
|
||||
# (the ×134 rollout; tools/rollout_whale_o0.py). All share src/shared/func_80144B9C.h.
|
||||
# (the ×134 rollout; tools/rollout_whale_o0.py — one-shot, retired S45). All share src/shared/func_80144B9C.h.
|
||||
#
|
||||
# P30 T2: the glob is `_o0?` (was `_o0b`) so ANY lettered -O0 sub-split is covered by this one rule.
|
||||
# A 4th -O0 region was found inside an -O2 jr split (0x80183CF0..0x80184920, 15 contiguous fns in
|
||||
@@ -674,7 +674,7 @@ WHALE_O0B_OBJS := $(patsubst src/%.c,build/src/%.o,$(wildcard src/ov_*/ov_*_o0?.
|
||||
$(WHALE_O0B_OBJS): CC1FLAGS := -quiet -O0 -G0 -mips1 -mcpu=3000 -mgas -msoft-float -fgnu-linker
|
||||
|
||||
# Phase-29 T2 Arm A: the -O0 cluster (0x8013B568..0x8013C98C) carved per single-file overlay into
|
||||
# <ov>_o0.o (tools/rollout_o0_cluster.py) — same per-file -O0 mechanism so its h_seq family members
|
||||
# <ov>_o0.o (tools/rollout_o0_cluster.py — one-shot, retired S45; the generic driver is tools/rollout_o0.py) — same per-file -O0 mechanism so its h_seq family members
|
||||
# bank whole-binary (the Task-1 swing verdict: they masked-MATCH only at -O0). One wildcard rule
|
||||
# -O0-compiles every overlay's _o0.o; ov_SC01_077_o0.o already has its explicit rule above (filtered
|
||||
# out to avoid a duplicate target-specific assignment). `*_o0.c` never matches the whale's `*_o0b.c`.
|
||||
|
||||
+25
-3
@@ -571,6 +571,27 @@ The reusable **flat-blob recipe** (every Gen2 overlay follows it):
|
||||
- Per-binary `<bin>_GHIDRA_PROG` → `make sig-refresh BINARY=<bin>`; `diff_settings.py` + the three
|
||||
report scripts gain a `<bin>` entry; `make expected` is per-binary-safe (merge-copy, no sibling clobber).
|
||||
|
||||
**Module-class binaries — `md_*` (P30 S44/S45; the §S44 loader table, `docs/memory-map.md`):**
|
||||
the small type-1 payloads (per-actor modules, the SC07 endgame pair) load at their OWN statically
|
||||
derived slots (A `0x800CAE08` · B `0x800CCB1C` · boot/resident `0x800CEDF8` · SC07 `0x801A00D8`),
|
||||
not the shared overlay slot. Onboard with **`tools/new_binary.sh <alias> <payload> <VRAM>
|
||||
[TEXT_LO]`** (the generalized `new_overlay.sh`; registry `config/modules.mk` / `MODULE_BINARIES`).
|
||||
Module-specific facts the recipe encodes:
|
||||
- **TEXT_LO ≠ 0** (the §154 module-id law: payload word0 is a global module id, sometimes followed
|
||||
by a fn-ptr table and/or data): derive per payload from the first-prologue scan (`27BDxxxx`) and
|
||||
the min fn-ptr-table target — NOT min-table alone (functions can precede the lowest table entry:
|
||||
the SC07 pair's real code start is 0xFC/0x158, their min table targets 0x930/0x370).
|
||||
- **The header carve is a dot-typed `.rodata` PAIRED with the c segment** (same name), never a
|
||||
standalone `rodata, hdr` object and never `bin`: a module header can hold a function's JUMP
|
||||
TABLE, whose `.L` labels only resolve when jtbl and function assemble in the SAME object (the
|
||||
EXE `[0x63238,.rodata,800]` precedent); `bin` assets link in the data block (wrong placement).
|
||||
- **A4 symbol-window law:** a module whose window lies INSIDE another binary's symbol region must
|
||||
NOT stack that binary's symbol file — the boot trio (`0x800CEDF8`) omits `symbols.resident.txt`
|
||||
(DsMix @0x800D1BD8 minted a phantom fn boundary in md_MAIN_011 before this).
|
||||
- **`make sig-modules`** signs every module at its own vram/TEXT_LO, seeding from the built ELF's
|
||||
`func_*` symbols when a build exists (bootstrap's linear partition glues adjacent functions
|
||||
around jtbl dispatch); fresh-clone fallback is `--bootstrap`, self-healing on the next run.
|
||||
|
||||
### §6.8 Cross-binary dedup & code-sharing (Phase 11) — "one match unlocks many"
|
||||
Full how-to in `docs/matching-cookbook.md` §11. Command crib:
|
||||
- **`make sig-overlays`** — Ghidra-FREE sign all 134 location overlays (`SCxx 0.4.dec`) at the shared overlay
|
||||
@@ -642,8 +663,8 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `ExportSymbols.java` | Dump curated symbols (feeds `config/symbols.us.txt`, R15). |
|
||||
| | `DumpProgramInfo.java` | Dump program metadata (loader, language, ImageBase, function count). |
|
||||
| | `DumpFunctionSignatures.java` | Dump function signatures (feeds `make sig-refresh`). |
|
||||
| | `ImportOverlay.java` | Import an overlay segment into the project. |
|
||||
| | `VerifyOverlay.java` | Verify an imported overlay against expected bytes. |
|
||||
| | `ImportOverlay.java` | **RETIRED (S45, R33)** — 1-overlay-era hardcoded import; `tools/ghidra_import_raw.sh` is the live path. |
|
||||
| | `VerifyOverlay.java` | **RETIRED (S45, R33)** — companion of ImportOverlay.java; retired with it. |
|
||||
| | `GetSymbolAt.java` | Read the symbol at a given address (scripted lookup). |
|
||||
| | `DecompileAt.java` | Decompile the function at a given address (scripted scaffold). |
|
||||
| | `DefineFunctions.java` | Disassemble + create functions at splat's validated entry points (`.run/<prog>_funcs.txt`) — completes a raw-blob program's function set (Phase 10). |
|
||||
@@ -664,7 +685,8 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/permuter/` | decomp-permuter harness (PERM_ recipes/weights) for stubborn near-misses. |
|
||||
| | `diff_settings.py` *(repo root)* | asm-differ config (arch `mipsel`, object mode vs `expected/`). |
|
||||
| | `tools/new_overlay.sh` | One-command location-overlay onboarding: `<SCxx> <FILE_nnn> [ENTRY]` — instantiate `config/splat.<ov>.yaml` from the template (+ non-4-aligned `bin` carve), register the binary in `config/overlays.mk` + the report/diff dicts, `make extract && build` byte-check. Idempotent (Phase 13, cookbook §13). **Phase-27 T7:** the optional `ENTRY` arg (default `0.4`) reaches a non-`0.4.dec` payload — the 4 SC07 overlays put code at PAC entry 1 (`1.4`). difficulty.py dropped from the insertion set (it derives now, T6). |
|
||||
| | `tools/disc_code_sweep.py` | **(Phase-27 T7)** Disc-completeness audit: decode every extracted PAC payload (reusing `sig_image.make_insn`) and flag code-bearing ones by BOTH `valid ≥ 0.90` AND `jr $ra` density `≥ 0.01` (the `jr $ra` gate is decisive — type-0/2 data decodes ~100% "valid" but has 0 returns). Reconciles the onboarded set against the disc — the R34 answer to "what code did nobody onboard". Findings → `docs/disc-completeness.md` (138/138 type-4 complete; **39 un-onboarded type-1 resident-class modules** pending load-address RE). |
|
||||
| | `tools/disc_code_sweep.py` | **RETIRED (S45, R33)** — superseded by `tools/disc_audit.py` / `make audit-disc` (whole-payload, BOTH raw+LZSS layers, residue-0 partition, claimed-by derived from `config/check.<bin>.sha`). The sweep read only the RAW layer through a 4,096-word window and had no notion of a claim; its historical findings are preserved in `docs/disc-completeness.md`. |
|
||||
| | `tools/new_binary.sh` | **(P30 S44/S45)** One-command onboarding for ANY flat-blob binary class: `<alias> <payload> <VRAM> [TEXT_LO]` — ov_* (overlay slot, registry `overlays.mk`) or md_* (own §S44 slot, registry `modules.mk`). Signs (TEXT_LO-aware §154), instantiates the shared template (non-zero TEXT_LO ⇒ paired-`.rodata` header carve), writes check.sha + symbols, registers in the registry + the 3 report/diff dicts (sentinel-anchored, ast-checked), extract+build byte-check. Idempotent. `new_overlay.sh` is now a thin wrapper over it. §6.7 module recipe. |
|
||||
| **PsyQ library linking** (cookbook §8/§9) | `tools/psyq_lib_split.py` | Split a PsyQ `.LIB` into per-object members. |
|
||||
| | `tools/psyq_build_libs.sh` | Build the PsyQ libs from split members. |
|
||||
| | `tools/psyq_identify.py` | Identify which SDK objects a region's functions belong to. |
|
||||
|
||||
@@ -82,7 +82,11 @@ mostly-unmatched code to the denominator; every "×134" family propagation is no
|
||||
honest direction — the prior number was measured over an incomplete disc.
|
||||
|
||||
## Reproduce
|
||||
> **⚠️ S45: `tools/disc_code_sweep.py` is RETIRED (R33)** — superseded by `make audit-disc`
|
||||
> (`tools/disc_audit.py`): whole-payload classification at BOTH the raw and LZSS layers, a
|
||||
> residue-0 partition over the disc, and `claimed-by` derived from `config/check.<bin>.sha`.
|
||||
> The commands below are kept for provenance of the 2026-07-15 findings; they no longer run.
|
||||
```
|
||||
tools/disc_code_sweep.py # the table above + the ranked hidden list
|
||||
tools/disc_code_sweep.py --types 1 # just the type-1 code modules
|
||||
tools/disc_code_sweep.py # (retired) the table above + the ranked hidden list
|
||||
tools/disc_code_sweep.py --types 1 # (retired) just the type-1 code modules
|
||||
```
|
||||
|
||||
@@ -224,7 +224,17 @@ stub on a named wall/behemoth/queue ledger** — 140/140 byte-identical througho
|
||||
(12,682,962/13,487,439) · 95.96% fn-count · 87.6% distinct (78,596/90,143 uniq).** The drop
|
||||
from S44's 94.4% is the denominator growing +52k module ins (honest direction, P27/S39/S44
|
||||
precedent).
|
||||
- **II.2** (retirements + SETUP module recipe) — next.
|
||||
- **II.2 ✅ — retirements (R33) + SETUP module recipe.** Deleted: `disc_code_sweep.py` (superseded
|
||||
by `disc_audit.py`/`make audit-disc`), `reconcile_decls.py` (superseded by `reconcile_tu`;
|
||||
its incumbent row removed from cdecl's differential — the audit's purpose was to enable exactly
|
||||
this deletion), the 3 one-shot rollout drivers (`rollout_801457a4_o0` / `rollout_whale_o0` /
|
||||
`rollout_o0_cluster` — `rollout_o0.py` is the live generic), `ImportOverlay.java` +
|
||||
`VerifyOverlay.java` (`ghidra_import_raw.sh` is the live path). Reference check first: the
|
||||
plan's "zero build refs" was wrong for 3 of them (comments + one live cdecl import — handled,
|
||||
gate re-proven green). SETUP §6.7 gained the **module-class recipe** (TEXT_LO derivation law,
|
||||
paired-.rodata header carve, A4 symbol-window law, ELF-seeded sig-modules) + inventory rows
|
||||
(R21); disc-completeness Reproduce marked retired; Makefile comments annotated.
|
||||
- **II.3** (metrics re-baseline + roadmap delta + decision-log) — next.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+2
-2
@@ -1239,7 +1239,8 @@ def audit_differential():
|
||||
tool. The reverse (symbols only this one sees) is the measured size of the hole."""
|
||||
sys.path.insert(0, os.path.join(REPO, 'tools'))
|
||||
import gen_harvest_targets as ght
|
||||
import reconcile_decls as rd
|
||||
# reconcile_decls RETIRED (S45, R33): its DATA_DECL_LINE_RE row proved the superset for
|
||||
# 26-A..S44; the incumbent is deleted, so the differential now compares the survivors only.
|
||||
import sig_unify as su
|
||||
|
||||
ec = os.path.join(REPO, 'src/shared/engine_core.h')
|
||||
@@ -1268,7 +1269,6 @@ def audit_differential():
|
||||
mine_d = {n for n in mine if n.startswith('D_')}
|
||||
rows = []
|
||||
for name, rx in (('gen_harvest_targets.DATA_DECL_RE', ght.DATA_DECL_RE),
|
||||
('reconcile_decls.DATA_DECL_LINE_RE', rd.DATA_DECL_LINE_RE),
|
||||
('sig_unify.DATA_DECL_RE', su.DATA_DECL_RE)):
|
||||
theirs = set()
|
||||
for m in rx.finditer(text):
|
||||
|
||||
@@ -1,166 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""disc_code_sweep.py — find code-bearing PAC payloads the onboarding may have missed (Phase-27 T7).
|
||||
|
||||
The disc-completeness question the byte-gate is structurally blind to (R34): the build only touches
|
||||
binaries someone ONBOARDED, so a code payload no one onboarded is invisible to `make check-all` no
|
||||
matter how green it is. The Phase-27 audit found four such overlays (SC07 FILE_006/007/010/011, code
|
||||
at PAC entry 1 not 0). This sweeps EVERY extracted PAC payload and reports which decode as MIPS code,
|
||||
so the onboarded set can be reconciled against the disc rather than trusted.
|
||||
|
||||
Method (reuses sig_image's exact rabbitizer decode): decode the first `--window` words of each raw
|
||||
payload as MIPS-LE, report the fraction rabbitizer calls valid. Overlay/EXE code runs ~0.97-1.00
|
||||
valid; data/graphics/audio sit far lower. A payload above `--threshold` that is NOT already onboarded
|
||||
is a candidate the audit must explain (onboard it, or record why it is not a build binary — e.g. a
|
||||
type-1 blob that loads at its own address like the resident, needing load-address analysis first).
|
||||
|
||||
COVERAGE-ASSERTED (R32): every extracted `{index}.{type}` payload is classified; the onboarded set is
|
||||
cross-checked so a hit that is already a binary is labelled, not re-flagged.
|
||||
|
||||
tools/disc_code_sweep.py # sweep all types, print the report
|
||||
tools/disc_code_sweep.py --types 1,6,7 # only the non-type-4 unknowns
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "bfm_extract"))
|
||||
import sig_image # make_insn — the shared rabbitizer decode (GTE-aware)
|
||||
import lzss # decompress — the SAME game-semantics decoder the extractor uses
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
# a raw payload is <dir>/<index>.<type> with NO .dec suffix; type-4 also has a .dec sibling
|
||||
_PAYLOAD = re.compile(r"/(\d+)\.(\d+)$")
|
||||
|
||||
|
||||
def onboarded_payloads():
|
||||
"""The set of payload paths already wired as a build binary (main, resident, every overlay).
|
||||
Read from the Makefile/overlays.mk *_EXE assignments — the single source of truth (R33).
|
||||
|
||||
KEYED BY THE RAW PAYLOAD PATH (Phase-28 T7). A type-4 overlay's `_EXE` is the DECOMPRESSED path
|
||||
(`…/0.4.dec`), but this sweep iterates the RAW payloads (`…/0.4`) so it can decode the compressed
|
||||
layer itself. Store the `.dec`-stripped form too, or all 138 type-4 overlays — now correctly seen
|
||||
as code via decompression — would fail the onboarded match and flag as false HIDDEN hits."""
|
||||
paths = set()
|
||||
for mk in ("Makefile", "config/overlays.mk"):
|
||||
p = os.path.join(REPO, mk)
|
||||
if not os.path.exists(p):
|
||||
continue
|
||||
for m in re.finditer(r"^\w+_EXE\s*:=\s*(\S+)", open(p).read(), re.M):
|
||||
norm = os.path.normpath(m.group(1))
|
||||
paths.add(norm)
|
||||
if norm.endswith(".dec"):
|
||||
paths.add(norm[:-len(".dec")]) # the raw payload this .dec came from
|
||||
return paths
|
||||
|
||||
|
||||
_JR_RA = 0x03E00008 # `jr $ra` — a function return; the discriminator rabbitizer.isValid() lacks
|
||||
|
||||
|
||||
def _signals(data, window):
|
||||
n = min(window, len(data) // 4)
|
||||
if n == 0:
|
||||
return 0.0, 0.0, 0
|
||||
ok = jr = 0
|
||||
for k in range(n):
|
||||
word = struct.unpack_from("<I", data, k * 4)[0]
|
||||
if sig_image.make_insn(word, 0x80000000 + k * 4).isValid():
|
||||
ok += 1
|
||||
if word == _JR_RA:
|
||||
jr += 1
|
||||
return ok / n, jr / n, n
|
||||
|
||||
|
||||
def code_signals(path, window):
|
||||
"""(valid_ratio, jr_ra_density, nwords, layer). isValid() alone is too permissive — structured
|
||||
DATA decodes ~100% valid (type-0/type-2 blobs hit 1.00 with ZERO returns). Real MIPS code carries
|
||||
a `jr $ra` roughly once per function (~2.9-3.4% of words across the onboarded overlays + resident);
|
||||
data carries ~0%. Requiring BOTH separates code from valid-looking data.
|
||||
|
||||
DECODES BOTH LAYERS (Phase-28 T7 — the fix). The original decoded only the RAW payload, which is
|
||||
STRUCTURALLY BLIND to compressed code: type-4 overlay code is LZSS-compressed, so raw bytes are
|
||||
noise and every type-4 row read "code 0" — a VACUOUS row for 138 known-code binaries. That is not
|
||||
a curiosity: this tool is the disc-completeness oracle (R34), and it could NOT have found the 4
|
||||
hidden SC07 overlays (their code is compressed like every type-4) — they were caught by
|
||||
hand-reconciling 138-vs-134. It found the 39 type-1 modules ONLY because those happen to be
|
||||
uncompressed. So decode both and take the stronger signal:
|
||||
* uncompressed code (type-1 resident-class): lives in the RAW bytes.
|
||||
* compressed code (type-4 overlays): lives in the LZSS-DECOMPRESSED bytes.
|
||||
A payload is code if EITHER layer is code; `layer` records which, so the report is auditable."""
|
||||
raw = open(path, "rb").read()
|
||||
r_ratio, r_jr, r_n = _signals(raw, window)
|
||||
best = (r_ratio, r_jr, r_n, "raw")
|
||||
try:
|
||||
res = lzss.decompress(raw)
|
||||
dec = res.data if hasattr(res, "data") else bytes(res)
|
||||
except Exception:
|
||||
dec = b""
|
||||
if len(dec) >= 64:
|
||||
d_ratio, d_jr, d_n = _signals(dec, window)
|
||||
# "stronger" = the layer that clears the code bar, or (if neither/both) the higher jr density
|
||||
# (the discriminator). This never downgrades a raw-code hit to a decompressed non-hit.
|
||||
if (d_jr, d_ratio) > (best[1], best[0]):
|
||||
best = (d_ratio, d_jr, d_n, "dec")
|
||||
return best
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--root", default="extracted/retail")
|
||||
ap.add_argument("--types", default=None, help="comma list of PAC types to sweep (default: all)")
|
||||
ap.add_argument("--window", type=int, default=4096, help="words decoded per payload")
|
||||
ap.add_argument("--threshold", type=float, default=0.90, help="valid-ratio floor")
|
||||
ap.add_argument("--jr-min", type=float, default=0.01, help="jr-$ra density floor (code ~0.03, data ~0)")
|
||||
a = ap.parse_args()
|
||||
|
||||
want = set(a.types.split(",")) if a.types else None
|
||||
onboard = onboarded_payloads()
|
||||
|
||||
rows, seen = [], 0
|
||||
for p in sorted(glob.glob(os.path.join(REPO, a.root, "**"), recursive=True)):
|
||||
if not os.path.isfile(p) or p.endswith(".dec"):
|
||||
continue
|
||||
m = _PAYLOAD.search(p)
|
||||
if not m:
|
||||
continue
|
||||
typ = m.group(2)
|
||||
if want and typ not in want:
|
||||
continue
|
||||
seen += 1
|
||||
rel = os.path.relpath(p, REPO)
|
||||
ratio, jr, nwords, layer = code_signals(p, a.window)
|
||||
is_code = ratio >= a.threshold and jr >= a.jr_min # BOTH signals (jr is the discriminator)
|
||||
rows.append((typ, ratio, jr, nwords, rel, os.path.normpath(rel) in onboard, is_code, layer))
|
||||
|
||||
assert len(rows) == seen, f"classified {len(rows)} of {seen} payloads — a silent skip (R32)"
|
||||
|
||||
hidden = [r for r in rows if r[6] and not r[5]]
|
||||
|
||||
by_type = {}
|
||||
for typ, ratio, jr, _, _, onb, is_code, _layer in rows:
|
||||
d = by_type.setdefault(typ, [0, 0, 0])
|
||||
d[0] += 1
|
||||
d[1] += is_code
|
||||
d[2] += is_code and onb
|
||||
|
||||
print(f"disc code sweep — {seen} payloads under {a.root}"
|
||||
+ (f" (types {a.types})" if want else "")
|
||||
+ f", window={a.window}w, valid>={a.threshold}, jr_ra>={a.jr_min}")
|
||||
print(f"{'type':>4} {'payloads':>9} {'code':>5} {'onboarded':>9} {'HIDDEN':>7}")
|
||||
for typ in sorted(by_type):
|
||||
tot, c, onb = by_type[typ]
|
||||
print(f"{typ:>4} {tot:>9} {c:>5} {onb:>9} {c - onb:>7}")
|
||||
|
||||
print(f"\nHIDDEN code-bearing payloads (valid>={a.threshold} AND jr_ra>={a.jr_min}, NOT onboarded): {len(hidden)}")
|
||||
for typ, ratio, jr, nwords, rel, _, _, layer in sorted(hidden, key=lambda r: -r[2]):
|
||||
print(f" type {typ} valid {ratio:5.1%} jr_ra {jr:5.2%} ({nwords}w, {layer}) {rel}")
|
||||
if not hidden:
|
||||
print(" (none — the onboarded set accounts for every code-bearing payload the sweep sees)")
|
||||
return 0 if not hidden else 3
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,62 +0,0 @@
|
||||
// ImportOverlay.java — headless postScript: load the T6b-proven resident blob + one location
|
||||
// overlay into the program at their byte-verified vaddrs, so the overlay/resident code becomes
|
||||
// disassemblable in Ghidra. The MCP tools cannot create memory blocks / bulk-load file bytes, so
|
||||
// this is done headless (analyzeHeadless opens the project directly and saves on completion).
|
||||
//
|
||||
// ~/ghidra_12.1_PUBLIC/support/analyzeHeadless ~/bfm-decomp/ghidra bfm \
|
||||
// -process SLUS_007.26 -noanalysis \
|
||||
// -scriptPath ~/bfm-decomp/tools/ghidra_scripts -postScript ImportOverlay.java
|
||||
//
|
||||
// Idempotent: re-runs write into the blocks created on the first run.
|
||||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.Address;
|
||||
import ghidra.program.model.mem.Memory;
|
||||
import ghidra.program.model.mem.MemoryBlock;
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Paths;
|
||||
|
||||
public class ImportOverlay extends GhidraScript {
|
||||
public void run() throws Exception {
|
||||
String home = System.getProperty("user.home");
|
||||
// {block name, vaddr, extracted file}
|
||||
String[][] targets = {
|
||||
{"residentBlob", "0x800CEDF8", home + "/bfm-decomp/extracted/MAIN.CD.dir/FILE_010.dir/1.1"},
|
||||
{"ovl_tutForest", "0x80128158", home + "/bfm-decomp/extracted/SC01.CD.dir/FILE_077.dir/0.4.dec"},
|
||||
};
|
||||
Memory mem = currentProgram.getMemory();
|
||||
println("BFMOVL == existing memory blocks ==");
|
||||
for (MemoryBlock b : mem.getBlocks())
|
||||
println("BFMOVL " + b.getName() + " " + b.getStart() + "-" + b.getEnd()
|
||||
+ " init=" + b.isInitialized());
|
||||
|
||||
for (String[] t : targets) {
|
||||
String name = t[0];
|
||||
Address addr = toAddr(t[1]);
|
||||
byte[] bytes = Files.readAllBytes(Paths.get(t[2]));
|
||||
MemoryBlock existing = mem.getBlock(addr);
|
||||
if (existing == null) {
|
||||
mem.createInitializedBlock(name, addr, new ByteArrayInputStream(bytes),
|
||||
bytes.length, monitor, false);
|
||||
println("BFMOVL CREATED " + name + " @ " + addr + " len=" + bytes.length);
|
||||
} else {
|
||||
println("BFMOVL into existing block '" + existing.getName()
|
||||
+ "' init=" + existing.isInitialized());
|
||||
if (!existing.isInitialized())
|
||||
mem.convertToInitialized(existing, (byte) 0);
|
||||
mem.setBytes(addr, bytes);
|
||||
println("BFMOVL WROTE " + bytes.length + " bytes @ " + addr);
|
||||
}
|
||||
}
|
||||
// sanity: read back first 8 bytes at each vaddr
|
||||
for (String[] t : targets) {
|
||||
Address a = toAddr(t[1]);
|
||||
byte[] c = new byte[8];
|
||||
mem.getBytes(a, c);
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (byte x : c) sb.append(String.format("%02x", x));
|
||||
println("BFMOVL VERIFY " + a + " first8=" + sb);
|
||||
}
|
||||
println("BFMOVL done.");
|
||||
}
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
// VerifyOverlay.java — read-only R9 check that the imported overlay/resident bytes persisted.
|
||||
// analyzeHeadless ... -process SLUS_007.26 -noanalysis -readOnly -postScript VerifyOverlay.java
|
||||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.Address;
|
||||
|
||||
public class VerifyOverlay extends GhidraScript {
|
||||
public void run() throws Exception {
|
||||
String[][] t = { {"residentBlob","0x800CEDF8"}, {"ovl_tutForest","0x80128158"} };
|
||||
for (String[] e : t) {
|
||||
Address a = toAddr(e[1]);
|
||||
byte[] c = new byte[8];
|
||||
currentProgram.getMemory().getBytes(a, c);
|
||||
StringBuilder sb = new StringBuilder();
|
||||
for (byte x : c) sb.append(String.format("%02x", x));
|
||||
boolean init = currentProgram.getMemory().getBlock(a).isInitialized();
|
||||
println("BFMVERIFY " + e[0] + " @" + a + " init=" + init + " first8=" + sb);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,259 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""RETIRED (Phase 26-A, R33) — superseded by tools/reconcile_tu.py. DO NOT RE-WIRE.
|
||||
|
||||
This tool asked the wrong question, and no regex fix could have saved it:
|
||||
|
||||
reconcile_decls asks "what does the FLEET call this symbol?"
|
||||
C asks "what does THIS TRANSLATION UNIT declare?"
|
||||
|
||||
The engine is loosely typed — the same address is legitimately declared with incompatible types in
|
||||
different overlays — so a single fleet-wide answer is **wrong for some TU by construction**. And it
|
||||
is worse than a silent skip: it writes an ACTIVELY WRONG declaration into the draft, which then
|
||||
collides with the very TU it was meant to conform to. Measured across ov_SC01_077's 12 TUs:
|
||||
|
||||
agrees with the TU's own declaration ....... 2883
|
||||
CONFLICTS with it (cc1 REJECTS the result) . 548 <- 16%
|
||||
TU declares it, this oracle has NO answer .. 357
|
||||
|
||||
and it was live on both banking paths: it rewrote 60 of 196 drafts in gate_stage, and every sibling
|
||||
in the ×134 jtbl_family_bank sweep — the project's economic engine.
|
||||
|
||||
Its own docstring already knew the symbols are PER-OVERLAY. That is exactly why a FLEET oracle
|
||||
could never have been right.
|
||||
|
||||
KEPT AS EVIDENCE, NOT AS CODE. Its `data_access_subs` also has no fn-ptr kind, so teaching its parser
|
||||
to see `extern void (*D_x[])(void);` would have ARMED it to rewrite a call-through `D_x[i]()` into
|
||||
`((u8 *)D_x)[i]()` — a dormant transform that "fixing the regex" would have detonated. reconcile_tu
|
||||
handles the kind natively, which is why it SUPERSEDES this rather than patching it.
|
||||
|
||||
Original docstring follows.
|
||||
"""
|
||||
"""reconcile_decls.py — the DATA-symbol analog of cast_call_sites.py (Phase 24 T7b, cookbook §33).
|
||||
|
||||
A freshly-matched giant / wave draft byte-matches STANDALONE with its own guessed decls, but to bank
|
||||
×1 in the overlay TU (and then propagate ×134 via a shared `DEFINE_func_*` macro) its callee/data
|
||||
externs must be FLEET-CANONICAL — otherwise `conflicting types` vs engine_core.h / sibling decls
|
||||
(a COMPILE error, not a byte miss). `cast_call_sites.py` handles this for callee FUNCTIONS
|
||||
(decl->canonical + fn-ptr cast at the call). This tool does the same for DATA symbols `D_XXXX`:
|
||||
rewrite the draft's `extern <T_draft> D_x...;` -> the fleet-canonical decl, and inject a byte-neutral
|
||||
cast at each ACCESS so the load/store opcode is unchanged. gcc-2.7.2 folds the compile-time
|
||||
pointer/scalar cast of a known symbol, so the body bytes are identical.
|
||||
|
||||
This is a PURE draft-text transform (like sig_unify / cast_call_sites / canon_resident_calls): the
|
||||
whole-binary `harvest_verify` byte-gate remains the sole arbiter (G3/P9) — a wrong reconcile just
|
||||
fails the gate and reverts, so it trial-and-errors safely. It does NOT touch callee FUNCTION decls
|
||||
(that's cast_call_sites/sig_unify — compose them). It changes only the draft's OWN data externs +
|
||||
the access sites; it never edits the fleet's canonical decls.
|
||||
|
||||
Byte-neutral cast taxonomy (proven by hand on func_80129CF8; the intended type is what the DRAFT
|
||||
declared, the canonical is the fleet's — cast every use to reproduce the intended access):
|
||||
draft array Ed[] vs canonical array Ec[] -> decl Ec[]; D_x[i] -> ((Ed*)D_x)[i]; D_x -> (Ed*)D_x
|
||||
draft array Ed[] vs canonical struct/scalar -> decl canon; D_x -> (Ed*)&D_x (address-of the object)
|
||||
draft scalar Td vs canonical scalar Tc -> decl Tc; D_x -> *(Td*)&D_x (forces the Td-width opcode)
|
||||
draft ptr P* D_x vs canonical scalar Tc -> decl Tc; write D_x=v -> D_x=(Tc)(v); read D_x -> (P*)D_x
|
||||
|
||||
Pipeline (cookbook §33):
|
||||
draft -> canon_resident_calls -> cast_call_sites -> reconcile_decls -> sig_unify -> harvest_verify --chunk 1
|
||||
(callee funcs) (data symbols) (def sig) (the byte-gate)
|
||||
|
||||
Usage:
|
||||
tools/reconcile_decls.py --overlay ov_SC01_077 --src-file src/ov_SC01_077/ov_SC01_077_a.c \
|
||||
--in .run/drafts-giant --out .run/drafts-giant-rc
|
||||
tools/reconcile_decls.py --print-canon D_80126948 # inspect the oracle's pick for one symbol
|
||||
"""
|
||||
import argparse, os, re, glob, importlib.util, collections
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
|
||||
def _load(mod, rel):
|
||||
spec = importlib.util.spec_from_file_location(mod, os.path.join(REPO, rel))
|
||||
m = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(m)
|
||||
return m
|
||||
|
||||
|
||||
_ght = _load('ght', 'tools/gen_harvest_targets.py') # reuse collect_data_decls / the D_ regexes
|
||||
|
||||
# `extern <type> D_XXXX;` or `extern <type> D_XXXX[];` (optional trailing /* comment */). <type> may
|
||||
# carry a trailing `*` (pointer) which we fold into the element type.
|
||||
DATA_DECL_LINE_RE = re.compile(
|
||||
r'^([ \t]*)extern\s+([A-Za-z_][\w \t]*?)\s*(\*?)\s*\b(D_[0-9A-Fa-f]+)\b\s*(\[\s*\])?\s*;'
|
||||
r'[ \t]*(?:/\*[^\n]*\*/)?[ \t]*$')
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the canonical-type ORACLE (T2)
|
||||
def canonical_data_map(extra_src=None):
|
||||
"""Fleet-majority canonical decl per data symbol. Returns {D_name: 'extern <decl>;'}.
|
||||
|
||||
Precedence: an engine_core.h macro decl is AUTHORITATIVE (it's the shared/propagated set every
|
||||
overlay co-instantiates — conforming to it is what avoids the per-overlay macro conflict). Else
|
||||
the plurality vote across all overlays + resident (ties -> the lexicographically-first spelling,
|
||||
deterministic). This is the picker the design-doc flagged as missing (gen_harvest_targets read
|
||||
only engine_core.h + ONE overlay, first-seen-wins, and never saw a fleet disagreement)."""
|
||||
ec = os.path.join(REPO, 'src/shared/engine_core.h')
|
||||
# 1. engine_core.h — authoritative (a symbol declared inside any DEFINE_ macro)
|
||||
ec_decls = _ght.collect_data_decls([ec]) # {D_name: 'extern ...;'} first-seen
|
||||
# 2. plurality across the fleet (all overlays + resident) for symbols engine_core.h lacks
|
||||
votes = collections.defaultdict(collections.Counter)
|
||||
fleet = (glob.glob(os.path.join(REPO, 'src/ov_*/*.c'))
|
||||
+ [os.path.join(REPO, 'src/resident/resident.c')])
|
||||
if extra_src:
|
||||
fleet.append(os.path.join(REPO, extra_src))
|
||||
for p in fleet:
|
||||
if not os.path.exists(p):
|
||||
continue
|
||||
txt = open(p).read()
|
||||
for m in _ght.DATA_DECL_RE.finditer(txt):
|
||||
decl = re.sub(r'\s+', ' ', m.group(1)).strip()
|
||||
nm = re.search(r'D_[0-9A-Fa-f]+', decl)
|
||||
if nm:
|
||||
votes[nm.group(0)][f'extern {decl};'] += 1
|
||||
canon = {}
|
||||
names = set(ec_decls) | set(votes)
|
||||
for nm in names:
|
||||
if nm in ec_decls:
|
||||
canon[nm] = ec_decls[nm] # authoritative
|
||||
else:
|
||||
# plurality; tie -> lexicographically-first decl spelling (deterministic)
|
||||
best = sorted(votes[nm].items(), key=lambda kv: (-kv[1], kv[0]))[0][0]
|
||||
canon[nm] = best
|
||||
return canon
|
||||
|
||||
|
||||
def parse_data_decl(decl):
|
||||
"""'extern u8 D_x[];' -> (name, elem='u8', is_array=True, is_ptr=False).
|
||||
'extern struct BigCopy D_x;' -> (name, 'struct BigCopy', False, False).
|
||||
'extern s32 *D_x;' -> (name, 's32', False, is_ptr=True)."""
|
||||
m = DATA_DECL_LINE_RE.match(decl) or DATA_DECL_LINE_RE.match(' ' + decl.strip())
|
||||
if not m:
|
||||
return None
|
||||
_indent, base, star, name, arr = m.group(1), m.group(2), m.group(3), m.group(4), m.group(5)
|
||||
return name, re.sub(r'\s+', ' ', base).strip(), bool(arr), bool(star)
|
||||
|
||||
|
||||
def _norm_type(elem, is_array, is_ptr):
|
||||
"""Spelling-insensitive key for 'do these decls differ' — collapse int-family aliases (same
|
||||
width+conv, casting them is a no-op), normalise whitespace. Array vs ptr vs scalar are distinct."""
|
||||
e = re.sub(r'\s+', ' ', elem).strip()
|
||||
e = re.sub(r'\b(s32|u32|int|unsigned int|unsigned|long|unsigned long|u_long)\b', 'int', e)
|
||||
kind = 'arr' if is_array else ('ptr' if is_ptr else 'sca')
|
||||
return (e, kind)
|
||||
|
||||
|
||||
def _base_ptr_expr(name, ielem, canon_is_array):
|
||||
"""The draft's intended array/base pointer, expressed under the canonical storage decl.
|
||||
canonical array -> the array decays: (Ed*)D_x
|
||||
canonical struct/scalar -> take its address: (Ed*)&D_x"""
|
||||
return f'(({ielem} *){name})' if canon_is_array else f'(({ielem} *)&{name})'
|
||||
|
||||
|
||||
def data_access_subs(name, idecl, cdecl):
|
||||
"""One (compiled-regex, replacement-fn) that reproduces the DRAFT's intended access to `name`
|
||||
under the CANONICAL storage decl in a SINGLE pass (re.sub never re-scans its own output, so a
|
||||
symbol appearing in several forms on one line can't double-wrap). [] if type-compatible, None if
|
||||
unparseable. The regex captures an optional leading `&` and a following `[` to pick the form."""
|
||||
ip = parse_data_decl(idecl)
|
||||
cp = parse_data_decl(cdecl)
|
||||
if not ip or not cp:
|
||||
return None # unparseable -> caller logs, leaves as-is
|
||||
_, ielem, iarr, iptr = ip
|
||||
_, celem, carr, cptr = cp
|
||||
if _norm_type(ielem, iarr, iptr) == _norm_type(celem, carr, cptr):
|
||||
return [] # already compatible
|
||||
rx = re.compile(rf'(&?)\b{re.escape(name)}\b(\s*\[)?')
|
||||
if iarr:
|
||||
base = _base_ptr_expr(name, ielem, carr) # (Ed*)D_x or (Ed*)&D_x (canon array vs not)
|
||||
def repl(m):
|
||||
amp, idx = m.group(1), m.group(2)
|
||||
if idx: # D_x[i] / &D_x[i] -> [&]base[i]
|
||||
return f'{amp}{base}{idx}'
|
||||
return base # bare D_x / &D_x -> base (array address IS the base ptr)
|
||||
else:
|
||||
# scalar (signedness/width) OR ptr-vs-scalar: force the intended-width/type access via a
|
||||
# cast-lvalue `*(<intended> *)&D_x` (valid as both lvalue and rvalue; ptr-intended -> `ielem **`).
|
||||
star = ' *' if iptr else ''
|
||||
acc = f'(*({ielem}{star} *)&{name})'
|
||||
def repl(m):
|
||||
amp, idx = m.group(1), m.group(2)
|
||||
return f'{amp}{acc}{idx or ""}'
|
||||
return [(rx, repl)]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- the transform (T3)
|
||||
def transform(text, canon):
|
||||
"""Return (new_text, n_data_reconciled, notes). For each data symbol the draft declares with a
|
||||
type differing from the fleet-canonical: rewrite the decl -> canonical + cast every access."""
|
||||
lines = text.split('\n')
|
||||
plan = {} # D_name -> (canonical_decl_line, [subs])
|
||||
decl_idx = {} # D_name -> line index of its decl
|
||||
notes = []
|
||||
for i, ln in enumerate(lines):
|
||||
m = DATA_DECL_LINE_RE.match(ln)
|
||||
if not m:
|
||||
continue
|
||||
indent, name = m.group(1), m.group(4)
|
||||
idecl = ln.strip()
|
||||
if name not in canon:
|
||||
continue # no fleet decl -> draft's is the only one, keep
|
||||
cdecl = canon[name]
|
||||
subs = data_access_subs(name, idecl, cdecl)
|
||||
if subs is None:
|
||||
notes.append(f'{name}: UNPARSEABLE decl, left as-is'); continue
|
||||
if not subs:
|
||||
continue # already compatible -> nothing to do
|
||||
plan[name] = (f'{indent}{cdecl}', subs)
|
||||
decl_idx[name] = i
|
||||
|
||||
if not plan:
|
||||
return text, 0, notes
|
||||
|
||||
out = []
|
||||
for i, ln in enumerate(lines):
|
||||
dname = next((n for n, j in decl_idx.items() if j == i), None)
|
||||
if dname is not None:
|
||||
out.append(plan[dname][0]) # replace the decl line with the canonical decl
|
||||
continue
|
||||
if DATA_DECL_LINE_RE.match(ln): # some OTHER data decl -> never cast in a decl
|
||||
out.append(ln); continue
|
||||
for name, (_c, subs) in plan.items(): # body line -> rewrite this symbol's accesses
|
||||
for rx, rep in subs:
|
||||
ln = rx.sub(rep, ln)
|
||||
out.append(ln)
|
||||
return '\n'.join(out), len(plan), notes
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--overlay', default='ov_SC01_077')
|
||||
ap.add_argument('--src-file', dest='src_file', default=None,
|
||||
help='overlay .c whose local decls also vote (default: none; the fleet scan '
|
||||
'already covers all overlays)')
|
||||
ap.add_argument('--in', dest='indir')
|
||||
ap.add_argument('--out', dest='outdir')
|
||||
ap.add_argument('--print-canon', metavar='D_XXXX', help='print the oracle pick for one symbol and exit')
|
||||
a = ap.parse_args()
|
||||
|
||||
canon = canonical_data_map(a.src_file)
|
||||
if a.print_canon:
|
||||
print(canon.get(a.print_canon, f'(no fleet decl for {a.print_canon})'))
|
||||
return
|
||||
if not (a.indir and a.outdir):
|
||||
ap.error('need --in and --out (or --print-canon)')
|
||||
|
||||
os.makedirs(os.path.join(REPO, a.outdir), exist_ok=True)
|
||||
drafts = touched = total = 0
|
||||
for p in sorted(glob.glob(os.path.join(REPO, a.indir, '*.c'))):
|
||||
new, k, notes = transform(open(p).read(), canon)
|
||||
open(os.path.join(REPO, a.outdir, os.path.basename(p)), 'w').write(new)
|
||||
drafts += 1
|
||||
if k:
|
||||
touched += 1; total += k
|
||||
for n in notes:
|
||||
print(f' [{os.path.basename(p)}] {n}')
|
||||
print(f'canonical data decls: {len(canon)}; drafts: {drafts}; '
|
||||
f'reconciled: {touched} draft(s), {total} data symbol(s)')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,133 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bank func_801457A4 across the fleet by moving its DEFINITION into <ov>_o0b.c (the -O0 object).
|
||||
|
||||
§116: `func_801457A4` is an -O0 function. In ov_SC01_077 its definition lives in
|
||||
ov_SC01_077_o0b.c, which the Makefile's WHALE_O0B_OBJS wildcard compiles -O0. In every other
|
||||
overlay the same function's stub sits in <ov>_after.c, which is -O2 — so `family_sweep --hseq`
|
||||
templated the -O0-matched body into an -O2 TU and the byte-gate correctly rejected all 137.
|
||||
|
||||
WHY THIS TOOL EXISTS AND family_sweep CANNOT DO IT (Phase 29 T80, the refuted shortcut):
|
||||
the obvious move is to relocate the member's `INCLUDE_ASM(...)` line into <ov>_o0b.c and let the
|
||||
existing sweep stage there. That is NOT byte-neutral — it is unbuildable. splat emits
|
||||
`asm/<ov>/nonmatchings/<seg>/<fn>.s` for EXACTLY the functions the segment's own .c marks with
|
||||
INCLUDE_ASM (measured: 12 lines <-> 12 .s files, identical sets). Delete the line from
|
||||
<ov>_after.c and the .s stops being generated, so the relocated reference cannot assemble.
|
||||
`asm/` follows the SEGMENT; object membership follows the .c FILE.
|
||||
|
||||
So the substitution has to be ATOMIC ACROSS TWO FILES — append the remapped body to <ov>_o0b.c
|
||||
AND drop the INCLUDE_ASM from <ov>_after.c in one edit — which `harvest_verify`/`family_sweep`
|
||||
do not do (they substitute a draft for a stub *in the stub's own file*). Hence this driver.
|
||||
Deliberately no splat change: a re-carve is the Phase-29 Arm-A wall (+0x20 data-symbol shift on
|
||||
3 of 4 sampled overlays).
|
||||
|
||||
The whole-binary byte-gate stays the sole arbiter (G3/P9): per overlay, build and compare against
|
||||
config/check.<ov>.sha; on any mismatch BOTH files are restored from their snapshots.
|
||||
|
||||
tools/rollout_801457a4_o0.py [--apply] [--limit N] [--jobs N]
|
||||
|
||||
Out of scope: ov_SC07_{006,007,010,011} have no _o0b.c (onboarded in Phase 27, never whale-carved).
|
||||
"""
|
||||
import argparse
|
||||
import concurrent.futures as futures
|
||||
import glob
|
||||
import hashlib
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, os.path.join(REPO, "tools"))
|
||||
import family_remap as FR # noqa: E402
|
||||
|
||||
FN = "func_801457A4"
|
||||
ADDR = 0x801457A4
|
||||
EX_OV = "ov_SC01_077"
|
||||
STUB_RE = re.compile(rf'^INCLUDE_ASM\("([^"]+)",\s*{FN}\);\s*$', re.M)
|
||||
|
||||
NOTE = (f"/* {FN} (@0x{ADDR:08X}) is an -O0 function; its definition lives in {{o0b}} (the -O0\n"
|
||||
f" * whale object, whose .text ends exactly at this address). Mirrors ov_SC01_077. §116 */")
|
||||
|
||||
|
||||
def sha1(p):
|
||||
h = hashlib.sha1()
|
||||
with open(p, "rb") as fh:
|
||||
for b in iter(lambda: fh.read(1 << 20), b""):
|
||||
h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def good_sha(ov):
|
||||
return open(os.path.join(REPO, f"config/check.{ov}.sha")).read().split()[0]
|
||||
|
||||
|
||||
def build_ok(ov):
|
||||
r = subprocess.run(["make", "build", f"BINARY={ov}"], cwd=REPO,
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
out = os.path.join(REPO, f"build/{ov}/{ov}")
|
||||
return r.returncode == 0 and os.path.exists(out) and sha1(out) == good_sha(ov)
|
||||
|
||||
|
||||
def candidates():
|
||||
out = []
|
||||
for after in sorted(glob.glob(os.path.join(REPO, "src/ov_*/ov_*_after.c"))):
|
||||
ov = os.path.basename(os.path.dirname(after))
|
||||
o0b = os.path.join(REPO, f"src/{ov}/{ov}_o0b.c")
|
||||
if not os.path.exists(o0b):
|
||||
continue # never whale-carved (the 4 SC07 tail overlays)
|
||||
if not STUB_RE.search(open(after).read()):
|
||||
continue # already banked, or no stub here
|
||||
out.append((ov, after, o0b))
|
||||
return out
|
||||
|
||||
|
||||
def attempt(ov, after, o0b, apply):
|
||||
draft, info = FR.remap_hseq(ADDR, EX_OV, ov, ADDR)
|
||||
if draft is None:
|
||||
return ov, "remap-refused", str(info)[:60]
|
||||
a_txt, b_txt = open(after).read(), open(o0b).read()
|
||||
m = STUB_RE.search(a_txt)
|
||||
new_after = a_txt[:m.start()] + NOTE.format(o0b=os.path.basename(o0b)) + a_txt[m.end():]
|
||||
new_o0b = b_txt.rstrip("\n") + "\n\n" + draft.rstrip("\n") + "\n"
|
||||
if not apply:
|
||||
return ov, "would-try", ""
|
||||
open(after, "w").write(new_after)
|
||||
open(o0b, "w").write(new_o0b)
|
||||
if build_ok(ov):
|
||||
return ov, "BANKED", ""
|
||||
open(after, "w").write(a_txt) # restore BOTH, always (§61)
|
||||
open(o0b, "w").write(b_txt)
|
||||
return ov, "gate-reject", ""
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--apply", action="store_true", help="write + gate (default: dry run)")
|
||||
ap.add_argument("--limit", type=int, default=0)
|
||||
ap.add_argument("--jobs", type=int, default=1, help="overlays gated in parallel (distinct binaries)")
|
||||
a = ap.parse_args()
|
||||
|
||||
cands = candidates()
|
||||
if a.limit:
|
||||
cands = cands[:a.limit]
|
||||
print(f"in scope: {len(cands)} overlays", flush=True)
|
||||
|
||||
res = []
|
||||
if a.jobs > 1 and a.apply:
|
||||
with futures.ThreadPoolExecutor(max_workers=a.jobs) as ex:
|
||||
for r in ex.map(lambda c: attempt(*c, a.apply), cands):
|
||||
res.append(r)
|
||||
print(f" {r[0]}: {r[1]} {r[2]}", flush=True)
|
||||
else:
|
||||
for c in cands:
|
||||
r = attempt(*c, a.apply)
|
||||
res.append(r)
|
||||
print(f" {r[0]}: {r[1]} {r[2]}", flush=True)
|
||||
|
||||
from collections import Counter
|
||||
print(Counter(r[1] for r in res))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,132 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase-29 T2 Arm A: carve the -O0 cluster (vram 0x8013B568..0x8013C98C, file 0x13410..0x14834)
|
||||
into its own -O0 object per single-file overlay, so the h_seq family members of those 9 matched
|
||||
-O0 exemplars can bank whole-binary (they masked-MATCH only at -O0 — the Task-1 swing verdict).
|
||||
|
||||
Background (the swing verdict, 2026-07-16): `family_sweep --hseq` compiled the -O0 cluster members
|
||||
at -O2 (member stub files are -O2), so their remapped bodies masked-MATCH but fail the whole-binary
|
||||
gate. gcc-2.7.2 has no per-function optimize pragma (opt is per-FILE), so — exactly like the whale
|
||||
(rollout_whale_o0.py, §38) and ov_SC01_077's own _o0 split (Phase-19 §18) — each overlay must
|
||||
compile the cluster in its own -O0 TU. This tool builds that split; the family sweep then fills it.
|
||||
|
||||
The cluster is a contiguous run of 16 fns (all INCLUDE_ASM stubs in an unswept overlay). We carve
|
||||
<ov>.c into 3 objects (a single object's .text can't be split around a middle object):
|
||||
- <ov>.c = before the cluster (0x0..0x13410), keeps name + nonmatchings/<ov> asm paths
|
||||
- <ov>_o0.c = the -O0 cluster (0x13410..0x14834), asm paths -> nonmatchings/<ov>_o0
|
||||
- <ov>_o2b.c = after the cluster (0x14834..data tail), asm paths -> nonmatchings/<ov>_o2b
|
||||
The Makefile O0_CLUSTER_OBJS wildcard -O0-compiles every <ov>_o0.o. Byte-NEUTRAL by construction:
|
||||
INCLUDE_ASM pastes the ORIGINAL bytes regardless of -O0/-O2, so a freshly-carved (all-stub) overlay
|
||||
stays byte-identical — gate it before sweeping. Idempotent (skips carved overlays). The whole-binary
|
||||
byte-gate (make check-all) is the sole arbiter (G3/P9). Usage:
|
||||
rollout_o0_cluster.py <ov> [<ov> ...] # single-file overlays only (files=1); whale/jr-carved
|
||||
# overlays keep the cluster inside jr_801380E0 (not handled)
|
||||
"""
|
||||
import sys, re, os
|
||||
|
||||
BASE = 0x80128158
|
||||
LO_FOFF, HI_FOFF = 0x13410, 0x14834 # cluster vram 0x8013B568 .. 0x8013C98C
|
||||
START_FN = "func_8013B568" # first fn of the cluster (0x13410)
|
||||
LAST_FN = "func_8013C964" # last fn of the cluster (just below 0x14834)
|
||||
CLUSTER_FNS = [ # the 16 contiguous -O0 fns (from ov_SC01_077_o0.c)
|
||||
"func_8013B568", "func_8013B598", "func_8013B6A0", "func_8013B7AC",
|
||||
"func_8013B7F4", "func_8013B83C", "func_8013BC7C", "func_8013BCDC",
|
||||
"func_8013BD34", "func_8013BD74", "func_8013C08C", "func_8013C0F8",
|
||||
"func_8013C360", "func_8013C414", "func_8013C938", "func_8013C964",
|
||||
]
|
||||
|
||||
|
||||
def header_end(lines):
|
||||
"""Header = the leading #include block (+ trailing blanks). Stops at the first non-#include,
|
||||
non-blank line (a DEFINE_func macro / decl), so the low-address dedup macros stay in `before`."""
|
||||
last_inc = -1
|
||||
for i, l in enumerate(lines):
|
||||
s = l.strip()
|
||||
if s.startswith("#include"):
|
||||
last_inc = i
|
||||
elif s and last_inc >= 0:
|
||||
break
|
||||
he = last_inc + 1
|
||||
while he < len(lines) and lines[he].strip() == "":
|
||||
he += 1
|
||||
return he
|
||||
|
||||
|
||||
def unit_start(lines, fn):
|
||||
"""Line index where `fn`'s definition/stub begins (an INCLUDE_ASM stub or a `... fn(` def line)."""
|
||||
for i, l in enumerate(lines):
|
||||
if re.search(rf'INCLUDE_ASM\("[^"]*",\s*{fn}\)\s*;', l):
|
||||
return i, "stub"
|
||||
if re.search(rf'\b{fn}\s*\(', l) and not l.lstrip().startswith(("//", "*", "extern")) \
|
||||
and "INCLUDE_ASM" not in l and (l.rstrip().endswith("{") or ")" in l and ";" not in l):
|
||||
return i, "def"
|
||||
return None, None
|
||||
|
||||
|
||||
def split_overlay(ov):
|
||||
cdir = f"src/{ov}"
|
||||
main_c = f"{cdir}/{ov}.c"
|
||||
o0_c, o2b_c = f"{cdir}/{ov}_o0.c", f"{cdir}/{ov}_o2b.c"
|
||||
yaml = f"config/splat.{ov}.yaml"
|
||||
if os.path.exists(o0_c):
|
||||
return f"SKIP {ov} (o0 exists)"
|
||||
if not (os.path.exists(main_c) and os.path.exists(yaml)):
|
||||
return f"SKIP {ov} (missing .c or yaml)"
|
||||
lines = open(main_c).read().split("\n")
|
||||
|
||||
# every cluster fn must be present in THIS file (single-file overlay) — else the cluster is
|
||||
# carved elsewhere (jr-embedded overlay) and this tool must not touch it.
|
||||
missing = [fn for fn in CLUSTER_FNS if not any(re.search(rf'\b{fn}\b', l) for l in lines)]
|
||||
if missing:
|
||||
return f"SKIP {ov} (cluster fns not in base .c: {missing[:3]}... — jr-embedded?)"
|
||||
|
||||
si, _ = unit_start(lines, START_FN)
|
||||
li, lkind = unit_start(lines, LAST_FN)
|
||||
if si is None or li is None:
|
||||
return f"ERROR {ov}: boundary fn not found (start={si}, last={li})"
|
||||
if lkind != "stub":
|
||||
return f"SKIP {ov} ({LAST_FN} is matched inline — cluster not all-stub; carve by hand)"
|
||||
# end = first non-blank line after the LAST_FN stub (keeps the after-region fn's forward externs
|
||||
# with the after-region, not the cluster).
|
||||
ei = li + 1
|
||||
while ei < len(lines) and lines[ei].strip() == "":
|
||||
ei += 1
|
||||
if not (si < li < ei):
|
||||
return f"ERROR {ov}: bad boundary order si={si} li={li} ei={ei}"
|
||||
|
||||
header = "\n".join(lines[:header_end(lines)]).rstrip("\n")
|
||||
before, cluster, after = lines[:si], lines[si:ei], lines[ei:]
|
||||
|
||||
OLD = f'nonmatchings/{ov}"'
|
||||
open(main_c, "w").write("\n".join(before).rstrip("\n") + "\n")
|
||||
cluster_body = "\n".join(cluster).replace(OLD, f'nonmatchings/{ov}_o0"')
|
||||
open(o0_c, "w").write(header + "\n\n" + cluster_body.rstrip("\n") + "\n")
|
||||
after_body = "\n".join(after).replace(OLD, f'nonmatchings/{ov}_o2b"')
|
||||
open(o2b_c, "w").write(header + "\n\n" + after_body.rstrip("\n") + "\n")
|
||||
|
||||
# carve the yaml code subseg [0x0, c, <ov>] -> before / o0 cluster / o2b after
|
||||
y = open(yaml).read()
|
||||
pat = re.compile(r'^(\s*)- \[0x0, c, ' + re.escape(ov) + r'\].*$', re.M)
|
||||
m = pat.search(y)
|
||||
if not m:
|
||||
return f"ERROR {ov}: '- [0x0, c, {ov}]' not found in yaml"
|
||||
ind = m.group(1)
|
||||
repl = (f'{ind}- [0x0, c, {ov}] # -O2 before -O0 cluster (file 0x0..0x{LO_FOFF:X})\n'
|
||||
f'{ind}- [0x{LO_FOFF:X}, c, {ov}_o0] # -O0 cluster 0x8013B568..0x8013C98C (0x{LO_FOFF:X}..0x{HI_FOFF:X})\n'
|
||||
f'{ind}- [0x{HI_FOFF:X}, c, {ov}_o2b] # -O2 after -O0 cluster (0x{HI_FOFF:X}..data tail)')
|
||||
open(yaml, "w").write(pat.sub(lambda _m: repl, y, count=1))
|
||||
ncl = sum(1 for l in cluster if "INCLUDE_ASM" in l)
|
||||
return f"OK {ov}: before {len(before)} / cluster {len(cluster)} ({ncl} stubs) / after {len(after)} lines"
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
targets = sys.argv[1:]
|
||||
if not targets:
|
||||
print(__doc__)
|
||||
sys.exit(2)
|
||||
n_ok = 0
|
||||
for ov in targets:
|
||||
r = split_overlay(ov)
|
||||
if r.startswith("OK"):
|
||||
n_ok += 1
|
||||
print(r)
|
||||
print(f"--- {n_ok}/{len(targets)} carved ---")
|
||||
@@ -1,91 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase-24 W9: roll out the -O0 whale (func_80144B9C) to single-file overlays for the ×134 bank.
|
||||
|
||||
The whale is byte-identical in all 134 overlays (reach-134) but only matches at -O0 (gcc-2.7.2 has
|
||||
no per-function optimize pragma), so each overlay must compile it in its own -O0 TU. Per overlay
|
||||
(<ov>.c is splat-emitted in vram order, so a line-split at the whale is safe):
|
||||
- carve the code subseg into before / o0b(-O0 whale) / after in config/splat.<ov>.yaml
|
||||
- <ov>.c = before-whale (keeps name + nonmatchings/<ov> asm paths)
|
||||
- <ov>_o0b.c = a thin wrapper: #include common.h + the shared src/shared/func_80144B9C.h
|
||||
- <ov>_after.c = header (includes) + after-whale, asm paths -> nonmatchings/<ov>_after
|
||||
The Makefile WHALE_O0B_OBJS wildcard rule -O0-compiles every <ov>_o0b.o. Idempotent (skips carved
|
||||
overlays). The whole-binary byte-gate (make check-all) is the sole arbiter. Usage:
|
||||
rollout_whale_o0.py <ov> [<ov> ...] or rollout_whale_o0.py --all
|
||||
"""
|
||||
import sys, re, os, glob
|
||||
|
||||
WHALE = "func_80144B9C"
|
||||
WHALE_FOFF = 0x1CA44 # vram 0x80144B9C - base 0x80128158
|
||||
AFTER_FOFF = 0x1D64C # vram 0x801457A4 (next fn) - base 0x80128158
|
||||
|
||||
|
||||
def header_end(lines):
|
||||
"""Header = the leading #include block (+ trailing blanks). Single-file overlays have no
|
||||
file-scope shared externs (each fn declares its own), so this captures common.h + engine_core.h."""
|
||||
last_inc = -1
|
||||
for i, l in enumerate(lines):
|
||||
s = l.strip()
|
||||
if s.startswith("#include"):
|
||||
last_inc = i
|
||||
elif s and last_inc >= 0:
|
||||
break
|
||||
he = last_inc + 1
|
||||
while he < len(lines) and lines[he].strip() == "":
|
||||
he += 1
|
||||
return he
|
||||
|
||||
|
||||
def split_overlay(ov):
|
||||
cdir = f"src/{ov}"
|
||||
main_c, o0b_c, after_c = f"{cdir}/{ov}.c", f"{cdir}/{ov}_o0b.c", f"{cdir}/{ov}_after.c"
|
||||
yaml = f"config/splat.{ov}.yaml"
|
||||
if os.path.exists(o0b_c):
|
||||
return f"SKIP {ov} (o0b exists)"
|
||||
if not (os.path.exists(main_c) and os.path.exists(yaml)):
|
||||
return f"SKIP {ov} (missing .c or yaml)"
|
||||
lines = open(main_c).read().split("\n")
|
||||
wi = [i for i, l in enumerate(lines) if WHALE in l and "INCLUDE_ASM" in l]
|
||||
if len(wi) != 1:
|
||||
return f"SKIP {ov} (whale INCLUDE_ASM count={len(wi)} — not a single-file stub)"
|
||||
wi = wi[0]
|
||||
he = header_end(lines)
|
||||
header = "\n".join(lines[:he]).rstrip("\n")
|
||||
before, after = lines[:wi], lines[wi + 1:]
|
||||
OLD = f'nonmatchings/{ov}"'
|
||||
open(main_c, "w").write("\n".join(before).rstrip("\n") + "\n")
|
||||
open(o0b_c, "w").write('#include "common.h"\n#include "../shared/func_80144B9C.h"\n')
|
||||
after_body = "\n".join(after).replace(OLD, f'nonmatchings/{ov}_after"')
|
||||
open(after_c, "w").write(header + "\n\n" + after_body.rstrip("\n") + "\n")
|
||||
# carve the yaml code subseg
|
||||
y = open(yaml).read()
|
||||
pat = re.compile(r'^(\s*)- \[0x0, c, ' + re.escape(ov) + r'\].*$', re.M)
|
||||
m = pat.search(y)
|
||||
if not m:
|
||||
return f"ERROR {ov}: '- [0x0, c, {ov}]' not found in yaml"
|
||||
ind = m.group(1)
|
||||
repl = (f'{ind}- [0x0, c, {ov}] # -O2 before whale (file 0x0..0x{WHALE_FOFF:X})\n'
|
||||
f'{ind}- [0x{WHALE_FOFF:X}, c, {ov}_o0b] # -O0 whale func_80144B9C (0x{WHALE_FOFF:X}..0x{AFTER_FOFF:X})\n'
|
||||
f'{ind}- [0x{AFTER_FOFF:X}, c, {ov}_after] # -O2 after whale (0x{AFTER_FOFF:X}..data tail)')
|
||||
open(yaml, "w").write(pat.sub(lambda _m: repl, y, count=1))
|
||||
return f"OK {ov}: before {len(before)} / after {len(after)} lines"
|
||||
|
||||
|
||||
def all_overlays():
|
||||
ovs = []
|
||||
for p in sorted(glob.glob("config/splat.ov_*.yaml")):
|
||||
ov = os.path.basename(p)[len("splat."):-len(".yaml")]
|
||||
if ov != "ov_SC01_077": # already carved (the harvest source, different structure)
|
||||
ovs.append(ov)
|
||||
return ovs
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
args = sys.argv[1:]
|
||||
targets = all_overlays() if args == ["--all"] else args
|
||||
n_ok = 0
|
||||
for ov in targets:
|
||||
r = split_overlay(ov)
|
||||
if r.startswith("OK"):
|
||||
n_ok += 1
|
||||
print(r)
|
||||
print(f"--- {n_ok}/{len(targets)} carved ---")
|
||||
Reference in New Issue
Block a user