From 2f7e06564b9d14ecbc41b096dcf3c1959a9fa2e6 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Thu, 13 Aug 2026 23:41:36 -0600 Subject: [PATCH] =?UTF-8?q?feat(phase-30=20S50):=20the=20stale=20seed=20sy?= =?UTF-8?q?mbol=20=E2=80=94=20A-prop's=2057%=20was=20ours,=20not=20the=20l?= =?UTF-8?q?ane's=20(=C2=A7171)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - REFUTES §170's open hypothesis (batched cards concentrate members into one TU ⇒ §169 collision): 5-draft groups banked 5/5; 11 of 35 unbanked drafts were already one-per-TU; and the two "concentrated" groups banked 12/12 and 10/10 once the real defect was fixed. - The cause: a per-location data symbol carried out of the seed body unrebased. match_one compares instruction ENCODINGS and is blind to a relocation's target NAME, so it scores MATCH standalone and dies at link in the host TU. 24 of 24 concentrated failures, all 1:1 rewritable at one constant vram delta (0x4128). - tools/aprop_symfix.py: audit + --fix, emits a gate_lane-shaped slate; deterministic and build-free, so it runs BEFORE the gate. The R34 second oracle for the class match_one cannot see. - family_cousins.py --aprop-cards: members now carry sym_map, the explicit {seed -> member} renames, read from the seed's C BODY (a matched seed has no .s of its own) vs the member's .s. Two case-mismatch defects fixed while wiring it (sig lowercase vs splat uppercase). - 23/24 banked. Stubs 12,468 -> 12,445. Fleet 95.2% instr / 89.9% distinct / 96.57% fn. R22 clean rebuild: check-all 213 passed, 0 failed of 213. dedup 2,043/0. - A-prop's true conversion is 87% (79/91); the 320 batched members are unblocked. - Cookbook §171 + §170 struck in place; SETUP row; decision-log (R31). --- docs/SETUP.md | 1 + docs/decision-log.md | 37 ++++++++ docs/matching-cookbook.md | 68 +++++++++++-- docs/progress.fleet.md | 14 +-- phase-ends/CURRENT_PHASE.md | 51 +++++++++- tools/aprop_symfix.py | 185 ++++++++++++++++++++++++++++++++++++ tools/family_cousins.py | 33 ++++++- 7 files changed, 374 insertions(+), 15 deletions(-) create mode 100644 tools/aprop_symfix.py diff --git a/docs/SETUP.md b/docs/SETUP.md index 1942840afc..b6dec1fcc2 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -673,6 +673,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/ghidra_import_raw.sh` | Headless import of a RAW flat blob — `BinaryLoader` + `--loader-baseAddr ` + `PSX:LE:32:default` (resident blob / Gen2 overlays; no PS-X EXE header). | | | `tools/prefetch_fleet.py` | **The fleet Ghidra-C prefetch batch** (P30 T0.5): one representative per remaining h_seq distinct class + ALL main/resident stubs → `.run/ghidra_c/`, resumable (skips cached), serial on the project lock; auto-stops a serving MCP (R23 — R29 applies at next MCP use); imports a missing overlay program on demand (`ghidra_import_raw.sh`, blob via `family_remap.img_path`, vram via the splat yaml). `--dry-run` / `--limit-programs N` (probe-first). R32: per-program outcome report, batch continues past failures. | | | `tools/family_cousins.py` | **(P30 S49, cookbook §168)** The COUSIN-UNIT survey — the similarity tier ABOVE h_seq. h_seq is an exact skeleton hash, so ±1 instruction (li-expansion, table-size drift) fragments same-source families into "singletons"; this tool clusters the distinct open skeletons by mnemonic-stream similarity (≥0.85, union-find) and attaches the best MATCHED-skeleton seed per unit → `.run/family_cousins.json` + `docs/family-cousins.md`. Categories A-prop / seeded / cousin-multi / cold. `--targets N --wave waveN` emits the crack_wave.js slate ranked by whole-UNIT open ins, with each seed's C body location (engine_core.h macro or inline src file) resolved. R32 both ways: fails loud on a stale family map (independent stub recount) and on a partition defect. RANKS AND SEEDS ONLY — cousins need a per-member seeded CRACK (skeleton drift ⇒ recompile), never a `family_sweep` remap; the whole-binary byte-gate stays the sole arbiter. | +| | `tools/aprop_symfix.py` | **(P30 S50, cookbook §171)** The stale-SEED-SYMBOL guard for the adapt/A-prop lanes — the second, DISAGREEING oracle (R34) for the one class `match_one` is structurally blind to. A per-location data symbol (`D_8018xxxx`) carried out of a seed body unrebased scores MATCH standalone (encodings identical; the scorer cannot see a relocation's target NAME) and dies in the host TU at link with `undefined reference`. Audits every draft's vram-suffixed symbols against the symbols the TARGET's own `.s` relocates; `--fix` rewrites the 1:1 cases and emits a `gate_lane`-shaped slate. Deterministic, no build — run it BEFORE the gate, never after. Measured on the S49 A-prop residue: 24 of 24 concentrated failures were this class, 23 banked after the rebase (A-prop conversion 57% -> 87%). Its primitives are imported by `family_cousins.py`, whose `--aprop-cards` members now carry an explicit `sym_map` of `{seed -> member}` renames. | | **Disc/.CD extraction** (`tools/bfm_extract/`) | `extract.py` | Walk the disc / extract root files (`make extract`). | | | `extract_exe.py` | Extract & verify `SLUS_007.26` (`--verify-disc`, owns `EXPECTED_EXE_SHA1`). | | | `extract_proto_exe.py` | Extract the prototype/demo EXE for cross-checking. | diff --git a/docs/decision-log.md b/docs/decision-log.md index 6ef824520e..86ea307535 100644 --- a/docs/decision-log.md +++ b/docs/decision-log.md @@ -2318,3 +2318,40 @@ DEBUG-MENU-LIST was the session's force multiplier. (2) Negative results with ev product: six payloads now carry "never loads in X/Y/Z" instead of "unknown". (3) The write-API no-op control was a real R35 miss — verify instruments with a VISIBLE effect. (4) R22 catching md_MAIN_003 three times in one evening is the rule working exactly as designed. + +## 2026-08-13 (P30 S50) — the A-prop conversion gap: the experiment the artifacts had already run + +**Context and belief.** S49's closing checkpoint made one thing the session's first job, on the +grounds that it "prices everything else": re-gate the 42 (truly 35) unbanked A-prop drafts ONE PER +TU, to test §170's hypothesis that family-batched cards concentrate members into a single +destination TU and die of the §169 collision. 320 further batched members — roughly 15M tokens of +wave — were explicitly held behind that measurement. The belief was that A-prop's 91%-agent / +57%-gate conversion was an integration-topology problem. + +**What failed.** The hypothesis, and the framing that made it look like it needed an experiment. +Three artifacts already in `.run/` answered it: `gate_aprop1.json` records **5-draft single-TU +groups banking 5/5** (batch size is not the discriminator); **11 of the 35** unbanked drafts were +already single-draft groups, i.e. the proposed test had already been run on them; and +`harvest_failed.ov_SC03_107.classified.txt` names the actual failure verbatim — `PLUMBING: +undefined reference to 'D_80181900'`, eleven times. §169's own law ("read the classified file +before theorising about any sweep failure") was written for exactly this and was not applied to the +sibling lane. + +**The pivot.** Diagnose from the recorded verdicts first; run the experiment only for what they do +not answer. A 40-line static audit — compare each draft's vram-suffixed symbols against the symbols +the target's own `.s` relocates — classified all 35 in under a second: **24 stale-seed-symbol, 11 +genuine DIFF**, zero ambiguity. One probe banked, then 22 of the remaining 23. + +**Why (byte-grounded).** A per-location data symbol is the seed's ENVIRONMENT, not its logic. +`match_one` compares instruction encodings and is blind to a relocation's target NAME, so a carried +symbol scores MATCH standalone and fails at link inside the host TU. A-prop's real conversion is +**87% (79/91)**, not 57% — the lane was never the problem, and the 320 held-back members are worth +substantially more than they were priced at. + +**Hindsight — the better path.** The cheap deterministic audit should have been part of the wave's +verification step from the start: it needs no build, it is the second oracle (R34) for the one class +`match_one` structurally cannot see, and it would have converted these 24 in S49 instead of leaving +them to be re-measured a session later. Generalized rule: **before designing an experiment to +explain a failure rate, grep the failure verdicts the tools already wrote** — and when a checkpoint +declares a test the top priority, that is a hypothesis with a plan attached, not a finding (R14/R35 +applied to my own handoff notes). diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index 6ed6dfe576..1e8e248d90 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -16325,12 +16325,21 @@ best agent rate of any wave type — **→ 56 banked (57%)**, ≈ **80k tok per per-member cousin card's 157k and a crack wave's 400k+. The agent notes are substitutions, not decompilations (`"src struct = D_801D0358"`; `"cb=func_8017FBA4; 0x12C, 0x4B0, 0x38E…"`). -**⚠ The conversion gap, and the test before scaling.** 91% agent → 57% gate is the WORST conversion -measured (the per-member cousin wave ran 81% → 92%). 14 of ~50 groups banked zero. **Hypothesis, -not yet proven:** family-batched cards CONCENTRATE members into one destination TU by construction — -precisely the §169 collision the spread law names, which held the 7a wave to 64%. **Re-gate the -unbanked drafts ONE PER TU before scaling a batched A-prop wave**; if the spread law is the cause the -same drafts bank, and the fix is to interleave families per gate batch rather than to redraft. +**⚠ The conversion gap.** 91% agent → 57% gate is the WORST conversion measured (the per-member +cousin wave ran 81% → 92%). 14 of ~50 groups banked zero. + +> ~~**Hypothesis, not yet proven:** family-batched cards CONCENTRATE members into one destination TU +> by construction — precisely the §169 collision the spread law names, which held the 7a wave to 64%. +> **Re-gate the unbanked drafts ONE PER TU before scaling a batched A-prop wave**; if the spread law +> is the cause the same drafts bank, and the fix is to interleave families per gate batch rather than +> to redraft.~~ +> +> **STRUCK 2026-08-13 (S50) — REFUTED. Batching was never the discriminator (5-draft groups banked +> 5/5; the two "concentrated" groups banked 12/12 and 10/10 once fixed). The cause was a STALE SEED +> SYMBOL: `match_one` is blind to a relocation's target NAME, so a carried-over `D_8018xxxx` scores +> MATCH standalone and dies at link. 23 of 24 banked after a mechanical rebase; A-prop's true +> conversion is 87%, not 57%. See §171 — and note the test itself was never needed: the answer was +> already sitting in `.run/harvest_failed..classified.txt`.** **Three ops notes paid for on the head:** (1) `family_sweep --hseq` defaults to `--band substantial`, so tiny/mid families are silently OUT OF SCOPE — a "never attempted" verdict can be a band default, @@ -16338,3 +16347,50 @@ not a difficulty (5 of 13 heads); (2) `dedup_extend` extends MACRO-backed groups INLINE def use `dedup_propagate --addr`; (3) that tool names its own blocker precisely (`missing file-scope extern (CARRY-FIXABLE): D_…`) — a data symbol the matched body's own TU never declares at file scope blocks every sibling until it is carried. + +## §171 — THE STALE SEED SYMBOL (P30 S50, 2026-08-13): why §170's 91%→57% was never codegen + +**§170 left one hypothesis open — that family-batched A-prop cards CONCENTRATE members into one +destination TU and die of the §169 collision. It is REFUTED, by data that already existed and by a +direct test.** Refuted three ways: (1) the recorded S49 verdicts show 5-draft single-TU groups +banking **5/5** twice, and 4/4 twice — batch size was never the discriminator; (2) 11 of the 35 +unbanked drafts were **single-draft groups**, i.e. already gated one-per-TU, and all 11 classify +`DIFF`; (3) after the real fix below, the two "concentrated" groups §170 blamed — 12 drafts and 10 +drafts into ONE `.c` each — banked **12/12 and 10/10**. + +**The real defect, and it is one line per draft.** A per-location data symbol (`D_8018xxxx` — a +function-pointer table, a jump table, a state array) is part of the seed's **ENVIRONMENT, not its +logic**. The adapt/A-prop lanes hand an agent a proven seed body plus a word-diff; the agent edits +the logic and carries the seed's symbol across unrebased. Then: + +- **`match_one` scores it MATCH.** It compiles standalone and compares instruction ENCODINGS. + `%hi(D_seed)` and `%hi(D_target)` are the same instruction with a different relocation TARGET + NAME, and the scorer is blind to the name. +- **The whole-binary gate kills it at LINK:** `undefined reference to 'D_80181900'`. + +Standalone-MATCH / host-TU-link-failure. That class was **24 of the 24** concentrated A-prop +failures — the entire conversion gap. In the word-diff card the rename is visible only as an opaque +**IMM** site (the low half of a `lui`/`lw` `%hi`/`%lo` pair), which is exactly the site class an +agent reads as "an immediate to copy", not "a symbol to rebase". + +**Measured:** all 24 were 1:1 rewritable, every one at the same seed→target vram delta (`0x4128` — +one binary's data section offset from the seed's). Rebased mechanically: **23/24 banked** (the 24th +is a genuine `DIFF`). A-prop's real conversion is **56 → 79 of 91 = 87%**, not 57%; the agents' +91% MATCH claim was very nearly right and the gap was ours. + +**Two tools, one primitive** (`tools/aprop_symfix.py`, imported by `family_cousins.py`): +- **Post-hoc guard** — `aprop_symfix.py --fix` audits every draft's vram-suffixed + symbols against the symbols the TARGET's own `.s` relocates, rewrites the 1:1 cases, and emits a + `gate_lane`-shaped slate. Deterministic, no build, so it belongs BEFORE the gate, never after. +- **Pre-hoc annotation** — `--aprop-cards` members now carry `sym_map`: the explicit + `{seed → member}` renames, computed from the seed's C BODY (a matched seed has no `.s` of its + own — it is compiled from C) versus the member's `.s`. It turns a puzzle into an instruction. + +This is R34 in one line: **`match_one` is a perfect codegen oracle and a NULL linkage oracle.** The +symbol audit is the second oracle that can disagree with it, and it costs no build. + +**Two spellings, one bug, twice:** `sig_image` writes hex lowercase, splat writes it uppercase, and +`seed_body_ref` builds `func_%08X`. Both defects found while wiring this (a NO_SEED_BODY on every +macro seed, an AMBIGUOUS on every 1:1 rename) were case mismatches — the same class §128/R35 keeps +naming. Compare function identities case-insensitively, and strip the `DEFINE_` prefix, or the +seed's own name reads as a stale symbol. diff --git a/docs/progress.fleet.md b/docs/progress.fleet.md index 44261cfe29..efec78fd3c 100644 --- a/docs/progress.fleet.md +++ b/docs/progress.fleet.md @@ -4,16 +4,16 @@ # cross-binary collapsible-byte leverage: docs/duplicates.cross.md. # THREE progress metrics (all matter — see the labels): -FLEET fn-count byte-ident: 350688 / 363163 = 96.56% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay) -FLEET instr-weighted : 12872722 / 13523856 = 95.2% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number) -FLEET distinct-code(uniq): 5260480 / 5851963 = 89.9% (81090/90928 unique fns; the DISTINCT-RE number) +FLEET fn-count byte-ident: 350711 / 363163 = 96.57% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay) +FLEET instr-weighted : 12873067 / 13523856 = 95.2% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number) +FLEET distinct-code(uniq): 5260690 / 5851963 = 89.9% (81104/90928 unique fns; the DISTINCT-RE number) MAIN game-code weighted : 436 / 79510 = 0.5% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-08-05; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift) - (fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 12872286 / 13444346 = 95.7%) + (fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 12872631 / 13444346 = 95.7%) -FLEET REAL substantive : 348518 (of which dedup-shared 254400 via 2043 groups / 254474 instances) +FLEET REAL substantive : 348541 (of which dedup-shared 254400 via 2043 groups / 254474 instances) FLEET LINKED PsyQ objs : 959 FLEET NON_MATCHING : 7 (0 in any default build — G4) -FLEET INCLUDE_ASM stubs : 12468 +FLEET INCLUDE_ASM stubs : 12445 FLEET matchable : 363163 | binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % | @@ -153,7 +153,7 @@ FLEET matchable : 363163 | ov_SC03_103 | 2497 | 1808 | 0 | 2500 | 2510 | 99.6% | | ov_SC03_104 | 2549 | 1811 | 0 | 2556 | 2616 | 97.7% | | ov_SC03_105 | 2451 | 1807 | 0 | 2458 | 2597 | 94.6% | -| ov_SC03_107 | 2268 | 1690 | 0 | 2351 | 2497 | 94.2% | +| ov_SC03_107 | 2291 | 1690 | 0 | 2374 | 2497 | 95.1% | | ov_SC03_108 | 2403 | 1806 | 0 | 2403 | 2443 | 98.4% | | ov_SC03_109 | 2409 | 1807 | 0 | 2411 | 2424 | 99.5% | | ov_SC03_110 | 2443 | 1830 | 0 | 2443 | 2468 | 99.0% | diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index a47ad9c480..886e5c21c6 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -181,7 +181,56 @@ stub on a named wall/behemoth/queue ledger** — 140/140 byte-identical througho --- -# 🛑 SESSION S49 CHECKPOINT — FINAL (2026-08-13) — FRESH SESSION SAFE HERE +# 🛑 SESSION S50 CHECKPOINT (2026-08-13) — FRESH SESSION SAFE HERE +> **Tree CLEAN** but for R23 `db.*.gbf` churn, a regenerable `.run/backlog.jsonl` edit, and two +> stray `gccdump.*` cc1 dumps at the repo root — never stage any of them. **Nothing running.** +> Gate: **`check-all` 213 passed / 0 failed of 213** from a CLEAN rebuild (`make clean && +> make extract-all && make check-all`, R22). +> **Fleet: 95.2% instr · 89.9% distinct · 96.57% fn-count · INCLUDE_ASM stubs 12,445** (was +> 12,468). **23 instances banked.** dedup 2,043/0. Cookbook **§171**; §170's open hypothesis +> STRUCK. Decision-log entry filed (R31). + +## ✅ S50 T6/T7 — the S49 resume-block step 1 is CLOSED, and it closed by REFUTATION +**The A-prop TU-spread test was never needed — the artifacts had already run it.** §170's +hypothesis (family-batched cards concentrate members into one destination TU ⇒ the §169 collision) +is **refuted three ways**: `gate_aprop1.json` shows 5-draft single-TU groups banking **5/5**; +**11 of the 35** unbanked drafts were already single-draft groups (all classify `DIFF`); and after +the real fix the two "concentrated" groups banked **12/12 and 10/10**. + +**The real cause (cookbook §171): the STALE SEED SYMBOL.** A per-location data symbol +(`D_8018xxxx`) is the seed's ENVIRONMENT, not its logic. Carried out of a seed body unrebased it +scores **MATCH in `match_one`** — which compares instruction ENCODINGS and is blind to a +relocation's target NAME — and dies at link inside the host TU (`undefined reference`). It was +**24 of the 24** concentrated A-prop failures, all 1:1 rewritable at one constant vram delta +(`0x4128`). Rebased mechanically → **23/24 banked** (the 24th, `func_801659DC`, is a genuine DIFF). +**A-prop's true conversion is 87% (79/91), not 57%.** + +**Shipped:** `tools/aprop_symfix.py` (audit + `--fix`, emits a `gate_lane`-shaped slate; no build, +so it runs BEFORE the gate) · `family_cousins.py --aprop-cards` members now carry an explicit +`sym_map` of `{seed → member}` renames, computed from the seed's C BODY (a matched seed has no `.s` +of its own) vs the member's `.s` · SETUP row · cookbook §171 · decision-log. + +## ▶ RESUME HERE +1. **The 320 batched A-prop members are re-priced and UNBLOCKED** — the 57% that held them back was + ours, not the lane's. Run the wave, and put `aprop_symfix --fix` in its verification step before + the gate. Regen chain first (the tools fail loud on a stale map, by design): + `tools/family_hseq.py` → `tools/family_cousins.py` → `--aprop-cards`. +2. **The ≥16 head, still open:** the **`D_801ED98C` carry fix** (one missing file-scope extern gates + **83 PURE members** — `dedup_propagate` names it itself) · **`func_8017C294`** (×16, 246 ins, + NEAR at 2 instructions after three seeded attempts — the board's biggest single crack and free + permuter fuel). +3. **12 A-prop drafts stay genuine `DIFF`** (11 singletons + `func_801659DC`) — redraft/permuter + fuel, not recovery fuel. + +## 🔑 THE ONE THING TO CARRY FORWARD +**Before designing an experiment to explain a failure rate, grep the failure verdicts the tools +already wrote.** §169's own law ("read the classified file before theorising") was written for this +and was not applied to the sibling lane; `harvest_failed..classified.txt` named the cause +verbatim, eleven times over, while the checkpoint called for a multi-hour measurement campaign. +Second: **`match_one` is a perfect codegen oracle and a NULL linkage oracle** — the symbol audit is +the R34 second oracle for the one class it cannot see, and it costs no build. + +# ▶ (superseded by S50) SESSION S49 CHECKPOINT — FINAL (2026-08-13) > **Tree CLEAN** but for R23 `db.*.gbf` churn + a regenerable `.run/backlog.jsonl` edit — never > stage either. **Nothing running.** 179 commits this session (`commit:1776`..`commit:1891`). > Gate at close: **`check-all` 213 passed / 0 failed of 213** from a CLEAN rebuild (R22 run **5×** diff --git a/tools/aprop_symfix.py b/tools/aprop_symfix.py new file mode 100644 index 0000000000..897d50d62e --- /dev/null +++ b/tools/aprop_symfix.py @@ -0,0 +1,185 @@ +"""Rebase stale SEED symbols in adapt/A-prop drafts onto the TARGET's own symbols. + +WHY THIS EXISTS (S50, cookbook §171). The adapt/A-prop lanes hand an agent a *proven seed body* +from one location plus a small token-diff, and the agent adapts the code — but a per-location data +symbol (`D_8018xxxx`, a function-pointer table, a jump table) is part of the seed's ENVIRONMENT, +not its logic. Carry it over unrebased and: + + - `match_one` still scores MATCH. It compiles standalone and compares instruction encodings; + a `%hi(D_seed)` and a `%hi(D_target)` are the same instruction with a different relocation + TARGET NAME, and the scorer is blind to the name. + - the whole-binary gate then fails at LINK: `undefined reference to 'D_seed'`. + +That gap — standalone-MATCH, link-undefined — was the entire A-prop 91%-agent-vs-57%-gate delta +measured in S49 (24 of 24 concentrated failures; §170's TU-collision hypothesis was refuted by +the same data: 5-draft groups banked 5/5). + +This is the SECOND, DISAGREEING ORACLE (R34) for that class: match_one says MATCH, the symbol +audit says the draft references a symbol the target's own .s never mentions. It is deterministic +and costs no build, so it belongs BEFORE the gate, never after. + + audit: tools/aprop_symfix.py (report only) + fix: tools/aprop_symfix.py --fix -> .run/aprop_symfix//.c + slate + +A slate record is {fn|name, binary, draft}; the emitted slate points `draft` at the fixed copy so +`tools/gate_lane.py` consumes it unchanged. +""" +import argparse, collections, json, os, re, sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__))) +import corpus + +# vram-suffixed symbols: D_80185A28, func_801727D0, jtbl_8018xxxx, ... +SYM = re.compile(r'\b[A-Za-z_][A-Za-z0-9_]*_?8[0-9A-Fa-f]{7}\b') +ASM_OPS = re.compile(r'%hi\(([^)]+)\)|%lo\(([^)]+)\)|\bjal\s+([A-Za-z_]\w*)|\.word\s+([A-Za-z_]\w*)') + + +def _strip_comments(txt): + return re.sub(r'//[^\n]*', '', re.sub(r'/\*.*?\*/', '', txt, flags=re.S)) + + +def syms_in_text(txt): + return set(SYM.findall(_strip_comments(txt))) + + +def draft_syms(path): + return syms_in_text(open(path).read()) + + +def body_text(path, name): + """The C body of ONE function out of a file (a draft is one function; a seed's file is not). + + Matches an `DEFINE_(` engine_core macro or a plain definition, then brace-matches. A + whole-file scan would drag every OTHER function's symbols into the comparison and drown the + signal, so callers annotating a SEED must pass through here.""" + if not path or not os.path.isfile(path): + return None + txt = _strip_comments(open(path).read()) + m = (re.search(rf'\bDEFINE_{re.escape(name)}\s*\(', txt) or + re.search(rf'^[A-Za-z_][^\n=;]*\b{re.escape(name)}\s*\(', txt, re.M)) + if not m: + return None + i = txt.find('{', m.start()) + if i < 0: + return None + depth = 0 + for j in range(i, len(txt)): + if txt[j] == '{': + depth += 1 + elif txt[j] == '}': + depth -= 1 + if depth == 0: + return txt[m.start():j + 1] + return None + + +def asm_syms(path): + out = set() + for m in ASM_OPS.finditer(open(path).read()): + for g in m.groups(): + if g: + out.add(g.split('+')[0].strip()) + return out + + +def diff_syms(c_path, asm_path, self_name): + """Compare a C body's symbols against the symbols its target .s actually relocates. + + Shared primitive: `aprop_symfix` uses it on a DRAFT (post-hoc guard), `family_cousins.sym_map` + uses it on the SEED BODY (pre-hoc card annotation). A matched seed has no `.s` of its own — + it is compiled from C — so the seed side must always be read from the C text (R33: one + implementation, two callers, rather than two drifting re-parsers). + + -> (status, stale[], asm_only[]). status: clean | STALE | AMBIGUOUS | NO_ASM | NO_DRAFT. + """ + if not c_path or not os.path.isfile(c_path): + return 'NO_DRAFT', [], [] + return diff_syms_text(open(c_path).read(), asm_path, self_name) + + +def diff_syms_text(c_text, asm_path, self_name, ignore=()): + """`ignore` names FUNCTION IDENTITIES to exclude from both sides — the caller's own name plus, + for a seed body, the SEED's name. Compared case-insensitively (sig_image spells hex lowercase, + splat uppercase) and with the `DEFINE_` macro prefix stripped, or a macro-bodied seed reports + its own name as a stale symbol and every 1:1 rename degrades to AMBIGUOUS.""" + if c_text is None: + return 'NO_DRAFT', [], [] + if not asm_path or not os.path.exists(asm_path): + return 'NO_ASM', [], [] + ign = {n.lower() for n in (self_name, *ignore) if n} + def own(s): + return re.sub(r'^DEFINE_', '', s).lower() in ign + d, a = syms_in_text(c_text), asm_syms(asm_path) + stale = sorted(s for s in d - a if not own(s)) + asm_only = sorted(s for s in a - d if not own(s) and not s.startswith('.')) + if not stale: + return 'clean', [], asm_only + # 1:1 is the mechanically-safe case (one seed symbol, one target symbol to take its place). + return ('STALE' if len(stale) == 1 and len(asm_only) == 1 else 'AMBIGUOUS'), stale, asm_only + + +def audit_one(fn, binary, draft): + return diff_syms(draft, corpus.asm_path(binary, fn), fn) + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument('slate') + ap.add_argument('--fix', action='store_true', help='write rebased copies + a gateable slate') + ap.add_argument('--outdir', default='.run/aprop_symfix') + ap.add_argument('--out-slate', default='.run/aprop_symfix_slate.json') + ap.add_argument('--skip', help='json list of already-banked fn names to exclude') + a = ap.parse_args() + + recs = [r for r in json.load(open(a.slate)) if isinstance(r, dict) and (r.get('fn') or r.get('name'))] + skip = set(json.load(open(a.skip))) if a.skip else set() + seen, rows = set(), [] + for r in recs: + fn = r.get('fn') or r['name'] + if fn in skip or fn in seen: + continue + seen.add(fn) + draft = r.get('draft') or f".run/aprop1/{fn}/{fn}.c" + st, stale, asm_only = audit_one(fn, r['binary'], draft) + rows.append(dict(fn=fn, binary=r['binary'], draft=draft, status=st, + stale=stale, asm_only=asm_only, sub=r.get('sub'))) + + cls = collections.Counter(x['status'] for x in rows) + if not rows: # R32: a silent skip is a DEFECT + sys.exit(f"R32: {a.slate} yielded 0 auditable records — wrong shape or everything skipped?") + print(f"{len(rows)} drafts audited: {dict(cls)}") + for x in rows: + if x['status'] in ('STALE', 'AMBIGUOUS'): + print(f" {x['status']:9} {x['fn']:16} {x['binary']:12} " + f"draft-only={x['stale']} asm-only={x['asm_only']}") + + if not a.fix: + json.dump(rows, open('.run/aprop_symcheck.json', 'w'), indent=1) + print("audit only — wrote .run/aprop_symcheck.json") + return + + slate, deltas = [], collections.Counter() + for x in rows: + if x['status'] != 'STALE': + continue + old, new = x['stale'][0], x['asm_only'][0] + txt = open(x['draft']).read() + fixed, n = re.subn(rf'\b{re.escape(old)}\b', new, txt) + if n == 0: # R32: prove the edit ran (R37) rather than assuming it did + print(f" !! {x['fn']}: 0 substitutions of {old} — skipped") + continue + d = os.path.join(a.outdir, x['fn']) + os.makedirs(d, exist_ok=True) + p = os.path.join(d, f"{x['fn']}.c") + open(p, 'w').write(fixed) + deltas[int(new[2:], 16) - int(old[2:], 16)] += 1 + slate.append(dict(fn=x['fn'], binary=x['binary'], sub=x['sub'], draft=p, + rebased=f"{old}->{new}", subs=n)) + json.dump(slate, open(a.out_slate, 'w'), indent=1) + print(f"\nrebased {len(slate)}/{cls['STALE']} STALE drafts -> {a.outdir}") + print(f"seed->target vram deltas: {[(hex(k), v) for k, v in deltas.items()]}") + print(f"gateable slate: {a.out_slate}") + + +if __name__ == '__main__': # importable: family_cousins reuses the primitives above + main() diff --git a/tools/family_cousins.py b/tools/family_cousins.py index 7e1c59a67f..e80c53bd47 100644 --- a/tools/family_cousins.py +++ b/tools/family_cousins.py @@ -39,6 +39,7 @@ from difflib import SequenceMatcher sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) import corpus +import aprop_symfix as ASF import family_remap as FR FAMILY_MAP = ".run/family_hseq.json" @@ -400,6 +401,34 @@ def _disasm(words, base_vram): return ["?"] * len(words) +def sym_map(seed_body, seed_name, member_asm, member_name): + """The per-location symbol renames an adapting agent MUST apply — or a loud status. + + S50 (cookbook §171). A per-location data symbol (`D_8018xxxx` — a function-pointer table, a + jump table) is part of the seed's ENVIRONMENT, not its logic. Carried into a sibling + unrebased it survives `match_one` — which compares instruction ENCODINGS and is blind to a + relocation's target NAME — and then dies at link with `undefined reference`. Standalone + MATCH, host-TU link failure: that single class was 24 of 24 of the concentrated A-prop gate + failures in S49, the whole 91%-agent-vs-57%-gate delta. + + In the word-diff card such a rename appears only as an opaque IMM site (the low half of a + `lui`/`lw` %hi/%lo pair). Naming it turns a puzzle into an instruction.""" + # R35: `sig_image` spells names in lowercase hex, `seed_body_ref` in uppercase — try both + # spellings or every macro-bodied seed silently reports NO_SEED_BODY. + txt = None + for nm in (seed_body.get("name"), seed_name): + txt = txt or (ASF.body_text(seed_body.get("path"), nm) if nm else None) + if txt is None: + return dict(status="NO_SEED_BODY", renames=[]) + st, stale, asm_only = ASF.diff_syms_text(txt, member_asm, member_name, + ignore=(seed_name, seed_body.get("name"))) + if st == 'STALE': # 1:1 — the mechanically-safe, agent-actionable case + return dict(status="ok", renames=[dict(seed=stale[0], member=asm_only[0])]) + if st == 'AMBIGUOUS': # R32: fail loud rather than align garbage + return dict(status="AMBIGUOUS", renames=[], seed_only=stale, member_only=asm_only) + return dict(status=st, renames=[]) + + def emit_adapt_cards(): """Per open member of a SEEDED unit, classify drift vs the seed's token stream and emit a micro-adapt card for the LI-ONLY / SMALL-EDIT classes (the ~1,100-member wave-7a pool). @@ -532,7 +561,9 @@ def emit_aprop_cards(only=None, limit_members=0): seed=f"0x{s:08x}", seed_dis=_disasm([s], sa + 4 * i)[0])) members.append(dict(name=name, binary=b, addr=f"0x{a:08x}", sub=sub, n_sites=len(sites), sites=sites[:40], - kinds=dict(collections.Counter(x["kind"] for x in sites)))) + kinds=dict(collections.Counter(x["kind"] for x in sites)), + sym_map=sym_map(sbody, sname, + corpus.asm_path(b, name), name))) if not members: continue if limit_members: