mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-10-04 16:28:11 -04:00
feat(phase-30 S50): the stale seed symbol — A-prop's 57% was ours, not the lane's (§171)
- REFUTES §170's open hypothesis (batched cards concentrate members into one TU ⇒ §169
collision): 5-draft groups banked 5/5; 11 of 35 unbanked drafts were already one-per-TU;
and the two "concentrated" groups banked 12/12 and 10/10 once the real defect was fixed.
- The cause: a per-location data symbol carried out of the seed body unrebased. match_one
compares instruction ENCODINGS and is blind to a relocation's target NAME, so it scores
MATCH standalone and dies at link in the host TU. 24 of 24 concentrated failures, all 1:1
rewritable at one constant vram delta (0x4128).
- tools/aprop_symfix.py: audit + --fix, emits a gate_lane-shaped slate; deterministic and
build-free, so it runs BEFORE the gate. The R34 second oracle for the class match_one
cannot see.
- family_cousins.py --aprop-cards: members now carry sym_map, the explicit {seed -> member}
renames, read from the seed's C BODY (a matched seed has no .s of its own) vs the member's
.s. Two case-mismatch defects fixed while wiring it (sig lowercase vs splat uppercase).
- 23/24 banked. Stubs 12,468 -> 12,445. Fleet 95.2% instr / 89.9% distinct / 96.57% fn.
R22 clean rebuild: check-all 213 passed, 0 failed of 213. dedup 2,043/0.
- A-prop's true conversion is 87% (79/91); the 320 batched members are unblocked.
- Cookbook §171 + §170 struck in place; SETUP row; decision-log (R31).
This commit is contained in:
Drew T
@@ -673,6 +673,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/ghidra_import_raw.sh` | Headless import of a RAW flat blob — `BinaryLoader` + `--loader-baseAddr <vram>` + `PSX:LE:32:default` (resident blob / Gen2 overlays; no PS-X EXE header). |
|
||||
| | `tools/prefetch_fleet.py` | **The fleet Ghidra-C prefetch batch** (P30 T0.5): one representative per remaining h_seq distinct class + ALL main/resident stubs → `.run/ghidra_c/`, resumable (skips cached), serial on the project lock; auto-stops a serving MCP (R23 — R29 applies at next MCP use); imports a missing overlay program on demand (`ghidra_import_raw.sh`, blob via `family_remap.img_path`, vram via the splat yaml). `--dry-run` / `--limit-programs N` (probe-first). R32: per-program outcome report, batch continues past failures. |
|
||||
| | `tools/family_cousins.py` | **(P30 S49, cookbook §168)** The COUSIN-UNIT survey — the similarity tier ABOVE h_seq. h_seq is an exact skeleton hash, so ±1 instruction (li-expansion, table-size drift) fragments same-source families into "singletons"; this tool clusters the distinct open skeletons by mnemonic-stream similarity (≥0.85, union-find) and attaches the best MATCHED-skeleton seed per unit → `.run/family_cousins.json` + `docs/family-cousins.md`. Categories A-prop / seeded / cousin-multi / cold. `--targets N --wave waveN` emits the crack_wave.js slate ranked by whole-UNIT open ins, with each seed's C body location (engine_core.h macro or inline src file) resolved. R32 both ways: fails loud on a stale family map (independent stub recount) and on a partition defect. RANKS AND SEEDS ONLY — cousins need a per-member seeded CRACK (skeleton drift ⇒ recompile), never a `family_sweep` remap; the whole-binary byte-gate stays the sole arbiter. |
|
||||
| | `tools/aprop_symfix.py` | **(P30 S50, cookbook §171)** The stale-SEED-SYMBOL guard for the adapt/A-prop lanes — the second, DISAGREEING oracle (R34) for the one class `match_one` is structurally blind to. A per-location data symbol (`D_8018xxxx`) carried out of a seed body unrebased scores MATCH standalone (encodings identical; the scorer cannot see a relocation's target NAME) and dies in the host TU at link with `undefined reference`. Audits every draft's vram-suffixed symbols against the symbols the TARGET's own `.s` relocates; `--fix` rewrites the 1:1 cases and emits a `gate_lane`-shaped slate. Deterministic, no build — run it BEFORE the gate, never after. Measured on the S49 A-prop residue: 24 of 24 concentrated failures were this class, 23 banked after the rebase (A-prop conversion 57% -> 87%). Its primitives are imported by `family_cousins.py`, whose `--aprop-cards` members now carry an explicit `sym_map` of `{seed -> member}` renames. |
|
||||
| **Disc/.CD extraction** (`tools/bfm_extract/`) | `extract.py` | Walk the disc / extract root files (`make extract`). |
|
||||
| | `extract_exe.py` | Extract & verify `SLUS_007.26` (`--verify-disc`, owns `EXPECTED_EXE_SHA1`). |
|
||||
| | `extract_proto_exe.py` | Extract the prototype/demo EXE for cross-checking. |
|
||||
|
||||
@@ -2318,3 +2318,40 @@ DEBUG-MENU-LIST was the session's force multiplier. (2) Negative results with ev
|
||||
product: six payloads now carry "never loads in X/Y/Z" instead of "unknown". (3) The write-API
|
||||
no-op control was a real R35 miss — verify instruments with a VISIBLE effect. (4) R22 catching
|
||||
md_MAIN_003 three times in one evening is the rule working exactly as designed.
|
||||
|
||||
## 2026-08-13 (P30 S50) — the A-prop conversion gap: the experiment the artifacts had already run
|
||||
|
||||
**Context and belief.** S49's closing checkpoint made one thing the session's first job, on the
|
||||
grounds that it "prices everything else": re-gate the 42 (truly 35) unbanked A-prop drafts ONE PER
|
||||
TU, to test §170's hypothesis that family-batched cards concentrate members into a single
|
||||
destination TU and die of the §169 collision. 320 further batched members — roughly 15M tokens of
|
||||
wave — were explicitly held behind that measurement. The belief was that A-prop's 91%-agent /
|
||||
57%-gate conversion was an integration-topology problem.
|
||||
|
||||
**What failed.** The hypothesis, and the framing that made it look like it needed an experiment.
|
||||
Three artifacts already in `.run/` answered it: `gate_aprop1.json` records **5-draft single-TU
|
||||
groups banking 5/5** (batch size is not the discriminator); **11 of the 35** unbanked drafts were
|
||||
already single-draft groups, i.e. the proposed test had already been run on them; and
|
||||
`harvest_failed.ov_SC03_107.classified.txt` names the actual failure verbatim — `PLUMBING:
|
||||
undefined reference to 'D_80181900'`, eleven times. §169's own law ("read the classified file
|
||||
before theorising about any sweep failure") was written for exactly this and was not applied to the
|
||||
sibling lane.
|
||||
|
||||
**The pivot.** Diagnose from the recorded verdicts first; run the experiment only for what they do
|
||||
not answer. A 40-line static audit — compare each draft's vram-suffixed symbols against the symbols
|
||||
the target's own `.s` relocates — classified all 35 in under a second: **24 stale-seed-symbol, 11
|
||||
genuine DIFF**, zero ambiguity. One probe banked, then 22 of the remaining 23.
|
||||
|
||||
**Why (byte-grounded).** A per-location data symbol is the seed's ENVIRONMENT, not its logic.
|
||||
`match_one` compares instruction encodings and is blind to a relocation's target NAME, so a carried
|
||||
symbol scores MATCH standalone and fails at link inside the host TU. A-prop's real conversion is
|
||||
**87% (79/91)**, not 57% — the lane was never the problem, and the 320 held-back members are worth
|
||||
substantially more than they were priced at.
|
||||
|
||||
**Hindsight — the better path.** The cheap deterministic audit should have been part of the wave's
|
||||
verification step from the start: it needs no build, it is the second oracle (R34) for the one class
|
||||
`match_one` structurally cannot see, and it would have converted these 24 in S49 instead of leaving
|
||||
them to be re-measured a session later. Generalized rule: **before designing an experiment to
|
||||
explain a failure rate, grep the failure verdicts the tools already wrote** — and when a checkpoint
|
||||
declares a test the top priority, that is a hypothesis with a plan attached, not a finding (R14/R35
|
||||
applied to my own handoff notes).
|
||||
|
||||
@@ -16325,12 +16325,21 @@ best agent rate of any wave type — **→ 56 banked (57%)**, ≈ **80k tok per
|
||||
per-member cousin card's 157k and a crack wave's 400k+. The agent notes are substitutions, not
|
||||
decompilations (`"src struct = D_801D0358"`; `"cb=func_8017FBA4; 0x12C, 0x4B0, 0x38E…"`).
|
||||
|
||||
**⚠ The conversion gap, and the test before scaling.** 91% agent → 57% gate is the WORST conversion
|
||||
measured (the per-member cousin wave ran 81% → 92%). 14 of ~50 groups banked zero. **Hypothesis,
|
||||
not yet proven:** family-batched cards CONCENTRATE members into one destination TU by construction —
|
||||
precisely the §169 collision the spread law names, which held the 7a wave to 64%. **Re-gate the
|
||||
unbanked drafts ONE PER TU before scaling a batched A-prop wave**; if the spread law is the cause the
|
||||
same drafts bank, and the fix is to interleave families per gate batch rather than to redraft.
|
||||
**⚠ The conversion gap.** 91% agent → 57% gate is the WORST conversion measured (the per-member
|
||||
cousin wave ran 81% → 92%). 14 of ~50 groups banked zero.
|
||||
|
||||
> ~~**Hypothesis, not yet proven:** family-batched cards CONCENTRATE members into one destination TU
|
||||
> by construction — precisely the §169 collision the spread law names, which held the 7a wave to 64%.
|
||||
> **Re-gate the unbanked drafts ONE PER TU before scaling a batched A-prop wave**; if the spread law
|
||||
> is the cause the same drafts bank, and the fix is to interleave families per gate batch rather than
|
||||
> to redraft.~~
|
||||
>
|
||||
> **STRUCK 2026-08-13 (S50) — REFUTED. Batching was never the discriminator (5-draft groups banked
|
||||
> 5/5; the two "concentrated" groups banked 12/12 and 10/10 once fixed). The cause was a STALE SEED
|
||||
> SYMBOL: `match_one` is blind to a relocation's target NAME, so a carried-over `D_8018xxxx` scores
|
||||
> MATCH standalone and dies at link. 23 of 24 banked after a mechanical rebase; A-prop's true
|
||||
> conversion is 87%, not 57%. See §171 — and note the test itself was never needed: the answer was
|
||||
> already sitting in `.run/harvest_failed.<binary>.classified.txt`.**
|
||||
|
||||
**Three ops notes paid for on the head:** (1) `family_sweep --hseq` defaults to `--band substantial`,
|
||||
so tiny/mid families are silently OUT OF SCOPE — a "never attempted" verdict can be a band default,
|
||||
@@ -16338,3 +16347,50 @@ not a difficulty (5 of 13 heads); (2) `dedup_extend` extends MACRO-backed groups
|
||||
INLINE def use `dedup_propagate --addr`; (3) that tool names its own blocker precisely
|
||||
(`missing file-scope extern (CARRY-FIXABLE): D_…`) — a data symbol the matched body's own TU never
|
||||
declares at file scope blocks every sibling until it is carried.
|
||||
|
||||
## §171 — THE STALE SEED SYMBOL (P30 S50, 2026-08-13): why §170's 91%→57% was never codegen
|
||||
|
||||
**§170 left one hypothesis open — that family-batched A-prop cards CONCENTRATE members into one
|
||||
destination TU and die of the §169 collision. It is REFUTED, by data that already existed and by a
|
||||
direct test.** Refuted three ways: (1) the recorded S49 verdicts show 5-draft single-TU groups
|
||||
banking **5/5** twice, and 4/4 twice — batch size was never the discriminator; (2) 11 of the 35
|
||||
unbanked drafts were **single-draft groups**, i.e. already gated one-per-TU, and all 11 classify
|
||||
`DIFF`; (3) after the real fix below, the two "concentrated" groups §170 blamed — 12 drafts and 10
|
||||
drafts into ONE `.c` each — banked **12/12 and 10/10**.
|
||||
|
||||
**The real defect, and it is one line per draft.** A per-location data symbol (`D_8018xxxx` — a
|
||||
function-pointer table, a jump table, a state array) is part of the seed's **ENVIRONMENT, not its
|
||||
logic**. The adapt/A-prop lanes hand an agent a proven seed body plus a word-diff; the agent edits
|
||||
the logic and carries the seed's symbol across unrebased. Then:
|
||||
|
||||
- **`match_one` scores it MATCH.** It compiles standalone and compares instruction ENCODINGS.
|
||||
`%hi(D_seed)` and `%hi(D_target)` are the same instruction with a different relocation TARGET
|
||||
NAME, and the scorer is blind to the name.
|
||||
- **The whole-binary gate kills it at LINK:** `undefined reference to 'D_80181900'`.
|
||||
|
||||
Standalone-MATCH / host-TU-link-failure. That class was **24 of the 24** concentrated A-prop
|
||||
failures — the entire conversion gap. In the word-diff card the rename is visible only as an opaque
|
||||
**IMM** site (the low half of a `lui`/`lw` `%hi`/`%lo` pair), which is exactly the site class an
|
||||
agent reads as "an immediate to copy", not "a symbol to rebase".
|
||||
|
||||
**Measured:** all 24 were 1:1 rewritable, every one at the same seed→target vram delta (`0x4128` —
|
||||
one binary's data section offset from the seed's). Rebased mechanically: **23/24 banked** (the 24th
|
||||
is a genuine `DIFF`). A-prop's real conversion is **56 → 79 of 91 = 87%**, not 57%; the agents'
|
||||
91% MATCH claim was very nearly right and the gap was ours.
|
||||
|
||||
**Two tools, one primitive** (`tools/aprop_symfix.py`, imported by `family_cousins.py`):
|
||||
- **Post-hoc guard** — `aprop_symfix.py <slate.json> --fix` audits every draft's vram-suffixed
|
||||
symbols against the symbols the TARGET's own `.s` relocates, rewrites the 1:1 cases, and emits a
|
||||
`gate_lane`-shaped slate. Deterministic, no build, so it belongs BEFORE the gate, never after.
|
||||
- **Pre-hoc annotation** — `--aprop-cards` members now carry `sym_map`: the explicit
|
||||
`{seed → member}` renames, computed from the seed's C BODY (a matched seed has no `.s` of its
|
||||
own — it is compiled from C) versus the member's `.s`. It turns a puzzle into an instruction.
|
||||
|
||||
This is R34 in one line: **`match_one` is a perfect codegen oracle and a NULL linkage oracle.** The
|
||||
symbol audit is the second oracle that can disagree with it, and it costs no build.
|
||||
|
||||
**Two spellings, one bug, twice:** `sig_image` writes hex lowercase, splat writes it uppercase, and
|
||||
`seed_body_ref` builds `func_%08X`. Both defects found while wiring this (a NO_SEED_BODY on every
|
||||
macro seed, an AMBIGUOUS on every 1:1 rename) were case mismatches — the same class §128/R35 keeps
|
||||
naming. Compare function identities case-insensitively, and strip the `DEFINE_` prefix, or the
|
||||
seed's own name reads as a stale symbol.
|
||||
|
||||
@@ -4,16 +4,16 @@
|
||||
# cross-binary collapsible-byte leverage: docs/duplicates.cross.md.
|
||||
|
||||
# THREE progress metrics (all matter — see the labels):
|
||||
FLEET fn-count byte-ident: 350688 / 363163 = 96.56% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
|
||||
FLEET instr-weighted : 12872722 / 13523856 = 95.2% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number)
|
||||
FLEET distinct-code(uniq): 5260480 / 5851963 = 89.9% (81090/90928 unique fns; the DISTINCT-RE number)
|
||||
FLEET fn-count byte-ident: 350711 / 363163 = 96.57% (REAL+LINKED+empties; FUNCTION-count, ×134-inflated — one crack counts per overlay)
|
||||
FLEET instr-weighted : 12873067 / 13523856 = 95.2% (shipped .text across main + resident + 211 overlays; the decomp.dev-DISPLAY number)
|
||||
FLEET distinct-code(uniq): 5260690 / 5851963 = 89.9% (81104/90928 unique fns; the DISTINCT-RE number)
|
||||
MAIN game-code weighted : 436 / 79510 = 0.5% (INCLUDED in the fleet numbers above since 2026-07-22 — roadmap §1 metrics contract; LINKED-excluding Ghidra sig dated 2026-08-05; caveat is R34: no independent second oracle for a PS-X EXE, NOT drift)
|
||||
(fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 12872286 / 13444346 = 95.7%)
|
||||
(fleet EXCLUDING main, for continuity with pre-2026-07-22 readings: 12872631 / 13444346 = 95.7%)
|
||||
|
||||
FLEET REAL substantive : 348518 (of which dedup-shared 254400 via 2043 groups / 254474 instances)
|
||||
FLEET REAL substantive : 348541 (of which dedup-shared 254400 via 2043 groups / 254474 instances)
|
||||
FLEET LINKED PsyQ objs : 959
|
||||
FLEET NON_MATCHING : 7 (0 in any default build — G4)
|
||||
FLEET INCLUDE_ASM stubs : 12468
|
||||
FLEET INCLUDE_ASM stubs : 12445
|
||||
FLEET matchable : 363163
|
||||
|
||||
| binary | REAL | shared | LINKED | byte-ident | matchable | byte-ident % |
|
||||
@@ -153,7 +153,7 @@ FLEET matchable : 363163
|
||||
| ov_SC03_103 | 2497 | 1808 | 0 | 2500 | 2510 | 99.6% |
|
||||
| ov_SC03_104 | 2549 | 1811 | 0 | 2556 | 2616 | 97.7% |
|
||||
| ov_SC03_105 | 2451 | 1807 | 0 | 2458 | 2597 | 94.6% |
|
||||
| ov_SC03_107 | 2268 | 1690 | 0 | 2351 | 2497 | 94.2% |
|
||||
| ov_SC03_107 | 2291 | 1690 | 0 | 2374 | 2497 | 95.1% |
|
||||
| ov_SC03_108 | 2403 | 1806 | 0 | 2403 | 2443 | 98.4% |
|
||||
| ov_SC03_109 | 2409 | 1807 | 0 | 2411 | 2424 | 99.5% |
|
||||
| ov_SC03_110 | 2443 | 1830 | 0 | 2443 | 2468 | 99.0% |
|
||||
|
||||
@@ -181,7 +181,56 @@ stub on a named wall/behemoth/queue ledger** — 140/140 byte-identical througho
|
||||
|
||||
---
|
||||
|
||||
# 🛑 SESSION S49 CHECKPOINT — FINAL (2026-08-13) — FRESH SESSION SAFE HERE
|
||||
# 🛑 SESSION S50 CHECKPOINT (2026-08-13) — FRESH SESSION SAFE HERE
|
||||
> **Tree CLEAN** but for R23 `db.*.gbf` churn, a regenerable `.run/backlog.jsonl` edit, and two
|
||||
> stray `gccdump.*` cc1 dumps at the repo root — never stage any of them. **Nothing running.**
|
||||
> Gate: **`check-all` 213 passed / 0 failed of 213** from a CLEAN rebuild (`make clean &&
|
||||
> make extract-all && make check-all`, R22).
|
||||
> **Fleet: 95.2% instr · 89.9% distinct · 96.57% fn-count · INCLUDE_ASM stubs 12,445** (was
|
||||
> 12,468). **23 instances banked.** dedup 2,043/0. Cookbook **§171**; §170's open hypothesis
|
||||
> STRUCK. Decision-log entry filed (R31).
|
||||
|
||||
## ✅ S50 T6/T7 — the S49 resume-block step 1 is CLOSED, and it closed by REFUTATION
|
||||
**The A-prop TU-spread test was never needed — the artifacts had already run it.** §170's
|
||||
hypothesis (family-batched cards concentrate members into one destination TU ⇒ the §169 collision)
|
||||
is **refuted three ways**: `gate_aprop1.json` shows 5-draft single-TU groups banking **5/5**;
|
||||
**11 of the 35** unbanked drafts were already single-draft groups (all classify `DIFF`); and after
|
||||
the real fix the two "concentrated" groups banked **12/12 and 10/10**.
|
||||
|
||||
**The real cause (cookbook §171): the STALE SEED SYMBOL.** A per-location data symbol
|
||||
(`D_8018xxxx`) is the seed's ENVIRONMENT, not its logic. Carried out of a seed body unrebased it
|
||||
scores **MATCH in `match_one`** — which compares instruction ENCODINGS and is blind to a
|
||||
relocation's target NAME — and dies at link inside the host TU (`undefined reference`). It was
|
||||
**24 of the 24** concentrated A-prop failures, all 1:1 rewritable at one constant vram delta
|
||||
(`0x4128`). Rebased mechanically → **23/24 banked** (the 24th, `func_801659DC`, is a genuine DIFF).
|
||||
**A-prop's true conversion is 87% (79/91), not 57%.**
|
||||
|
||||
**Shipped:** `tools/aprop_symfix.py` (audit + `--fix`, emits a `gate_lane`-shaped slate; no build,
|
||||
so it runs BEFORE the gate) · `family_cousins.py --aprop-cards` members now carry an explicit
|
||||
`sym_map` of `{seed → member}` renames, computed from the seed's C BODY (a matched seed has no `.s`
|
||||
of its own) vs the member's `.s` · SETUP row · cookbook §171 · decision-log.
|
||||
|
||||
## ▶ RESUME HERE
|
||||
1. **The 320 batched A-prop members are re-priced and UNBLOCKED** — the 57% that held them back was
|
||||
ours, not the lane's. Run the wave, and put `aprop_symfix --fix` in its verification step before
|
||||
the gate. Regen chain first (the tools fail loud on a stale map, by design):
|
||||
`tools/family_hseq.py` → `tools/family_cousins.py` → `--aprop-cards`.
|
||||
2. **The ≥16 head, still open:** the **`D_801ED98C` carry fix** (one missing file-scope extern gates
|
||||
**83 PURE members** — `dedup_propagate` names it itself) · **`func_8017C294`** (×16, 246 ins,
|
||||
NEAR at 2 instructions after three seeded attempts — the board's biggest single crack and free
|
||||
permuter fuel).
|
||||
3. **12 A-prop drafts stay genuine `DIFF`** (11 singletons + `func_801659DC`) — redraft/permuter
|
||||
fuel, not recovery fuel.
|
||||
|
||||
## 🔑 THE ONE THING TO CARRY FORWARD
|
||||
**Before designing an experiment to explain a failure rate, grep the failure verdicts the tools
|
||||
already wrote.** §169's own law ("read the classified file before theorising") was written for this
|
||||
and was not applied to the sibling lane; `harvest_failed.<binary>.classified.txt` named the cause
|
||||
verbatim, eleven times over, while the checkpoint called for a multi-hour measurement campaign.
|
||||
Second: **`match_one` is a perfect codegen oracle and a NULL linkage oracle** — the symbol audit is
|
||||
the R34 second oracle for the one class it cannot see, and it costs no build.
|
||||
|
||||
# ▶ (superseded by S50) SESSION S49 CHECKPOINT — FINAL (2026-08-13)
|
||||
> **Tree CLEAN** but for R23 `db.*.gbf` churn + a regenerable `.run/backlog.jsonl` edit — never
|
||||
> stage either. **Nothing running.** 179 commits this session (`commit:1776`..`commit:1891`).
|
||||
> Gate at close: **`check-all` 213 passed / 0 failed of 213** from a CLEAN rebuild (R22 run **5×**
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
"""Rebase stale SEED symbols in adapt/A-prop drafts onto the TARGET's own symbols.
|
||||
|
||||
WHY THIS EXISTS (S50, cookbook §171). The adapt/A-prop lanes hand an agent a *proven seed body*
|
||||
from one location plus a small token-diff, and the agent adapts the code — but a per-location data
|
||||
symbol (`D_8018xxxx`, a function-pointer table, a jump table) is part of the seed's ENVIRONMENT,
|
||||
not its logic. Carry it over unrebased and:
|
||||
|
||||
- `match_one` still scores MATCH. It compiles standalone and compares instruction encodings;
|
||||
a `%hi(D_seed)` and a `%hi(D_target)` are the same instruction with a different relocation
|
||||
TARGET NAME, and the scorer is blind to the name.
|
||||
- the whole-binary gate then fails at LINK: `undefined reference to 'D_seed'`.
|
||||
|
||||
That gap — standalone-MATCH, link-undefined — was the entire A-prop 91%-agent-vs-57%-gate delta
|
||||
measured in S49 (24 of 24 concentrated failures; §170's TU-collision hypothesis was refuted by
|
||||
the same data: 5-draft groups banked 5/5).
|
||||
|
||||
This is the SECOND, DISAGREEING ORACLE (R34) for that class: match_one says MATCH, the symbol
|
||||
audit says the draft references a symbol the target's own .s never mentions. It is deterministic
|
||||
and costs no build, so it belongs BEFORE the gate, never after.
|
||||
|
||||
audit: tools/aprop_symfix.py <slate.json> (report only)
|
||||
fix: tools/aprop_symfix.py <slate.json> --fix -> .run/aprop_symfix/<fn>/<fn>.c + slate
|
||||
|
||||
A slate record is {fn|name, binary, draft}; the emitted slate points `draft` at the fixed copy so
|
||||
`tools/gate_lane.py` consumes it unchanged.
|
||||
"""
|
||||
import argparse, collections, json, os, re, sys
|
||||
|
||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__)))
|
||||
import corpus
|
||||
|
||||
# vram-suffixed symbols: D_80185A28, func_801727D0, jtbl_8018xxxx, ...
|
||||
SYM = re.compile(r'\b[A-Za-z_][A-Za-z0-9_]*_?8[0-9A-Fa-f]{7}\b')
|
||||
ASM_OPS = re.compile(r'%hi\(([^)]+)\)|%lo\(([^)]+)\)|\bjal\s+([A-Za-z_]\w*)|\.word\s+([A-Za-z_]\w*)')
|
||||
|
||||
|
||||
def _strip_comments(txt):
|
||||
return re.sub(r'//[^\n]*', '', re.sub(r'/\*.*?\*/', '', txt, flags=re.S))
|
||||
|
||||
|
||||
def syms_in_text(txt):
|
||||
return set(SYM.findall(_strip_comments(txt)))
|
||||
|
||||
|
||||
def draft_syms(path):
|
||||
return syms_in_text(open(path).read())
|
||||
|
||||
|
||||
def body_text(path, name):
|
||||
"""The C body of ONE function out of a file (a draft is one function; a seed's file is not).
|
||||
|
||||
Matches an `DEFINE_<name>(` engine_core macro or a plain definition, then brace-matches. A
|
||||
whole-file scan would drag every OTHER function's symbols into the comparison and drown the
|
||||
signal, so callers annotating a SEED must pass through here."""
|
||||
if not path or not os.path.isfile(path):
|
||||
return None
|
||||
txt = _strip_comments(open(path).read())
|
||||
m = (re.search(rf'\bDEFINE_{re.escape(name)}\s*\(', txt) or
|
||||
re.search(rf'^[A-Za-z_][^\n=;]*\b{re.escape(name)}\s*\(', txt, re.M))
|
||||
if not m:
|
||||
return None
|
||||
i = txt.find('{', m.start())
|
||||
if i < 0:
|
||||
return None
|
||||
depth = 0
|
||||
for j in range(i, len(txt)):
|
||||
if txt[j] == '{':
|
||||
depth += 1
|
||||
elif txt[j] == '}':
|
||||
depth -= 1
|
||||
if depth == 0:
|
||||
return txt[m.start():j + 1]
|
||||
return None
|
||||
|
||||
|
||||
def asm_syms(path):
|
||||
out = set()
|
||||
for m in ASM_OPS.finditer(open(path).read()):
|
||||
for g in m.groups():
|
||||
if g:
|
||||
out.add(g.split('+')[0].strip())
|
||||
return out
|
||||
|
||||
|
||||
def diff_syms(c_path, asm_path, self_name):
|
||||
"""Compare a C body's symbols against the symbols its target .s actually relocates.
|
||||
|
||||
Shared primitive: `aprop_symfix` uses it on a DRAFT (post-hoc guard), `family_cousins.sym_map`
|
||||
uses it on the SEED BODY (pre-hoc card annotation). A matched seed has no `.s` of its own —
|
||||
it is compiled from C — so the seed side must always be read from the C text (R33: one
|
||||
implementation, two callers, rather than two drifting re-parsers).
|
||||
|
||||
-> (status, stale[], asm_only[]). status: clean | STALE | AMBIGUOUS | NO_ASM | NO_DRAFT.
|
||||
"""
|
||||
if not c_path or not os.path.isfile(c_path):
|
||||
return 'NO_DRAFT', [], []
|
||||
return diff_syms_text(open(c_path).read(), asm_path, self_name)
|
||||
|
||||
|
||||
def diff_syms_text(c_text, asm_path, self_name, ignore=()):
|
||||
"""`ignore` names FUNCTION IDENTITIES to exclude from both sides — the caller's own name plus,
|
||||
for a seed body, the SEED's name. Compared case-insensitively (sig_image spells hex lowercase,
|
||||
splat uppercase) and with the `DEFINE_` macro prefix stripped, or a macro-bodied seed reports
|
||||
its own name as a stale symbol and every 1:1 rename degrades to AMBIGUOUS."""
|
||||
if c_text is None:
|
||||
return 'NO_DRAFT', [], []
|
||||
if not asm_path or not os.path.exists(asm_path):
|
||||
return 'NO_ASM', [], []
|
||||
ign = {n.lower() for n in (self_name, *ignore) if n}
|
||||
def own(s):
|
||||
return re.sub(r'^DEFINE_', '', s).lower() in ign
|
||||
d, a = syms_in_text(c_text), asm_syms(asm_path)
|
||||
stale = sorted(s for s in d - a if not own(s))
|
||||
asm_only = sorted(s for s in a - d if not own(s) and not s.startswith('.'))
|
||||
if not stale:
|
||||
return 'clean', [], asm_only
|
||||
# 1:1 is the mechanically-safe case (one seed symbol, one target symbol to take its place).
|
||||
return ('STALE' if len(stale) == 1 and len(asm_only) == 1 else 'AMBIGUOUS'), stale, asm_only
|
||||
|
||||
|
||||
def audit_one(fn, binary, draft):
|
||||
return diff_syms(draft, corpus.asm_path(binary, fn), fn)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('slate')
|
||||
ap.add_argument('--fix', action='store_true', help='write rebased copies + a gateable slate')
|
||||
ap.add_argument('--outdir', default='.run/aprop_symfix')
|
||||
ap.add_argument('--out-slate', default='.run/aprop_symfix_slate.json')
|
||||
ap.add_argument('--skip', help='json list of already-banked fn names to exclude')
|
||||
a = ap.parse_args()
|
||||
|
||||
recs = [r for r in json.load(open(a.slate)) if isinstance(r, dict) and (r.get('fn') or r.get('name'))]
|
||||
skip = set(json.load(open(a.skip))) if a.skip else set()
|
||||
seen, rows = set(), []
|
||||
for r in recs:
|
||||
fn = r.get('fn') or r['name']
|
||||
if fn in skip or fn in seen:
|
||||
continue
|
||||
seen.add(fn)
|
||||
draft = r.get('draft') or f".run/aprop1/{fn}/{fn}.c"
|
||||
st, stale, asm_only = audit_one(fn, r['binary'], draft)
|
||||
rows.append(dict(fn=fn, binary=r['binary'], draft=draft, status=st,
|
||||
stale=stale, asm_only=asm_only, sub=r.get('sub')))
|
||||
|
||||
cls = collections.Counter(x['status'] for x in rows)
|
||||
if not rows: # R32: a silent skip is a DEFECT
|
||||
sys.exit(f"R32: {a.slate} yielded 0 auditable records — wrong shape or everything skipped?")
|
||||
print(f"{len(rows)} drafts audited: {dict(cls)}")
|
||||
for x in rows:
|
||||
if x['status'] in ('STALE', 'AMBIGUOUS'):
|
||||
print(f" {x['status']:9} {x['fn']:16} {x['binary']:12} "
|
||||
f"draft-only={x['stale']} asm-only={x['asm_only']}")
|
||||
|
||||
if not a.fix:
|
||||
json.dump(rows, open('.run/aprop_symcheck.json', 'w'), indent=1)
|
||||
print("audit only — wrote .run/aprop_symcheck.json")
|
||||
return
|
||||
|
||||
slate, deltas = [], collections.Counter()
|
||||
for x in rows:
|
||||
if x['status'] != 'STALE':
|
||||
continue
|
||||
old, new = x['stale'][0], x['asm_only'][0]
|
||||
txt = open(x['draft']).read()
|
||||
fixed, n = re.subn(rf'\b{re.escape(old)}\b', new, txt)
|
||||
if n == 0: # R32: prove the edit ran (R37) rather than assuming it did
|
||||
print(f" !! {x['fn']}: 0 substitutions of {old} — skipped")
|
||||
continue
|
||||
d = os.path.join(a.outdir, x['fn'])
|
||||
os.makedirs(d, exist_ok=True)
|
||||
p = os.path.join(d, f"{x['fn']}.c")
|
||||
open(p, 'w').write(fixed)
|
||||
deltas[int(new[2:], 16) - int(old[2:], 16)] += 1
|
||||
slate.append(dict(fn=x['fn'], binary=x['binary'], sub=x['sub'], draft=p,
|
||||
rebased=f"{old}->{new}", subs=n))
|
||||
json.dump(slate, open(a.out_slate, 'w'), indent=1)
|
||||
print(f"\nrebased {len(slate)}/{cls['STALE']} STALE drafts -> {a.outdir}")
|
||||
print(f"seed->target vram deltas: {[(hex(k), v) for k, v in deltas.items()]}")
|
||||
print(f"gateable slate: {a.out_slate}")
|
||||
|
||||
|
||||
if __name__ == '__main__': # importable: family_cousins reuses the primitives above
|
||||
main()
|
||||
+32
-1
@@ -39,6 +39,7 @@ from difflib import SequenceMatcher
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import corpus
|
||||
import aprop_symfix as ASF
|
||||
import family_remap as FR
|
||||
|
||||
FAMILY_MAP = ".run/family_hseq.json"
|
||||
@@ -400,6 +401,34 @@ def _disasm(words, base_vram):
|
||||
return ["?"] * len(words)
|
||||
|
||||
|
||||
def sym_map(seed_body, seed_name, member_asm, member_name):
|
||||
"""The per-location symbol renames an adapting agent MUST apply — or a loud status.
|
||||
|
||||
S50 (cookbook §171). A per-location data symbol (`D_8018xxxx` — a function-pointer table, a
|
||||
jump table) is part of the seed's ENVIRONMENT, not its logic. Carried into a sibling
|
||||
unrebased it survives `match_one` — which compares instruction ENCODINGS and is blind to a
|
||||
relocation's target NAME — and then dies at link with `undefined reference`. Standalone
|
||||
MATCH, host-TU link failure: that single class was 24 of 24 of the concentrated A-prop gate
|
||||
failures in S49, the whole 91%-agent-vs-57%-gate delta.
|
||||
|
||||
In the word-diff card such a rename appears only as an opaque IMM site (the low half of a
|
||||
`lui`/`lw` %hi/%lo pair). Naming it turns a puzzle into an instruction."""
|
||||
# R35: `sig_image` spells names in lowercase hex, `seed_body_ref` in uppercase — try both
|
||||
# spellings or every macro-bodied seed silently reports NO_SEED_BODY.
|
||||
txt = None
|
||||
for nm in (seed_body.get("name"), seed_name):
|
||||
txt = txt or (ASF.body_text(seed_body.get("path"), nm) if nm else None)
|
||||
if txt is None:
|
||||
return dict(status="NO_SEED_BODY", renames=[])
|
||||
st, stale, asm_only = ASF.diff_syms_text(txt, member_asm, member_name,
|
||||
ignore=(seed_name, seed_body.get("name")))
|
||||
if st == 'STALE': # 1:1 — the mechanically-safe, agent-actionable case
|
||||
return dict(status="ok", renames=[dict(seed=stale[0], member=asm_only[0])])
|
||||
if st == 'AMBIGUOUS': # R32: fail loud rather than align garbage
|
||||
return dict(status="AMBIGUOUS", renames=[], seed_only=stale, member_only=asm_only)
|
||||
return dict(status=st, renames=[])
|
||||
|
||||
|
||||
def emit_adapt_cards():
|
||||
"""Per open member of a SEEDED unit, classify drift vs the seed's token stream and emit a
|
||||
micro-adapt card for the LI-ONLY / SMALL-EDIT classes (the ~1,100-member wave-7a pool).
|
||||
@@ -532,7 +561,9 @@ def emit_aprop_cards(only=None, limit_members=0):
|
||||
seed=f"0x{s:08x}", seed_dis=_disasm([s], sa + 4 * i)[0]))
|
||||
members.append(dict(name=name, binary=b, addr=f"0x{a:08x}", sub=sub,
|
||||
n_sites=len(sites), sites=sites[:40],
|
||||
kinds=dict(collections.Counter(x["kind"] for x in sites))))
|
||||
kinds=dict(collections.Counter(x["kind"] for x in sites)),
|
||||
sym_map=sym_map(sbody, sname,
|
||||
corpus.asm_path(b, name), name)))
|
||||
if not members:
|
||||
continue
|
||||
if limit_members:
|
||||
|
||||
Reference in New Issue
Block a user