From ee4b3a02e84254d91f22113fb5842f6c8eee02e9 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Wed, 15 Jul 2026 19:20:56 -0600 Subject: [PATCH] =?UTF-8?q?feat(phase-27=20T5):=20extract=5Funit=20carries?= =?UTF-8?q?=20file-scope=20#define=20macros=20=E2=80=94=20honest=200x8017B?= =?UTF-8?q?EBC=20probe=20+=20the=20pin-crash=20wall=20dissolved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit family_remap.extract_unit dropped the file-scope function-like #define macros a body references (the gte_* C inline-asm GTE-op macros live ABOVE the function; the backward preamble walk stopped at the first #define/continuation line). A staged sibling saw every GTE op as an implicit-declaration CALL. Two consequences in one bug: - staging failed: 0x8017BEBC's 112 members all CC1-FAIL'd -> a FAKE 0% probe that reads "mechanical harvest dead" when the tool was broken (a 4th phantom exhaustion proof, exactly the 26-A audit class). - the SIGABRT: with a caller-saved register PIN present, the phantom call pushes cc1's sched1 into create_reg_dead_note's abort (sched.c:2725) — the §42e "pin-crash wall". The wave-2 SIGABRT agent proved this IS the cause (.run/giants/pin_crash_sigabrt.md): pinned families stage 133/133 clean once their macros ride along. A propagation wall recorded as a compiler limit for phases = a staging-tool artefact. - _carry_macros: prepend the function-like #define macros the unit body references (file order), not already inside the unit. Safe by construction: feeds only the templating path (remap_hseq), never make_macro's engine_core.h lift (no #define embedded in a DEFINE_func_*() macro); gather_externs only scans func_/D_ so no bogus extern; regression-verified non-GTE exemplars carry 0 macros (a no-op where it should be). THE HONEST PROBE (R14): staging 0 -> 106/112 (6 skip = IMM tier-2). Bounded 8-member gate sample = 0 banked / 8, ALL genuine DIFF (T4 classifier: compiled, wrong bytes — NOT plumbing). 0x8017BEBC is BYTE-PROVEN NOT TEMPLATABLE: the roadmap's "largest cheap win left" (B2) is REFUTED. The h_seq match is necessary, not sufficient. This 0% MEANS something because the tool is fixed first. Carried to T8: harvest the now-unblocked pin families (verify the 133/133 claim + bank). --- phase-ends/CURRENT_PHASE.md | 3 ++- tools/family_remap.py | 40 ++++++++++++++++++++++++++++++++++++- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 1a76ab03f..285e52293 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -39,7 +39,7 @@ Phase 26 closed on an honest pivot — the mechanical/templating harvest is byte - [x] **Task 2 — Makefile fail-closed (the enabling fix)** `[xHigh]` — **DONE.** `.SHELLFLAGS := -ec` (global fail-closed) with ONE documented opt-out: `check-env` (`set +e` — its contract is accumulate-every-failure). Fixed the `check-all:610` `grep -c` landmine (`|| true` — grep -c exits 1 on 0 matches, which `-e` would treat as fatal → check-all would fail when nothing failed). Strengthened `check-all`/`extract-all` from `fail == 0` → **`pass == N`** (coverage assertion, R32 — the old form was a vacuous pass on an empty pipeline). Gave the two audit oracles a dependent: **new `make tools-health`** = `audit-corpus` + `audit-cdecl` + `report`, fail-closed (NOT a `report`/`build` prereq — audit-cdecl is ~minutes). SETUP §6.3 documents it (R21). **VERIFIED:** (1) known-answer — a broken `lint_symbol_refs` makes `make report` exit non-zero, and a **negative control** proves it: the *identical* break exits **0** under old `.SHELLFLAGS=-c`, **2** under `-ec`; (2) the `grep -c` landmine and the vacuous-pass both reproduced + fixed in isolation; (3) `check-env` still exits 0 (opt-out works); (4) **`make check-all` → 136/136 byte-identical**, and a forced `main` re-extract+rebuild exercised the full splat→cpp→cc1→maspsx→as→ld→objcopy→check pipeline under `-e` → `143dbb89…`; (5) `audit-corpus` (7s) + `audit-cdecl` (green) + `tools-health` dry-run all wired. Recipe scan found the Makefile was already `-e`-aware (`set -o pipefail`, explicit `|| true`, guarded `@` lines) — line 610 was the only real hazard. *(completes with this commit)* - [x] **Task 3 — Curated `.run/` preservation** `[xHigh]` — **DONE** (pulled ahead of Task 1 — it de-risks the sprint's inputs). `.gitignore` `/.run/` → contents-exclude form (`/.run/*` + `!` exceptions, the `/tools/bin/*.sha256` precedent). **Refined at execution against the bytes:** the naive "commit the dirs" would have been **12.3 MB of regenerable gcc RTL scratch**; the genuinely irreplaceable set is **~2.2 MB / 31 files** — the 6 Phase-25 `*.opus.{c,md}` seed recons (49K), the `func_80178004` gdb-on-cc1 **harness + `ORACLE_PROOF.md` + the v00–v07 draft ladder + the sched/combine `.lst` evidence** (~110K), and the two frontier ledgers (`backlog.jsonl` 1.9M, `fuel_manifest.json` 67K). `dumps_v00..v07/` + `d_pf*.i.*` stay ignored — **regenerable via `runorc.sh` + the `.gdb` scripts** (R33: commit what a rerun cannot reproduce). **VERIFIED:** `git add --dry-run .run/` stages exactly the 30 intended files, 0 bulk; negative control — `.run/ghidra-mcp.log`, `dumps_v00`, `d_pf.i.sched`, `d_pf.s` all still `IGNORED`; no `db.*.gbf` staged (R23). *(completes with this commit)* - [x] **Task 4 — The cdecl strip primitive + surface cc1 stderr** `[xHigh]` — **DONE.** Found the defect is **six** copied scalar-name regexes, not two (`harvest_verify._TD`, `masked_diff.SCALAR_TYPEDEF_RE`, `canon_sig_reconcile`'s own, `eval_lora`, `format_finetune`, + the 2 masked_diff consumers). Added **one primitive to `cdecl`**: `typedef_names(tu_path)` + `strip_provided_typedefs(draft, provided)` — built on `tu_statements` (robust) **not** `tu_scope` (which coverage-asserts → would crash the byte-gate on any unrelated unparseable file-scope statement; a deliberate refinement of the plan). Split multi-typedef lines via `split_statements` (depth-aware); covers scalar AND struct typedefs; keeps draft-local types. **`harvest_verify`:** per-TU strip-set (unblocks the 39 struct-typedef drafts) + **cc1 stderr surfaced** — `build()` stashes it, a single-draft failure is classified **DIFF / PLUMBING:… / CC1-FAIL / SKIP** (`.run/harvest_failed.classified.txt`), so a `redefinition` is no longer recorded as a byte miss. **`masked_diff.strip_scalar_typedefs()`** (common.h set derived once, R33) wired into `match_one` + `p16_permute`. Unblocks B4's `func_8015C32C` (`redefinition of 's16'`). **VERIFIED:** (1) headline known-answer — `func_8015C030` → **`MATCH (23 ins)` UNEDITED** (was CC1-FAIL; multi-line split alone fixes it); (2) unit — 7/7 scalars stripped, a local struct KEPT, a TU-provided `Blk16` stripped; (3) classifier unit — DIFF/PLUMBING/CC1-FAIL/SKIP all correct; (4) all 5 tools import + parse; (5) **R22 clean-fleet 136/136** + main clean-rebuild `143dbb89` (a mid-test `c4546248` "mismatch" was a stale-incremental artifact from concurrent compiles — resolved by a clean rebuild, the R22 lesson; my edits touch only `tools/`, `src/` stayed git-clean). A strip bug can only fail-to-bank, never falsely bank (the audit invariant). SETUP §6.3 + cdecl inventory updated (R21). *(completes with this commit)* -- [ ] **Task 5 — `extract_unit` macro-carry → the `0x8017BEBC` probe** `[Max]` — carry file-scope macro deps; then `--stage-only` sizing (minutes) → a **bounded ~10-member gate sample** for the rate → leave the full ~112 gated builds to P28's gate farm. Verify: the staged unit compiles standalone (today it cannot); report the realized bank-rate as a measured number. **A 0% here is only meaningful AFTER the macro fix.** +- [x] **Task 5 — `extract_unit` macro-carry → the `0x8017BEBC` probe** `[xHigh]` — **DONE.** Fixed `family_remap.extract_unit` to carry the file-scope function-like `#define` macros the body references (`_carry_macros`) — the gte_* C inline-asm macros live above the function and the backward walk dropped them, so every staged sibling saw undefined GTE ops → CC1-FAIL. **Now staging works: 0 → 106/112** members stage (6 skip = IMM tier-2, a separate class). Safe by construction: it only feeds the templating path (`remap_hseq`), never `make_macro`'s engine_core.h lift; `gather_externs` only scans func_/D_ so no bogus extern; **regression-verified** non-GTE exemplars carry 0 macros. **THE HONEST PROBE (R14):** bounded 8-member gate sample = **0 banked / 8, all genuine DIFF** (T4 classifier — compiled, wrong bytes; NOT plumbing). **`0x8017BEBC` is byte-proven NOT templatable → the roadmap's "largest cheap win left" (B2) is REFUTED** — the h_seq match is necessary, not sufficient; Phase-26's mechanical-exhaustion extends here. **This is why the fix had to come first: a pre-fix 0% was a tooling artifact; this 0% is a real byte-gate refusal.** **🔑 BONUS (major):** the same macro-carry is the fix the wave-2 SIGABRT agent proved dissolves the **§42e pin-crash wall** — the SIGABRT (`sched.c:2725 create_reg_dead_note`) came from dropped macros turning GTE ops into implicit calls that push a caller-saved pin into the fatal shape; **pinned families stage 133/133 clean once macros ride along** (`.run/giants/pin_crash_sigabrt.md`). A propagation wall that capped phases is down → **carried to T8** (harvest the pin families). *(completes with this commit)* - [x] **Task 6 — Scanner migration** `[xHigh]` — **DONE.** `exemplar_miner`: replaced the `dp.registered_addrs()` proxy (config/dedup.us.yaml — ~60% wrong: a matched-but-unregistered fn stayed in the residual pool) with `corpus.stubs(source)` — "is it still work?" = "is it still INCLUDE_ASM" (R33). `difficulty`: replaced the **136-entry hand-dict** with `cfg_for(alias)` (the layout is mechanical: `src/` + `asm//nonmatchings`; main/resident the two specials) — **proven byte-exact** for all 136 (0 mismatches derivation-vs-dict), validated against the tree (`src/` must exist, R32/R33) not a hand-list. **Also removed difficulty from `new_overlay.sh`'s sentinel-insertion set** (T6 made it obsolete — otherwise onboarding would insert a dead dict entry into a file with no dict; the other 3 tools' hand-lists stay, migrated one-at-a-time per the audit). **VERIFIED:** (1) both tools + new_overlay.sh parse; (2) **airtight known-answer** — old `difficulty.py` vs new produce **byte-identical** `.md` AND `.csv` on the same tree (the vs-committed diff was pure staleness — committed doc is 2026-06-20); (3) unknown alias → clean error, not silent-empty; (4) `exemplar_miner` runs → 223 residual stubs (the corrected count; not in `make report`, so no known-answer constraint — the change is the fix); (5) new_overlay.sh bash+embedded-python valid, difficulty absent. Now a new overlay (T7) needs zero difficulty hand-registration. *(completes with this commit)* - [x] **Task 7 — Disc-completeness audit + onboard + type-sweep** `[xHigh]` — **DONE.** Generalized `new_overlay.sh` with an optional `[ENTRY]` arg (default `0.4`); onboarded `ov_SC07_{006,007,010,011}` from `1.4.dec` — each **byte-identical** (`7ca772be`/`b3b95547`/`d7b5875d`/`9885af74`). **Fleet 136 → 140**, `check-all` **140/140** (T2's `pass==N` correctly re-baselined). The sweep (`tools/disc_code_sweep.py`, committed) revealed the initial `isValid()`-only threshold was far too weak (389 false "hits"; type-0/2 data decodes ~100% valid) — **fixed with a `jr $ra` density gate** (code ~2.9-3.4%, data 0.000%, validated on positive+negative controls). **Honest result:** type-4 is **COMPLETE (138/138)**; all other types are data EXCEPT **type-1 = 40 code payloads, 1 onboarded (resident), 39 HIDDEN** — resident-class modules (mostly `MAIN.CD/FILE_XXX/1.1`) that load at **unknown addresses**, so they are **not mechanically onboardable** (P9: can't byte-verify without the address; needs Phase-3-style runtime RE). Documented in `docs/disc-completeness.md`. **This is a bigger re-baselining than +4:** the true code surface is 140 onboarded **+ 39 type-1 modules pending RE** — the completion contract's binary count and the "100%" bar both move (→ T10/T11). SETUP §6.3 tool inventory updated (R21). *(completes with this commit)* - [ ] **Task 8 — The byte-gate-honest re-scan + partition + ledger rebuild** `[Ultracode (triage breadth — PROMPT, R26/R27) → Max (synthesis)]` — refresh surveys on the fixed tools; `worklist --assert-partition` **scoped honestly** (worklist's universe is ONE overlay ~263 stubs vs the fleet's 53,371 — a genuine fleet partition is a scope change, not a flag); triage the 1,670 untriaged; rebuild the wall ledger from data; fix the ledger corruption above. Verify: partition passes against `corpus.stubs()`; retracted claims gone. @@ -62,6 +62,7 @@ Phase 26 closed on an honest pivot — the mechanical/templating harvest is byte - **2026-07-15 · Task 3 — Curated `.run/` preservation.** **Pulled ahead of Task 1** (a 5-minute deviation from plan order, P3 autonomy): Task 1's Fable5 agents work *inside* `.run/`, and its Phase-25 seed recons were untracked — an agent overwriting `.run/giants/func_8014D820.opus.c` would have destroyed irreplaceable input. Five minutes out of a four-day window is a trivial price for removing that. **Execution refined the plan against the bytes (R33):** the plan said "track `.run/giants/*.opus.{c,md}` + `.run/fable_80178004/`", but those directories are **8.5M and 3.8M — almost entirely gcc RTL dump scratch** (`d_pf.i.combine/.sched/.lreg`, `dumps_v00..v07`) that `runorc.sh` + the `.gdb` scripts regenerate. The irreplaceable core is **~2.2 MB**: 49K of seed recon, ~110K of oracle harness + proof + draft ladder, and the two ledgers. Committed that; left the regenerable bulk ignored. Verified both directions (intended set stages; bulk still `IGNORED`). **Carried to Task 1:** the sprint's *outputs* must be added to the allowlist as they land — the same reasoning that motivated this task. - **2026-07-15 · Task 2 — Makefile fail-closed.** The roadmap §5 asserted `make report` is fail-closed; it was not (`.ONESHELL` + no `-e` → only the last command's exit survives; `dedup-check` "gated" purely by being last). Set `.SHELLFLAGS := -ec` globally + `check-env` opt-out; fixed the `grep -c` landmine; upgraded `check-all`/`extract-all` to coverage assertions (`pass == N`); added `make tools-health` as the audits' dependent. The **negative control** is the proof that mattered — same broken gate, exit 0 under `-c` vs exit 2 under `-ec` — turning "the swallow is real" from a claim into a measurement (R14 discipline applied to my own fix). Full clean-fleet R22 held (136/136 + a forced main rebuild under `-e`). **This unblocks every downstream R32 assertion**: until now, any gate added to a report-invoked tool was swallowed on arrival. SETUP §6.3 updated (R21). +- **2026-07-15 · Task 5 — the macro-carry, and why a tool fix precedes a probe.** The plan's whole point was that the `0x8017BEBC` probe would lie without the fix — and it would have: pre-fix, 112/112 members CC1-FAIL on undefined gte_ macros and the probe reads "0% templatable, mechanical harvest dead," a fourth phantom exhaustion proof. Post-fix, 106/112 stage and the byte-gate gives the HONEST 0/8 (all genuine DIFF) — the family really isn't templatable, and now I can say so with evidence. The convergence with the wave-2 SIGABRT agent is the striking part: it independently traced the "pin-crash wall" to this exact `extract_unit` macro-drop (dropped macros → implicit-call GTE ops → a caller-saved pin trips `sched.c:2725`'s abort), so one fix both makes the probe honest AND dissolves a propagation wall the project recorded as a compiler limit for phases. The 26-A audit thesis, a third time: our tool was the wall. I verified the fix is scoped (templating path only, not the macro-lift) and regression-clean (non-GTE families untouched) before trusting it. **Carried to T8:** harvest the now-unblocked pin families (the agent claims 133/133 clean staging — verify + bank). - **2026-07-15 · Task 10 — completion dashboard + the second oracle.** The instructive part was resisting the plan's own framing. The plan said "add main via corpus.stubs" as if a one-liner; the reality is three layers — the src_stubs landmine (main→100%), the LINKED-fallback over-count in corpus.stubs, and main's only sig being a month-stale Ghidra sig that excludes LINKED. The honest resolution: main's Ghidra sig EXCLUDES LINKED, which turns out to be exactly right for a game-code weighted metric (LINKED is complete, lives in fn-count), so iterating it against corpus.stubs is clean — but the number is provisional (stale sig), so I report it SEPARATE and un-folded rather than corrupt the decomp.dev headline (P9 over the metrics contract's literal wording). The resident second oracle was the clean win (probed 0-phantom before wiring, R14). And I caught a T7 straggler — `sig-overlays`'s `0.4.dec` glob would have silently dropped the 4 new SC07 sigs on any regen — the same silent-skip class the whole audit exists to kill. `docs/second-oracle.md` is the honest deferral for main: a half-oracle (splat-seeded) would be worse than a documented gap. - **2026-07-15 · Task 7 — disc-completeness audit (a scope-expanding finding).** Onboarding the 4 SC07 overlays was the easy, mechanical half (same class as the 134, byte-verified). The sweep was where the discipline mattered: my first pass flagged 389 "hidden code" payloads, which a moment's skepticism (type-2 at 201/201 100% valid?) exposed as false positives — `rabbitizer.isValid()` is far too permissive on structured data. The `jr $ra`-density discriminator (validated against positive AND negative controls before I trusted a single count, R14) collapsed it to the honest answer: **only type-1 carries hidden code, 39 modules.** The consequential finding is that these are resident-class (unknown load address), so they're NOT mechanically onboardable — and reporting them as "found but deferred to load-address RE" rather than force-onboarding at a guessed address is the P9 call. **⚠️ This meaningfully expands the endgame: game-code TRUE 100% now spans 140 binaries PLUS ~39 type-1 modules pending RE — the roadmap assumed 136.** Surfaced to Drew in the progress report; the contract update flows through T10 (dashboard) + T11 (Roadmap delta). Also: T6's difficulty derivation proved itself here — the onboard touched only 3 tool dicts, not 4. - **2026-07-15 · Task 6 — scanner migration (before T7's onboarding).** Two R33 migrations: `exemplar_miner`'s "is it still work?" now asks the invariant (`corpus.stubs` = still-INCLUDE_ASM) instead of the dedup registry (a ~60%-wrong proxy), and `difficulty` derives its per-binary paths from the alias instead of a 136-entry hand-dict. The discipline that made this safe: **prove the derivation byte-exact against the thing it replaces before deleting it** — I checked `cfg_for(alias) == BINARIES[alias]` for all 136, then confirmed old-tool-vs-new-tool output byte-identical on the same tree (isolating my change from a month of doc staleness, R14). A coupling I had to catch: removing difficulty's dict made `new_overlay.sh`'s difficulty insertion obsolete → left as-is it would have corrupted difficulty on the next onboarding, so T6 also removed that entry. **Deliberately did NOT migrate `dup_report.BINARIES`** (corpus itself depends on it as the binary-list source) — that's a larger change the audit defers to per-bank byte-gated migration; T7 still hand-registers new overlays there. diff --git a/tools/family_remap.py b/tools/family_remap.py index 5753e3435..5c0bda0f7 100644 --- a/tools/family_remap.py +++ b/tools/family_remap.py @@ -427,6 +427,40 @@ def symbol_map(addr, from_ov, to_ov, to_addr=None): # walk must not cross it (the layer belongs to the file, not to the function beneath it). _DECL_LAYER_END = re.compile(r"end (?:§8b carried decl layer|canonical-sig layer)") +_DEFINE_FN = re.compile(r'^\s*#define\s+([A-Za-z_]\w*)\s*\(') + + +def _carry_macros(lines, start, end, unit_text): + r"""The file-scope function-like `#define` macros the unit BODY references but does not itself + contain — returned in file order, to be prepended to the unit (Phase-27 T5). + + A GTE-heavy per-location function calls C inline-asm macros (`gte_ldv0(...)`, `gte_rtps()`, …) + that are `#define`d at FILE scope, above the function, NOT pulled from common.h (common.h carries + the *assembler* gte_macros.inc for the INCLUDE_ASM path; the C macros live only in the exemplar's + .c). extract_unit's backward preamble walk stops at the first `#define`/continuation line, so it + dropped them — and a sibling staged without them saw every GTE op as an implicit-declaration CALL. + That is TWO failures in one: the sibling fails to compile, AND — if the body carries a caller-saved + register pin — the phantom call pushes cc1's sched1 into `create_reg_dead_note`'s abort + (sched.c:2725), the SIGABRT long mis-recorded as the §42e "pin-crash wall" (`.run/giants/ + pin_crash_sigabrt.md`: the wall is this staging drop, not a compiler limit — a pinned family + stages 133/133 clean once its macros ride along). Carried macros never reach dedup_propagate's + engine_core.h lift (that path builds from its own body, not extract_unit), so no `#define` is ever + embedded inside a `DEFINE_func_*()` macro.""" + carried, i = [], 0 + while i < len(lines): + m = _DEFINE_FN.match(lines[i]) + if not m: + i += 1 + continue + j = i + while j < len(lines) and lines[j].rstrip().endswith("\\"): + j += 1 + inside = start <= i and j <= end + if not inside and re.search(rf'\b{re.escape(m.group(1))}\s*\(', unit_text): + carried.append("\n".join(lines[i:j + 1])) + i = j + 1 + return carried + def extract_unit(ov, addr): """the matched inline def + its contiguous preceding extern/blank/comment lines, from the overlay src. @@ -469,7 +503,11 @@ def extract_unit(ov, addr): if started and depth <= 0: end = k break - return "\n".join(lines[start:end + 1]), cf + unit_text = "\n".join(lines[start:end + 1]) + macros = _carry_macros(lines, start, end, unit_text) # gte_* etc. the body needs (T5) + if macros: + unit_text = "\n".join(macros) + "\n" + unit_text + return unit_text, cf return _macro_unit(addr)