From aa0736a4f855b36bf851266ee17776ea3925e5a7 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Wed, 8 Jul 2026 13:20:36 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-25):=20T7.2=20log=20+=20cookbook=20?= =?UTF-8?q?=C2=A740a=20=E2=80=94=20decl-reconcile=20pass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - CURRENT_PHASE: T7.2 done (1,729 banked, fleet 71.32%, clean-fleet 136/136); resume points to T7.3 + Close; Phase-26 backlog (16 split-TU families, 683 diffs, 127 draftable) - cookbook §40a: the decl-reconcile pass — (1) --no-preclassify (match_one isolation can't see engine_types.h, false-negatives lifted types; real-TU byte-gate is arbiter), (2) split files are separate TUs → blind lift-all unsafe (Buf cross-TU layout collision, PsyQ MATRIX/VECTOR shadowing), (3) byte-neutral verify per lift --- docs/matching-cookbook.md | 33 ++++++++++++++++++ phase-ends/CURRENT_PHASE.md | 68 ++++++++++++++++++++++++++----------- 2 files changed, 81 insertions(+), 20 deletions(-) diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index c70bd24cc1..1a723c3b2a 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -2276,3 +2276,36 @@ CC1-FAIL in isolation → defer them, else harvest_verify bisection explodes (~3 **136/136**, **fleet 66.02% → 70.82%**. The 11.1 MB structural-family frontier is CHEAPLY recoverable: crack ONE exemplar per family (the only real work — agent/wave), then `family_sweep` fills the ~133 members free. **Transferable** to ANY overlay/bank-based decomp where a fn recurs per-region with per-region symbols (→ cross-project idea). + +### §40a — The DECL-RECONCILE pass (type-lift so remapped drafts compile in the sibling TU) (Phase 25 T7.2, 2026-07-08) + +When the matched exemplar's body references a type defined INLINE in the source overlay's `.c`, the mechanical remap +(§40) reproduces that reference — but the type isn't in the SIBLING overlay's TU, so the sibling draft won't compile +(the sweep's `match_one` pre-classify tags it `type/decl`). Fix = lift the type into the shared header every overlay +includes (`src/shared/engine_types.h` via `engine_core.h`), then re-sweep. **+1,729 members** this way (fleet +70.82→71.32%, clean-fleet 136/136). Three load-bearing gotchas: + +1. **The pre-classify is a FALSE-NEGATIVE for lifted types.** `family_sweep`'s `match_one` pre-classify compiles the + draft in ISOLATION (`mipsel-cpp -Iinclude` + a prepended `common.h`) — it does **not** see `src/shared/engine_types.h` + (that's pulled only by the real overlay TU, via `../shared/engine_core.h`). So a type-lifted family still + pre-classifies as `type/decl` even though it compiles + byte-matches in the REAL TU. Verify against the real gate, + not the proxy (R14): a single `harvest_verify` of the remap into one sibling = byte-identical. Use + **`family_sweep --no-preclassify`** to route every remappable exemplar straight to the real-TU byte-gate (still the + sole arbiter — a wrong draft is compile-failed/reverted with bounded bisection cost, since the lift makes most compile). + +2. **Overlay SPLIT files (`_a`/`_after`/`_o0`) are SEPARATE .o TUs — a blind "lift all splits" is UNSAFE.** Each split + has its OWN local type namespace, so two splits can define the same-named type with a DIFFERENT layout and never clash + (different TUs). Lifting such a type fleet-wide `conflicting types for ` in the other split's TU. Also, a split may + locally `typedef … MATRIX/VECTOR` (PsyQ SDK names) — lifting those fleet-wide SHADOWS the real SDK types. So: + `build_engine_types --file src//_after.c --exclude --strip` lifts one split's types, + leaving conflicting/​shadowing names TU-local. Detect collisions FIRST (the tool's own `find_defs`/`find_typedefs` + across all splits + the existing header; a same-name-DIFFERENT-body pair is the landmine). ov077's only cross-split + collision was `Buf` (`_a`≠`_after`); `_a`'s `MATRIX`/`VECTOR` were the PsyQ shadows → both deferred. Safe mechanical + ceiling = base types + one non-shadowing split minus its colliders. The rest (cross-TU renames, PsyQ-layout + verification, -O0 clusters) is genuine per-type reconciliation, NOT mechanical → backlog. + +3. **Every lift must be byte-NEUTRAL.** Type defs emit no code, so `--strip` (move def → header) leaves the source + binary identical — but VERIFY: rebuild the WHOLE source overlay (all splits) and confirm its locked SHA + (ov077 `d19c9580`) before sweeping. A broken strip / mis-ordered header surfaces as a compile failure or SHA drift + at this cheap ~30s gate, before any expensive sweep. Then the full R22 clean-fleet 136/136 confirms no fleet-wide + header collision. Tools: `build_engine_types.py --file/--exclude`, `family_sweep.py --no-preclassify`. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index d97e98549c..4a0936cc08 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -33,32 +33,40 @@ idiom curriculum → sweep one exemplar per family (curriculum-ordered), each cr - [ ] T7 — Sweep one exemplar per family, curriculum-ordered (validate top ~3-5 first, then scale) *(Ultracode harvests / Fable5 discovery — prompt)* - [ ] Close — clean-fleet verify · PhaseEnd synthesis · plain-English recap (R25) · Phase-26 backlog -## FRESH SESSION — RESUME HERE (T7 in progress; finish T7 → Close; do NOT start T4 yet, per Drew 2026-07-08) -**Done + committed:** T0–T3 + the **T7 matched-free MECHANICAL sweep** — fleet **66.02% → 70.82%** (+16,512 -member-matches; R22 clean-fleet **136/136**; commit `commit:0476`). Tools PROVEN + committed: `tools/family_remap.py` -(the remap lever), `tools/family_sweep.py` (sweep driver), `tools/family_manifest.py` (T2), split-aware `recover_integration`. +## FRESH SESSION — RESUME HERE (T7 nearly done; finish T7.3 → Close; do NOT start T4 yet, per Drew 2026-07-08) +**Done + committed:** T0–T3 + T7-part-1 (matched-free sweep, `commit:0476`) + **T7.2 decl-reconcile** (`commit:0479`) — +fleet **66.02% → 70.82% → 71.32%**. T7.2 banked **1,729** member-matches (base-type families 532 + `_after`-type +families 1,197) via type-lift + mechanical remap; **R22 clean-fleet 136/136**, dedup-check 1813/0. **The mechanical family method (the phase's engine — T3):** h_norm families are TEMPLATES, not free dedup. Per family: crack ONE exemplar → `family_remap` builds each member's C by positionally substituting the per-overlay symbols (read from each member's image) → **plain `harvest_verify`** byte-gate (NOT gate_stage's transforms — they perturb a correct remap). ~0 agent tokens/member. `func_` names are UPPERCASE-hex. -**NEXT (finish T7, then Close):** -1. **Re-run surveys first:** `make sig-overlays` (idempotent) → `.venv/bin/python tools/family_manifest.py` — the - matched set grew 16.5k, so the draftable/matched-free split updates. -2. **Decl-reconcile the 29 `type/decl` deferred** (`.run/sweep_deferred.txt`): exemplars use LOCAL types (e.g. - `MatEntry`, defined in ov_SC01_077.c not engine_types.h) so the remapped draft won't compile. Fix: - `tools/build_engine_types.py --source ov_SC01_077 --strip` (lift local types to `src/shared/engine_types.h`, - byte-neutral — verify ov_SC01_077 still builds `d19c9580`), then `tools/family_sweep.py --only ` - → ~+3,800 banks. Clean-fleet verify (R22) + commit. -3. **8 edge cases** (`.run/sweep_deferred.txt`: 4 diff + 4 remap-fail): `0x8013c360` = -O0 cluster; `0x80165ca0`/ - `0x8012a018` = known Phase-24 stragglers. Brief look or defer. -4. **683 whole-binary-diff simple failures** (~4%: match_one-MATCH but TU-context diff, like the T1 memcpy idiom): - defer as a small residual or batch-investigate. -5. **The 127 DRAFTABLE families / 6.7 MB** (`docs/family-manifest.md`): their ov077 exemplar is a STUB → must be - CRACKED first — **this is where T4 (v4) + T5 (the Ultracode wave) come back**: crack the 127 exemplars, then - `family_sweep` remaps their members ×134 free. **Prompt for `/effort ultracode` before the wave (R27).** -6. **Close:** clean-fleet verify, PhaseEnd (R25 recap). Byte-weighted % (~31%→) rises materially now (big families). +**T7.2 findings (durable — cookbook §40 refinement):** (1) `family_sweep --no-preclassify` — the match_one +pre-classify compiles in ISOLATION (`-Iinclude` only) so it CANNOT see `src/shared/engine_types.h` (pulled by the +real overlay TUs via `../shared/engine_core.h`); it false-negatives every type-lifted family. The flag routes +remappable exemplars straight to the real-TU `harvest_verify` byte-gate (the sole arbiter). (2) The overlay SPLIT +files (`_a`/`_after`/`_o0`) are SEPARATE TUs with conflicting/​shadowing local types → a blind "lift all splits" is +UNSAFE: `Buf` has a DIFFERENT layout in `_a` vs `_after` (fleet-wide lift → `conflicting types` in the other TU), +and `_a` defines `MATRIX`/`VECTOR` = PsyQ SDK names (fleet-wide lift SHADOWS the real types). Safe mechanical ceiling += base types + `_after` minus `Buf`. New tool knobs: `build_engine_types --file ` + `--exclude `. + +**NEXT (finish T7.3, then Close):** +1. ~~Re-run surveys~~ ✅ T7.1. ~~Decl-reconcile the 29~~ ✅ T7.2 (1,729 banked; committed `commit:0479`). +2. **T7.3 — 8 edge cases** (`.run/sweep_deferred.preT72.txt`: 4 diff + 4 remap-fail): `0x8013c360` = -O0 cluster; + `0x80165ca0`/`0x8012a018` = known Phase-24 stragglers. Brief look or defer. +3. **Close:** clean-fleet verify, PhaseEnd (R25 recap), Phase-26 backlog. + +**Phase-26 backlog (deferred, evidence logged):** +- **T7.2 residual: 16 h_norm families / ~2,128 members** — 15 need `_a.c` types (PsyQ MATRIX/VECTOR shadowing) or + `_o0.c` E_3B7AC (-O0 cluster); 1 needs `_after`'s `Buf` (cross-TU collision). Need per-type reconciliation + (namespaced/per-TU header, rename the colliding `Buf`, or verify `_a`'s MATRIX/VECTOR are PsyQ-layout-compatible + and use the real SDK headers) — NOT mechanical. +- **683 whole-binary-diff simple failures** (~4%: match_one-MATCH but TU-context diff, like the T1 memcpy idiom). +- **The 127 DRAFTABLE families / 6.7 MB** (`docs/family-manifest.md`): ov077 exemplar is a STUB → must be CRACKED + first — this is where **T4 (v4) + T5 (the Ultracode wave)** come back: crack the 127 exemplars, then + `family_sweep` remaps their members ×134. **Prompt for `/effort ultracode` before the wave (R27).** ## Blockers None. @@ -143,3 +151,23 @@ None. - Deferred (`.run/sweep_deferred.txt`): 29 type/decl (type-lift recoverable), 4 diff, 4 remap-fail. - 683 / 17,195 simple drafts failed the whole-binary gate (match_one-MATCH but TU-context diff, ~4%; auto-reverted). - **Reproduce:** `family_sweep.py --limit 0` (after `make sig-overlays`). The remap is `family_remap.py`. + +### T7.2 — decl-reconcile the 29 type/decl families ✅ (2026-07-08, `commit:0479`) — 1,729 banked; split-TU wall found +- **T7.1 refresh:** `make sig-overlays` (134 re-signed) + `family_manifest.py` — levers unchanged (2,764 families; + 127 draftable / 156 matched-free) because they key on ov077 membership, which the 16.5k sibling-banks didn't change. +- **Base type-lift:** `build_engine_types --source ov_SC01_077 --strip` lifted 4 base-local types (`MatEntry`, + `P_TAG`, `OtBlk`, `packed_word`/`word_bytes`) → `engine_types.h`; byte-neutral (ov077 d19c9580). Banked **532**. +- **The pre-classify false-negative (R14):** after the lift, all 29 STILL pre-classified as `type/decl` (0 simple). + Root cause: `match_one` isolation compiles with `-Iinclude` + a prepended `common.h` — it does NOT see + `src/shared/engine_types.h` (the real overlay TU pulls it via `../shared/engine_core.h`). PROVEN false-negative: + single real-TU gate `func_80142A80`→ov_SC01_000 = `9052dc0e` BYTE-IDENTICAL. FIX: `family_sweep --no-preclassify`. +- **The split-TU wall:** the 25 remaining families need types from the overlay SPLIT files, which are SEPARATE .o TUs + with conflicting/​shadowing local types. Blind "lift all splits" FAILED: `typedef Buf` has a DIFFERENT layout in + `_a.c` vs `_after.c` (`conflicting types for Buf` when shared); `_a.c` defines `MATRIX`/`VECTOR` (PsyQ SDK names → + fleet-wide SHADOW). Safe ceiling = base + `_after` **minus Buf** (`build_engine_types --file … --exclude Buf`): + byte-neutral, banked **+1,197** (9 of 10 `_after` families/overlay; the 1 miss is the Buf-user). +- **T7.2 TOTAL 1,729** (532 + 1,197). **R22 clean-fleet 136/136** from clean tree; dedup-check 1813/0; fleet **71.32%**. +- **Deferred → Phase 26:** 16 families / ~2,128 members (see backlog above). Old deferred snapshot preserved at + `.run/sweep_deferred.preT72.txt` (the 29 addrs). +- **Reproduce:** `build_engine_types --source ov_SC01_077 --strip` ; `build_engine_types --file + src/ov_SC01_077/ov_SC01_077_after.c --exclude Buf --strip` ; `family_sweep --only <29 addrs> --no-preclassify`.