From 881dd5a37ccb36eec9e407cf73ecdced3cbcc0cc Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 1 Sep 2026 12:20:59 -0600 Subject: [PATCH] =?UTF-8?q?docs:=20the=20NEAR-TWIN=20BAND=20=E2=80=94=20?= =?UTF-8?q?=C2=A7389,=20accelerator=20#17,=20SETUP,=20playbook=20=C2=A72a,?= =?UTF-8?q?=20generic-decomp-package?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The exact-hash twin tier found 22 of 352 reachable open stubs (6%). The edit-distance band added by `seed_ref --near` finds 75 of 352 (21%) — 3.4x — on a corpus we believed fully mined. 31 of the new rows were PURE reloc-only twins of already-banked bodies; 8 banked the same day at ~0 agent tokens, one 94-ins exemplar serving five open copies. * cookbook §389: the h_norm hole (norm_stream drops its pending lui-hi on an intervening R-type, so indexed-global reloc twins hash differently and vanish from seed_ref/twin_sweep/dedup/family-maps at once). Do NOT fix h_norm — every stored calibration keys on it; the near tier reads through it. * accelerators #17: the generalisable law. A similarity hash built for DEDUP under-matches by design, which is correct for dedup and silently lossy as a FRONTIER join — the two questions want opposite error directions, and the frontier failure looks exactly like "this function is unique". * generic-decomp-package §2b: build the near band at the same time as the exact tier, with the three verifications. It pays from the first bank for a new project, where we paid a session to recover the debt. * SETUP inventory row + playbook §2a (run it before believing any "no twin" verdict; never send a RELOC-ONLY row to a drafting agent). --- docs/SETUP.md | 1 + docs/accelerators.md | 39 +++++++++ docs/cookbook-index.md | 37 ++++++--- docs/generic-decomp-package.md | 30 ++++++- docs/matching-cookbook.md | 141 +++++++++++++++++++++++++++++++++ docs/wave-playbook.md | 12 +++ 6 files changed, 250 insertions(+), 10 deletions(-) diff --git a/docs/SETUP.md b/docs/SETUP.md index b5b6a66f9..85b6ba538 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -780,6 +780,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | **S68 tooling — the gater lane + the -O0 route** (P31 S68) | **`tools/gater_lane.py` (NEW)** — the continuous gater: drains a wave's finished drafts into `parallel_gate`, grouped by binary, `--r22` by default. Ledger AND verdicts keyed **`binary:fn:arm`** (R48 — and an escalation ALWAYS has a prior verdict, so arm-keying is what stops an in-flight fable draft riding the opus one). `--extra BINARY:PATH` for non-wave drafts, `--skip-binary` for lanes that may be writing, and **main is routed IN-TREE via `harvest_verify`** because `parallel_gate`'s worktree cannot stage main's psyq_integrate link inputs. **`tools/workflows/escalate_fable.js` (NEW)** — warm-started escalation: passes the prior draft + its measured closeness + its ruled-out levers, and demands a reusable `new_idiom`. **`tools/o0_boundary.py` (NEW)** — the stranded-boundary -O0 sweep (141 binaries / 288 boundaries / 0 candidates: the class is EXHAUSTED, and that null is negative-controlled). | | | **S69 tooling — the TRIAGE LADDER** (P31 S69) | **`tools/triage_ladder.py` (NEW)** — the zero-token pass that answers *"does this target need an agent at all?"* before one is spent. **The PRE/POST split is the point:** `--pre ` runs the TARGET-SIDE tiers only (BANKED · WALL-332 · PARKED) — no draft, no build, milliseconds — and writes `/triage.json` + `triage_exclude.txt`; `--post` runs the full `residual_rules_b` residual routing, which needs a draft and runs `match_one`. `--escalate B:FN` exits 2 on a walled/banked target (the check S68 was missing when it escalated a §332 wall at closeness 8). `--acceptance` is the R39/R32 harness: false-skip over EVERY open stub, recall over sampled matched fns, and a wall negative control over already-banked code — all pure filesystem work, so it runs in seconds over the whole corpus. **It REFUSES on a non-quiescent tree** (`pgrep -af` rows for gate/build lanes + `lane_inflight`): a live gate makes the stub oracle transiently wrong in both directions (§377). Wired into `wave_args.py` (drops walled/parked targets at draw time, reusing `pre_classify` — one implementation, R33) and into `tools/workflows/escalate_fable.js`, which now REFUSES any target that does not carry `triage:'DRAFT'`. **Routing correction (§376):** the `INTEG-STANDALONE-MATCH` / `NOCOMPILE-UNDECLARED-*` tiers are GATE-FIRST candidates, never free banks — S69 gated that class **0/28** raw; the route is `fix_arity_callers --any-proto` then the gate. | | | **S69 tooling — the §378 SELF-CALLER CAST** (P31 S69) | **`tools/cast_self_callers.py` (NEW)** — the mirror of `cast_call_sites.py`. That one fixes the DRAFT calling a conflicting CALLEE; this one fixes the TU's OWN already-banked code calling the function the draft is about to DEFINE, which nothing handled and which is the terminal blocker of the §376 pile. **Never run it alone** — it answers the error that `fix_arity_callers --any-proto` CREATES: no-protoing the conflicting decl makes the draft's definition the prototype in scope, so the TU's own call fails anew with `too few arguments`. Order is `arity -> self-cast -> [--sync-decls] -> gate`. Byte-neutral because gcc-2.7.2 folds a cast of a known function symbol back to a direct `jal`. `--sync-decls` covers the narrow-param case a no-proto decl CANNOT legally reach (C89 requires promotion-stable parameter types when one declaration has no prototype, so `void f(s16)` is illegal against `extern void f();` — which is exactly why `fix_arity_callers` skips it as 'narrow-param'); safe only once the call sites are cast, because a declaration then emits no code. It REFUSES a function whose return type it cannot read off the draft (R43). **Journals every edit, and `--undo-journal --keep ` after the gate is MANDATORY** — a leftover cast made `ov_SC07_000` stop compiling and every later gate verdict on it measured a broken baseline. Wired into `recover_integration.py --stages arity,self-cast` (tier `binary`), prescribed by `residual_rules_b`'s decl-conflict tiers, and in the playbook §4b. Measured: **8 banked of 28** including `main/func_80036D58` at zero agent tokens; generalises to the callee the diagnostic NAMES (banked `main/func_80021D38` that way). | +| | **S69 tooling — the NEAR-TWIN BAND** (P31 S69) | **`tools/seed_ref.py --near [--max-d N] [--near-control]` (WIDENED, not a new tool — R33)** — the fleet-wide twin oracle gained an edit-distance tier over reloc-normalized instruction streams, because the exact-hash tier answers only *"is there a byte-identical copy?"* while a frontier needs *"is there anything CLOSE?"*. **Measured widening: 22 of 352 reachable open stubs had a d=0 hash twin; 75 of 352 (21%) have a banked match at d<=25 — 3.4x.** Root cause of the gap is §389: `h_norm`'s normalizer drops its pending lui-hi on an intervening R-type, so indexed-global reloc twins hash differently and are invisible to seed_ref/twin_sweep/dedup/family-maps simultaneously. **Do NOT fix `h_norm`** — every stored map and calibration keys on it; the near tier reads through the hole. Verification built in: sound prefilters that cannot lose a true pair, R32 population assertion, R34 cross-check reproducing all 22 exact twins every run, R39 controls (positive 200/200 at d=0; random-pair base rate 1.17%). Classes emitted: HASH-TWIN · RELOC-ONLY (mechanical — remap via `family_remap` and gate; **8 of 10 banked at ~0 agent tokens on first use**) · NEAR-COUSIN (seeded crack). Known remaining gap: `family_sweep.load_sigs` globs `ov_` only, so 38 md_ + 5 resident + 67 main are structurally invisible to it (a 407-ins md_SC05_026 PURE twin of banked ov_MAIN_012 code was found in the wild). Full audit: `.run/S69_fable/report.md`. | | | **S68 fixes — six instances of the overlay-layout assumption** (cookbook **§363**) | `dedup_propagate` could not even IMPORT (`os.` at module level in the one module that imports `os as _os`). `seed_ref` offered main's LINKED-subseg DEAD TEXT as bankable twins (43 of 82 hits — a draft there gates GREEN while wrong); now refuses and COUNTS the refusal. `parallel_gate.stage_generated` hard-coded `build//.ld`; now asks the Makefile for `_LD_SCRIPT`/`_UNDEF_SYMS`/`_UNDEF_FUNCS` and REFUSES when absent. `rtu_match` gained **`--tu`** (+ `blocker_probe` passes `stub.path` and `stub.asm_dir`) — it reconstructed `src//.c`, which is the overlay layout; main's sources are LOOSE FILES in `src/`. `gate_stage` no longer synthesises `--out`/`--good-sha` — for main those resolved to a nonexistent path and then **ov_SC01_077's SHA** via `DEF_SHA`. **`psyq_integrate`**: the `*_externals.ld` map is now MONOTONIC — it was re-derived against the CURRENT `.ld`, so `firstfile = 0x80061FA8;` was DROPPED on every incremental relink and main was 2 bytes red before any draft was spliced (**the true identity of the 2026-08-15 'main link defect'**). | | | **S68 — the module-binary -O0 carve route** (cookbook **§371**) | `jr_isolate_all` + `overlay_src_split` + the **Makefile -O0 glob widened to `src/md_*/md_*_o0?.c`** open carving for the single-object `md_*` binaries. Three stacked causes behind one `unaddressable content` message (interior-YAML-comment symbol-list truncation; a trailing verbatim-asm chunk with no region; bare tag forward decls), then the **spimdisasm rodata-migration trap**: migrated rodata follows its function ONLY within the same subseg, so a carve silently drops it and `INCLUDE_RODATA` cannot bring it back — rename the `.rodata` subseg to the object its emitters moved to. **The Makefile hunk MUST be committed with the carve** or a fresh clone loses -O0 on the region and every draft banked there mystery-fails. | | | `tools/recover_rejects.py` | **(P31 S59)** Free recovery of PRE-GATE rejects, wired into the maintenance lane. Two paths exist for a draft that does not bank and only one was recorded: a gate failure gets a backlog row (closeness/class/best draft), while a draft the reloc pre-filter drops reached nothing — **569 of 1,261 drafts over eight waves, 45%**. Of the `MISMATCH?` rejects, **13% carry `shape: MATCH`** — right body, wrong symbol names, i.e. the §171 stale-seed class `aprop_symfix` rebases deterministically. Reads `.run/reloc_rejects.jsonl` (written by `ox_campaign.reloc_filter`), keeps shape-MATCH rows that are STILL open stubs, runs `aprop_symfix --fix`, and STAGES the rebased bodies into `.run/sweep_maint//` for the lane's existing free gate. It never substitutes, gates or commits — a bad recovery can waste a build, never a bank. Tried-once is remembered in `.run/recover_rejects_seen.json`. Zero model tokens. | diff --git a/docs/accelerators.md b/docs/accelerators.md index f9a491996..921e9d23f 100644 --- a/docs/accelerators.md +++ b/docs/accelerators.md @@ -383,3 +383,42 @@ any per-function matching oracle compiles in isolation, and every real bank comp no-proto the conflicting decl and the draft's own definition becomes the prototype in scope, so the same call site now fails with `too few arguments`. A class can look dead after one fix and be three mechanical steps from banking. Drive the chain to a byte verdict before writing anything off. + +## #17 — YOUR SIMILARITY JOIN IS PROBABLY A POINT WHEN IT SHOULD BE A BAND (P31 S69) + +**What happened.** This project's twin oracle keyed on an exact signature hash for sixty-plus +sessions. It found **22 of 352** reachable open stubs with a banked twin — 6% — and everything else +read as a singleton. One agent-run added an edit-distance tier over reloc-normalized streams and the +number went to **75 of 352 — 21%, a 3.4x widening on a corpus we believed was fully mined.** Thirty-one +of the new rows were PURE twins of already-banked bodies differing only in relocation slots; ten were +clean, and **eight banked the same day for ~0 agent tokens**. One 94-instruction exemplar served five +open copies. + +**Why the hash missed them.** The normalizer under-matched by design: it dropped a pending `lui`-hi +whenever an R-type intervened, so the indexed-global triad kept its `%lo` in the hash (§389). Two +per-overlay copies of one function differing only in a data symbol's ADDRESS hashed differently. + +**The generalisable law — and it is about ERROR DIRECTION, not accuracy.** + +| the question | wants | a wrong answer costs | +|---|---|---| +| **dedup**: "are these certainly the same?" | under-matching | a duplicated bank; cheap, visible | +| **frontier join**: "is anything close to this?" | over-matching | free work becomes an invisible singleton; expensive, silent | + +**One hash cannot serve both.** If you built your similarity layer for dedup — almost everyone does, +because dedup is the first use — then your frontier join is silently lossy, and the loss looks exactly +like "this function is unique", which nobody ever investigates. + +**What to do, in any decomp:** build the near tier at the SAME time as the exact tier, not later. +Normalize relocations out of the stream; prefilter soundly on length and opcode histogram so no true +pair can be lost; edit-distance the survivors. Then verify it three ways: assert the population it +scanned (R32), require the band to reproduce every exact-hash pair on every run (R34), and control +against random pairs to get the base rate (1.17% here, so the signal is real). Do NOT retrofit the +normalizer itself — every stored map, ledger and calibration keys on it; read *through* the hole with +a new tier instead. + +**Why it matters more for a NEW project than it did for us.** The band pays from the very first bank: +every function you crack becomes an exemplar for everything within a few instructions of it, +immediately. Build it late and you accumulate invisible-singleton debt that costs a whole session to +recover — and you will never know how much you left on the floor, because the tool reports a +confident, true, useless number. diff --git a/docs/cookbook-index.md b/docs/cookbook-index.md index 4a9c5e3f0..8adff5746 100644 --- a/docs/cookbook-index.md +++ b/docs/cookbook-index.md @@ -2,7 +2,7 @@ > **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section. > -> `docs/matching-cookbook.md` is ~716 KB / 1039 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. +> `docs/matching-cookbook.md` is ~716 KB / 1044 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. **How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win. @@ -101,7 +101,7 @@ - **§346** — `c ? X : -X` TAKES expand_expr's COND_EXPR **SINGLETON** PATH (copy, then negate IN PLACE) — AN if/else STATEMENT GIVES THE TWO-ARM FORM (P31 S67; byte-proven ov_SC03_102/func_80180C38, closed the last instruction) L31322 - **§370** — ★★ — A **HARD BOUND** FROM sched.c, AND THE reorg SLOT-STEAL DIAGNOSTIC (P31 S68; main/func_8001BC6C, 69 ins, NOT closed — 33 → 28 over ~45 measured compiles) L31838 -### instruction scheduling (83) +### instruction scheduling (84) - **§3-T2** — Source statement order drives instruction scheduling L78 - **§3** — When a diff is pure scheduling → decomp-permuter (harness built, Phase 6) L107 @@ -186,6 +186,7 @@ - **§3-3.** — HARD FACT FOR THE SCHEDULING MAP — an `asm` ALWAYS has priority 1 L32010 - **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) L32164 - **§381** — THE `insn_count` HOIST THRESHOLD IS A DIAL YOU CAN READ WITH `cc1 -dL` (P31 S69; four independent uses in one wave) L32216 +- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) L32309 ### register allocation & pins (127) @@ -317,7 +318,7 @@ - **§374** — A `register … __asm__("$30")` RESERVATION IS **NOT HONOURED** BY `move_movables` (P31 S68; main/func_80015608, 86 ins) L32026 - **§375** — AN `$a0`-`$a3` PIN USED LATE RELOCATES AN **EARLIER** OUTGOING-CALL USE OF THAT REGISTER (P31 S68; main/func_8005F0C8, 88 ins) L32042 -### CSE / redundancy / rematerialization (43) +### CSE / redundancy / rematerialization (44) - **§46** — The `func_80178D40` crack (890 ins ×134, the heaviest core in the game): four LOOP-STRUCTURE levers cheap-Opus found by reading loop.c/jump.c/cse.c (Phase 26 session 8, 2026-07-13) L3347 - **§83d** — CSE's quantity budget is WHOLE-FUNCTION, so a local rewrite cannot fix a local symptom L6486 @@ -362,8 +363,9 @@ - **§368** — ★★★ — THE **RELOAD-REMAT CONSTANT**: REACH A REGISTER NO PIN CAN REACH (P31 S68; ov_SC03_105/func_80187A30, 339 ins, fable escalation closed 8 → 0 in ONE edit) L31792 - **§373** — ★★★ — THE **DEAD-RESET CSE-BREAKER**, THE PIN THAT BREAKS A sched2 ANTI-DEP, AND WHY AN `asm` CAN NEVER RAISE PRIORITY (P31 S68; byte-proven ov_SC06_010/func_8017E764, 438 ins, fable escalation closed 8 → 0) L31977 - **§3-1.** — DEAD-RESET CSE-BREAKER — the zero-footprint replacement for a §195-I asm re-tie L31979 +- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) L32309 -### loops & induction variables (43) +### loops & induction variables (45) - **§3-T1** — Loop pointer: top-of-body for `addu` induction, not constant-folded `addiu` L71 - **§34** — The `func_80138ED0` giant crack: gcc-2.7.2's **3-qty sort bug** + the **zero-byte asm allocation toolkit** + the **giv-init fence** (Phase 24 T5; Opus→close=21, Fable5→MATCH ×134) L2473 @@ -408,6 +410,8 @@ - **§357** — ONE STRUCT POINTER, NOT TWO: A SECOND SOURCE VARIABLE BUILDS A THIRD IV (P31 S68; byte-proven ov_SC06_029/func_80181DF8, 335 ins, 330 → 13) L31589 - **§361** — ★ — A LOOP-TAIL BYTE SIGNATURE THAT NAMES ITS SOURCE SHAPE — AND THE LAW THAT A "SCHEDULING TIE" MAY BE YOUR OWN EARLIER LEVER (P31 S68; byte-proven main/func_800241C0, fable escalation, 19 → 0 in 3 iterations) L31637 - **§366** — ★★ — `group_case_nodes` MERGES **STACKED CONSECUTIVE** CASE LABELS: GIVE EVERY CASE ITS OWN BODY (P31 S68; ov_SC01_001/func_8017EC28, **first-try MATCH 360/360**, 96/96 relocs audited) L31757 +- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) L32309 +- **§386** — ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MASK, NEVER A DEREF (P31 S69; byte-proven main/func_80020598, 292 ins, escalation 1 → 0) L32335 ### structs, block moves & memcpy (84) @@ -704,7 +708,7 @@ - **§361** — ★ — A LOOP-TAIL BYTE SIGNATURE THAT NAMES ITS SOURCE SHAPE — AND THE LAW THAT A "SCHEDULING TIE" MAY BE YOUR OWN EARLIER LEVER (P31 S68; byte-proven main/func_800241C0, fable escalation, 19 → 0 in 3 iterations) L31637 - **§367** — RECONCILING A DECL CONFLICT BETWEEN TWO DRAFTS FOR THE SAME TU (P31 S68; main) L31777 -### jump tables & switches (54) +### jump tables & switches (55) - **§8** — rodata island (compiler jump tables) — the `.data→.rodata→.data` sandwich (Phase 7) L339 - **§8a** — rodata island in a flat OVERLAY — the tail sandwich, per matched jr-function (Phase 26 — PoC PROVEN) L361 @@ -760,8 +764,9 @@ - **§362** — TWO TRAPS WHEN A CARVE MOVES A STUB INTO THE `-O0` TU (P31 S68; byte-proven, 6 fns / 2,547 ins across ov_MAIN_012 / ov_SC02_037 / ov_SC03_107) L31682 - **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) L31890 - **§384** — ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE BANK (P31 S69; measured twice, cost one destroyed match) L32259 +- **§387** — ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep the fold (P31 S69; byte-proven main/func_80030F80, 343 ins, escalation 3 → 0) L32361 -### optimisation level (-O0/-O2) (21) +### optimisation level (-O0/-O2) (22) - **§6** — Per-module optimization mixing — the -O0 boot module (Phase 7) L265 - **Detecting** — the opt level (do this first) L273 @@ -784,6 +789,7 @@ - **Addendum** — Addendum to §261a — the -O0 increment-operator copy also fires on the POINTER LOCAL itself (func_8001212C) L30473 - **Addendum** — Addendum to §164-XX/§165-06 — at -O0 a bare local IS a memory lvalue, so the increment ope (func_8001212C) L30477 - **§362** — TWO TRAPS WHEN A CARVE MOVES A STUB INTO THE `-O0` TU (P31 S68; byte-proven, 6 fns / 2,547 ins across ov_MAIN_012 / ov_SC02_037 / ov_SC03_107) L31682 +- **§388** — ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6) L32384 ### family propagation & sweeps (116) @@ -904,7 +910,7 @@ - **§368** — ★★★ — THE **RELOAD-REMAT CONSTANT**: REACH A REGISTER NO PIN CAN REACH (P31 S68; ov_SC03_105/func_80187A30, 339 ins, fable escalation closed 8 → 0 in ONE edit) L31792 - **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) L32164 -### integration / TU plumbing (71) +### integration / TU plumbing (72) - **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) L456 - **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) L502 @@ -977,8 +983,9 @@ - **§367** — RECONCILING A DECL CONFLICT BETWEEN TWO DRAFTS FOR THE SAME TU (P31 S68; main) L31777 - **§376** — ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT THE **TU** (P31 S69; measured 0/28) L32055 - **§378** — ★★★ — THE **SELF-CALLER CAST**: LET A TU KEEP CALLING THE FUNCTION IT IS ABOUT TO DEFINE (P31 S69; byte-proven ov_SC04_010/func_8017D6CC) L32114 +- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) L32309 -### build graph, splat & the harness (186) +### build graph, splat & the harness (187) - **§4** — Flag/toolchain gotchas L190 - **Build** — mechanism — per-file opt override (splat resegmentation) L307 @@ -1166,6 +1173,7 @@ - **§358** — (sharpens §333) — AN **UNREFERENCED** FIXED-SIZE AGGREGATE LOCAL IS LOAD-BEARING (P31 S68; same function) L31598 - **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) L31890 - **§383** — TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) L32244 +- **§388** — ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6) L32384 ### process, measurement & doctrine (127) @@ -1297,7 +1305,7 @@ - **§376** — ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT THE **TU** (P31 S69; measured 0/28) L32055 - **§384** — ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE BANK (P31 S69; measured twice, cost one destroyed match) L32259 -### (unbucketed — title matched no symptom vocabulary) (307) +### (unbucketed — title matched no symptom vocabulary) (308) - **§3-How** — to use this L30 - **§1** — Idiom catalog (asm pattern → C that produces it) L39 @@ -1606,6 +1614,7 @@ - **§377** — THREE HARNESS DEFECTS FOUND IN ONE GATING SESSION, ALL "A CONFIDENT NUMBER ABOUT A SMALLER WORLD" (P31 S69) L32097 - **§380** — ★★★ — **A SECOND SET OF A PSEUDO DISQUALIFIES IT FROM `move_movables`** (P31 S69; main/func_800215F4, 465 ins, closeness 106 → 59 → 39) L32195 - **§382** — TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) L32232 +- **§389** — ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLETON (P31 S69; 31 stubs / 4,811 ins recovered, 8 banked same day) L32409 ## All sections, in order @@ -2649,6 +2658,11 @@ - **§382** — TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) L32232 - **§383** — TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) L32244 - **§384** — ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE BANK (P31 S69; measured twice, cost one destroyed match) L32259 +- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) L32309 +- **§386** — ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MASK, NEVER A DEREF (P31 S69; byte-proven main/func_80020598, 292 ins, escalation 1 → 0) L32335 +- **§387** — ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep the fold (P31 S69; byte-proven main/func_80030F80, 343 ins, escalation 3 → 0) L32361 +- **§388** — ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6) L32384 +- **§389** — ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLETON (P31 S69; 31 stubs / 4,811 ins recovered, 8 banked same day) L32409 --- @@ -3700,3 +3714,8 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: ` | L32232 | §382 | TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) | | L32244 | §383 | TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) | | L32259 | §384 | ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE | +| L32309 | §385 | ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init prehead | +| L32335 | §386 | ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MA | +| L32361 | §387 | ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep th | +| L32384 | §388 | ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 | +| L32409 | §389 | ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLE | diff --git a/docs/generic-decomp-package.md b/docs/generic-decomp-package.md index e75d0bad2..2d8d8e988 100644 --- a/docs/generic-decomp-package.md +++ b/docs/generic-decomp-package.md @@ -30,11 +30,39 @@ tool existed — and that were invisible until an idiom taught us to look. **2. The structural tooling, before the first crack.** The families/twins/dedup layer is what converts one crack into N banks. In BFM this arrived late and retroactively harvested thousands of instructions. Port it first: -`corpus` (the coverage oracle) · `seed_ref` / twin join on signature hashes · `family_remap` / +`corpus` (the coverage oracle) · `seed_ref` (exact-hash twin join **AND its `--near` edit-distance +band — see 2b**) · `family_remap` / `family_sweep` · `dedup_propagate` (position-locked overlay sharing) · the `-O0`/opt-level carve chain (`o0_detect`, `o0_subsplit`, `o0_boundary`) · `wall_sweep` (toolchain walls) · the draw filter · the byte-gate + clean-fleet verifier. +**2b. THE SIMILARITY JOIN MUST BE A BAND, NOT A POINT (P31 S69 — port this, it is cheap and it +compounds).** A twin oracle keyed on an exact signature hash answers only *"is there a byte-identical +copy?"*. That is the wrong question for a frontier. The right one is *"is there anything CLOSE?"*, and +the difference is not marginal: + +| tier | reachable open stubs with a banked match | +|---|---| +| exact hash (d=0) — where this project sat for 60+ sessions | **22 of 352 (6%)** | +| edit-distance band to d<=25 (`seed_ref --near`) | **75 of 352 (21%)** | + +**A 3.4x widening, found in one agent-run, on a corpus we thought was fully mined.** 31 of the new +rows were PURE reloc-only twins of already-banked bodies — free work that had been sitting invisible; +8 banked the same day at ~0 agent tokens. The root cause was a normalizer that under-matched by +design (§389): safe for dedup, silently lossy as a frontier join. + +For a NEW decomp this matters *more* than it did here, because the band pays from the very first +banks: every function you crack immediately becomes a potential exemplar for everything within a few +instructions of it, and you never accumulate the invisible-singleton debt this project spent a +session recovering. Build the near tier at the same time as the exact tier — not sixty sessions +later. Concretely: normalize relocations out of the instruction stream, prefilter soundly on +length/opcode-histogram so no true pair can be lost, then edit-distance the survivors; assert the +population (R32), cross-check that the band reproduces every exact-hash pair (R34), and control +against random pairs for the base rate (R39: 1.17% here). + +**And audit every hash you own for BOTH questions.** Dedup wants under-matching; a frontier join +wants over-matching. One hash cannot serve both error directions, and the failure is silent. + **3. The differential-oracle harness (accelerators #15) — the one that works at 0%.** Two independent paths per question, disagreement fails loudly, on a schedule. diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index a60ad7224..5e06a5ae4 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -32305,3 +32305,144 @@ This is the R22 corollary aimed the other way. R22 says a *reverted* config need (a build over stale extract state) was broken, and I attributed the failure to the subject, reverted a legitimate 96-line match, and wrote a checkpoint calling it a false bank. The give-away I ignored: the commit's own diffstat showed `config/overlays.mk` and a splat yaml right next to the `.c`. + +## §385 ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) + +**Closes cookbook §5's open class.** Symptom: two independent one-instruction inits after a mult (or +any latency chain) emit in swapped order, and the swap is invariant under source order, register +pins, and zero-byte barriers. + +**Root cause, read out of the pinned compiler** (`sched.c` `priority()`:1425 + `rank_for_schedule()`:2385): +sched2 schedules BACKWARD. The init that writes a register the mult READS carries an anti-dependence +that inherits the mult's priority (anti cost clamps to 1, so `+cost-1 = +0`), while the dependence-free +init stays at priority 1. **Priority beats the LUID tie-break**, so no C-level reordering can flip it. + +**The fix — donate priority to the loser** via a NON-volatile single-instruction asm with a dead +extra input reading the winner's destination: + +```c +c2 = 0; +__asm__("addiu %0,%1,18" : "=r"(ptr) : "r"(base), "r"(c2)); +``` + +The true dep on the `c2` init donates priority 2 AND delays the asm's backward release until the +donor is placed — target order, identical bytes. + +**Two traps.** A **volatile** asm cannot be used: volatile is a full scheduling barrier, so it +inherits the whole chain's priority and glues itself behind the `mflo`. And a bare ghost asm whose +output is immediately overwritten gets deleted. + +## §386 ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MASK, NEVER A DEREF (P31 S69; byte-proven main/func_80020598, 292 ins, escalation 1 → 0) + +**Diff tell:** a one-row `reg+imm` residual where yours reads offset-from-**giv** and the target reads +offset-from-**biv** *at the same address* — and the twin expression in another arm IS giv-based in the +target. + +**Why no deref spelling works.** A byte deref is a `DEST_ADDR` giv, and `loop.c`'s `combine_givs` +folds it onto the reduced register **unconditionally**: identity or `express_from`, then +`memory_address_p` and equal MIPS `ADDRESS_COST`. There is no per-instance guard, so no temp, no +ordering and no spelling can split two identical `(mult, add)` givs. + +**The lever — defer the load's birth past `loop.c`:** + +```c +b1 = (w >> 8) & 0xFF; /* NOT ((u8 *)p)[1] */ +``` + +Post-loop `combine` then narrows `zero_extend(lshiftrt(mem))` into an `lbu` at base+k off the +ORIGINAL register. + +**Two corollaries.** (1) Temp-pointer escapes are impossible: `cse1`'s `find_best_addr` tie-break at +equal `ADDRESS_COST` prefers the MORE complex address (it frees a register) and folds any same-ebb +temp back into `(plus base k)` before loop — and anything opaque enough to survive cse1 is equally +opaque to cse2/combine. (2) The narrowed spelling changes spill demand, so expect the §333/§334 +dead-aggregate pad to need retuning (here `pad[6]` → `pad[4]` at frame 0x88). + +## §387 ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep the fold (P31 S69; byte-proven main/func_80030F80, 343 ins, escalation 3 → 0) + +When one case needs a cse-forced RELOAD of the switch-index load while another must keep FOLDING that +same value into a call argument: + +* **Do NOT put the invalidator in the case body** — `reorg.c`'s `stop_search_p` halts + `fill_slots_from_thread` at ANY asm insn (`asm_noperands >= 0`), so branch-delay-slot hoists into + that thread die (+1 length). +* **Do NOT volatile-cast the load** — `loop.c:339` `init_recog_no_volatile` fails `recog` on every + `DEST_ADDR` giv rewrite of a volatile mem, so its address stays biv-based forever. + +**Instead:** name the index in a local; load the dispatch byte into an `s32` local (a `u8` costs an +`andi`); place a **non-volatile** `__asm__("" ::: "memory")` AFTER that `lbu` and BEFORE the switch — +zero bytes, invalidates cse's memory table on every path, and sits outside every case's delay-slot +thread. Never between a load and its first use: it eats the final-pass load-delay nop. Fold-cases +then pass the named local; reload-cases re-read memory. + +**Side-effect law, and it is general:** adding or removing pseudos flips razor-thin global-alloc ties +among pre-loop invariant pointers — which one loses its callee-saved register and gets reload- +rematerialised as inline `lui`/`addiu`. **This is INVISIBLE under `match_one`'s HI16/LO16 mask** and +surfaces as a bogus "schedule" diff at the remat site. Audit with `objdump -r`, and flip it back by +swapping the two invariants' init order. + +## §388 ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6) + +At `-O0` gcc-2.7.2 allocates with **`stupid.c`**, not local/global-alloc. It is simple enough to +simulate exactly, so predict the colours instead of guessing at spellings: + +* every insn **including NOTES** gets a sequential `suid`; +* a pseudo set at suid `s` dies at `max(last_use, s+2)` and occupies `[born, dead-1]`; +* pseudos are allocated **longest-first**, then **fewer-refs-first**, then lowest regno, first-fit + `$v0,$v1,$a0,$a1,…`; +* a `register` declaration emits a zero-byte head-(use) insn making that var ONE convex interval. + +**Two consequences you can diagnose by inspection.** (1) An `-O0` `copy;sll` multiply chain +2-colours ping-pong *because* the `+2` rule makes each link conflict with the next — so a NOTE inside +the chain (any statement-expression's `BLOCK_BEG`/`BLOCK_END`) lapses the slack, one copy becomes a +same-register move, and `final.c` deletes it: **a length −1 next to a `({...})` is THIS, not +scheduling.** (2) A `register` decl recolours the whole chain via its head-use interval, so +exact-length REGALLOC rotations in `-O0` code are decl-lifetime artifacts. `cc1 -dr` suid arithmetic +predicts the colours before you compile. + +**Also from the same run — the SYMBOL-ADDEND SHIELD.** In a pointer index, `(s32)(idx + K)`: the +same-mode cast hides the `PLUS` from `pointer_int_sum`'s distributive rewrite, yet `EXPAND_SUM` still +absorbs `K` into the relocation, emitting `la SYM+K*size` with the scale applied to `idx` alone. Use +it when the target shows a symbol+constant base feeding a scaled index — no plain spelling +reproduces it, because the frontend otherwise distributes `K` into the pointer before scaling. + +## §389 ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLETON (P31 S69; 31 stubs / 4,811 ins recovered, 8 banked same day) + +**The hole.** `sig_image.norm_stream` tracks a pending `lui`-hi so it can neutralise `%hi`/`%lo` +pairs — but it DROPS that pending hi the moment an R-type intervenes. The indexed-global triad is +exactly that shape: + +``` +lui $at, %hi(arr) +addu $at, $at, idx <-- R-type; the pending hi is dropped here +lw r, %lo(arr)($at) <-- %lo survives into the hash +``` + +So two per-overlay copies of ONE function that differ only in a data symbol's ADDRESS hash to +**different** `h_norm`. They are byte-identical modulo relocation and they are invisible to every +hash-keyed consumer at once: `seed_ref` (d=0 tier), `twin_sweep`, `dup_report`, +`config/dedup.us.yaml`, and the family maps' structural tier. + +**Measured consequence (2026-09-01).** Beyond the 22 stubs with a d=0 hash twin, **31 more reachable +open stubs (4,811 ins) were PURE twins of already-banked bodies at instruction edit distance 1–5** — +`family_remap.classify_member` seconds all 31 as PURE. Ten were clean of jtbl carve blockers; +remapping them and gating banked **8**, at ~0 agent tokens. One exemplar (`ov_SC06_033:0x80185f6c`, +94 ins) served **five** open copies; `ov_MAIN_012:0x8016ab6c` (188 ins) serves five more. + +**The fix is a NEW TIER, NOT A NEW NORMALIZER.** Do **not** change `h_norm`: every stored map, +ledger and calibration in the project keys on it, and a re-hash invalidates all of them. Instead read +*through* the hole with an edit-distance tier over reloc-normalized streams — +`tools/seed_ref.py --near [--max-d N]`, which scans all 213 binaries, reproduces all 22 hash twins as +an R34 cross-check on every run, and ships R39 controls (positive 200/200 at d=0; random-pair base +rate 1.17%). + +**THE LAW, and it generalises past this project.** *A conservative normalizer is SAFE for a DEDUP +claim and UNSAFE as a FRONTIER JOIN.* Dedup asks "are these certainly the same?" — under-matching +there is harmless. A frontier join asks "is there anything close to this?" — and there, every missed +match silently converts free mechanical work into an apparent singleton that a future session will +pay an agent to re-derive from scratch. Audit any hash you use for BOTH questions; they want opposite +error directions. + +**Diff tell:** an open stub your card calls "no banked twin — derive from the .s", whose body is a +per-location copy of engine code that exists in a sibling overlay. Run the near tier before believing +a singleton verdict. diff --git a/docs/wave-playbook.md b/docs/wave-playbook.md index ab4c472ab..4989a40da 100644 --- a/docs/wave-playbook.md +++ b/docs/wave-playbook.md @@ -100,6 +100,18 @@ target**, joined on the corpus signature hashes (`tools/seed_ref.py`). > neither tool could see them. A `mechanical_remap_refused` flag now tells the agent: copy the BODY, > expect a declaration blocker. +### 2a. RUN `seed_ref --near` BEFORE BELIEVING ANY "NO TWIN" VERDICT (S69) + +``` +.venv/bin/python tools/seed_ref.py --near --max-d 25 --json .run//near.json +``` + +The exact-hash tier finds 22 of 352 reachable open stubs; the near band finds **75 of 352 (21%)** — +3.4x. `h_norm` is blind to indexed-global relocs (§389), so true reloc-only twins of ALREADY-BANKED +bodies hash differently and read as singletons. A `RELOC-ONLY` row is mechanical: `family_remap` the +banked exemplar onto the open address, then gate — **8 of 10 banked at ~0 agent tokens on first use**, +one 94-ins exemplar serving five open copies. Never send a RELOC-ONLY row to a drafting agent. + ### 2b. RUN `neighbor_ref` FOR EVERY CARD — the biggest measured cost lever in the wave ```