mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
docs: the NEAR-TWIN BAND — §389, accelerator #17, SETUP, playbook §2a, generic-decomp-package
The exact-hash twin tier found 22 of 352 reachable open stubs (6%). The edit-distance band added by `seed_ref --near` finds 75 of 352 (21%) — 3.4x — on a corpus we believed fully mined. 31 of the new rows were PURE reloc-only twins of already-banked bodies; 8 banked the same day at ~0 agent tokens, one 94-ins exemplar serving five open copies. * cookbook §389: the h_norm hole (norm_stream drops its pending lui-hi on an intervening R-type, so indexed-global reloc twins hash differently and vanish from seed_ref/twin_sweep/dedup/family-maps at once). Do NOT fix h_norm — every stored calibration keys on it; the near tier reads through it. * accelerators #17: the generalisable law. A similarity hash built for DEDUP under-matches by design, which is correct for dedup and silently lossy as a FRONTIER join — the two questions want opposite error directions, and the frontier failure looks exactly like "this function is unique". * generic-decomp-package §2b: build the near band at the same time as the exact tier, with the three verifications. It pays from the first bank for a new project, where we paid a session to recover the debt. * SETUP inventory row + playbook §2a (run it before believing any "no twin" verdict; never send a RELOC-ONLY row to a drafting agent).
This commit is contained in:
Drew T
@@ -780,6 +780,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | **S68 tooling — the gater lane + the -O0 route** (P31 S68) | **`tools/gater_lane.py` (NEW)** — the continuous gater: drains a wave's finished drafts into `parallel_gate`, grouped by binary, `--r22` by default. Ledger AND verdicts keyed **`binary:fn:arm`** (R48 — and an escalation ALWAYS has a prior verdict, so arm-keying is what stops an in-flight fable draft riding the opus one). `--extra BINARY:PATH` for non-wave drafts, `--skip-binary` for lanes that may be writing, and **main is routed IN-TREE via `harvest_verify`** because `parallel_gate`'s worktree cannot stage main's psyq_integrate link inputs. **`tools/workflows/escalate_fable.js` (NEW)** — warm-started escalation: passes the prior draft + its measured closeness + its ruled-out levers, and demands a reusable `new_idiom`. **`tools/o0_boundary.py` (NEW)** — the stranded-boundary -O0 sweep (141 binaries / 288 boundaries / 0 candidates: the class is EXHAUSTED, and that null is negative-controlled). |
|
||||
| | **S69 tooling — the TRIAGE LADDER** (P31 S69) | **`tools/triage_ladder.py` (NEW)** — the zero-token pass that answers *"does this target need an agent at all?"* before one is spent. **The PRE/POST split is the point:** `--pre <wave>` runs the TARGET-SIDE tiers only (BANKED · WALL-332 · PARKED) — no draft, no build, milliseconds — and writes `<wave>/triage.json` + `triage_exclude.txt`; `--post` runs the full `residual_rules_b` residual routing, which needs a draft and runs `match_one`. `--escalate B:FN` exits 2 on a walled/banked target (the check S68 was missing when it escalated a §332 wall at closeness 8). `--acceptance` is the R39/R32 harness: false-skip over EVERY open stub, recall over sampled matched fns, and a wall negative control over already-banked code — all pure filesystem work, so it runs in seconds over the whole corpus. **It REFUSES on a non-quiescent tree** (`pgrep -af` rows for gate/build lanes + `lane_inflight`): a live gate makes the stub oracle transiently wrong in both directions (§377). Wired into `wave_args.py` (drops walled/parked targets at draw time, reusing `pre_classify` — one implementation, R33) and into `tools/workflows/escalate_fable.js`, which now REFUSES any target that does not carry `triage:'DRAFT'`. **Routing correction (§376):** the `INTEG-STANDALONE-MATCH` / `NOCOMPILE-UNDECLARED-*` tiers are GATE-FIRST candidates, never free banks — S69 gated that class **0/28** raw; the route is `fix_arity_callers --any-proto` then the gate. |
|
||||
| | **S69 tooling — the §378 SELF-CALLER CAST** (P31 S69) | **`tools/cast_self_callers.py` (NEW)** — the mirror of `cast_call_sites.py`. That one fixes the DRAFT calling a conflicting CALLEE; this one fixes the TU's OWN already-banked code calling the function the draft is about to DEFINE, which nothing handled and which is the terminal blocker of the §376 pile. **Never run it alone** — it answers the error that `fix_arity_callers --any-proto` CREATES: no-protoing the conflicting decl makes the draft's definition the prototype in scope, so the TU's own call fails anew with `too few arguments`. Order is `arity -> self-cast -> [--sync-decls] -> gate`. Byte-neutral because gcc-2.7.2 folds a cast of a known function symbol back to a direct `jal`. `--sync-decls` covers the narrow-param case a no-proto decl CANNOT legally reach (C89 requires promotion-stable parameter types when one declaration has no prototype, so `void f(s16)` is illegal against `extern void f();` — which is exactly why `fix_arity_callers` skips it as 'narrow-param'); safe only once the call sites are cast, because a declaration then emits no code. It REFUSES a function whose return type it cannot read off the draft (R43). **Journals every edit, and `--undo-journal --keep <banked>` after the gate is MANDATORY** — a leftover cast made `ov_SC07_000` stop compiling and every later gate verdict on it measured a broken baseline. Wired into `recover_integration.py --stages arity,self-cast` (tier `binary`), prescribed by `residual_rules_b`'s decl-conflict tiers, and in the playbook §4b. Measured: **8 banked of 28** including `main/func_80036D58` at zero agent tokens; generalises to the callee the diagnostic NAMES (banked `main/func_80021D38` that way). |
|
||||
| | **S69 tooling — the NEAR-TWIN BAND** (P31 S69) | **`tools/seed_ref.py --near [--max-d N] [--near-control]` (WIDENED, not a new tool — R33)** — the fleet-wide twin oracle gained an edit-distance tier over reloc-normalized instruction streams, because the exact-hash tier answers only *"is there a byte-identical copy?"* while a frontier needs *"is there anything CLOSE?"*. **Measured widening: 22 of 352 reachable open stubs had a d=0 hash twin; 75 of 352 (21%) have a banked match at d<=25 — 3.4x.** Root cause of the gap is §389: `h_norm`'s normalizer drops its pending lui-hi on an intervening R-type, so indexed-global reloc twins hash differently and are invisible to seed_ref/twin_sweep/dedup/family-maps simultaneously. **Do NOT fix `h_norm`** — every stored map and calibration keys on it; the near tier reads through the hole. Verification built in: sound prefilters that cannot lose a true pair, R32 population assertion, R34 cross-check reproducing all 22 exact twins every run, R39 controls (positive 200/200 at d=0; random-pair base rate 1.17%). Classes emitted: HASH-TWIN · RELOC-ONLY (mechanical — remap via `family_remap` and gate; **8 of 10 banked at ~0 agent tokens on first use**) · NEAR-COUSIN (seeded crack). Known remaining gap: `family_sweep.load_sigs` globs `ov_` only, so 38 md_ + 5 resident + 67 main are structurally invisible to it (a 407-ins md_SC05_026 PURE twin of banked ov_MAIN_012 code was found in the wild). Full audit: `.run/S69_fable/report.md`. |
|
||||
| | **S68 fixes — six instances of the overlay-layout assumption** (cookbook **§363**) | `dedup_propagate` could not even IMPORT (`os.` at module level in the one module that imports `os as _os`). `seed_ref` offered main's LINKED-subseg DEAD TEXT as bankable twins (43 of 82 hits — a draft there gates GREEN while wrong); now refuses and COUNTS the refusal. `parallel_gate.stage_generated` hard-coded `build/<b>/<b>.ld`; now asks the Makefile for `<b>_LD_SCRIPT`/`<b>_UNDEF_SYMS`/`<b>_UNDEF_FUNCS` and REFUSES when absent. `rtu_match` gained **`--tu`** (+ `blocker_probe` passes `stub.path` and `stub.asm_dir`) — it reconstructed `src/<source>/<split>.c`, which is the overlay layout; main's sources are LOOSE FILES in `src/`. `gate_stage` no longer synthesises `--out`/`--good-sha` — for main those resolved to a nonexistent path and then **ov_SC01_077's SHA** via `DEF_SHA`. **`psyq_integrate`**: the `*_externals.ld` map is now MONOTONIC — it was re-derived against the CURRENT `.ld`, so `firstfile = 0x80061FA8;` was DROPPED on every incremental relink and main was 2 bytes red before any draft was spliced (**the true identity of the 2026-08-15 'main link defect'**). |
|
||||
| | **S68 — the module-binary -O0 carve route** (cookbook **§371**) | `jr_isolate_all` + `overlay_src_split` + the **Makefile -O0 glob widened to `src/md_*/md_*_o0?.c`** open carving for the single-object `md_*` binaries. Three stacked causes behind one `unaddressable content` message (interior-YAML-comment symbol-list truncation; a trailing verbatim-asm chunk with no region; bare tag forward decls), then the **spimdisasm rodata-migration trap**: migrated rodata follows its function ONLY within the same subseg, so a carve silently drops it and `INCLUDE_RODATA` cannot bring it back — rename the `.rodata` subseg to the object its emitters moved to. **The Makefile hunk MUST be committed with the carve** or a fresh clone loses -O0 on the region and every draft banked there mystery-fails. |
|
||||
| | `tools/recover_rejects.py` | **(P31 S59)** Free recovery of PRE-GATE rejects, wired into the maintenance lane. Two paths exist for a draft that does not bank and only one was recorded: a gate failure gets a backlog row (closeness/class/best draft), while a draft the reloc pre-filter drops reached nothing — **569 of 1,261 drafts over eight waves, 45%**. Of the `MISMATCH?` rejects, **13% carry `shape: MATCH`** — right body, wrong symbol names, i.e. the §171 stale-seed class `aprop_symfix` rebases deterministically. Reads `.run/reloc_rejects.jsonl` (written by `ox_campaign.reloc_filter`), keeps shape-MATCH rows that are STILL open stubs, runs `aprop_symfix --fix`, and STAGES the rebased bodies into `.run/sweep_maint/<bin>/` for the lane's existing free gate. It never substitutes, gates or commits — a bad recovery can waste a build, never a bank. Tried-once is remembered in `.run/recover_rejects_seen.json`. Zero model tokens. |
|
||||
|
||||
@@ -383,3 +383,42 @@ any per-function matching oracle compiles in isolation, and every real bank comp
|
||||
no-proto the conflicting decl and the draft's own definition becomes the prototype in scope, so the
|
||||
same call site now fails with `too few arguments`. A class can look dead after one fix and be three
|
||||
mechanical steps from banking. Drive the chain to a byte verdict before writing anything off.
|
||||
|
||||
## #17 — YOUR SIMILARITY JOIN IS PROBABLY A POINT WHEN IT SHOULD BE A BAND (P31 S69)
|
||||
|
||||
**What happened.** This project's twin oracle keyed on an exact signature hash for sixty-plus
|
||||
sessions. It found **22 of 352** reachable open stubs with a banked twin — 6% — and everything else
|
||||
read as a singleton. One agent-run added an edit-distance tier over reloc-normalized streams and the
|
||||
number went to **75 of 352 — 21%, a 3.4x widening on a corpus we believed was fully mined.** Thirty-one
|
||||
of the new rows were PURE twins of already-banked bodies differing only in relocation slots; ten were
|
||||
clean, and **eight banked the same day for ~0 agent tokens**. One 94-instruction exemplar served five
|
||||
open copies.
|
||||
|
||||
**Why the hash missed them.** The normalizer under-matched by design: it dropped a pending `lui`-hi
|
||||
whenever an R-type intervened, so the indexed-global triad kept its `%lo` in the hash (§389). Two
|
||||
per-overlay copies of one function differing only in a data symbol's ADDRESS hashed differently.
|
||||
|
||||
**The generalisable law — and it is about ERROR DIRECTION, not accuracy.**
|
||||
|
||||
| the question | wants | a wrong answer costs |
|
||||
|---|---|---|
|
||||
| **dedup**: "are these certainly the same?" | under-matching | a duplicated bank; cheap, visible |
|
||||
| **frontier join**: "is anything close to this?" | over-matching | free work becomes an invisible singleton; expensive, silent |
|
||||
|
||||
**One hash cannot serve both.** If you built your similarity layer for dedup — almost everyone does,
|
||||
because dedup is the first use — then your frontier join is silently lossy, and the loss looks exactly
|
||||
like "this function is unique", which nobody ever investigates.
|
||||
|
||||
**What to do, in any decomp:** build the near tier at the SAME time as the exact tier, not later.
|
||||
Normalize relocations out of the stream; prefilter soundly on length and opcode histogram so no true
|
||||
pair can be lost; edit-distance the survivors. Then verify it three ways: assert the population it
|
||||
scanned (R32), require the band to reproduce every exact-hash pair on every run (R34), and control
|
||||
against random pairs to get the base rate (1.17% here, so the signal is real). Do NOT retrofit the
|
||||
normalizer itself — every stored map, ledger and calibration keys on it; read *through* the hole with
|
||||
a new tier instead.
|
||||
|
||||
**Why it matters more for a NEW project than it did for us.** The band pays from the very first bank:
|
||||
every function you crack becomes an exemplar for everything within a few instructions of it,
|
||||
immediately. Build it late and you accumulate invisible-singleton debt that costs a whole session to
|
||||
recover — and you will never know how much you left on the floor, because the tool reports a
|
||||
confident, true, useless number.
|
||||
|
||||
+28
-9
@@ -2,7 +2,7 @@
|
||||
|
||||
> **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section.
|
||||
>
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1039 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
> `docs/matching-cookbook.md` is ~716 KB / 1044 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses.
|
||||
|
||||
**How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win.
|
||||
|
||||
@@ -101,7 +101,7 @@
|
||||
- **§346** — `c ? X : -X` TAKES expand_expr's COND_EXPR **SINGLETON** PATH (copy, then negate IN PLACE) — AN if/else STATEMENT GIVES THE TWO-ARM FORM (P31 S67; byte-proven ov_SC03_102/func_80180C38, closed the last instruction) <sub>L31322</sub>
|
||||
- **§370** — ★★ — A **HARD BOUND** FROM sched.c, AND THE reorg SLOT-STEAL DIAGNOSTIC (P31 S68; main/func_8001BC6C, 69 ins, NOT closed — 33 → 28 over ~45 measured compiles) <sub>L31838</sub>
|
||||
|
||||
### instruction scheduling (83)
|
||||
### instruction scheduling (84)
|
||||
|
||||
- **§3-T2** — Source statement order drives instruction scheduling <sub>L78</sub>
|
||||
- **§3** — When a diff is pure scheduling → decomp-permuter (harness built, Phase 6) <sub>L107</sub>
|
||||
@@ -186,6 +186,7 @@
|
||||
- **§3-3.** — HARD FACT FOR THE SCHEDULING MAP — an `asm` ALWAYS has priority 1 <sub>L32010</sub>
|
||||
- **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) <sub>L32164</sub>
|
||||
- **§381** — THE `insn_count` HOIST THRESHOLD IS A DIAL YOU CAN READ WITH `cc1 -dL` (P31 S69; four independent uses in one wave) <sub>L32216</sub>
|
||||
- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) <sub>L32309</sub>
|
||||
|
||||
### register allocation & pins (127)
|
||||
|
||||
@@ -317,7 +318,7 @@
|
||||
- **§374** — A `register … __asm__("$30")` RESERVATION IS **NOT HONOURED** BY `move_movables` (P31 S68; main/func_80015608, 86 ins) <sub>L32026</sub>
|
||||
- **§375** — AN `$a0`-`$a3` PIN USED LATE RELOCATES AN **EARLIER** OUTGOING-CALL USE OF THAT REGISTER (P31 S68; main/func_8005F0C8, 88 ins) <sub>L32042</sub>
|
||||
|
||||
### CSE / redundancy / rematerialization (43)
|
||||
### CSE / redundancy / rematerialization (44)
|
||||
|
||||
- **§46** — The `func_80178D40` crack (890 ins ×134, the heaviest core in the game): four LOOP-STRUCTURE levers cheap-Opus found by reading loop.c/jump.c/cse.c (Phase 26 session 8, 2026-07-13) <sub>L3347</sub>
|
||||
- **§83d** — CSE's quantity budget is WHOLE-FUNCTION, so a local rewrite cannot fix a local symptom <sub>L6486</sub>
|
||||
@@ -362,8 +363,9 @@
|
||||
- **§368** — ★★★ — THE **RELOAD-REMAT CONSTANT**: REACH A REGISTER NO PIN CAN REACH (P31 S68; ov_SC03_105/func_80187A30, 339 ins, fable escalation closed 8 → 0 in ONE edit) <sub>L31792</sub>
|
||||
- **§373** — ★★★ — THE **DEAD-RESET CSE-BREAKER**, THE PIN THAT BREAKS A sched2 ANTI-DEP, AND WHY AN `asm` CAN NEVER RAISE PRIORITY (P31 S68; byte-proven ov_SC06_010/func_8017E764, 438 ins, fable escalation closed 8 → 0) <sub>L31977</sub>
|
||||
- **§3-1.** — DEAD-RESET CSE-BREAKER — the zero-footprint replacement for a §195-I asm re-tie <sub>L31979</sub>
|
||||
- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) <sub>L32309</sub>
|
||||
|
||||
### loops & induction variables (43)
|
||||
### loops & induction variables (45)
|
||||
|
||||
- **§3-T1** — Loop pointer: top-of-body for `addu` induction, not constant-folded `addiu` <sub>L71</sub>
|
||||
- **§34** — The `func_80138ED0` giant crack: gcc-2.7.2's **3-qty sort bug** + the **zero-byte asm allocation toolkit** + the **giv-init fence** (Phase 24 T5; Opus→close=21, Fable5→MATCH ×134) <sub>L2473</sub>
|
||||
@@ -408,6 +410,8 @@
|
||||
- **§357** — ONE STRUCT POINTER, NOT TWO: A SECOND SOURCE VARIABLE BUILDS A THIRD IV (P31 S68; byte-proven ov_SC06_029/func_80181DF8, 335 ins, 330 → 13) <sub>L31589</sub>
|
||||
- **§361** — ★ — A LOOP-TAIL BYTE SIGNATURE THAT NAMES ITS SOURCE SHAPE — AND THE LAW THAT A "SCHEDULING TIE" MAY BE YOUR OWN EARLIER LEVER (P31 S68; byte-proven main/func_800241C0, fable escalation, 19 → 0 in 3 iterations) <sub>L31637</sub>
|
||||
- **§366** — ★★ — `group_case_nodes` MERGES **STACKED CONSECUTIVE** CASE LABELS: GIVE EVERY CASE ITS OWN BODY (P31 S68; ov_SC01_001/func_8017EC28, **first-try MATCH 360/360**, 96/96 relocs audited) <sub>L31757</sub>
|
||||
- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) <sub>L32309</sub>
|
||||
- **§386** — ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MASK, NEVER A DEREF (P31 S69; byte-proven main/func_80020598, 292 ins, escalation 1 → 0) <sub>L32335</sub>
|
||||
|
||||
### structs, block moves & memcpy (84)
|
||||
|
||||
@@ -704,7 +708,7 @@
|
||||
- **§361** — ★ — A LOOP-TAIL BYTE SIGNATURE THAT NAMES ITS SOURCE SHAPE — AND THE LAW THAT A "SCHEDULING TIE" MAY BE YOUR OWN EARLIER LEVER (P31 S68; byte-proven main/func_800241C0, fable escalation, 19 → 0 in 3 iterations) <sub>L31637</sub>
|
||||
- **§367** — RECONCILING A DECL CONFLICT BETWEEN TWO DRAFTS FOR THE SAME TU (P31 S68; main) <sub>L31777</sub>
|
||||
|
||||
### jump tables & switches (54)
|
||||
### jump tables & switches (55)
|
||||
|
||||
- **§8** — rodata island (compiler jump tables) — the `.data→.rodata→.data` sandwich (Phase 7) <sub>L339</sub>
|
||||
- **§8a** — rodata island in a flat OVERLAY — the tail sandwich, per matched jr-function (Phase 26 — PoC PROVEN) <sub>L361</sub>
|
||||
@@ -760,8 +764,9 @@
|
||||
- **§362** — TWO TRAPS WHEN A CARVE MOVES A STUB INTO THE `-O0` TU (P31 S68; byte-proven, 6 fns / 2,547 ins across ov_MAIN_012 / ov_SC02_037 / ov_SC03_107) <sub>L31682</sub>
|
||||
- **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) <sub>L31890</sub>
|
||||
- **§384** — ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE BANK (P31 S69; measured twice, cost one destroyed match) <sub>L32259</sub>
|
||||
- **§387** — ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep the fold (P31 S69; byte-proven main/func_80030F80, 343 ins, escalation 3 → 0) <sub>L32361</sub>
|
||||
|
||||
### optimisation level (-O0/-O2) (21)
|
||||
### optimisation level (-O0/-O2) (22)
|
||||
|
||||
- **§6** — Per-module optimization mixing — the -O0 boot module (Phase 7) <sub>L265</sub>
|
||||
- **Detecting** — the opt level (do this first) <sub>L273</sub>
|
||||
@@ -784,6 +789,7 @@
|
||||
- **Addendum** — Addendum to §261a — the -O0 increment-operator copy also fires on the POINTER LOCAL itself (func_8001212C) <sub>L30473</sub>
|
||||
- **Addendum** — Addendum to §164-XX/§165-06 — at -O0 a bare local IS a memory lvalue, so the increment ope (func_8001212C) <sub>L30477</sub>
|
||||
- **§362** — TWO TRAPS WHEN A CARVE MOVES A STUB INTO THE `-O0` TU (P31 S68; byte-proven, 6 fns / 2,547 ins across ov_MAIN_012 / ov_SC02_037 / ov_SC03_107) <sub>L31682</sub>
|
||||
- **§388** — ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6) <sub>L32384</sub>
|
||||
|
||||
### family propagation & sweeps (116)
|
||||
|
||||
@@ -904,7 +910,7 @@
|
||||
- **§368** — ★★★ — THE **RELOAD-REMAT CONSTANT**: REACH A REGISTER NO PIN CAN REACH (P31 S68; ov_SC03_105/func_80187A30, 339 ins, fable escalation closed 8 → 0 in ONE edit) <sub>L31792</sub>
|
||||
- **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) <sub>L32164</sub>
|
||||
|
||||
### integration / TU plumbing (71)
|
||||
### integration / TU plumbing (72)
|
||||
|
||||
- **§8c** — Splitting a TU means rebuilding its DECLARATION ENVIRONMENT, not moving text (Phase 26 session 6) <sub>L456</sub>
|
||||
- **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) <sub>L502</sub>
|
||||
@@ -977,8 +983,9 @@
|
||||
- **§367** — RECONCILING A DECL CONFLICT BETWEEN TWO DRAFTS FOR THE SAME TU (P31 S68; main) <sub>L31777</sub>
|
||||
- **§376** — ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT THE **TU** (P31 S69; measured 0/28) <sub>L32055</sub>
|
||||
- **§378** — ★★★ — THE **SELF-CALLER CAST**: LET A TU KEEP CALLING THE FUNCTION IT IS ABOUT TO DEFINE (P31 S69; byte-proven ov_SC04_010/func_8017D6CC) <sub>L32114</sub>
|
||||
- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) <sub>L32309</sub>
|
||||
|
||||
### build graph, splat & the harness (186)
|
||||
### build graph, splat & the harness (187)
|
||||
|
||||
- **§4** — Flag/toolchain gotchas <sub>L190</sub>
|
||||
- **Build** — mechanism — per-file opt override (splat resegmentation) <sub>L307</sub>
|
||||
@@ -1166,6 +1173,7 @@
|
||||
- **§358** — (sharpens §333) — AN **UNREFERENCED** FIXED-SIZE AGGREGATE LOCAL IS LOAD-BEARING (P31 S68; same function) <sub>L31598</sub>
|
||||
- **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) <sub>L31890</sub>
|
||||
- **§383** — TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) <sub>L32244</sub>
|
||||
- **§388** — ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6) <sub>L32384</sub>
|
||||
|
||||
### process, measurement & doctrine (127)
|
||||
|
||||
@@ -1297,7 +1305,7 @@
|
||||
- **§376** — ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT THE **TU** (P31 S69; measured 0/28) <sub>L32055</sub>
|
||||
- **§384** — ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE BANK (P31 S69; measured twice, cost one destroyed match) <sub>L32259</sub>
|
||||
|
||||
### (unbucketed — title matched no symptom vocabulary) (307)
|
||||
### (unbucketed — title matched no symptom vocabulary) (308)
|
||||
|
||||
- **§3-How** — to use this <sub>L30</sub>
|
||||
- **§1** — Idiom catalog (asm pattern → C that produces it) <sub>L39</sub>
|
||||
@@ -1606,6 +1614,7 @@
|
||||
- **§377** — THREE HARNESS DEFECTS FOUND IN ONE GATING SESSION, ALL "A CONFIDENT NUMBER ABOUT A SMALLER WORLD" (P31 S69) <sub>L32097</sub>
|
||||
- **§380** — ★★★ — **A SECOND SET OF A PSEUDO DISQUALIFIES IT FROM `move_movables`** (P31 S69; main/func_800215F4, 465 ins, closeness 106 → 59 → 39) <sub>L32195</sub>
|
||||
- **§382** — TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) <sub>L32232</sub>
|
||||
- **§389** — ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLETON (P31 S69; 31 stubs / 4,811 ins recovered, 8 banked same day) <sub>L32409</sub>
|
||||
|
||||
|
||||
## All sections, in order
|
||||
@@ -2649,6 +2658,11 @@
|
||||
- **§382** — TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) <sub>L32232</sub>
|
||||
- **§383** — TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) <sub>L32244</sub>
|
||||
- **§384** — ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE BANK (P31 S69; measured twice, cost one destroyed match) <sub>L32259</sub>
|
||||
- **§385** — ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0) <sub>L32309</sub>
|
||||
- **§386** — ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MASK, NEVER A DEREF (P31 S69; byte-proven main/func_80020598, 292 ins, escalation 1 → 0) <sub>L32335</sub>
|
||||
- **§387** — ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep the fold (P31 S69; byte-proven main/func_80030F80, 343 ins, escalation 3 → 0) <sub>L32361</sub>
|
||||
- **§388** — ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6) <sub>L32384</sub>
|
||||
- **§389** — ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLETON (P31 S69; 31 stubs / 4,811 ins recovered, 8 banked same day) <sub>L32409</sub>
|
||||
|
||||
|
||||
---
|
||||
@@ -3700,3 +3714,8 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: `
|
||||
| L32232 | §382 | TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) |
|
||||
| L32244 | §383 | TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) |
|
||||
| L32259 | §384 | ★★★ — A CARVE-CONFIG BANK IS RED UNTIL YOU RE-EXTRACT, AND THAT LOOKS EXACTLY LIKE A FALSE |
|
||||
| L32309 | §385 | ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init prehead |
|
||||
| L32335 | §386 | ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MA |
|
||||
| L32361 | §387 | ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep th |
|
||||
| L32384 | §388 | ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 |
|
||||
| L32409 | §389 | ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLE |
|
||||
|
||||
@@ -30,11 +30,39 @@ tool existed — and that were invisible until an idiom taught us to look.
|
||||
**2. The structural tooling, before the first crack.**
|
||||
The families/twins/dedup layer is what converts one crack into N banks. In BFM this arrived late and
|
||||
retroactively harvested thousands of instructions. Port it first:
|
||||
`corpus` (the coverage oracle) · `seed_ref` / twin join on signature hashes · `family_remap` /
|
||||
`corpus` (the coverage oracle) · `seed_ref` (exact-hash twin join **AND its `--near` edit-distance
|
||||
band — see 2b**) · `family_remap` /
|
||||
`family_sweep` · `dedup_propagate` (position-locked overlay sharing) · the `-O0`/opt-level carve
|
||||
chain (`o0_detect`, `o0_subsplit`, `o0_boundary`) · `wall_sweep` (toolchain walls) · the draw
|
||||
filter · the byte-gate + clean-fleet verifier.
|
||||
|
||||
**2b. THE SIMILARITY JOIN MUST BE A BAND, NOT A POINT (P31 S69 — port this, it is cheap and it
|
||||
compounds).** A twin oracle keyed on an exact signature hash answers only *"is there a byte-identical
|
||||
copy?"*. That is the wrong question for a frontier. The right one is *"is there anything CLOSE?"*, and
|
||||
the difference is not marginal:
|
||||
|
||||
| tier | reachable open stubs with a banked match |
|
||||
|---|---|
|
||||
| exact hash (d=0) — where this project sat for 60+ sessions | **22 of 352 (6%)** |
|
||||
| edit-distance band to d<=25 (`seed_ref --near`) | **75 of 352 (21%)** |
|
||||
|
||||
**A 3.4x widening, found in one agent-run, on a corpus we thought was fully mined.** 31 of the new
|
||||
rows were PURE reloc-only twins of already-banked bodies — free work that had been sitting invisible;
|
||||
8 banked the same day at ~0 agent tokens. The root cause was a normalizer that under-matched by
|
||||
design (§389): safe for dedup, silently lossy as a frontier join.
|
||||
|
||||
For a NEW decomp this matters *more* than it did here, because the band pays from the very first
|
||||
banks: every function you crack immediately becomes a potential exemplar for everything within a few
|
||||
instructions of it, and you never accumulate the invisible-singleton debt this project spent a
|
||||
session recovering. Build the near tier at the same time as the exact tier — not sixty sessions
|
||||
later. Concretely: normalize relocations out of the instruction stream, prefilter soundly on
|
||||
length/opcode-histogram so no true pair can be lost, then edit-distance the survivors; assert the
|
||||
population (R32), cross-check that the band reproduces every exact-hash pair (R34), and control
|
||||
against random pairs for the base rate (R39: 1.17% here).
|
||||
|
||||
**And audit every hash you own for BOTH questions.** Dedup wants under-matching; a frontier join
|
||||
wants over-matching. One hash cannot serve both error directions, and the failure is silent.
|
||||
|
||||
**3. The differential-oracle harness (accelerators #15) — the one that works at 0%.**
|
||||
Two independent paths per question, disagreement fails loudly, on a schedule.
|
||||
|
||||
|
||||
@@ -32305,3 +32305,144 @@ This is the R22 corollary aimed the other way. R22 says a *reverted* config need
|
||||
(a build over stale extract state) was broken, and I attributed the failure to the subject, reverted
|
||||
a legitimate 96-line match, and wrote a checkpoint calling it a false bank. The give-away I ignored:
|
||||
the commit's own diffstat showed `config/overlays.mk` and a splat yaml right next to the `.c`.
|
||||
|
||||
## §385 ★★★ — THE **SCHED2 PRIORITY-DONOR ASM**: closing the "hoisted-invariant vs IV-init preheader swap" class (P31 S69; byte-proven main/func_80038A58, 347 ins, fable escalation 2 → 0)
|
||||
|
||||
**Closes cookbook §5's open class.** Symptom: two independent one-instruction inits after a mult (or
|
||||
any latency chain) emit in swapped order, and the swap is invariant under source order, register
|
||||
pins, and zero-byte barriers.
|
||||
|
||||
**Root cause, read out of the pinned compiler** (`sched.c` `priority()`:1425 + `rank_for_schedule()`:2385):
|
||||
sched2 schedules BACKWARD. The init that writes a register the mult READS carries an anti-dependence
|
||||
that inherits the mult's priority (anti cost clamps to 1, so `+cost-1 = +0`), while the dependence-free
|
||||
init stays at priority 1. **Priority beats the LUID tie-break**, so no C-level reordering can flip it.
|
||||
|
||||
**The fix — donate priority to the loser** via a NON-volatile single-instruction asm with a dead
|
||||
extra input reading the winner's destination:
|
||||
|
||||
```c
|
||||
c2 = 0;
|
||||
__asm__("addiu %0,%1,18" : "=r"(ptr) : "r"(base), "r"(c2));
|
||||
```
|
||||
|
||||
The true dep on the `c2` init donates priority 2 AND delays the asm's backward release until the
|
||||
donor is placed — target order, identical bytes.
|
||||
|
||||
**Two traps.** A **volatile** asm cannot be used: volatile is a full scheduling barrier, so it
|
||||
inherits the whole chain's priority and glues itself behind the `mflo`. And a bare ghost asm whose
|
||||
output is immediately overwritten gets deleted.
|
||||
|
||||
## §386 ★★★ — A BYTE LOAD ON THE **BIV** BASE WAS BORN IN THE COMBINE PASS: SPELL IT AS A SHIFT-MASK, NEVER A DEREF (P31 S69; byte-proven main/func_80020598, 292 ins, escalation 1 → 0)
|
||||
|
||||
**Diff tell:** a one-row `reg+imm` residual where yours reads offset-from-**giv** and the target reads
|
||||
offset-from-**biv** *at the same address* — and the twin expression in another arm IS giv-based in the
|
||||
target.
|
||||
|
||||
**Why no deref spelling works.** A byte deref is a `DEST_ADDR` giv, and `loop.c`'s `combine_givs`
|
||||
folds it onto the reduced register **unconditionally**: identity or `express_from`, then
|
||||
`memory_address_p` and equal MIPS `ADDRESS_COST`. There is no per-instance guard, so no temp, no
|
||||
ordering and no spelling can split two identical `(mult, add)` givs.
|
||||
|
||||
**The lever — defer the load's birth past `loop.c`:**
|
||||
|
||||
```c
|
||||
b1 = (w >> 8) & 0xFF; /* NOT ((u8 *)p)[1] */
|
||||
```
|
||||
|
||||
Post-loop `combine` then narrows `zero_extend(lshiftrt(mem))` into an `lbu` at base+k off the
|
||||
ORIGINAL register.
|
||||
|
||||
**Two corollaries.** (1) Temp-pointer escapes are impossible: `cse1`'s `find_best_addr` tie-break at
|
||||
equal `ADDRESS_COST` prefers the MORE complex address (it frees a register) and folds any same-ebb
|
||||
temp back into `(plus base k)` before loop — and anything opaque enough to survive cse1 is equally
|
||||
opaque to cse2/combine. (2) The narrowed spelling changes spill demand, so expect the §333/§334
|
||||
dead-aggregate pad to need retuning (here `pad[6]` → `pad[4]` at frame 0x88).
|
||||
|
||||
## §387 ★★ — **SPLIT-FOLD DISPATCH CLOBBER**: one switch case needs a reload, another must keep the fold (P31 S69; byte-proven main/func_80030F80, 343 ins, escalation 3 → 0)
|
||||
|
||||
When one case needs a cse-forced RELOAD of the switch-index load while another must keep FOLDING that
|
||||
same value into a call argument:
|
||||
|
||||
* **Do NOT put the invalidator in the case body** — `reorg.c`'s `stop_search_p` halts
|
||||
`fill_slots_from_thread` at ANY asm insn (`asm_noperands >= 0`), so branch-delay-slot hoists into
|
||||
that thread die (+1 length).
|
||||
* **Do NOT volatile-cast the load** — `loop.c:339` `init_recog_no_volatile` fails `recog` on every
|
||||
`DEST_ADDR` giv rewrite of a volatile mem, so its address stays biv-based forever.
|
||||
|
||||
**Instead:** name the index in a local; load the dispatch byte into an `s32` local (a `u8` costs an
|
||||
`andi`); place a **non-volatile** `__asm__("" ::: "memory")` AFTER that `lbu` and BEFORE the switch —
|
||||
zero bytes, invalidates cse's memory table on every path, and sits outside every case's delay-slot
|
||||
thread. Never between a load and its first use: it eats the final-pass load-delay nop. Fold-cases
|
||||
then pass the named local; reload-cases re-read memory.
|
||||
|
||||
**Side-effect law, and it is general:** adding or removing pseudos flips razor-thin global-alloc ties
|
||||
among pre-loop invariant pointers — which one loses its callee-saved register and gets reload-
|
||||
rematerialised as inline `lui`/`addiu`. **This is INVISIBLE under `match_one`'s HI16/LO16 mask** and
|
||||
surfaces as a bogus "schedule" diff at the remat site. Audit with `objdump -r`, and flip it back by
|
||||
swapping the two invariants' init order.
|
||||
|
||||
## §388 ★★★ — THE **-O0 COLOURING ORACLE**: simulate `stupid.c` instead of grinding spellings (P31 S69; main/func_80011380 proved a C-level WALL at 6)
|
||||
|
||||
At `-O0` gcc-2.7.2 allocates with **`stupid.c`**, not local/global-alloc. It is simple enough to
|
||||
simulate exactly, so predict the colours instead of guessing at spellings:
|
||||
|
||||
* every insn **including NOTES** gets a sequential `suid`;
|
||||
* a pseudo set at suid `s` dies at `max(last_use, s+2)` and occupies `[born, dead-1]`;
|
||||
* pseudos are allocated **longest-first**, then **fewer-refs-first**, then lowest regno, first-fit
|
||||
`$v0,$v1,$a0,$a1,…`;
|
||||
* a `register` declaration emits a zero-byte head-(use) insn making that var ONE convex interval.
|
||||
|
||||
**Two consequences you can diagnose by inspection.** (1) An `-O0` `copy;sll` multiply chain
|
||||
2-colours ping-pong *because* the `+2` rule makes each link conflict with the next — so a NOTE inside
|
||||
the chain (any statement-expression's `BLOCK_BEG`/`BLOCK_END`) lapses the slack, one copy becomes a
|
||||
same-register move, and `final.c` deletes it: **a length −1 next to a `({...})` is THIS, not
|
||||
scheduling.** (2) A `register` decl recolours the whole chain via its head-use interval, so
|
||||
exact-length REGALLOC rotations in `-O0` code are decl-lifetime artifacts. `cc1 -dr` suid arithmetic
|
||||
predicts the colours before you compile.
|
||||
|
||||
**Also from the same run — the SYMBOL-ADDEND SHIELD.** In a pointer index, `(s32)(idx + K)`: the
|
||||
same-mode cast hides the `PLUS` from `pointer_int_sum`'s distributive rewrite, yet `EXPAND_SUM` still
|
||||
absorbs `K` into the relocation, emitting `la SYM+K*size` with the scale applied to `idx` alone. Use
|
||||
it when the target shows a symbol+constant base feeding a scaled index — no plain spelling
|
||||
reproduces it, because the frontend otherwise distributes `K` into the pointer before scaling.
|
||||
|
||||
## §389 ★★★ — `h_norm` IS BLIND TO INDEXED-GLOBAL RELOCS, SO FREE WORK BECOMES AN INVISIBLE SINGLETON (P31 S69; 31 stubs / 4,811 ins recovered, 8 banked same day)
|
||||
|
||||
**The hole.** `sig_image.norm_stream` tracks a pending `lui`-hi so it can neutralise `%hi`/`%lo`
|
||||
pairs — but it DROPS that pending hi the moment an R-type intervenes. The indexed-global triad is
|
||||
exactly that shape:
|
||||
|
||||
```
|
||||
lui $at, %hi(arr)
|
||||
addu $at, $at, idx <-- R-type; the pending hi is dropped here
|
||||
lw r, %lo(arr)($at) <-- %lo survives into the hash
|
||||
```
|
||||
|
||||
So two per-overlay copies of ONE function that differ only in a data symbol's ADDRESS hash to
|
||||
**different** `h_norm`. They are byte-identical modulo relocation and they are invisible to every
|
||||
hash-keyed consumer at once: `seed_ref` (d=0 tier), `twin_sweep`, `dup_report`,
|
||||
`config/dedup.us.yaml`, and the family maps' structural tier.
|
||||
|
||||
**Measured consequence (2026-09-01).** Beyond the 22 stubs with a d=0 hash twin, **31 more reachable
|
||||
open stubs (4,811 ins) were PURE twins of already-banked bodies at instruction edit distance 1–5** —
|
||||
`family_remap.classify_member` seconds all 31 as PURE. Ten were clean of jtbl carve blockers;
|
||||
remapping them and gating banked **8**, at ~0 agent tokens. One exemplar (`ov_SC06_033:0x80185f6c`,
|
||||
94 ins) served **five** open copies; `ov_MAIN_012:0x8016ab6c` (188 ins) serves five more.
|
||||
|
||||
**The fix is a NEW TIER, NOT A NEW NORMALIZER.** Do **not** change `h_norm`: every stored map,
|
||||
ledger and calibration in the project keys on it, and a re-hash invalidates all of them. Instead read
|
||||
*through* the hole with an edit-distance tier over reloc-normalized streams —
|
||||
`tools/seed_ref.py --near [--max-d N]`, which scans all 213 binaries, reproduces all 22 hash twins as
|
||||
an R34 cross-check on every run, and ships R39 controls (positive 200/200 at d=0; random-pair base
|
||||
rate 1.17%).
|
||||
|
||||
**THE LAW, and it generalises past this project.** *A conservative normalizer is SAFE for a DEDUP
|
||||
claim and UNSAFE as a FRONTIER JOIN.* Dedup asks "are these certainly the same?" — under-matching
|
||||
there is harmless. A frontier join asks "is there anything close to this?" — and there, every missed
|
||||
match silently converts free mechanical work into an apparent singleton that a future session will
|
||||
pay an agent to re-derive from scratch. Audit any hash you use for BOTH questions; they want opposite
|
||||
error directions.
|
||||
|
||||
**Diff tell:** an open stub your card calls "no banked twin — derive from the .s", whose body is a
|
||||
per-location copy of engine code that exists in a sibling overlay. Run the near tier before believing
|
||||
a singleton verdict.
|
||||
|
||||
@@ -100,6 +100,18 @@ target**, joined on the corpus signature hashes (`tools/seed_ref.py`).
|
||||
> neither tool could see them. A `mechanical_remap_refused` flag now tells the agent: copy the BODY,
|
||||
> expect a declaration blocker.
|
||||
|
||||
### 2a. RUN `seed_ref --near` BEFORE BELIEVING ANY "NO TWIN" VERDICT (S69)
|
||||
|
||||
```
|
||||
.venv/bin/python tools/seed_ref.py --near --max-d 25 --json .run/<wave>/near.json
|
||||
```
|
||||
|
||||
The exact-hash tier finds 22 of 352 reachable open stubs; the near band finds **75 of 352 (21%)** —
|
||||
3.4x. `h_norm` is blind to indexed-global relocs (§389), so true reloc-only twins of ALREADY-BANKED
|
||||
bodies hash differently and read as singletons. A `RELOC-ONLY` row is mechanical: `family_remap` the
|
||||
banked exemplar onto the open address, then gate — **8 of 10 banked at ~0 agent tokens on first use**,
|
||||
one 94-ins exemplar serving five open copies. Never send a RELOC-ONLY row to a drafting agent.
|
||||
|
||||
### 2b. RUN `neighbor_ref` FOR EVERY CARD — the biggest measured cost lever in the wave
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user