From 9fc27960e9f18f13f4cffa69fe1bc1eb182b8ea8 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Tue, 1 Sep 2026 11:02:52 -0600 Subject: [PATCH] =?UTF-8?q?feat(integration):=20teach=20the=20rest=20of=20?= =?UTF-8?q?the=20toolkit=20about=20=C2=A7378=20(self-caller=20cast)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The lever existed but nothing downstream applied it. Proof it mattered: a wave agent this session diagnosed its own blocker as "§378 THE SELF-CALLER CAST, a TU-level fix (cast_self_callers.py) that requires editing src/, which I'm not permitted to touch" — the knowledge propagated, the automation did not. * recover_integration.py: NEW "self-cast" stage (tier=binary), so the driver can run the whole chain as --stages arity,self-cast. The docstring states WHY the order is not arbitrary: self-cast answers the error that "arity" CREATES. * residual_rules_b.py: both decl-conflict tiers now prescribe the full chain instead of "route to integration / budget for banking", and NOCOMPILE-UNDECLARED-FIXED now says outright NOT to gate the autodecl arm (it is a second conflicting declaration in the real TU). * wave-playbook §4b: replaced the stale two-step recipe with the three-step chain, the one-driver form, the callee variant, and the MANDATORY --undo-journal. * SETUP.md: full inventory row (R21) — it had zero mentions. Not wired, deliberately: gate_stage's ladder rewrites DRAFTS via _xform, while this edits the TU; a src-side edit inside the automatic gate needs revert-on-failure, which recover_integration already owns. Still open: a draft_prechecks rule to catch the self-decl conflict statically, before a build is spent. The new stage's plumbing is verified (CLI + candidate selection); its functional end-to-end run is NOT — gate12 held the tree. --- docs/SETUP.md | 1 + docs/cookbook-index.md | 32 +++++++++--- docs/matching-cookbook.md | 95 ++++++++++++++++++++++++++++++++++++ docs/wave-playbook.md | 23 ++++++++- tools/recover_integration.py | 27 +++++++++- tools/residual_rules_b.py | 16 ++++-- 6 files changed, 180 insertions(+), 14 deletions(-) diff --git a/docs/SETUP.md b/docs/SETUP.md index 0cf2ae26e..b5b6a66f9 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -779,6 +779,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | **S68 tooling, round 2 — retrieval, triage, walls, liveness** (P31 S68) | **`tools/neighbor_ref.py` (NEW)** — for an OPEN stub, the already-MATCHED functions worth READING as worked examples, ranked SAME-TU first, then same binary, then shape (li-normalised skeleton / call-sequence hash / reloc-kind sequence / CFG counts / opcode cosine, all precomputed in `.run/feat.*.jsonl`), with a hard penalty for opt-level mismatch (§116). Surfaces the neighbour's HEADER COMMENT — the payload agents actually consume. Answers the question `seed_ref` cannot: seed_ref needs a hash-identical twin, this finds a near-twin. **Run it for every card.** Measured motivation: S68's cheapest large matches all came from a neighbour (555 ins/72k, 657 ins/122k first compile, 753 ins/177k) while neighbour-less main fns ran 200-350k for ~80 ins. **`tools/wall_sweep.py` (NEW)** — enumerates the §332 delay-slot macro walls (10 fns / 1,027 ins); `--emit-exclude` feeds `draw_waves --exclude`. **Run before every draw AND before every escalation** — an escalation cannot beat the toolchain. **`tools/residual_rules.py` + `residual_rules_b.py` (NEW, experimental)** — residual→cookbook-lever classifiers; `_b` is the stronger (113/113 classified, 63% certain/high). NOT yet wired into the pipeline: see `docs/next-session-triage-ladder.md`. **`tools/lane_inflight.py` (NEW)** — the RECORDED liveness ledger (`add` on launch, `done` on verdict); `list` exits non-zero while any agent is live and IS the guard. **`tools/r22_verify.sh` (NEW)** — the clean-fleet verify, EXCLUSIVE by construction: it refuses while `lane_inflight` shows live agents, because `make clean` deletes `asm/` and drafting agents READ it. Clears `.run/R22_DEBT` only on a genuinely green run. | | | **S68 tooling — the gater lane + the -O0 route** (P31 S68) | **`tools/gater_lane.py` (NEW)** — the continuous gater: drains a wave's finished drafts into `parallel_gate`, grouped by binary, `--r22` by default. Ledger AND verdicts keyed **`binary:fn:arm`** (R48 — and an escalation ALWAYS has a prior verdict, so arm-keying is what stops an in-flight fable draft riding the opus one). `--extra BINARY:PATH` for non-wave drafts, `--skip-binary` for lanes that may be writing, and **main is routed IN-TREE via `harvest_verify`** because `parallel_gate`'s worktree cannot stage main's psyq_integrate link inputs. **`tools/workflows/escalate_fable.js` (NEW)** — warm-started escalation: passes the prior draft + its measured closeness + its ruled-out levers, and demands a reusable `new_idiom`. **`tools/o0_boundary.py` (NEW)** — the stranded-boundary -O0 sweep (141 binaries / 288 boundaries / 0 candidates: the class is EXHAUSTED, and that null is negative-controlled). | | | **S69 tooling — the TRIAGE LADDER** (P31 S69) | **`tools/triage_ladder.py` (NEW)** — the zero-token pass that answers *"does this target need an agent at all?"* before one is spent. **The PRE/POST split is the point:** `--pre ` runs the TARGET-SIDE tiers only (BANKED · WALL-332 · PARKED) — no draft, no build, milliseconds — and writes `/triage.json` + `triage_exclude.txt`; `--post` runs the full `residual_rules_b` residual routing, which needs a draft and runs `match_one`. `--escalate B:FN` exits 2 on a walled/banked target (the check S68 was missing when it escalated a §332 wall at closeness 8). `--acceptance` is the R39/R32 harness: false-skip over EVERY open stub, recall over sampled matched fns, and a wall negative control over already-banked code — all pure filesystem work, so it runs in seconds over the whole corpus. **It REFUSES on a non-quiescent tree** (`pgrep -af` rows for gate/build lanes + `lane_inflight`): a live gate makes the stub oracle transiently wrong in both directions (§377). Wired into `wave_args.py` (drops walled/parked targets at draw time, reusing `pre_classify` — one implementation, R33) and into `tools/workflows/escalate_fable.js`, which now REFUSES any target that does not carry `triage:'DRAFT'`. **Routing correction (§376):** the `INTEG-STANDALONE-MATCH` / `NOCOMPILE-UNDECLARED-*` tiers are GATE-FIRST candidates, never free banks — S69 gated that class **0/28** raw; the route is `fix_arity_callers --any-proto` then the gate. | +| | **S69 tooling — the §378 SELF-CALLER CAST** (P31 S69) | **`tools/cast_self_callers.py` (NEW)** — the mirror of `cast_call_sites.py`. That one fixes the DRAFT calling a conflicting CALLEE; this one fixes the TU's OWN already-banked code calling the function the draft is about to DEFINE, which nothing handled and which is the terminal blocker of the §376 pile. **Never run it alone** — it answers the error that `fix_arity_callers --any-proto` CREATES: no-protoing the conflicting decl makes the draft's definition the prototype in scope, so the TU's own call fails anew with `too few arguments`. Order is `arity -> self-cast -> [--sync-decls] -> gate`. Byte-neutral because gcc-2.7.2 folds a cast of a known function symbol back to a direct `jal`. `--sync-decls` covers the narrow-param case a no-proto decl CANNOT legally reach (C89 requires promotion-stable parameter types when one declaration has no prototype, so `void f(s16)` is illegal against `extern void f();` — which is exactly why `fix_arity_callers` skips it as 'narrow-param'); safe only once the call sites are cast, because a declaration then emits no code. It REFUSES a function whose return type it cannot read off the draft (R43). **Journals every edit, and `--undo-journal --keep ` after the gate is MANDATORY** — a leftover cast made `ov_SC07_000` stop compiling and every later gate verdict on it measured a broken baseline. Wired into `recover_integration.py --stages arity,self-cast` (tier `binary`), prescribed by `residual_rules_b`'s decl-conflict tiers, and in the playbook §4b. Measured: **8 banked of 28** including `main/func_80036D58` at zero agent tokens; generalises to the callee the diagnostic NAMES (banked `main/func_80021D38` that way). | | | **S68 fixes — six instances of the overlay-layout assumption** (cookbook **§363**) | `dedup_propagate` could not even IMPORT (`os.` at module level in the one module that imports `os as _os`). `seed_ref` offered main's LINKED-subseg DEAD TEXT as bankable twins (43 of 82 hits — a draft there gates GREEN while wrong); now refuses and COUNTS the refusal. `parallel_gate.stage_generated` hard-coded `build//.ld`; now asks the Makefile for `_LD_SCRIPT`/`_UNDEF_SYMS`/`_UNDEF_FUNCS` and REFUSES when absent. `rtu_match` gained **`--tu`** (+ `blocker_probe` passes `stub.path` and `stub.asm_dir`) — it reconstructed `src//.c`, which is the overlay layout; main's sources are LOOSE FILES in `src/`. `gate_stage` no longer synthesises `--out`/`--good-sha` — for main those resolved to a nonexistent path and then **ov_SC01_077's SHA** via `DEF_SHA`. **`psyq_integrate`**: the `*_externals.ld` map is now MONOTONIC — it was re-derived against the CURRENT `.ld`, so `firstfile = 0x80061FA8;` was DROPPED on every incremental relink and main was 2 bytes red before any draft was spliced (**the true identity of the 2026-08-15 'main link defect'**). | | | **S68 — the module-binary -O0 carve route** (cookbook **§371**) | `jr_isolate_all` + `overlay_src_split` + the **Makefile -O0 glob widened to `src/md_*/md_*_o0?.c`** open carving for the single-object `md_*` binaries. Three stacked causes behind one `unaddressable content` message (interior-YAML-comment symbol-list truncation; a trailing verbatim-asm chunk with no region; bare tag forward decls), then the **spimdisasm rodata-migration trap**: migrated rodata follows its function ONLY within the same subseg, so a carve silently drops it and `INCLUDE_RODATA` cannot bring it back — rename the `.rodata` subseg to the object its emitters moved to. **The Makefile hunk MUST be committed with the carve** or a fresh clone loses -O0 on the region and every draft banked there mystery-fails. | | | `tools/recover_rejects.py` | **(P31 S59)** Free recovery of PRE-GATE rejects, wired into the maintenance lane. Two paths exist for a draft that does not bank and only one was recorded: a gate failure gets a backlog row (closeness/class/best draft), while a draft the reloc pre-filter drops reached nothing — **569 of 1,261 drafts over eight waves, 45%**. Of the `MISMATCH?` rejects, **13% carry `shape: MATCH`** — right body, wrong symbol names, i.e. the §171 stale-seed class `aprop_symfix` rebases deterministically. Reads `.run/reloc_rejects.jsonl` (written by `ox_campaign.reloc_filter`), keeps shape-MATCH rows that are STILL open stubs, runs `aprop_symfix --fix`, and STAGES the rebased bodies into `.run/sweep_maint//` for the lane's existing free gate. It never substitutes, gates or commits — a bad recovery can waste a build, never a bank. Tried-once is remembered in `.run/recover_rejects_seen.json`. Zero model tokens. | diff --git a/docs/cookbook-index.md b/docs/cookbook-index.md index a3b1a8527..bb7c16c47 100644 --- a/docs/cookbook-index.md +++ b/docs/cookbook-index.md @@ -2,7 +2,7 @@ > **Generated by `tools/cookbook_index.py` — do not hand-edit** (R33). Regenerate after adding a cookbook section. > -> `docs/matching-cookbook.md` is ~716 KB / 1033 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. +> `docs/matching-cookbook.md` is ~716 KB / 1038 sections. Grepping it blind is how three P30 wave-1 agents each "discovered" an idiom that was already written down. **Start here, then read the section.** A section appears under every symptom it addresses. **How to use:** name what you SEE in the diff (a stolen delay slot, an extra `la`, a swapped register pair, a `conflicting types` error), find that symptom below, read those sections first. If nothing fits, THEN grind — and add a section when you win. @@ -101,7 +101,7 @@ - **§346** — `c ? X : -X` TAKES expand_expr's COND_EXPR **SINGLETON** PATH (copy, then negate IN PLACE) — AN if/else STATEMENT GIVES THE TWO-ARM FORM (P31 S67; byte-proven ov_SC03_102/func_80180C38, closed the last instruction) L31322 - **§370** — ★★ — A **HARD BOUND** FROM sched.c, AND THE reorg SLOT-STEAL DIAGNOSTIC (P31 S68; main/func_8001BC6C, 69 ins, NOT closed — 33 → 28 over ~45 measured compiles) L31838 -### instruction scheduling (81) +### instruction scheduling (83) - **§3-T2** — Source statement order drives instruction scheduling L78 - **§3** — When a diff is pure scheduling → decomp-permuter (harness built, Phase 6) L107 @@ -184,6 +184,8 @@ - **§373** — ★★★ — THE **DEAD-RESET CSE-BREAKER**, THE PIN THAT BREAKS A sched2 ANTI-DEP, AND WHY AN `asm` CAN NEVER RAISE PRIORITY (P31 S68; byte-proven ov_SC06_010/func_8017E764, 438 ins, fable escalation closed 8 → 0) L31977 - **§3-2.** — A REGISTER PIN THAT DELETES A sched2 ANTI-DEPENDENCE L31997 - **§3-3.** — HARD FACT FOR THE SCHEDULING MAP — an `asm` ALWAYS has priority 1 L32010 +- **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) L32164 +- **§381** — THE `insn_count` HOIST THRESHOLD IS A DIAL YOU CAN READ WITH `cc1 -dL` (P31 S69; four independent uses in one wave) L32216 ### register allocation & pins (127) @@ -407,7 +409,7 @@ - **§361** — ★ — A LOOP-TAIL BYTE SIGNATURE THAT NAMES ITS SOURCE SHAPE — AND THE LAW THAT A "SCHEDULING TIE" MAY BE YOUR OWN EARLIER LEVER (P31 S68; byte-proven main/func_800241C0, fable escalation, 19 → 0 in 3 iterations) L31637 - **§366** — ★★ — `group_case_nodes` MERGES **STACKED CONSECUTIVE** CASE LABELS: GIVE EVERY CASE ITS OWN BODY (P31 S68; ov_SC01_001/func_8017EC28, **first-try MATCH 360/360**, 96/96 relocs audited) L31757 -### structs, block moves & memcpy (83) +### structs, block moves & memcpy (84) - **§3-T2** — Source statement order drives instruction scheduling L78 - **§5** — Known hard-residual classes (instruction-identical, one byte-exact blocker) L199 @@ -492,8 +494,9 @@ - **§351** — `/s` (MEM_IN_STRUCT_P) IS A DIAL YOU CHOOSE PER ACCESS: A COMPONENT_REF GRANTS IT AND LETS cse KEEP AN INDEX ACROSS THE STORE; A PLAIN CAST DENIES IT (P31 S67; ov_SC01_000/func_8017DD04, 186 -> 5) L31456 - **§357** — ONE STRUCT POINTER, NOT TWO: A SECOND SOURCE VARIABLE BUILDS A THIRD IV (P31 S68; byte-proven ov_SC06_029/func_80181DF8, 335 ins, 330 → 13) L31589 - **§364** — ★ — THE libgpu `P_TAG` BITFIELD SPELLING IS **OPT-LEVEL DEPENDENT** (P31 S68; two functions, opposite verdicts, same session) L31734 +- **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) L32164 -### types, signedness & load/store width (91) +### types, signedness & load/store width (92) - **§3-I1** — Unsigned range check: `(x - lo) < (hi-lo)` → `addiu`+`sltiu` L41 - **§3-I2** — Byte mask forces `andi` even after `lbu` L47 @@ -586,6 +589,7 @@ - **§345** — A VOLATILE **STORE** EVICTS THE MEM FROM cse AND KEEPS `sh`; A VOLATILE **LOAD** BLOCKS combine AND DEGRADES `lh` INTO `lhu+sll+sra` (P31 S67; byte-proven ov_SC01_084/func_80181A7C) L31312 - **§351** — `/s` (MEM_IN_STRUCT_P) IS A DIAL YOU CHOOSE PER ACCESS: A COMPONENT_REF GRANTS IT AND LETS cse KEEP AN INDEX ACROSS THE STORE; A PLAIN CAST DENIES IT (P31 S67; ov_SC01_000/func_8017DD04, 186 -> 5) L31456 - **§378** — ★★★ — THE **SELF-CALLER CAST**: LET A TU KEEP CALLING THE FUNCTION IT IS ABOUT TO DEFINE (P31 S69; byte-proven ov_SC04_010/func_8017D6CC) L32114 +- **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) L32164 ### declarations, prototypes & K&R (110) @@ -780,7 +784,7 @@ - **Addendum** — Addendum to §164-XX/§165-06 — at -O0 a bare local IS a memory lvalue, so the increment ope (func_8001212C) L30477 - **§362** — TWO TRAPS WHEN A CARVE MOVES A STUB INTO THE `-O0` TU (P31 S68; byte-proven, 6 fns / 2,547 ins across ov_MAIN_012 / ov_SC02_037 / ov_SC03_107) L31682 -### family propagation & sweeps (115) +### family propagation & sweeps (116) - **§8d** — Templating a body INTO a TU must not CHANGE its declaration environment — demote the carried data externs (Phase 26 session 8, byte-proven on `func_8015AE2C` ×133) L502 - **§11** — Cross-binary dedup & code-sharing (Phase 11 — "one match unlocks many") L927 @@ -897,6 +901,7 @@ - **§341** — AN HImode STORE TEMP REWEIGHTS A sched2 TIE-BREAK THAT NO STATEMENT ORDER CAN REACH (P31 S67; byte-proven ov_SC03_006/func_801823B8, last 4 ins) L31243 - **§355** — A REMAPPED SIBLING'S **SOURCE BIAS IS NOT ITS EMITTED BIAS** — DO NOT HAND-SHIFT OFFSETS TO MATCH THE ASM (P31 S68; byte-proven ov_SC07_007/func_8013DD68, 187/187 in 2 iterations) L31560 - **§368** — ★★★ — THE **RELOAD-REMAT CONSTANT**: REACH A REGISTER NO PIN CAN REACH (P31 S68; ov_SC03_105/func_80187A30, 339 ins, fable escalation closed 8 → 0 in ONE edit) L31792 +- **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) L32164 ### integration / TU plumbing (71) @@ -972,7 +977,7 @@ - **§376** — ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT THE **TU** (P31 S69; measured 0/28) L32055 - **§378** — ★★★ — THE **SELF-CALLER CAST**: LET A TU KEEP CALLING THE FUNCTION IT IS ABOUT TO DEFINE (P31 S69; byte-proven ov_SC04_010/func_8017D6CC) L32114 -### build graph, splat & the harness (185) +### build graph, splat & the harness (186) - **§4** — Flag/toolchain gotchas L190 - **Build** — mechanism — per-file opt override (splat resegmentation) L307 @@ -1159,6 +1164,7 @@ - **§357** — ONE STRUCT POINTER, NOT TWO: A SECOND SOURCE VARIABLE BUILDS A THIRD IV (P31 S68; byte-proven ov_SC06_029/func_80181DF8, 335 ins, 330 → 13) L31589 - **§358** — (sharpens §333) — AN **UNREFERENCED** FIXED-SIZE AGGREGATE LOCAL IS LOAD-BEARING (P31 S68; same function) L31598 - **§371** — ★★ — CARVING A **SINGLE-OBJECT MODULE BINARY**, AND THE spimdisasm RODATA-MIGRATION TRAP THAT FOLLOWS (P31 S68; byte-proven md_MAIN_003, func_800D0D6C 345 ins) L31890 +- **§383** — TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) L32244 ### process, measurement & doctrine (126) @@ -1289,7 +1295,7 @@ - **§370** — ★★ — A **HARD BOUND** FROM sched.c, AND THE reorg SLOT-STEAL DIAGNOSTIC (P31 S68; main/func_8001BC6C, 69 ins, NOT closed — 33 → 28 over ~45 measured compiles) L31838 - **§376** — ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT THE **TU** (P31 S69; measured 0/28) L32055 -### (unbucketed — title matched no symptom vocabulary) (305) +### (unbucketed — title matched no symptom vocabulary) (307) - **§3-How** — to use this L30 - **§1** — Idiom catalog (asm pattern → C that produces it) L39 @@ -1596,6 +1602,8 @@ - **§372** — ★★★ — THE **COPY-CAPTURE PAIR**, AND THE ONE ZERO-BYTE EDIT THAT DEFEATS BOTH (P31 S68; byte-proven main/func_8003491C, 78 ins, fable escalation closed 5 → 0) L31939 - **CROSS-REFERENCE** — TO §370 — checked and found INAPPLICABLE here, which is the point L32018 - **§377** — THREE HARNESS DEFECTS FOUND IN ONE GATING SESSION, ALL "A CONFIDENT NUMBER ABOUT A SMALLER WORLD" (P31 S69) L32097 +- **§380** — ★★★ — **A SECOND SET OF A PSEUDO DISQUALIFIES IT FROM `move_movables`** (P31 S69; main/func_800215F4, 465 ins, closeness 106 → 59 → 39) L32195 +- **§382** — TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) L32232 ## All sections, in order @@ -2633,6 +2641,11 @@ - **§376** — ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT THE **TU** (P31 S69; measured 0/28) L32055 - **§377** — THREE HARNESS DEFECTS FOUND IN ONE GATING SESSION, ALL "A CONFIDENT NUMBER ABOUT A SMALLER WORLD" (P31 S69) L32097 - **§378** — ★★★ — THE **SELF-CALLER CAST**: LET A TU KEEP CALLING THE FUNCTION IT IS ABOUT TO DEFINE (P31 S69; byte-proven ov_SC04_010/func_8017D6CC) L32114 +- **§379** — ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) L32164 +- **§380** — ★★★ — **A SECOND SET OF A PSEUDO DISQUALIFIES IT FROM `move_movables`** (P31 S69; main/func_800215F4, 465 ins, closeness 106 → 59 → 39) L32195 +- **§381** — THE `insn_count` HOIST THRESHOLD IS A DIAL YOU CAN READ WITH `cc1 -dL` (P31 S69; four independent uses in one wave) L32216 +- **§382** — TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) L32232 +- **§383** — TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) L32244 --- @@ -3678,3 +3691,8 @@ Notes routinely quote that as a section id. This table resolves it. Grep bait: ` | L32055 | §376 | ★★★ — A STANDALONE `match_one` CLOSENESS OF 0 IS A CLAIM ABOUT THE **BODY**, NEVER ABOUT T | | L32097 | §377 | THREE HARNESS DEFECTS FOUND IN ONE GATING SESSION, ALL "A CONFIDENT NUMBER ABOUT A SMALLER | | L32114 | §378 | ★★★ — THE **SELF-CALLER CAST**: LET A TU KEEP CALLING THE FUNCTION IT IS ABOUT TO DEFINE ( | +| L32164 | §379 | ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CA | +| L32195 | §380 | ★★★ — **A SECOND SET OF A PSEUDO DISQUALIFIES IT FROM `move_movables`** (P31 S69; main/fun | +| L32216 | §381 | THE `insn_count` HOIST THRESHOLD IS A DIAL YOU CAN READ WITH `cc1 -dL` (P31 S69; four inde | +| L32232 | §382 | TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) | +| L32244 | §383 | TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) | diff --git a/docs/matching-cookbook.md b/docs/matching-cookbook.md index d9b9d0e7d..bf6f3bf03 100644 --- a/docs/matching-cookbook.md +++ b/docs/matching-cookbook.md @@ -32160,3 +32160,98 @@ changes the call's value category, so it REFUSES instead, R43), skips declaratio call site in that TU is checked against.* A draft is not just a body — it is a new prototype imposed on code that already compiled. Expect to fix the callers, and expect the fix to be a cast, because a cast is the only edit that changes typing without changing codegen. + +## §379 ★★★ — **MEM_IN_STRUCT_P**: THE SAME LOAD, WRITTEN AS A STRUCT MEMBER, SCHEDULES WHERE A CAST CANNOT (P31 S69; byte-proven main/func_80021284 220 ins and main/func_8002D904 217 ins, found INDEPENDENTLY by two agents) + +**The symptom.** A load sits below a store to a scalar global in your output and above it in the +target (or vice versa), and no statement permutation moves it. `func_80021284` sat at closeness 7 +across a **120-order statement-permutation plateau**; the fix took it straight to 0. + +**The mechanism.** gcc-2.7.2's alias machinery asks `true_dependence`/`anti_dependence` whether a +load may cross a store. Its answer turns on `MEM_IN_STRUCT_P` — the `/s` flag `expand_expr` sets on +a MEM. A raw `*(u32 *)(arg0 + N)` load is a plain MEM: it may alias the scalar global, so the +scheduler will not move it across that store. The **same address read through a struct pointer** +(`p->field`) is `/s`, declared non-aliasing against a fixed scalar store, and floats freely. + + *(u32 *)(arg0 + 0x18) /* plain MEM — pinned below the store to D_800xxxxx */ + p->attr /* MEM_IN_STRUCT_P — free to hoist to the block top */ + +**The complement, same pass, opposite direction** (`func_8002D904`, worth **26 of its 41 residuals**): +assigning an indexed table address to a **pointer variable first** makes the resulting `INDIRECT_REF` +*non*-`MEM_IN_STRUCT_P` — gcc sets `/s` whenever "the address was computed by addition", so every +array/cast spelling comes out `/s`+varying, and the load floats to the block top. Take the address +into a local pointer when you need the load to STAY. + +**So the dial has two ends and both are zero-byte:** + +| you want | write it as | +|---|---| +| the load to FLOAT (cross a scalar store) | a struct-member access through a pointer | +| the load to STAY | take the address into a local pointer variable first | + +Related store-side note at `src/800.c:21942` (W16). This is the alias-analysis sibling of §41b's +"global load hoisted above the RTL prologue" wall — that one is unreachable; this one is a spelling. + +## §380 ★★★ — **A SECOND SET OF A PSEUDO DISQUALIFIES IT FROM `move_movables`** (P31 S69; main/func_800215F4, 465 ins, closeness 106 → 59 → 39) + +`move_movables` hoists a loop invariant only while the pseudo has **exactly one set**. Give it a +second, and the hoist does not happen — a lever no register pin can reach, because a pin changes +*which* register, never *whether* the value is hoisted. + + pz = &otz; /* first set */ + … + pz = &dzp[i >> 1]; /* second set — pz is now ineligible for hoisting */ + + k = 0xC000; + … + k = 0xFFF; /* same trick on a constant */ + +Two loop invariants that a `$v0` pin could not un-hoist came down this way. Reach for it whenever +the target computes something INSIDE the loop that your draft computes in the preheader. + +**The related pin form** (`func_8001EFE0`, 468 ins): a **call-used hard-reg pin** makes `invariant_p` +return 0 when the loop contains a call, which is what stops `move_movables` hoisting an addPrim +address. Pin to `$a3` — a call-clobbered register — precisely *because* it is call-clobbered. + +## §381 — THE `insn_count` HOIST THRESHOLD IS A DIAL YOU CAN READ WITH `cc1 -dL` (P31 S69; four independent uses in one wave) + +`move_movables`' willingness to hoist scales with the loop's `insn_count`, and the edge is sharp +enough to sit on. Read the count from the `-dL` (loop) dump, then move ONE statement across it: + +* `func_80040DE8` — insn_count **232** hoisted `(s16)param_1` at the exact `58×2×2 = 232` edge; a + `key = param_1;` copy inside the loop lifted it to 233 and **un-hoisted** it. +* `func_80023BF0` — loop 2's insn_count is **39** and needs **≥41**: below the threshold gcc admits + one constant hoist too many (`li 0xFFFF` at threshold 40), which costs a fourth callee-saved + register and shifts the whole register file. Still open at 269 for exactly this reason. +* `func_8001DA34` — `29×1×9 ≥ 203` hoisted the `0xFF000000` OT mask and spilled `ot` (frame + 0xB8→0xC8); fixed with `$a1`/`$a2` pins. **BANKED, 408 ins.** + +**Read the dump before you theorise.** Three of these four were found by reading `-dL`, not by +guessing, and the fourth is documented as an open threshold miss rather than a mystery. + +## §382 — TWO FOLD REASSOCIATIONS THAT NEED THEIR OWN STATEMENT (P31 S69) + +gcc-2.7.2's `fold` rewrites these before you ever see RTL, so the fix is always "split the +expression", never "add a cast": + +* `A * (X / 0x7F * 0x3FFF)` folds to `(A * 0x3FFF) * X` — the 0x3FFF scaling needs its own statement + (`func_80040DE8`). +* `A + (B + C)` reassociates to `(A + C) + B` — an address split into two statements blocks it + (`func_80021284`). +* `(x << 9) | CONST | var` reassociates so that only the shift gets hoisted — the four `getTPage` + bases must be explicit pre-loop variables (`func_8001EFE0`). + +## §383 — TWO TOOLCHAIN FACTS THE PACKS DID NOT CARRY (P31 S69) + +1. **maspsx requires DECIMAL memory-operand offsets in verbatim asm.** `0x40($sp)` dies with + `invalid literal for int() with base 10`; write `64($sp)`. Hex is fine for plain immediates. + Found while banking `main/func_80026514` (239 ins) the §265 verbatim-asm way — that function is + genuinely HANDWRITTEN: its `D_800B99DA` fetch is a gas macro expansion of `lbu $v1,0xA3AA($t2)` + into `lui $at,1 / addu $at,$t2,$at / lbu $v1,-0x5C56($at)`, which gcc cannot emit. +2. **A 4-byte struct local is BLKmode**, so `assign_stack_local` gives it 8-byte + `BIGGEST_ALIGNMENT`. Declaring `xy0`/`xy1` as `s32` instead of a 4-byte struct is the difference + between frame 0x130 and 0x138 (`func_8001EFE0`). +3. **`convert_modes` returns x unchanged when `oldmode == GET_MODE(x)`**, so **no cast can emit the + `(s16)` `sll`/`sra` pair**. Only an `s16` local assigned in one block and consumed in a distant + one does — and the copy must sit ABOVE the guard, or it coalesces away against the target's + `addu $a1,$v0,$zero` (`func_8002D904`). diff --git a/docs/wave-playbook.md b/docs/wave-playbook.md index 41f79b494..ab4c472ab 100644 --- a/docs/wave-playbook.md +++ b/docs/wave-playbook.md @@ -179,7 +179,28 @@ a merging gate makes the stub oracle wrong in both directions (§377). If it ref **And do not treat its `INTEG-STANDALONE-MATCH` / `NOCOMPILE-UNDECLARED-*` output as banks.** That is the §376 correction: standalone closeness 0 proves the BODY, not that the TU accepts the SIGNATURE. -Measured S69: **0 of 28** of that class banked raw. Route them `fix_arity_callers --any-proto` → gate. +Measured S69: **0 of 28** of that class banked raw — then **8 banked** once the missing lever +existed. Run the full §378 chain, IN ORDER; each step only reveals the next, and stopping at step 1 +is how the class read as dead for half a session: + +``` +tools/fix_arity_callers.py --binary B --funcs FN --any-proto --apply --journal J1 # `conflicting types' +tools/cast_self_callers.py --binary B --funcs FN --drafts D --apply --journal J2 # `too few arguments' +tools/cast_self_callers.py … --sync-decls # narrow-param: C89 forbids no-proto vs `void f(s16)' + # the byte-gate is the sole arbiter +tools/cast_self_callers.py --undo-journal J2 --keep # MANDATORY — see below +``` + +Or in one driver: `recover_integration.py --binary B --stages arity,self-cast --max-tier fleet --r22`. + +> **THE UNDO IS NOT OPTIONAL.** The casts go in BEFORE the gate, in preparation. A cast left behind +> for a draft that did not bank made `ov_SC07_000` fail to COMPILE at HEAD, so every later gate +> verdict on that binary measured a broken baseline rather than a draft (found only because two +> drafting agents reported BASELINE-RED and I checked their claim). Run `--undo-journal --keep +> ` after EVERY gate. + +If the diagnostic names a DIFFERENT symbol (`conflicting types for func_8012AD44`), run the same +chain on THAT symbol — it is a callee, not your target. That banked `main:func_80021D38`. ## 5. Draft diff --git a/tools/recover_integration.py b/tools/recover_integration.py index ddb52b89b..8d36023e8 100644 --- a/tools/recover_integration.py +++ b/tools/recover_integration.py @@ -60,8 +60,16 @@ STAGE_TIER = {"arity": "fleet", # fix_arity_callers writes src/shared/eng "tu-scope": "binary", # §103 STU: moves a contested TU decl into its consumers # (P31 T6 — the sweep-only lever the recovery path lacked; # writes src//*.c, covered by the TU snapshot) - "macro-externs": "draft"} # §121: rewrite draft decls of DEFINE_-defined callees to + "macro-externs": "draft", # §121: rewrite draft decls of DEFINE_-defined callees to # the macro's own signature (draft text only) + "self-cast": "binary"} # §378 (P31 S69): cast the TU's OWN call sites of the + # function being defined. Writes src//*.c only. + # THE STAGE ORDER MATTERS AND IS NOT ARBITRARY: `arity` + # no-protos the conflicting forward decl, which makes the + # DRAFT'S DEFINITION the prototype in scope, so the TU's own + # call fails anew with `too few arguments'. self-cast is the + # answer to the error `arity` CREATES, so it must run AFTER + # it, never instead of it. def tier_ok(stage, max_tier): @@ -193,7 +201,7 @@ def main(): ap.add_argument("--no-propagate", action="store_true") ap.add_argument("--run-id", default="r1", help="run-local scratch under .run/recover// (§55b trap 4)") ap.add_argument("--stages", default="demacroize", - help="comma-separated: demacroize,arity (default: demacroize — the measured blocker)") + help="comma-separated: demacroize,arity,self-cast,tu-scope,macro-externs (default: demacroize). §378 chain = arity,self-cast — in that order.") ap.add_argument("--max-tier", default="binary", choices=("draft", "binary", "fleet"), help="refuse any stage whose blast radius exceeds this (default binary = no shared-state edits)") ap.add_argument("--r22", action="store_true", help="clean-fleet verify after banking (REQUIRED for fleet tier)") @@ -336,6 +344,21 @@ def main(): raise SystemExit(f"[recover] fix_arity_callers failed: {(r.stderr or r.stdout)[-300:]}") print(" " + (r.stdout.strip().splitlines()[-1] if r.stdout.strip() else "(fix_arity_callers: no output)")) + if "self-cast" in stages: # P31 S69 — §378: the TU's own call sites of the fn being + # DEFINED. `arity` fixes `conflicting types'; this fixes the `too few arguments' that + # fixing it produces, because the draft's definition is now the prototype in scope. + # Byte-neutral: gcc-2.7.2 folds a cast of a known function symbol back to a direct jal. + # --sync-decls additionally handles the narrow-param case (a promotion-affected parameter + # like s16 makes a no-proto decl ILLEGAL in C89, which is exactly why fix_arity_callers + # skips it), and is safe ONLY because the call sites are cast first. + jr = f"{run_dir}/selfcast.json" + r = sh([PY, "tools/cast_self_callers.py", "--apply", "--sync-decls", "--binary", a.binary, + "--funcs", ",".join(targets), "--drafts", dd, + "--journal", jr]) + if r.returncode: + raise SystemExit(f"[recover] cast_self_callers failed: {(r.stderr or r.stdout)[-300:]}") + print(" " + (r.stdout.strip().splitlines()[-1] if r.stdout.strip() else "(self-cast: no output)")) + if "macro-externs" in stages and draft_rewrite: # P31 T6 — §121: a draft's decl of a DEFINE_-defined callee # must match the macro's OWN definition head (a guessed `extern int f();` collides with # the macro's real `void f(s32)` — measured: ONE such draft poisoned an entire probe diff --git a/tools/residual_rules_b.py b/tools/residual_rules_b.py index 670110de8..6bfe17efb 100644 --- a/tools/residual_rules_b.py +++ b/tools/residual_rules_b.py @@ -836,8 +836,12 @@ def classify(binary, fn, draft, tag=None): "the ONLY defect was %d missing extern(s) (%s); with decls COMPUTED " "from the target .s access widths the body byte-matches standalone" % (len(undecl), ", ".join(note["decls"].values())), - "add the computed extern(s) and route to integration — the matching " - "work is already done.") + "GATE-FIRST, NOT A BANK (§376, measured 0/28 raw in P31 S69). Do NOT " + "gate the autodecl arm: the extern that satisfies the STANDALONE probe is " + "a SECOND conflicting declaration in the real TU. Gate the RAW draft after " + "the §378 chain, in this order: fix_arity_callers --any-proto --binary " + " --funcs -> cast_self_callers --binary --funcs " + "--drafts [--sync-decls] -> gate. Each step only reveals the next.") return _finish(out) if res2.get("status") in ("near", "fail") and st2 is not None: out["status"] = res2["status"] @@ -874,8 +878,12 @@ def classify(binary, fn, draft, tag=None): if out["status"] == "match": _fire(rules, "INTEG-STANDALONE-MATCH", "certain", "§58/§59/§367", "closeness 0: the draft is byte-correct standalone", - "the blocker is TU integration (decl reconcile ladder, §367 banked-spelling rule); " - "matching is done — budget for banking.") + "matching is done; BANKING IS NOT. A standalone closeness of 0 proves the BODY and " + "says nothing about the TU accepting the SIGNATURE (§376 — this class gated 0/28 raw " + "in P31 S69, then 8 banked once the chain existed). Route: fix_arity_callers " + "--any-proto -> cast_self_callers [--sync-decls] (§378/§378a) -> gate; or " + "recover_integration --stages arity,self-cast. If the diagnostic names a DIFFERENT " + "symbol, run the same chain on THAT symbol.") return _finish(out) if st is None: