mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-29 23:24:32 -04:00
docs: wire the S68 tools into SETUP.md (R21) and the playbook (when to use them)
A tool nobody knows about is invisible work. Audit found neighbor_ref (built an
hour ago), residual_rules, lane_inflight and r22_verify in NEITHER doc, and
wall_sweep in the playbook but not the inventory.
SETUP.md gains a tooling-inventory row for all five with what each is FOR.
wave-playbook gains §2b: run neighbor_ref for EVERY card, placed right after the
seed_ref step because it answers the weaker and far more common question ('which
matched function should this agent READ?') that seed_ref structurally cannot. It
carries the measurement that justifies it -- a ~20x token swing on that single
variable -- and the failure it prevents: func_8017BEBC's card said 'no banked twin'
while a matched 755-instruction near-twin sat 3,700 lines up IN ITS OWN FILE.
Also states the two honest limits: an opt-level mismatch is PENALISED not merely
ranked low (§116 -- an -O2 example misleads an -O0 target), and a neighbour is a
worked example to READ, never a body to copy (§168 law 1, cousin-remap 0/26).
This commit is contained in:
@@ -776,6 +776,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/audit_digest.py` + **`make audit-digest`** | **(P30 S1e, cookbook §140)** The **scoreboard** oracle: recomputes the three headline metrics from the CURRENT tree and fails if the committed `docs/progress.fleet.md` disagrees. Wired into `tools-health` AFTER `report`. Exists because a digest generated from a working tree that later changed (work reverted before the commit landed) is **byte-invisible** — `check-all` stays 140/140 over it forever (R34: the byte-gate is a null oracle for DOCUMENTS) — and the next honest regeneration then reads as a REGRESSION that never happened. That is exactly what the `commit:1426` digest did: overstated **+7,879 ins / +130 unique fns**, which parked the phase's best lever on a phantom for a session. Compares **integers, not the printed percentages** (the staleness rendered as "94.4%" on both sides). Negative-control-proven against that stale digest. Same task hardened `progress.py stub_addrs`, which wrapped the fail-closed `corpus.stubs` in a bare `except` → empty stub set → `matched = sig − stubs` credited EVERY function: byte-witnessed reporting **instr 100.00% / distinct 100.00%** in a tree with no `asm/`. The identical swallow was fixed in `cast_call_sites.tu_for` + `reconcile_tu.tu_for`, where it silently reconciled drafts against the default `<ov>.c` instead of the jr/-O0 split TU — the very bug `cast_call_sites`' docstring exists to fix. |
|
||||
| | `tools/cdecl.py` + **`make audit-cdecl`** | **THE C-declaration oracle (cookbook §51g).** ONE recursive-descent parser of C's **declarator grammar**, replacing fifteen tools' private regex models — models that disagreed with each other and were, all fifteen, blind to fn-ptr/jump-table decls (`extern void (*D_X[])(void);`), sized arrays (`[4]`), and multi-declarators (where the *whole line* was dropped). Total by construction, not by shape enumeration. **Two statement paths, because the inputs differ:** `tu_statements()` derives a TU's file scope from **`cpp`** (a decl inside a `DEFINE_func_*` macro body declares nothing until invoked — §8c; 54 ms/TU), and `split_statements()` is a **span-preserving** raw split for drafts (which get rewritten). API: `parse` / `scope` / `tu_scope` / `Declarator{name,kind,type,params,pnames,is_proto,is_definition}`. Verified: **2,952,246 depth-0 statements → 2,731,521 declarators, 0 parser defects**; **50,405 distinct declarations round-tripped through the real cross-gcc, 0 rejected**; residue adjudicated NOT-C *by gcc*, not by opinion. **Phase-27 T4 — the canonical draft-typedef strip:** `typedef_names(tu_path)` (the names a TU declares as typedefs, robust `tu_statements`-based so a coverage gap can't crash the byte-gate) + `strip_provided_typedefs(draft, provided)` (drop a draft's self-contained typedefs the target already supplies, splitting multi-typedef lines and covering scalar AND struct typedefs). Replaced **six** copied scalar-name regexes with complementary holes: `harvest_verify` now strips per-TU (unblocks the 39 struct-typedef drafts `_TD` dropped) and **surfaces cc1 stderr** so a `redefinition`/`conflicting types` failure reports as **PLUMBING**, not a byte mismatch (`.run/harvest_failed.classified.txt`); `masked_diff.strip_scalar_typedefs()` (used by `match_one`/`p16_permute`) fixes the multi-typedef-line skip that discarded 42 masked-MATCH drafts over whitespace (`func_8015C030` → `MATCH (23 ins)` unedited). `canon_sig_reconcile`/`eval_lora`/`format_finetune` keep their own copies for now (migrate per-bank, byte-gated — the audit-prescribed cadence). |
|
||||
| | Phase 26-A tool-hygiene close (A9d–A10) | **DELETED** (R33, dead Phase-17 chain): `tools/census_conflict_callees.py` + `tools/derive_canonical_sigs.py` — `reconcile_tu`/`cdecl` answer their question from the build. **`overlay_src_split.py`**: `scan_construct` force_decl latch fixed (no longer swallows a def sharing a line with leading externs) + `hidden_definitions()` R32 coverage oracle wired into `selftest`. **`jr_isolate_all.py` `jr_inventory`**: `banked` DERIVED FROM THE IMAGE (`family_remap.reloc_targets` owns-a-carve) not a gitignored roster (R33) + curated-name via `addr_of` + 1:1 carve-ownership assert. **`family_remap.reloc_targets`**: optional `data=` param (read the image once, pass to N calls). **`backlog.py`**: `BACKLOG_NO_RENDER` env so parallel `gate_stage` workers skip the render race (append is atomic). `reconcile_tu` confirmed live on BOTH banking paths (`gate_stage` + `jtbl_family_bank.recover`→`bank_exemplar`). |
|
||||
| | **S68 tooling, round 2 — retrieval, triage, walls, liveness** (P31 S68) | **`tools/neighbor_ref.py` (NEW)** — for an OPEN stub, the already-MATCHED functions worth READING as worked examples, ranked SAME-TU first, then same binary, then shape (li-normalised skeleton / call-sequence hash / reloc-kind sequence / CFG counts / opcode cosine, all precomputed in `.run/feat.*.jsonl`), with a hard penalty for opt-level mismatch (§116). Surfaces the neighbour's HEADER COMMENT — the payload agents actually consume. Answers the question `seed_ref` cannot: seed_ref needs a hash-identical twin, this finds a near-twin. **Run it for every card.** Measured motivation: S68's cheapest large matches all came from a neighbour (555 ins/72k, 657 ins/122k first compile, 753 ins/177k) while neighbour-less main fns ran 200-350k for ~80 ins. **`tools/wall_sweep.py` (NEW)** — enumerates the §332 delay-slot macro walls (10 fns / 1,027 ins); `--emit-exclude` feeds `draw_waves --exclude`. **Run before every draw AND before every escalation** — an escalation cannot beat the toolchain. **`tools/residual_rules.py` + `residual_rules_b.py` (NEW, experimental)** — residual→cookbook-lever classifiers; `_b` is the stronger (113/113 classified, 63% certain/high). NOT yet wired into the pipeline: see `docs/next-session-triage-ladder.md`. **`tools/lane_inflight.py` (NEW)** — the RECORDED liveness ledger (`add` on launch, `done` on verdict); `list` exits non-zero while any agent is live and IS the guard. **`tools/r22_verify.sh` (NEW)** — the clean-fleet verify, EXCLUSIVE by construction: it refuses while `lane_inflight` shows live agents, because `make clean` deletes `asm/` and drafting agents READ it. Clears `.run/R22_DEBT` only on a genuinely green run. |
|
||||
| | **S68 tooling — the gater lane + the -O0 route** (P31 S68) | **`tools/gater_lane.py` (NEW)** — the continuous gater: drains a wave's finished drafts into `parallel_gate`, grouped by binary, `--r22` by default. Ledger AND verdicts keyed **`binary:fn:arm`** (R48 — and an escalation ALWAYS has a prior verdict, so arm-keying is what stops an in-flight fable draft riding the opus one). `--extra BINARY:PATH` for non-wave drafts, `--skip-binary` for lanes that may be writing, and **main is routed IN-TREE via `harvest_verify`** because `parallel_gate`'s worktree cannot stage main's psyq_integrate link inputs. **`tools/workflows/escalate_fable.js` (NEW)** — warm-started escalation: passes the prior draft + its measured closeness + its ruled-out levers, and demands a reusable `new_idiom`. **`tools/o0_boundary.py` (NEW)** — the stranded-boundary -O0 sweep (141 binaries / 288 boundaries / 0 candidates: the class is EXHAUSTED, and that null is negative-controlled). |
|
||||
| | **S68 fixes — six instances of the overlay-layout assumption** (cookbook **§363**) | `dedup_propagate` could not even IMPORT (`os.` at module level in the one module that imports `os as _os`). `seed_ref` offered main's LINKED-subseg DEAD TEXT as bankable twins (43 of 82 hits — a draft there gates GREEN while wrong); now refuses and COUNTS the refusal. `parallel_gate.stage_generated` hard-coded `build/<b>/<b>.ld`; now asks the Makefile for `<b>_LD_SCRIPT`/`<b>_UNDEF_SYMS`/`<b>_UNDEF_FUNCS` and REFUSES when absent. `rtu_match` gained **`--tu`** (+ `blocker_probe` passes `stub.path` and `stub.asm_dir`) — it reconstructed `src/<source>/<split>.c`, which is the overlay layout; main's sources are LOOSE FILES in `src/`. `gate_stage` no longer synthesises `--out`/`--good-sha` — for main those resolved to a nonexistent path and then **ov_SC01_077's SHA** via `DEF_SHA`. **`psyq_integrate`**: the `*_externals.ld` map is now MONOTONIC — it was re-derived against the CURRENT `.ld`, so `firstfile = 0x80061FA8;` was DROPPED on every incremental relink and main was 2 bytes red before any draft was spliced (**the true identity of the 2026-08-15 'main link defect'**). |
|
||||
| | **S68 — the module-binary -O0 carve route** (cookbook **§371**) | `jr_isolate_all` + `overlay_src_split` + the **Makefile -O0 glob widened to `src/md_*/md_*_o0?.c`** open carving for the single-object `md_*` binaries. Three stacked causes behind one `unaddressable content` message (interior-YAML-comment symbol-list truncation; a trailing verbatim-asm chunk with no region; bare tag forward decls), then the **spimdisasm rodata-migration trap**: migrated rodata follows its function ONLY within the same subseg, so a carve silently drops it and `INCLUDE_RODATA` cannot bring it back — rename the `.rodata` subseg to the object its emitters moved to. **The Makefile hunk MUST be committed with the carve** or a fresh clone loses -O0 on the region and every draft banked there mystery-fails. |
|
||||
|
||||
@@ -100,6 +100,39 @@ target**, joined on the corpus signature hashes (`tools/seed_ref.py`).
|
||||
> neither tool could see them. A `mechanical_remap_refused` flag now tells the agent: copy the BODY,
|
||||
> expect a declaration blocker.
|
||||
|
||||
### 2b. RUN `neighbor_ref` FOR EVERY CARD — the biggest measured cost lever in the wave
|
||||
|
||||
```
|
||||
python3 tools/neighbor_ref.py --binary <bin> --fn <fn> --top 5
|
||||
```
|
||||
|
||||
`seed_ref` (step 2) answers *"is there a byte-identical twin?"*. This answers the weaker and far
|
||||
more common question: **"which already-MATCHED function should this agent READ first?"**
|
||||
|
||||
**The measurement, S68.** Every one of the cheapest large matches came from an agent finding a
|
||||
matched neighbour; the expensive ones had none:
|
||||
|
||||
| function | ins | tokens | what unlocked it |
|
||||
|---|---|---|---|
|
||||
| `func_800D1254` | 555 | **72k** | an `-O0` sibling in the same binary |
|
||||
| `func_800D12D0` | 657 | **122k** | the `-O0` sibling, FIRST COMPILE |
|
||||
| `func_8018AD9C` | 397 | **87k** | a banked twin, §193-A one-shot |
|
||||
| `func_8017BEBC` | 753 | **177k** | a near-twin IN THE SAME FILE |
|
||||
| `main` fns with no neighbour | ~80 | **200–350k** | — |
|
||||
|
||||
That is a ~20× swing on the one variable the card controls.
|
||||
|
||||
**The failure it exists to prevent:** `func_8017BEBC`'s card asserted **"no banked twin"** while a
|
||||
matched 755-instruction near-twin sat 3,700 lines up in its own destination file, its header comment
|
||||
listing the four levers the target needed. `seed_ref` joins on signature hashes and could not see it.
|
||||
Three other S68 agents found their unlock the same way, unprompted — so this is a supplied habit now,
|
||||
not an accidental one.
|
||||
|
||||
**Read the ranking honestly:** SAME-TU beats everything (same decl environment, same carve, and its
|
||||
header usually records the levers). An opt-level mismatch is PENALISED, not ranked low — an `-O2`
|
||||
example actively misleads an `-O0` target (§116). And a neighbour is a **worked example to read**,
|
||||
never a body to copy: §168 law 1 measured cousin-remap at 0/26.
|
||||
|
||||
## 3. Packs
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user