From 41b259e5289dbfc340e43166292b8f5aa96a0d1f Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Wed, 26 Aug 2026 20:19:41 -0600 Subject: [PATCH] =?UTF-8?q?fix(wave):=20cards=20keyed=20by=20(binary,fn)?= =?UTF-8?q?=20=E2=80=94=20t5a=20received=2048/48=20ANOTHER=20binary's=20ca?= =?UTF-8?q?rd=20(R48);=20t5=5Fcards.py=20builds=20the=20target's=20OWN=20f?= =?UTF-8?q?uel=20(tu=5Fref=2088%=20/=20decl=5Fprior=2098%)=20via=20wave=5F?= =?UTF-8?q?card=5Ffuel.py=20extracted=20from=20build=5Fwave=5Fatlas=20(R33?= =?UTF-8?q?);=20t5=5Fbank.sh=20recovers=20un-banked=20drafts=20via=20fix?= =?UTF-8?q?=5Ftu=5Fret=5Fdecls=20(2/2=20on=20t5a);=20distill=20workflow=20?= =?UTF-8?q?generalized=20(P31=20S63=20T5)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/SETUP.md | 2 +- tools/build_wave_atlas.py | 96 +---------------------- tools/claude_wave_packs.py | 33 ++++++-- tools/t5_bank.sh | 41 +++++++++- tools/t5_cards.py | 56 +++++++++++++ tools/t5_distill_args.py | 54 +++++++++++++ tools/wave_card_fuel.py | 104 +++++++++++++++++++++++++ tools/workflows/claude_wave_distill.js | 17 ++-- 8 files changed, 291 insertions(+), 112 deletions(-) create mode 100644 tools/t5_cards.py create mode 100644 tools/t5_distill_args.py create mode 100644 tools/wave_card_fuel.py diff --git a/docs/SETUP.md b/docs/SETUP.md index aeb3982b5..438d05cf8 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -737,7 +737,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/interleave_check.py` | **(Phase 31 S62 T2, cookbook §302)** Order ⇔ yaml consistency for an overlay's jtbl carve: the `_JTBL_INTERLEAVE` order must equal the yaml subseg sequence position by position. `--fix` regenerates the order from the yaml (never hand-edit it). Run after ANY restore of overlays.mk or a splat yaml (two of the five S61 reds and four of the six clean-sweep failures were exactly this). | | | `tools/pads_audit.py` | **(Phase 31 S62 T2, cookbook §302)** DERIVES each object's `JTBL_PADS` from the bytes: compiles the TU to count its tables, walks the retail words inside the yaml carve with those sizes (pad 4 ⇔ one zero word), reports SPEC-DRIFT / CARVE-DRIFT / ok with the derived spec. Reproduced 15/15 correct specs as its negative control; replaced S61's blind value searches. | | | `tools/claude_wave_packs.py` · `tools/workflows/claude_wave_draft.js` · `tools/wave_judge.py` | **(Phase 31 S62 T4)** The Claude-subagent wave harness that ran the model-ladder probe: identical packs from `api_agent` (SYS + user_msg + law-1c warm start + gate feedback) → a Workflow of drafting agents per (arm, fn) with CLI equivalents of grep/read_file/match_one/submit, writing only under `.run///` → `wave_judge.py` gates each arm with `sweep_parallel` (banked = removed INCLUDE_ASM lines in `git diff`; tree reset between arms; `--union` leaves the cheapest winning drafts in the tree). `tools/workflows/claude_wave_distill.js` = the R16 distill (extract per byte-proven transcript → verify vs the cookbook). Routing rule from T4: ≤50 Sonnet, 51–120 Sonnet, >120 Opus, Opus on residue; haiku retired; DeepSeek optional at ~$0.03/fn. | -| | `tools/t5_targets.py` · `tools/t5_bank.sh` | **(Phase 31 S63 T5)** The wave *draw* and the wave *bank* halves around the T4 harness: `t5_targets.py --wave .run/t5x --n 48 [--residue .run/t5w] [--main-only]` draws a routed slate (open per `corpus.stubs`, ledger `.run/t5/drawn.json` keyed (binary,fn), ≤120 ins → sonnet / >120 → opus, residue → opus once, stratified by band, ≤4 per binary, one fn NAME per wave — the harness keys packs/drafts by bare name, so `claude_wave_packs`/`wave_judge` now REFUSE duplicates, R43/R48); then `claude_wave_packs.py` → Workflow `tools/workflows/claude_wave_draft.js` `args={wave, targets}` (one agent per target, `model=target.arm`) → `tools/t5_bank.sh .run/t5x sonnet opus` (refuses if a lane/gate runs or src/ config/ is dirty; `wave_judge --union`; R22 clean fleet sweep parsed `N passed, 0 failed of N`; commit `— N banked`). | +| | `tools/t5_targets.py` · `tools/t5_cards.py` · `tools/wave_card_fuel.py` · `tools/t5_bank.sh` | **(Phase 31 S63 T5)** The wave *draw* and the wave *bank* halves around the T4 harness: `t5_targets.py --wave .run/t5x --n 48 [--residue .run/t5w] [--main-only]` draws a routed slate (open per `corpus.stubs`, ledger `.run/t5/drawn.json` keyed (binary,fn), ≤120 ins → sonnet / >120 → opus, residue → opus once, stratified by band, ≤4 per binary, one fn NAME per wave — the harness keys packs/drafts by bare name, so `claude_wave_packs`/`wave_judge` now REFUSE duplicates, R43/R48); then **`t5_cards.py --wave .run/t5x`** BUILDS the target's OWN card fuel (`tu_ref` = in-TU banked neighbours §194-E, `decl_prior` = fleet signatures §196) from `wave_card_fuel.py` — those helpers extracted VERBATIM out of `build_wave_atlas.py` (which parses argv at import and so was unusable as a library) so both consumers share ONE oracle (R33); measured 88% tu_ref / 98% decl_prior on a fresh K-class slate, against 0% correct fuel before the fix. Then `claude_wave_packs.py --cards /cards.json` → Workflow `tools/workflows/claude_wave_draft.js` `args={wave, targets}` (one agent per target, `model=target.arm`) → `tools/t5_bank.sh .run/t5x sonnet opus` (refuses if a lane/gate runs or src/ config/ is dirty; `wave_judge --union`; then a RECOVERY pass — every un-banked draft through `fix_tu_ret_decls`, which retypes a destination TU's `extern void` decl on a value-returning function and re-gates (wave t5a: 2/2 banked, both byte-exact bodies refused for that decl alone); then the R22 clean fleet sweep parsed `N passed, 0 failed of N`; commit `— N banked`). | | | `tools/jtbl_rodata_pads.py --derive` | **(Phase 31 S62 T3a, cookbook §303)** Module path of the §8e pads filter: the Makefile runs `--derive $(BINARY) --tu ` for every `md_*` object — jump-table pads derived at build time from the retail island + the emission stream (trailing pads `0t1`, table-aware, const data passes through). No stored spec; an anchor miss fails the build with the offset. | | | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. | | | `tools/symcheck.py` | **(Phase 29 SESSION-18, cookbook §67a)** The pre-gate SYMBOL-SET guard: diffs the symbols a draft's object references (reloc records) against the target `.s`'s `%hi`/`%lo`/`jal` set. Catches the class **every masked oracle is structurally blind to** — `match_one`/`masked_diff` compare relocation-MASKED words (object-vs-`.s` mode is symbol-agnostic by construction) and `rtu_match` **compiles without linking**, so a draft that invents an extern no symbol table defines reads MATCH in both and can never bank (the SESSION-17 `func_801463A0` `_s`-alias trap). Negative-control-proven: on a draft with one renamed data extern, `match_one` reports the SAME 14 mismatched as the correct draft while `symcheck` exits 1 naming both the MISSING and the INVENTED symbol. `--c` (compiles via `match_one`, so the triple can never drift) or `--obj`. A cheap necessary condition, NOT a match oracle — still finish on the byte-gate (G3/P9). | diff --git a/tools/build_wave_atlas.py b/tools/build_wave_atlas.py index ca467cf50..46a542a0e 100644 --- a/tools/build_wave_atlas.py +++ b/tools/build_wave_atlas.py @@ -293,20 +293,10 @@ if not PRIORS: levers = set(a.levers.split(',')) atlas = json.load(open(a.atlas)) -_open = {} -def _stubmap(binary): - if binary not in _open: - # corpus.stubs() is addr -> Stub; the NAME lives on the record - _open[binary] = {st.symbol: st for st in corpus.stubs(binary).values()} - return _open[binary] - -def is_open(binary, fn): - return fn in _stubmap(binary) - -def home_tu(binary, fn): - """The stub's home .c — this is the GATE GROUP KEY (gate_lane groups by (binary, src)).""" - st = _stubmap(binary).get(fn) - return st.path if st else None +# CARD FUEL LIVES IN ONE PLACE (R33, P31 S63): _stubmap/is_open/home_tu/_tu_bodies/tu_neighbours +# moved to tools/wave_card_fuel.py so the Claude wave packs (tools/t5_cards.py) build IDENTICAL +# fuel. They were unreachable as a library here: this module parses argv at import. +from wave_card_fuel import _stubmap, is_open, home_tu, _tu_bodies, tu_neighbours # noqa: F401 def model_for(nins): if nins <= 50: return 'haiku' @@ -315,84 +305,6 @@ def model_for(nins): _DPIDX = DP.load() # symbol -> fleet declaration evidence (§196) knn = atlas.get('knn') or {} # exemplar key -> neighbours; 'M:' entries are BANKED (§193-A) -_TU_BODIES = {} - - -def _tu_bodies(tu_path): - """{fn: body_text} for every function DEFINED in a TU (banked C only -- stubs are INCLUDE_ASM). - - Brace-matched from each definition so a symbol is attributed to the function that uses it, not - to the file. Memoized per TU: a wave draws many cards from one .c.""" - if tu_path in _TU_BODIES: - return _TU_BODIES[tu_path] - out = {} - try: - txt = open(tu_path, errors='replace').read() - except (OSError, TypeError): - _TU_BODIES[tu_path] = out - return out - import re as _re - for m in _re.finditer(r'^[A-Za-z_][\w \t\*]*?\b(\w+)\s*\([^;{]*\)\s*\{', txt, _re.M): - i, depth = m.end() - 1, 0 - while i < len(txt): - if txt[i] == '{': depth += 1 - elif txt[i] == '}': - depth -= 1 - if depth == 0: break - i += 1 - out[m.group(1)] = txt[m.start():i + 1] - _TU_BODIES[tu_path] = out - return out - - -_SYM = None - - -def tu_neighbours(binary, fn, tu_path, asm_file, topn=2): - """Banked functions in the card's OWN TU, ranked by symbols shared with the target's asm. - - The symbols are read from the TARGET's .s (its relocation operands), so this is evidence about - the function being drafted, not about the file. See §194-E for why the card needs this at all.""" - global _SYM - if not tu_path or not asm_file: - return [] - import re as _re - if _SYM is None: - # OPERANDS ONLY. A splat .s carries the encoded WORD in a comment column, so a naive - # "[A-Z]\w{3,}" reads `D8FFBD27` and `CC00228E` as symbol names and the overlap score - # becomes noise (measured: 34 "symbols" for one function, 31 of them hex words). - # Symbols reach the .s in exactly three shapes: a jal target, and %hi()/%lo() operands. - _SYM = _re.compile(r'\b(?:jal\s+(\w+)|%[hl][io]\(([\w+]+)\))') - try: - raw = _SYM.findall(open(asm_file, errors='replace').read()) - except OSError: - return [] - want = {(a or b).split('+')[0] for a, b in raw if (a or b)} - want.discard(fn) - if not want: - return [] - scored = [] - for other, body in _tu_bodies(tu_path).items(): - if other == fn: - continue - shared = [s for s in want if s in body] - if len(shared) >= 2: - scored.append((len(shared), other, sorted(shared)[:6])) - scored.sort(key=lambda x: -x[0]) - if not scored: - # FALL BACK TO THE BEST SINGLE SHARED SYMBOL rather than emitting nothing. One shared - # callee is weak evidence, but the card reports the count so the agent can weigh it, and a - # weak same-TU lead still beats the zero-locality state §194-E measured. - weak = sorted(((len([s for s in want if s in body]), other) - for other, body in _tu_bodies(tu_path).items() if other != fn), reverse=True) - if weak and weak[0][0] == 1: - o = weak[0][1] - body = _tu_bodies(tu_path)[o] - return [{'fn': o, 'shared': 1, 'symbols': [s for s in sorted(want) if s in body][:6]}] - return [{'fn': o, 'shared': n, 'symbols': syms} for n, o, syms in scored[:topn]] - - - def _o0_unbankable(spath): """True iff this target is an -O0 function sitting in a subseg the build compiles -O2. diff --git a/tools/claude_wave_packs.py b/tools/claude_wave_packs.py index df462505d..534b3c106 100644 --- a/tools/claude_wave_packs.py +++ b/tools/claude_wave_packs.py @@ -6,6 +6,12 @@ prior_draft (law-1c filtered: symbols must overlap the target .s) + gate_feedbac written once. Identical packs are what made the T4 arms comparable; keep it that way. Usage: claude_wave_packs.py [--cards cards.json] +CARDS ARE KEYED BY (binary, fn), NEVER BY BARE NAME (R48; byte-measured P31 S63). Overlays share +function NAMES at equal addresses, and a K-class target has never been carded, so a name-keyed +lookup over `.run/wave_*_cards.json` returns SOME OTHER overlay's same-named function every time: +in wave t5a it mis-carded 48 of 48 targets, and nine agents independently reported "the pack's +warm-start/twin/decl hints were a different function". A card whose binary does not match the +target is DROPPED and counted, never silently substituted (R32/R43). targets.json: [{name, binary, nins, sub, asm, tu}, ...] (api_agent's target shape; see tools/main_queue_rebuild.py / .run/t4/targets.json for how to derive one from corpus) out_dir/SYS.md + out_dir/packs/.md are produced; the workflow script @@ -29,22 +35,28 @@ def main(): dup = [n for n, c in __import__('collections').Counter(t['name'] for t in targets).items() if c > 1] if dup: # packs are keyed by bare fn name; two binaries' same-named fns would overwrite (R43/R48) sys.exit('REFUSED: %d duplicate fn name(s) in one wave (packs are name-keyed): %s' % (len(dup), ' '.join(dup))) - cards = {} + cards, wrong_bin = {}, 0 + def add(e): + nonlocal wrong_bin + if not (isinstance(e, dict) and e.get('fn')): + return + b = e.get('binary') or e.get('b') + if not b: # a card that cannot prove its binary cannot be trusted to a target + wrong_bin += 1; return + cards[(b, e['fn'])] = e if '--cards' in sys.argv: - for c in json.load(open(sys.argv[sys.argv.index('--cards') + 1])): - if c.get('fn'): cards[c['fn']] = c - else: # newest per-fn card across every wave card file + for c in json.load(open(sys.argv[sys.argv.index('--cards') + 1])): add(c) + else: # newest per-(binary,fn) card across every wave card file for f in sorted(glob.glob('.run/wave_*_cards.json'), key=os.path.getmtime): try: - for e in json.load(open(f)): - if isinstance(e, dict) and e.get('fn'): cards[e['fn']] = e + for e in json.load(open(f)): add(e) except Exception: pass os.makedirs(os.path.join(out, 'packs'), exist_ok=True) open(os.path.join(out, 'SYS.md'), 'w').write(A.SYS) n = 0 for t in targets: - um = A.user_msg(t, cards.get(t['name'])) + um = A.user_msg(t, cards.get((t['binary'], t['name']))) pd, where = A.prior_draft(t) if pd: um += ("\n\nA PREVIOUS ATTEMPT on this exact function left this body behind (%s). The whole-binary " @@ -55,7 +67,12 @@ def main(): if gf: um += '\n\n' + gf + '\n' open(os.path.join(out, 'packs', t['name'] + '.md'), 'w').write(um); n += 1 - print('packs: %d written to %s (cards matched %d/%d)' % (n, out, sum(1 for t in targets if t['name'] in cards), len(targets))) + hit = sum(1 for t in targets if (t['binary'], t['name']) in cards) + byname = sum(1 for t in targets if any(k[1] == t['name'] for k in cards)) + print('packs: %d written to %s' % (n, out)) + print('cards: %d/%d matched on (binary, fn); %d target(s) have a same-named card in ANOTHER binary ' + '(DROPPED, not substituted — R48); %d card(s) skipped for carrying no binary' + % (hit, len(targets), byname - hit, wrong_bin)) if __name__ == '__main__': main() diff --git a/tools/t5_bank.sh b/tools/t5_bank.sh index 54d5a39df..ac61b96c3 100644 --- a/tools/t5_bank.sh +++ b/tools/t5_bank.sh @@ -4,9 +4,14 @@ # src/ or config/ is already dirty (R42: never blind-revert; someone's banks may be uncommitted). # 2. WAVE= tools/wave_judge.py --union : whole-binary gate per arm (git-diff bank # detection, tree reset between arms), then the union gated once and LEFT IN THE TREE. -# 3. R22 clean fleet sweep (make clean -> extract-all -> check-all); the pass line must read +# 3. RECOVER the un-banked: a draft whose BODY is byte-correct but whose destination TU declares +# the function `extern void ...` when it returns a value is refused at closeness 0 with no +# codegen fault at all. `fix_tu_ret_decls` retypes those decls, re-gates, and restores every +# edit the gate does not pay for. Measured on wave t5a: 2 of 2 such refusals banked, both +# self-reported MATCH and both confirmed byte-exact by match_one in isolation. +# 4. R22 clean fleet sweep (make clean -> extract-all -> check-all); the pass line must read # "N passed, 0 failed of N" (exit codes checked, R53) or nothing is committed. -# 4. commit the union; the subject carries "— N banked" (the maintenance lane's exemplar counter). +# 5. commit the union; the subject carries "— N banked" (the maintenance lane's exemplar counter). # Drafting must be FINISHED before this runs: the judge rewrites src/ and the sweep deletes asm/. cd /home/musashi/bfm-decomp || exit 2 WAVE=${1:?usage: t5_bank.sh [arms...]}; shift @@ -23,8 +28,38 @@ WAVE="$WAVE" .venv/bin/python tools/wave_judge.py $ARMS --union 2>&1 | tee "$WAV [ "${PIPESTATUS[0]}" = 0 ] || { say "judge exited non-zero — stopping (tree left as is)"; exit 3; } N=$(python3 -c "import json;j=json.load(open('$WAVE/judge.json'));print(len(j.get('union',{}).get('banked',[])))") SUMMARY=$(python3 -c "import json;j=json.load(open('$WAVE/judge.json'));print(', '.join('%s %d/%d'%(a,v['n_banked'],v['drafts']) for a,v in j['arms'].items()))") +# ---- recovery pass over everything the union did not bank (see step 3) -------------------------- +UNB=$(.venv/bin/python - "$WAVE" <<'PY' +import json, os, shutil, sys +w = sys.argv[1] +tg = json.load(open(w + '/targets.json')); j = json.load(open(w + '/judge.json')) +banked = set(j.get('union', {}).get('banked', [])) +for v in j.get('arms', {}).values(): banked |= set(v.get('banked', [])) +best = j.get('union', {}).get('picked', {}) +arms = list(j.get('arms', {})) +rec = w + '/recover'; shutil.rmtree(rec, ignore_errors=True) +pairs = [] +for t in tg: + if t['name'] in banked: continue + for arm in [best.get(t['name'])] + arms: + src = w + '/%s/%s.c' % (arm, t['name']) if arm else '' + if src and os.path.exists(src) and os.path.getsize(src): + os.makedirs('%s/%s' % (rec, t['binary']), exist_ok=True) + shutil.copy(src, '%s/%s/%s.c' % (rec, t['binary'], t['name'])) + pairs.append([t['binary'], t['name']]); break +json.dump(pairs, open(w + '/ret_pairs.json', 'w')) +print(len(pairs)) +PY +) +if [ "${UNB:-0}" != 0 ]; then + say "recovery: $UNB un-banked draft(s) -> fix_tu_ret_decls (TU 'extern void' on a value-returning fn)" + flock .run/auto/draw.lock .venv/bin/python tools/fix_tu_ret_decls.py --pairs-file "$WAVE/ret_pairs.json" \ + --drafts "$WAVE/recover" -j 4 2>&1 | tee "$WAVE/recover.log" | tail -3 + N=$(git diff -U0 -- src/ | grep -cE '^-INCLUDE_ASM' || true) + SUMMARY="$SUMMARY, +recovery" +fi if [ "$N" = 0 ]; then say "0 banked ($SUMMARY) — nothing to commit"; git status --short -- src/ config/ | head; exit 0; fi -say "$N banked in the union ($SUMMARY) — R22 clean fleet sweep" +say "$N banked total ($SUMMARY) — R22 clean fleet sweep" ( make clean && make extract-all JOBS=32 && make check-all JOBS=32 ) > "$WAVE/sweep.log" 2>&1 RC=$? PASS=$(grep -E '^check-all: ' "$WAVE/sweep.log" | tail -1) diff --git a/tools/t5_cards.py b/tools/t5_cards.py new file mode 100644 index 000000000..753a04bf0 --- /dev/null +++ b/tools/t5_cards.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +"""tools/t5_cards.py — BUILD the card fuel for a wave's own targets (P31 S63 T5). + +Usage: t5_cards.py --wave .run/t5b [--out /cards.json] + +WHY (byte-measured, P31 S63 wave t5a). `claude_wave_packs.py` looked cards up by BARE FUNCTION NAME +across `.run/wave_*_cards.json`; overlays share names at equal addresses, so 48 of 48 t5a targets +(and 15 of T4's 20) got ANOTHER binary's card — wrong banked twin, wrong TU neighbours, wrong +declarations. Nine agents independently reported discarding it. Keying by (binary, fn) fixes the +mis-attribution but leaves K-class ("never touched") targets with NO fuel at all, and `_fuel()` +then emits nothing: no `tu_ref`, no `decl_prior`. Yet t5a's agents cracked function after function +by finding an already-banked sibling in their own TU BY HAND ("in-TU twin ~150 lines earlier", +"structural twin in the SAME TU") — which is exactly what `tu_ref` is (§194-E, 62% of targets). + +So this builds the two computable fields for the target ITSELF, from the same oracle +`build_wave_atlas` uses (`tools/wave_card_fuel.py`, R33): + tu_ref already-banked functions in the target's OWN TU, ranked by symbols shared with the + target's .s relocations + decl_prior the fleet's consensus signature for every symbol the .s references (§196) — kills + the two-arity A/B compile §195-A proved the asm cannot answer +`seed_ref` (the cross-TU banked twin) needs the atlas knn and is NOT built here: a card with no +seed_ref makes `_fuel` say "no banked twin — derive from the .s", which is TRUE, where the old +name-keyed card pointed confidently at an unrelated function. Coverage is printed (R32/R41).""" +import argparse, json, os, sys +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))); os.chdir(REPO) +sys.path.insert(0, 'tools') +import wave_card_fuel as F +import decl_prior as DP + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--wave', required=True) + ap.add_argument('--out', default='') + a = ap.parse_args() + targets = json.load(open(os.path.join(a.wave, 'targets.json'))) + idx = DP.load() + cards, n_tu, n_dp = [], 0, 0 + for t in targets: + tu = t.get('tu') or F.home_tu(t['binary'], t['name']) + tu_ref = F.tu_neighbours(t['binary'], t['name'], tu, t['asm']) + try: + dp = DP.for_asm(t['asm'], tu, idx=idx, binary=t['binary']) + except Exception as e: + print(' decl_prior failed for %s/%s: %s' % (t['binary'], t['name'], str(e)[:80])); dp = [] + n_tu += bool(tu_ref); n_dp += bool(dp) + cards.append({'fn': t['name'], 'binary': t['binary'], 'addr': t.get('addr'), 'nins': t['nins'], + 'sub': t['sub'], 'tu_ref': tu_ref, 'decl_prior': dp}) + p = a.out or os.path.join(a.wave, 'cards.json') + json.dump(cards, open(p, 'w'), indent=1) + print('cards: %d built for %s (tu_ref on %d/%d = %.0f%%; decl_prior on %d/%d = %.0f%%; seed_ref not built — ' + 'the pack will honestly say "no banked twin")' + % (len(cards), a.wave, n_tu, len(targets), 100.0 * n_tu / max(1, len(targets)), + n_dp, len(targets), 100.0 * n_dp / max(1, len(targets)))) + print('wrote %s' % p) +if __name__ == '__main__': + main() diff --git a/tools/t5_distill_args.py b/tools/t5_distill_args.py new file mode 100644 index 000000000..7a78e5fe6 --- /dev/null +++ b/tools/t5_distill_args.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python3 +"""tools/t5_distill_args.py — build the R16 distill slate for a Claude wave (P31 S63 T5). + +Usage: t5_distill_args.py --wave .run/t5a --wfdir [--label 'P31 S63 t5a'] [--all-arms] + +Only BYTE-PROVEN transcripts are distilled (the distiller's contract: "the whole-binary byte-gate +ACCEPTED the final draft, so the final body is ground truth"), so the slate = the wave's banked set +from `/judge.json`, joined to the drafting agents' transcripts via the workflow's +`journal.jsonl` (result rows carry fn/arm/note; the transcript is `agent-.jsonl`). +By default only the arm that BANKED each fn is distilled (--all-arms keeps every arm's attempt). +Coverage is asserted (R32): a banked fn with no transcript is reported and counted, never skipped +silently. Writes /distill_args.json for + Workflow(scriptPath=tools/workflows/claude_wave_distill.js, args=).""" +import argparse, json, os, sys +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))); os.chdir(REPO) + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--wave', required=True) + ap.add_argument('--wfdir', required=True, help="the drafting run's transcript dir (holds journal.jsonl + agent-*.jsonl)") + ap.add_argument('--label', default='', help="provenance string for the cookbook header, e.g. 'P31 S63 t5a'") + ap.add_argument('--all-arms', action='store_true') + ap.add_argument('--out', default='') + a = ap.parse_args() + jd = json.load(open(os.path.join(a.wave, 'judge.json'))) + banked_by_arm = {arm: set(v.get('banked', [])) for arm, v in jd.get('arms', {}).items()} + banked = set().union(*banked_by_arm.values()) if banked_by_arm else set() + union = set(jd.get('union', {}).get('banked', [])) + picked = jd.get('union', {}).get('picked', {}) + jp = os.path.join(a.wfdir, 'journal.jsonl') + if not os.path.exists(jp): sys.exit('no journal.jsonl in %s' % a.wfdir) + rows = [json.loads(l) for l in open(jp) if l.strip()] + tr = {} + for r in rows: + res = r.get('result') or {} + if isinstance(res, dict) and res.get('fn'): + tr[(res['fn'], res.get('arm'))] = (os.path.join(a.wfdir, 'agent-%s.jsonl' % r['agentId']), res.get('note', '')) + out, missing = [], [] + for fn in sorted(banked): + arms = sorted(banked_by_arm) if a.all_arms else [picked.get(fn) or next(x for x in banked_by_arm if fn in banked_by_arm[x])] + for arm in arms: + if fn not in banked_by_arm.get(arm, ()): continue + hit = tr.get((fn, arm)) + if not hit or not os.path.exists(hit[0]): missing.append('%s/%s' % (arm, fn)); continue + out.append({'fn': fn, 'arm': arm, 'transcript': hit[0], 'note': hit[1]}) + print('banked %d (union-gated %d); transcripts found %d; label %r' % (len(banked), len(union), len(out), a.label)) + if missing: + print('NO TRANSCRIPT for %d banked draft(s) — NOT distilled (R32, reported not skipped): %s' + % (len(missing), ' '.join(missing[:20]))) + p = a.out or os.path.join(a.wave, 'distill_args.json') + json.dump({'targets': out, 'label': a.label or os.path.basename(a.wave.rstrip('/'))}, open(p, 'w'), indent=1) + print('wrote %s' % p) +if __name__ == '__main__': + main() diff --git a/tools/wave_card_fuel.py b/tools/wave_card_fuel.py new file mode 100644 index 000000000..4d42287eb --- /dev/null +++ b/tools/wave_card_fuel.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""tools/wave_card_fuel.py — the per-target CARD FUEL an agent pack carries, as ONE oracle (R33). + +These helpers were defined inside `build_wave_atlas.py`, which parses argv at import and so cannot +be reused as a library. Extracted VERBATIM (P31 S63) so `build_wave_atlas` and the Claude wave's +`tools/t5_cards.py` compute the SAME fuel rather than two copies that drift: + + home_tu(binary, fn) the stub's home .c (the gate group key) + _tu_bodies(tu_path) {fn: body} for every function DEFINED (banked) in a TU + tu_neighbours(...) already-banked functions in the target's OWN TU, ranked by symbols + shared with the target's .s relocations (§194-E, the highest-yield + source measured; wave t5a's agents rediscovered it by hand and said so) + +Why it exists (byte-measured, P31 S63): `claude_wave_packs` looked cards up by BARE FUNCTION NAME, +and overlays share names at equal addresses — so ALL 48 of wave t5a's targets (and 15 of T4's 20) +received ANOTHER binary's card: wrong twin, wrong TU neighbours, wrong declarations. Nine agents +reported it independently. The fix is a (binary, fn) key — which, for never-carded K-class work, +means there is no card at all. So the fuel has to be BUILT for the target itself, from here.""" +import os, sys +REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(REPO, 'tools')) +import corpus + +_open = {} +def _stubmap(binary): + if binary not in _open: + # corpus.stubs() is addr -> Stub; the NAME lives on the record + _open[binary] = {st.symbol: st for st in corpus.stubs(binary).values()} + return _open[binary] +def is_open(binary, fn): + return fn in _stubmap(binary) +def home_tu(binary, fn): + """The stub's home .c — this is the GATE GROUP KEY (gate_lane groups by (binary, src)).""" + st = _stubmap(binary).get(fn) + return st.path if st else None +_TU_BODIES = {} +def _tu_bodies(tu_path): + """{fn: body_text} for every function DEFINED in a TU (banked C only -- stubs are INCLUDE_ASM). + + Brace-matched from each definition so a symbol is attributed to the function that uses it, not + to the file. Memoized per TU: a wave draws many cards from one .c.""" + if tu_path in _TU_BODIES: + return _TU_BODIES[tu_path] + out = {} + try: + txt = open(tu_path, errors='replace').read() + except (OSError, TypeError): + _TU_BODIES[tu_path] = out + return out + import re as _re + for m in _re.finditer(r'^[A-Za-z_][\w \t\*]*?\b(\w+)\s*\([^;{]*\)\s*\{', txt, _re.M): + i, depth = m.end() - 1, 0 + while i < len(txt): + if txt[i] == '{': depth += 1 + elif txt[i] == '}': + depth -= 1 + if depth == 0: break + i += 1 + out[m.group(1)] = txt[m.start():i + 1] + _TU_BODIES[tu_path] = out + return out +_SYM = None +def tu_neighbours(binary, fn, tu_path, asm_file, topn=2): + """Banked functions in the card's OWN TU, ranked by symbols shared with the target's asm. + + The symbols are read from the TARGET's .s (its relocation operands), so this is evidence about + the function being drafted, not about the file. See §194-E for why the card needs this at all.""" + global _SYM + if not tu_path or not asm_file: + return [] + import re as _re + if _SYM is None: + # OPERANDS ONLY. A splat .s carries the encoded WORD in a comment column, so a naive + # "[A-Z]\w{3,}" reads `D8FFBD27` and `CC00228E` as symbol names and the overlap score + # becomes noise (measured: 34 "symbols" for one function, 31 of them hex words). + # Symbols reach the .s in exactly three shapes: a jal target, and %hi()/%lo() operands. + _SYM = _re.compile(r'\b(?:jal\s+(\w+)|%[hl][io]\(([\w+]+)\))') + try: + raw = _SYM.findall(open(asm_file, errors='replace').read()) + except OSError: + return [] + want = {(a or b).split('+')[0] for a, b in raw if (a or b)} + want.discard(fn) + if not want: + return [] + scored = [] + for other, body in _tu_bodies(tu_path).items(): + if other == fn: + continue + shared = [s for s in want if s in body] + if len(shared) >= 2: + scored.append((len(shared), other, sorted(shared)[:6])) + scored.sort(key=lambda x: -x[0]) + if not scored: + # FALL BACK TO THE BEST SINGLE SHARED SYMBOL rather than emitting nothing. One shared + # callee is weak evidence, but the card reports the count so the agent can weigh it, and a + # weak same-TU lead still beats the zero-locality state §194-E measured. + weak = sorted(((len([s for s in want if s in body]), other) + for other, body in _tu_bodies(tu_path).items() if other != fn), reverse=True) + if weak and weak[0][0] == 1: + o = weak[0][1] + body = _tu_bodies(tu_path)[o] + return [{'fn': o, 'shared': 1, 'symbols': [s for s in sorted(want) if s in body][:6]}] + return [{'fn': o, 'shared': n, 'symbols': syms} for n, o, syms in scored[:topn]] diff --git a/tools/workflows/claude_wave_distill.js b/tools/workflows/claude_wave_distill.js index b4512bd4a..56f0c42fd 100644 --- a/tools/workflows/claude_wave_distill.js +++ b/tools/workflows/claude_wave_distill.js @@ -1,13 +1,14 @@ export const meta = { - name: 't4-idiom-distill', - description: 'R16 flywheel: distill the unlocking lesson from each byte-proven T4 transcript, verify novelty against the cookbook, return §-formatted candidates', + name: 'claude-wave-idiom-distill', + description: 'R16 flywheel: distill the unlocking lesson from each byte-proven wave transcript, verify novelty against the cookbook, return §-formatted candidates', phases: [ - { title: 'Extract', detail: '31 transcripts -> one lesson each' }, + { title: 'Extract', detail: 'one distiller per byte-proven transcript -> one lesson each' }, { title: 'Verify', detail: 'novel claims grepped against docs/matching-cookbook.md' }, ], } +// args: { targets: [{fn, arm, transcript, note}, ...], label: 'P31 S63 t5a' } — from tools/t5_distill_args.py const REPO = '/home/musashi/bfm-decomp' -const WFDIR = '/home/musashi/.claude/projects/-home-musashi-bfm-decomp/b0026d2d-2724-4aed-813a-3deef224d210/subagents/workflows/wf_c691ef44-320/' +const LABEL = args.label || 'P31 S63' const LESSON = { type: 'object', properties: { @@ -38,7 +39,7 @@ const lessons = await pipeline( args.targets, t => agent( `You are distilling ONE byte-proven decompilation transcript for the project's matching cookbook (docs/matching-cookbook.md, gcc-2.7.2 MIPS idioms). Repo: ${REPO}. -Transcript (JSONL, an agent's full tool trace): ${t.transcript.replace("WF/", WFDIR)} +Transcript (JSONL, an agent's full tool trace): ${t.transcript} Function: ${t.fn} (arm ${t.arm}); the whole-binary byte-gate ACCEPTED the final draft, so the final body is ground truth. The agent's own closing note was: "${t.note}" Read the transcript. Find the moment(s) the masked diff (match_one closeness) dropped to 0 and what C change caused it. Report ONE lesson — the most generalizable one — as: @@ -64,8 +65,8 @@ Read only; write nothing.`, refs the agent used: ${JSON.stringify(lesson.cookbook_refs_used)} Check docs/matching-cookbook.md (grep for the mechanism, the instruction pattern, the C construct; docs/cookbook-index.md is symptom-keyed) and decide: COVERED — an existing § already states this law (name it in covered_by) - ADDENDUM — an existing § is close but this adds a byte-proven refinement (name it; entry_markdown = a 3-8 line addendum paragraph starting "**Addendum (P31 S62 T4, ${t.fn}):**") - NEW — no § states it; entry_markdown = a full section in the book's style, header "## §NNN — (P31 S62 T4; byte-proven ${t.fn})" with NNN left literally as NNN, then the tell, the mechanism, the C shape, the evidence line + ADDENDUM — an existing § is close but this adds a byte-proven refinement (name it; entry_markdown = a 3-8 line addendum paragraph starting "**Addendum (${LABEL}, ${t.fn}):**") + NEW — no § states it; entry_markdown = a full section in the book's style, header "## §NNN — <TITLE IN CAPS> (${LABEL}; byte-proven ${t.fn})" with NNN left literally as NNN, then the tell, the mechanism, the C shape, the evidence line REFUTED — the claim contradicts a byte-proven § or the evidence does not support it (say why) Quote the § numbers you checked in why. Read only; write nothing.`, { label: `verify:${t.fn}`, phase: 'Verify', model: 'opus', schema: VERDICT }).then(v => ({ ...lesson, ...(v || { verdict: 'UNVERIFIED', why: 'verifier returned null' }) })) @@ -75,4 +76,4 @@ const out = lessons.filter(Boolean) const tally = {} for (const l of out) tally[l.verdict] = (tally[l.verdict] || 0) + 1 log('verdicts: ' + JSON.stringify(tally)) -return { lessons: out } \ No newline at end of file +return { label: LABEL, lessons: out } \ No newline at end of file