mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-26 13:33:34 -04:00
fix(wave): cards keyed by (binary,fn) — t5a received 48/48 ANOTHER binary's card (R48); t5_cards.py builds the target's OWN fuel (tu_ref 88% / decl_prior 98%) via wave_card_fuel.py extracted from build_wave_atlas (R33); t5_bank.sh recovers un-banked drafts via fix_tu_ret_decls (2/2 on t5a); distill workflow generalized (P31 S63 T5)
This commit is contained in:
Drew T
+1
-1
@@ -737,7 +737,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/interleave_check.py` | **(Phase 31 S62 T2, cookbook §302)** Order ⇔ yaml consistency for an overlay's jtbl carve: the `_JTBL_INTERLEAVE` order must equal the yaml subseg sequence position by position. `--fix` regenerates the order from the yaml (never hand-edit it). Run after ANY restore of overlays.mk or a splat yaml (two of the five S61 reds and four of the six clean-sweep failures were exactly this). |
|
||||
| | `tools/pads_audit.py` | **(Phase 31 S62 T2, cookbook §302)** DERIVES each object's `JTBL_PADS` from the bytes: compiles the TU to count its tables, walks the retail words inside the yaml carve with those sizes (pad 4 ⇔ one zero word), reports SPEC-DRIFT / CARVE-DRIFT / ok with the derived spec. Reproduced 15/15 correct specs as its negative control; replaced S61's blind value searches. |
|
||||
| | `tools/claude_wave_packs.py` · `tools/workflows/claude_wave_draft.js` · `tools/wave_judge.py` | **(Phase 31 S62 T4)** The Claude-subagent wave harness that ran the model-ladder probe: identical packs from `api_agent` (SYS + user_msg + law-1c warm start + gate feedback) → a Workflow of drafting agents per (arm, fn) with CLI equivalents of grep/read_file/match_one/submit, writing only under `.run/<wave>/<arm>/` → `wave_judge.py` gates each arm with `sweep_parallel` (banked = removed INCLUDE_ASM lines in `git diff`; tree reset between arms; `--union` leaves the cheapest winning drafts in the tree). `tools/workflows/claude_wave_distill.js` = the R16 distill (extract per byte-proven transcript → verify vs the cookbook). Routing rule from T4: ≤50 Sonnet, 51–120 Sonnet, >120 Opus, Opus on residue; haiku retired; DeepSeek optional at ~$0.03/fn. |
|
||||
| | `tools/t5_targets.py` · `tools/t5_bank.sh` | **(Phase 31 S63 T5)** The wave *draw* and the wave *bank* halves around the T4 harness: `t5_targets.py --wave .run/t5x --n 48 [--residue .run/t5w] [--main-only]` draws a routed slate (open per `corpus.stubs`, ledger `.run/t5/drawn.json` keyed (binary,fn), ≤120 ins → sonnet / >120 → opus, residue → opus once, stratified by band, ≤4 per binary, one fn NAME per wave — the harness keys packs/drafts by bare name, so `claude_wave_packs`/`wave_judge` now REFUSE duplicates, R43/R48); then `claude_wave_packs.py` → Workflow `tools/workflows/claude_wave_draft.js` `args={wave, targets}` (one agent per target, `model=target.arm`) → `tools/t5_bank.sh .run/t5x sonnet opus` (refuses if a lane/gate runs or src/ config/ is dirty; `wave_judge --union`; R22 clean fleet sweep parsed `N passed, 0 failed of N`; commit `— N banked`). |
|
||||
| | `tools/t5_targets.py` · `tools/t5_cards.py` · `tools/wave_card_fuel.py` · `tools/t5_bank.sh` | **(Phase 31 S63 T5)** The wave *draw* and the wave *bank* halves around the T4 harness: `t5_targets.py --wave .run/t5x --n 48 [--residue .run/t5w] [--main-only]` draws a routed slate (open per `corpus.stubs`, ledger `.run/t5/drawn.json` keyed (binary,fn), ≤120 ins → sonnet / >120 → opus, residue → opus once, stratified by band, ≤4 per binary, one fn NAME per wave — the harness keys packs/drafts by bare name, so `claude_wave_packs`/`wave_judge` now REFUSE duplicates, R43/R48); then **`t5_cards.py --wave .run/t5x`** BUILDS the target's OWN card fuel (`tu_ref` = in-TU banked neighbours §194-E, `decl_prior` = fleet signatures §196) from `wave_card_fuel.py` — those helpers extracted VERBATIM out of `build_wave_atlas.py` (which parses argv at import and so was unusable as a library) so both consumers share ONE oracle (R33); measured 88% tu_ref / 98% decl_prior on a fresh K-class slate, against 0% correct fuel before the fix. Then `claude_wave_packs.py --cards <wave>/cards.json` → Workflow `tools/workflows/claude_wave_draft.js` `args={wave, targets}` (one agent per target, `model=target.arm`) → `tools/t5_bank.sh .run/t5x sonnet opus` (refuses if a lane/gate runs or src/ config/ is dirty; `wave_judge --union`; then a RECOVERY pass — every un-banked draft through `fix_tu_ret_decls`, which retypes a destination TU's `extern void` decl on a value-returning function and re-gates (wave t5a: 2/2 banked, both byte-exact bodies refused for that decl alone); then the R22 clean fleet sweep parsed `N passed, 0 failed of N`; commit `— N banked`). |
|
||||
| | `tools/jtbl_rodata_pads.py --derive` | **(Phase 31 S62 T3a, cookbook §303)** Module path of the §8e pads filter: the Makefile runs `--derive $(BINARY) --tu <tu>` for every `md_*` object — jump-table pads derived at build time from the retail island + the emission stream (trailing pads `0t1`, table-aware, const data passes through). No stored spec; an anchor miss fails the build with the offset. |
|
||||
| | `tools/blocker_probe.py` | **(Phase 29 SESSION-16, cookbook §65)** WHY a byte-correct draft fails the whole-binary gate. Read-only; **two oracles** (R34): STATIC (`cdecl.parse` + `cdecl.compatible` — cc1's own acceptance question, never text equality) beside the REAL cc1 (via `rtu_match`), leading with the DISAGREEMENT table. Classes `self_decl_hdr`/`self_decl_tu`/`callee_decl`/`data_decl`/`local_type`, each mapped to a blast-radius tier (T0 draft-only / T1 binary-local / T2 fleet-shared). Blockers STACK, so a function's tier is the MAX over them. 36 drafts in ~9 s. Replaced+deleted `.run/diag_plumbing.py`. |
|
||||
| | `tools/symcheck.py` | **(Phase 29 SESSION-18, cookbook §67a)** The pre-gate SYMBOL-SET guard: diffs the symbols a draft's object references (reloc records) against the target `.s`'s `%hi`/`%lo`/`jal` set. Catches the class **every masked oracle is structurally blind to** — `match_one`/`masked_diff` compare relocation-MASKED words (object-vs-`.s` mode is symbol-agnostic by construction) and `rtu_match` **compiles without linking**, so a draft that invents an extern no symbol table defines reads MATCH in both and can never bank (the SESSION-17 `func_801463A0` `_s`-alias trap). Negative-control-proven: on a draft with one renamed data extern, `match_one` reports the SAME 14 mismatched as the correct draft while `symcheck` exits 1 naming both the MISSING and the INVENTED symbol. `--c` (compiles via `match_one`, so the triple can never drift) or `--obj`. A cheap necessary condition, NOT a match oracle — still finish on the byte-gate (G3/P9). |
|
||||
|
||||
@@ -293,20 +293,10 @@ if not PRIORS:
|
||||
levers = set(a.levers.split(','))
|
||||
atlas = json.load(open(a.atlas))
|
||||
|
||||
_open = {}
|
||||
def _stubmap(binary):
|
||||
if binary not in _open:
|
||||
# corpus.stubs() is addr -> Stub; the NAME lives on the record
|
||||
_open[binary] = {st.symbol: st for st in corpus.stubs(binary).values()}
|
||||
return _open[binary]
|
||||
|
||||
def is_open(binary, fn):
|
||||
return fn in _stubmap(binary)
|
||||
|
||||
def home_tu(binary, fn):
|
||||
"""The stub's home .c — this is the GATE GROUP KEY (gate_lane groups by (binary, src))."""
|
||||
st = _stubmap(binary).get(fn)
|
||||
return st.path if st else None
|
||||
# CARD FUEL LIVES IN ONE PLACE (R33, P31 S63): _stubmap/is_open/home_tu/_tu_bodies/tu_neighbours
|
||||
# moved to tools/wave_card_fuel.py so the Claude wave packs (tools/t5_cards.py) build IDENTICAL
|
||||
# fuel. They were unreachable as a library here: this module parses argv at import.
|
||||
from wave_card_fuel import _stubmap, is_open, home_tu, _tu_bodies, tu_neighbours # noqa: F401
|
||||
|
||||
def model_for(nins):
|
||||
if nins <= 50: return 'haiku'
|
||||
@@ -315,84 +305,6 @@ def model_for(nins):
|
||||
|
||||
_DPIDX = DP.load() # symbol -> fleet declaration evidence (§196)
|
||||
knn = atlas.get('knn') or {} # exemplar key -> neighbours; 'M:' entries are BANKED (§193-A)
|
||||
_TU_BODIES = {}
|
||||
|
||||
|
||||
def _tu_bodies(tu_path):
|
||||
"""{fn: body_text} for every function DEFINED in a TU (banked C only -- stubs are INCLUDE_ASM).
|
||||
|
||||
Brace-matched from each definition so a symbol is attributed to the function that uses it, not
|
||||
to the file. Memoized per TU: a wave draws many cards from one .c."""
|
||||
if tu_path in _TU_BODIES:
|
||||
return _TU_BODIES[tu_path]
|
||||
out = {}
|
||||
try:
|
||||
txt = open(tu_path, errors='replace').read()
|
||||
except (OSError, TypeError):
|
||||
_TU_BODIES[tu_path] = out
|
||||
return out
|
||||
import re as _re
|
||||
for m in _re.finditer(r'^[A-Za-z_][\w \t\*]*?\b(\w+)\s*\([^;{]*\)\s*\{', txt, _re.M):
|
||||
i, depth = m.end() - 1, 0
|
||||
while i < len(txt):
|
||||
if txt[i] == '{': depth += 1
|
||||
elif txt[i] == '}':
|
||||
depth -= 1
|
||||
if depth == 0: break
|
||||
i += 1
|
||||
out[m.group(1)] = txt[m.start():i + 1]
|
||||
_TU_BODIES[tu_path] = out
|
||||
return out
|
||||
|
||||
|
||||
_SYM = None
|
||||
|
||||
|
||||
def tu_neighbours(binary, fn, tu_path, asm_file, topn=2):
|
||||
"""Banked functions in the card's OWN TU, ranked by symbols shared with the target's asm.
|
||||
|
||||
The symbols are read from the TARGET's .s (its relocation operands), so this is evidence about
|
||||
the function being drafted, not about the file. See §194-E for why the card needs this at all."""
|
||||
global _SYM
|
||||
if not tu_path or not asm_file:
|
||||
return []
|
||||
import re as _re
|
||||
if _SYM is None:
|
||||
# OPERANDS ONLY. A splat .s carries the encoded WORD in a comment column, so a naive
|
||||
# "[A-Z]\w{3,}" reads `D8FFBD27` and `CC00228E` as symbol names and the overlap score
|
||||
# becomes noise (measured: 34 "symbols" for one function, 31 of them hex words).
|
||||
# Symbols reach the .s in exactly three shapes: a jal target, and %hi()/%lo() operands.
|
||||
_SYM = _re.compile(r'\b(?:jal\s+(\w+)|%[hl][io]\(([\w+]+)\))')
|
||||
try:
|
||||
raw = _SYM.findall(open(asm_file, errors='replace').read())
|
||||
except OSError:
|
||||
return []
|
||||
want = {(a or b).split('+')[0] for a, b in raw if (a or b)}
|
||||
want.discard(fn)
|
||||
if not want:
|
||||
return []
|
||||
scored = []
|
||||
for other, body in _tu_bodies(tu_path).items():
|
||||
if other == fn:
|
||||
continue
|
||||
shared = [s for s in want if s in body]
|
||||
if len(shared) >= 2:
|
||||
scored.append((len(shared), other, sorted(shared)[:6]))
|
||||
scored.sort(key=lambda x: -x[0])
|
||||
if not scored:
|
||||
# FALL BACK TO THE BEST SINGLE SHARED SYMBOL rather than emitting nothing. One shared
|
||||
# callee is weak evidence, but the card reports the count so the agent can weigh it, and a
|
||||
# weak same-TU lead still beats the zero-locality state §194-E measured.
|
||||
weak = sorted(((len([s for s in want if s in body]), other)
|
||||
for other, body in _tu_bodies(tu_path).items() if other != fn), reverse=True)
|
||||
if weak and weak[0][0] == 1:
|
||||
o = weak[0][1]
|
||||
body = _tu_bodies(tu_path)[o]
|
||||
return [{'fn': o, 'shared': 1, 'symbols': [s for s in sorted(want) if s in body][:6]}]
|
||||
return [{'fn': o, 'shared': n, 'symbols': syms} for n, o, syms in scored[:topn]]
|
||||
|
||||
|
||||
|
||||
def _o0_unbankable(spath):
|
||||
"""True iff this target is an -O0 function sitting in a subseg the build compiles -O2.
|
||||
|
||||
|
||||
@@ -6,6 +6,12 @@ prior_draft (law-1c filtered: symbols must overlap the target .s) + gate_feedbac
|
||||
written once. Identical packs are what made the T4 arms comparable; keep it that way.
|
||||
|
||||
Usage: claude_wave_packs.py <targets.json> <out_dir> [--cards cards.json]
|
||||
CARDS ARE KEYED BY (binary, fn), NEVER BY BARE NAME (R48; byte-measured P31 S63). Overlays share
|
||||
function NAMES at equal addresses, and a K-class target has never been carded, so a name-keyed
|
||||
lookup over `.run/wave_*_cards.json` returns SOME OTHER overlay's same-named function every time:
|
||||
in wave t5a it mis-carded 48 of 48 targets, and nine agents independently reported "the pack's
|
||||
warm-start/twin/decl hints were a different function". A card whose binary does not match the
|
||||
target is DROPPED and counted, never silently substituted (R32/R43).
|
||||
targets.json: [{name, binary, nins, sub, asm, tu}, ...] (api_agent's target shape; see
|
||||
tools/main_queue_rebuild.py / .run/t4/targets.json for how to derive one from corpus)
|
||||
out_dir/SYS.md + out_dir/packs/<fn>.md are produced; the workflow script
|
||||
@@ -29,22 +35,28 @@ def main():
|
||||
dup = [n for n, c in __import__('collections').Counter(t['name'] for t in targets).items() if c > 1]
|
||||
if dup: # packs are keyed by bare fn name; two binaries' same-named fns would overwrite (R43/R48)
|
||||
sys.exit('REFUSED: %d duplicate fn name(s) in one wave (packs are name-keyed): %s' % (len(dup), ' '.join(dup)))
|
||||
cards = {}
|
||||
cards, wrong_bin = {}, 0
|
||||
def add(e):
|
||||
nonlocal wrong_bin
|
||||
if not (isinstance(e, dict) and e.get('fn')):
|
||||
return
|
||||
b = e.get('binary') or e.get('b')
|
||||
if not b: # a card that cannot prove its binary cannot be trusted to a target
|
||||
wrong_bin += 1; return
|
||||
cards[(b, e['fn'])] = e
|
||||
if '--cards' in sys.argv:
|
||||
for c in json.load(open(sys.argv[sys.argv.index('--cards') + 1])):
|
||||
if c.get('fn'): cards[c['fn']] = c
|
||||
else: # newest per-fn card across every wave card file
|
||||
for c in json.load(open(sys.argv[sys.argv.index('--cards') + 1])): add(c)
|
||||
else: # newest per-(binary,fn) card across every wave card file
|
||||
for f in sorted(glob.glob('.run/wave_*_cards.json'), key=os.path.getmtime):
|
||||
try:
|
||||
for e in json.load(open(f)):
|
||||
if isinstance(e, dict) and e.get('fn'): cards[e['fn']] = e
|
||||
for e in json.load(open(f)): add(e)
|
||||
except Exception:
|
||||
pass
|
||||
os.makedirs(os.path.join(out, 'packs'), exist_ok=True)
|
||||
open(os.path.join(out, 'SYS.md'), 'w').write(A.SYS)
|
||||
n = 0
|
||||
for t in targets:
|
||||
um = A.user_msg(t, cards.get(t['name']))
|
||||
um = A.user_msg(t, cards.get((t['binary'], t['name'])))
|
||||
pd, where = A.prior_draft(t)
|
||||
if pd:
|
||||
um += ("\n\nA PREVIOUS ATTEMPT on this exact function left this body behind (%s). The whole-binary "
|
||||
@@ -55,7 +67,12 @@ def main():
|
||||
if gf:
|
||||
um += '\n\n' + gf + '\n'
|
||||
open(os.path.join(out, 'packs', t['name'] + '.md'), 'w').write(um); n += 1
|
||||
print('packs: %d written to %s (cards matched %d/%d)' % (n, out, sum(1 for t in targets if t['name'] in cards), len(targets)))
|
||||
hit = sum(1 for t in targets if (t['binary'], t['name']) in cards)
|
||||
byname = sum(1 for t in targets if any(k[1] == t['name'] for k in cards))
|
||||
print('packs: %d written to %s' % (n, out))
|
||||
print('cards: %d/%d matched on (binary, fn); %d target(s) have a same-named card in ANOTHER binary '
|
||||
'(DROPPED, not substituted — R48); %d card(s) skipped for carrying no binary'
|
||||
% (hit, len(targets), byname - hit, wrong_bin))
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
+38
-3
@@ -4,9 +4,14 @@
|
||||
# src/ or config/ is already dirty (R42: never blind-revert; someone's banks may be uncommitted).
|
||||
# 2. WAVE=<wave> tools/wave_judge.py <arms> --union : whole-binary gate per arm (git-diff bank
|
||||
# detection, tree reset between arms), then the union gated once and LEFT IN THE TREE.
|
||||
# 3. R22 clean fleet sweep (make clean -> extract-all -> check-all); the pass line must read
|
||||
# 3. RECOVER the un-banked: a draft whose BODY is byte-correct but whose destination TU declares
|
||||
# the function `extern void ...` when it returns a value is refused at closeness 0 with no
|
||||
# codegen fault at all. `fix_tu_ret_decls` retypes those decls, re-gates, and restores every
|
||||
# edit the gate does not pay for. Measured on wave t5a: 2 of 2 such refusals banked, both
|
||||
# self-reported MATCH and both confirmed byte-exact by match_one in isolation.
|
||||
# 4. R22 clean fleet sweep (make clean -> extract-all -> check-all); the pass line must read
|
||||
# "N passed, 0 failed of N" (exit codes checked, R53) or nothing is committed.
|
||||
# 4. commit the union; the subject carries "— N banked" (the maintenance lane's exemplar counter).
|
||||
# 5. commit the union; the subject carries "— N banked" (the maintenance lane's exemplar counter).
|
||||
# Drafting must be FINISHED before this runs: the judge rewrites src/ and the sweep deletes asm/.
|
||||
cd /home/musashi/bfm-decomp || exit 2
|
||||
WAVE=${1:?usage: t5_bank.sh <wave_dir> [arms...]}; shift
|
||||
@@ -23,8 +28,38 @@ WAVE="$WAVE" .venv/bin/python tools/wave_judge.py $ARMS --union 2>&1 | tee "$WAV
|
||||
[ "${PIPESTATUS[0]}" = 0 ] || { say "judge exited non-zero — stopping (tree left as is)"; exit 3; }
|
||||
N=$(python3 -c "import json;j=json.load(open('$WAVE/judge.json'));print(len(j.get('union',{}).get('banked',[])))")
|
||||
SUMMARY=$(python3 -c "import json;j=json.load(open('$WAVE/judge.json'));print(', '.join('%s %d/%d'%(a,v['n_banked'],v['drafts']) for a,v in j['arms'].items()))")
|
||||
# ---- recovery pass over everything the union did not bank (see step 3) --------------------------
|
||||
UNB=$(.venv/bin/python - "$WAVE" <<'PY'
|
||||
import json, os, shutil, sys
|
||||
w = sys.argv[1]
|
||||
tg = json.load(open(w + '/targets.json')); j = json.load(open(w + '/judge.json'))
|
||||
banked = set(j.get('union', {}).get('banked', []))
|
||||
for v in j.get('arms', {}).values(): banked |= set(v.get('banked', []))
|
||||
best = j.get('union', {}).get('picked', {})
|
||||
arms = list(j.get('arms', {}))
|
||||
rec = w + '/recover'; shutil.rmtree(rec, ignore_errors=True)
|
||||
pairs = []
|
||||
for t in tg:
|
||||
if t['name'] in banked: continue
|
||||
for arm in [best.get(t['name'])] + arms:
|
||||
src = w + '/%s/%s.c' % (arm, t['name']) if arm else ''
|
||||
if src and os.path.exists(src) and os.path.getsize(src):
|
||||
os.makedirs('%s/%s' % (rec, t['binary']), exist_ok=True)
|
||||
shutil.copy(src, '%s/%s/%s.c' % (rec, t['binary'], t['name']))
|
||||
pairs.append([t['binary'], t['name']]); break
|
||||
json.dump(pairs, open(w + '/ret_pairs.json', 'w'))
|
||||
print(len(pairs))
|
||||
PY
|
||||
)
|
||||
if [ "${UNB:-0}" != 0 ]; then
|
||||
say "recovery: $UNB un-banked draft(s) -> fix_tu_ret_decls (TU 'extern void' on a value-returning fn)"
|
||||
flock .run/auto/draw.lock .venv/bin/python tools/fix_tu_ret_decls.py --pairs-file "$WAVE/ret_pairs.json" \
|
||||
--drafts "$WAVE/recover" -j 4 2>&1 | tee "$WAVE/recover.log" | tail -3
|
||||
N=$(git diff -U0 -- src/ | grep -cE '^-INCLUDE_ASM' || true)
|
||||
SUMMARY="$SUMMARY, +recovery"
|
||||
fi
|
||||
if [ "$N" = 0 ]; then say "0 banked ($SUMMARY) — nothing to commit"; git status --short -- src/ config/ | head; exit 0; fi
|
||||
say "$N banked in the union ($SUMMARY) — R22 clean fleet sweep"
|
||||
say "$N banked total ($SUMMARY) — R22 clean fleet sweep"
|
||||
( make clean && make extract-all JOBS=32 && make check-all JOBS=32 ) > "$WAVE/sweep.log" 2>&1
|
||||
RC=$?
|
||||
PASS=$(grep -E '^check-all: ' "$WAVE/sweep.log" | tail -1)
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tools/t5_cards.py — BUILD the card fuel for a wave's own targets (P31 S63 T5).
|
||||
|
||||
Usage: t5_cards.py --wave .run/t5b [--out <wave>/cards.json]
|
||||
|
||||
WHY (byte-measured, P31 S63 wave t5a). `claude_wave_packs.py` looked cards up by BARE FUNCTION NAME
|
||||
across `.run/wave_*_cards.json`; overlays share names at equal addresses, so 48 of 48 t5a targets
|
||||
(and 15 of T4's 20) got ANOTHER binary's card — wrong banked twin, wrong TU neighbours, wrong
|
||||
declarations. Nine agents independently reported discarding it. Keying by (binary, fn) fixes the
|
||||
mis-attribution but leaves K-class ("never touched") targets with NO fuel at all, and `_fuel()`
|
||||
then emits nothing: no `tu_ref`, no `decl_prior`. Yet t5a's agents cracked function after function
|
||||
by finding an already-banked sibling in their own TU BY HAND ("in-TU twin ~150 lines earlier",
|
||||
"structural twin in the SAME TU") — which is exactly what `tu_ref` is (§194-E, 62% of targets).
|
||||
|
||||
So this builds the two computable fields for the target ITSELF, from the same oracle
|
||||
`build_wave_atlas` uses (`tools/wave_card_fuel.py`, R33):
|
||||
tu_ref already-banked functions in the target's OWN TU, ranked by symbols shared with the
|
||||
target's .s relocations
|
||||
decl_prior the fleet's consensus signature for every symbol the .s references (§196) — kills
|
||||
the two-arity A/B compile §195-A proved the asm cannot answer
|
||||
`seed_ref` (the cross-TU banked twin) needs the atlas knn and is NOT built here: a card with no
|
||||
seed_ref makes `_fuel` say "no banked twin — derive from the .s", which is TRUE, where the old
|
||||
name-keyed card pointed confidently at an unrelated function. Coverage is printed (R32/R41)."""
|
||||
import argparse, json, os, sys
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))); os.chdir(REPO)
|
||||
sys.path.insert(0, 'tools')
|
||||
import wave_card_fuel as F
|
||||
import decl_prior as DP
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--wave', required=True)
|
||||
ap.add_argument('--out', default='')
|
||||
a = ap.parse_args()
|
||||
targets = json.load(open(os.path.join(a.wave, 'targets.json')))
|
||||
idx = DP.load()
|
||||
cards, n_tu, n_dp = [], 0, 0
|
||||
for t in targets:
|
||||
tu = t.get('tu') or F.home_tu(t['binary'], t['name'])
|
||||
tu_ref = F.tu_neighbours(t['binary'], t['name'], tu, t['asm'])
|
||||
try:
|
||||
dp = DP.for_asm(t['asm'], tu, idx=idx, binary=t['binary'])
|
||||
except Exception as e:
|
||||
print(' decl_prior failed for %s/%s: %s' % (t['binary'], t['name'], str(e)[:80])); dp = []
|
||||
n_tu += bool(tu_ref); n_dp += bool(dp)
|
||||
cards.append({'fn': t['name'], 'binary': t['binary'], 'addr': t.get('addr'), 'nins': t['nins'],
|
||||
'sub': t['sub'], 'tu_ref': tu_ref, 'decl_prior': dp})
|
||||
p = a.out or os.path.join(a.wave, 'cards.json')
|
||||
json.dump(cards, open(p, 'w'), indent=1)
|
||||
print('cards: %d built for %s (tu_ref on %d/%d = %.0f%%; decl_prior on %d/%d = %.0f%%; seed_ref not built — '
|
||||
'the pack will honestly say "no banked twin")'
|
||||
% (len(cards), a.wave, n_tu, len(targets), 100.0 * n_tu / max(1, len(targets)),
|
||||
n_dp, len(targets), 100.0 * n_dp / max(1, len(targets))))
|
||||
print('wrote %s' % p)
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tools/t5_distill_args.py — build the R16 distill slate for a Claude wave (P31 S63 T5).
|
||||
|
||||
Usage: t5_distill_args.py --wave .run/t5a --wfdir <workflow transcript dir> [--label 'P31 S63 t5a'] [--all-arms]
|
||||
|
||||
Only BYTE-PROVEN transcripts are distilled (the distiller's contract: "the whole-binary byte-gate
|
||||
ACCEPTED the final draft, so the final body is ground truth"), so the slate = the wave's banked set
|
||||
from `<wave>/judge.json`, joined to the drafting agents' transcripts via the workflow's
|
||||
`journal.jsonl` (result rows carry fn/arm/note; the transcript is `agent-<agentId>.jsonl`).
|
||||
By default only the arm that BANKED each fn is distilled (--all-arms keeps every arm's attempt).
|
||||
Coverage is asserted (R32): a banked fn with no transcript is reported and counted, never skipped
|
||||
silently. Writes <wave>/distill_args.json for
|
||||
Workflow(scriptPath=tools/workflows/claude_wave_distill.js, args=<that file's content>)."""
|
||||
import argparse, json, os, sys
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__))); os.chdir(REPO)
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--wave', required=True)
|
||||
ap.add_argument('--wfdir', required=True, help="the drafting run's transcript dir (holds journal.jsonl + agent-*.jsonl)")
|
||||
ap.add_argument('--label', default='', help="provenance string for the cookbook header, e.g. 'P31 S63 t5a'")
|
||||
ap.add_argument('--all-arms', action='store_true')
|
||||
ap.add_argument('--out', default='')
|
||||
a = ap.parse_args()
|
||||
jd = json.load(open(os.path.join(a.wave, 'judge.json')))
|
||||
banked_by_arm = {arm: set(v.get('banked', [])) for arm, v in jd.get('arms', {}).items()}
|
||||
banked = set().union(*banked_by_arm.values()) if banked_by_arm else set()
|
||||
union = set(jd.get('union', {}).get('banked', []))
|
||||
picked = jd.get('union', {}).get('picked', {})
|
||||
jp = os.path.join(a.wfdir, 'journal.jsonl')
|
||||
if not os.path.exists(jp): sys.exit('no journal.jsonl in %s' % a.wfdir)
|
||||
rows = [json.loads(l) for l in open(jp) if l.strip()]
|
||||
tr = {}
|
||||
for r in rows:
|
||||
res = r.get('result') or {}
|
||||
if isinstance(res, dict) and res.get('fn'):
|
||||
tr[(res['fn'], res.get('arm'))] = (os.path.join(a.wfdir, 'agent-%s.jsonl' % r['agentId']), res.get('note', ''))
|
||||
out, missing = [], []
|
||||
for fn in sorted(banked):
|
||||
arms = sorted(banked_by_arm) if a.all_arms else [picked.get(fn) or next(x for x in banked_by_arm if fn in banked_by_arm[x])]
|
||||
for arm in arms:
|
||||
if fn not in banked_by_arm.get(arm, ()): continue
|
||||
hit = tr.get((fn, arm))
|
||||
if not hit or not os.path.exists(hit[0]): missing.append('%s/%s' % (arm, fn)); continue
|
||||
out.append({'fn': fn, 'arm': arm, 'transcript': hit[0], 'note': hit[1]})
|
||||
print('banked %d (union-gated %d); transcripts found %d; label %r' % (len(banked), len(union), len(out), a.label))
|
||||
if missing:
|
||||
print('NO TRANSCRIPT for %d banked draft(s) — NOT distilled (R32, reported not skipped): %s'
|
||||
% (len(missing), ' '.join(missing[:20])))
|
||||
p = a.out or os.path.join(a.wave, 'distill_args.json')
|
||||
json.dump({'targets': out, 'label': a.label or os.path.basename(a.wave.rstrip('/'))}, open(p, 'w'), indent=1)
|
||||
print('wrote %s' % p)
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env python3
|
||||
"""tools/wave_card_fuel.py — the per-target CARD FUEL an agent pack carries, as ONE oracle (R33).
|
||||
|
||||
These helpers were defined inside `build_wave_atlas.py`, which parses argv at import and so cannot
|
||||
be reused as a library. Extracted VERBATIM (P31 S63) so `build_wave_atlas` and the Claude wave's
|
||||
`tools/t5_cards.py` compute the SAME fuel rather than two copies that drift:
|
||||
|
||||
home_tu(binary, fn) the stub's home .c (the gate group key)
|
||||
_tu_bodies(tu_path) {fn: body} for every function DEFINED (banked) in a TU
|
||||
tu_neighbours(...) already-banked functions in the target's OWN TU, ranked by symbols
|
||||
shared with the target's .s relocations (§194-E, the highest-yield
|
||||
source measured; wave t5a's agents rediscovered it by hand and said so)
|
||||
|
||||
Why it exists (byte-measured, P31 S63): `claude_wave_packs` looked cards up by BARE FUNCTION NAME,
|
||||
and overlays share names at equal addresses — so ALL 48 of wave t5a's targets (and 15 of T4's 20)
|
||||
received ANOTHER binary's card: wrong twin, wrong TU neighbours, wrong declarations. Nine agents
|
||||
reported it independently. The fix is a (binary, fn) key — which, for never-carded K-class work,
|
||||
means there is no card at all. So the fuel has to be BUILT for the target itself, from here."""
|
||||
import os, sys
|
||||
REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
sys.path.insert(0, os.path.join(REPO, 'tools'))
|
||||
import corpus
|
||||
|
||||
_open = {}
|
||||
def _stubmap(binary):
|
||||
if binary not in _open:
|
||||
# corpus.stubs() is addr -> Stub; the NAME lives on the record
|
||||
_open[binary] = {st.symbol: st for st in corpus.stubs(binary).values()}
|
||||
return _open[binary]
|
||||
def is_open(binary, fn):
|
||||
return fn in _stubmap(binary)
|
||||
def home_tu(binary, fn):
|
||||
"""The stub's home .c — this is the GATE GROUP KEY (gate_lane groups by (binary, src))."""
|
||||
st = _stubmap(binary).get(fn)
|
||||
return st.path if st else None
|
||||
_TU_BODIES = {}
|
||||
def _tu_bodies(tu_path):
|
||||
"""{fn: body_text} for every function DEFINED in a TU (banked C only -- stubs are INCLUDE_ASM).
|
||||
|
||||
Brace-matched from each definition so a symbol is attributed to the function that uses it, not
|
||||
to the file. Memoized per TU: a wave draws many cards from one .c."""
|
||||
if tu_path in _TU_BODIES:
|
||||
return _TU_BODIES[tu_path]
|
||||
out = {}
|
||||
try:
|
||||
txt = open(tu_path, errors='replace').read()
|
||||
except (OSError, TypeError):
|
||||
_TU_BODIES[tu_path] = out
|
||||
return out
|
||||
import re as _re
|
||||
for m in _re.finditer(r'^[A-Za-z_][\w \t\*]*?\b(\w+)\s*\([^;{]*\)\s*\{', txt, _re.M):
|
||||
i, depth = m.end() - 1, 0
|
||||
while i < len(txt):
|
||||
if txt[i] == '{': depth += 1
|
||||
elif txt[i] == '}':
|
||||
depth -= 1
|
||||
if depth == 0: break
|
||||
i += 1
|
||||
out[m.group(1)] = txt[m.start():i + 1]
|
||||
_TU_BODIES[tu_path] = out
|
||||
return out
|
||||
_SYM = None
|
||||
def tu_neighbours(binary, fn, tu_path, asm_file, topn=2):
|
||||
"""Banked functions in the card's OWN TU, ranked by symbols shared with the target's asm.
|
||||
|
||||
The symbols are read from the TARGET's .s (its relocation operands), so this is evidence about
|
||||
the function being drafted, not about the file. See §194-E for why the card needs this at all."""
|
||||
global _SYM
|
||||
if not tu_path or not asm_file:
|
||||
return []
|
||||
import re as _re
|
||||
if _SYM is None:
|
||||
# OPERANDS ONLY. A splat .s carries the encoded WORD in a comment column, so a naive
|
||||
# "[A-Z]\w{3,}" reads `D8FFBD27` and `CC00228E` as symbol names and the overlap score
|
||||
# becomes noise (measured: 34 "symbols" for one function, 31 of them hex words).
|
||||
# Symbols reach the .s in exactly three shapes: a jal target, and %hi()/%lo() operands.
|
||||
_SYM = _re.compile(r'\b(?:jal\s+(\w+)|%[hl][io]\(([\w+]+)\))')
|
||||
try:
|
||||
raw = _SYM.findall(open(asm_file, errors='replace').read())
|
||||
except OSError:
|
||||
return []
|
||||
want = {(a or b).split('+')[0] for a, b in raw if (a or b)}
|
||||
want.discard(fn)
|
||||
if not want:
|
||||
return []
|
||||
scored = []
|
||||
for other, body in _tu_bodies(tu_path).items():
|
||||
if other == fn:
|
||||
continue
|
||||
shared = [s for s in want if s in body]
|
||||
if len(shared) >= 2:
|
||||
scored.append((len(shared), other, sorted(shared)[:6]))
|
||||
scored.sort(key=lambda x: -x[0])
|
||||
if not scored:
|
||||
# FALL BACK TO THE BEST SINGLE SHARED SYMBOL rather than emitting nothing. One shared
|
||||
# callee is weak evidence, but the card reports the count so the agent can weigh it, and a
|
||||
# weak same-TU lead still beats the zero-locality state §194-E measured.
|
||||
weak = sorted(((len([s for s in want if s in body]), other)
|
||||
for other, body in _tu_bodies(tu_path).items() if other != fn), reverse=True)
|
||||
if weak and weak[0][0] == 1:
|
||||
o = weak[0][1]
|
||||
body = _tu_bodies(tu_path)[o]
|
||||
return [{'fn': o, 'shared': 1, 'symbols': [s for s in sorted(want) if s in body][:6]}]
|
||||
return [{'fn': o, 'shared': n, 'symbols': syms} for n, o, syms in scored[:topn]]
|
||||
@@ -1,13 +1,14 @@
|
||||
export const meta = {
|
||||
name: 't4-idiom-distill',
|
||||
description: 'R16 flywheel: distill the unlocking lesson from each byte-proven T4 transcript, verify novelty against the cookbook, return §-formatted candidates',
|
||||
name: 'claude-wave-idiom-distill',
|
||||
description: 'R16 flywheel: distill the unlocking lesson from each byte-proven wave transcript, verify novelty against the cookbook, return §-formatted candidates',
|
||||
phases: [
|
||||
{ title: 'Extract', detail: '31 transcripts -> one lesson each' },
|
||||
{ title: 'Extract', detail: 'one distiller per byte-proven transcript -> one lesson each' },
|
||||
{ title: 'Verify', detail: 'novel claims grepped against docs/matching-cookbook.md' },
|
||||
],
|
||||
}
|
||||
// args: { targets: [{fn, arm, transcript, note}, ...], label: 'P31 S63 t5a' } — from tools/t5_distill_args.py
|
||||
const REPO = '/home/musashi/bfm-decomp'
|
||||
const WFDIR = '/home/musashi/.claude/projects/-home-musashi-bfm-decomp/b0026d2d-2724-4aed-813a-3deef224d210/subagents/workflows/wf_c691ef44-320/'
|
||||
const LABEL = args.label || 'P31 S63'
|
||||
const LESSON = {
|
||||
type: 'object',
|
||||
properties: {
|
||||
@@ -38,7 +39,7 @@ const lessons = await pipeline(
|
||||
args.targets,
|
||||
t => agent(
|
||||
`You are distilling ONE byte-proven decompilation transcript for the project's matching cookbook (docs/matching-cookbook.md, gcc-2.7.2 MIPS idioms). Repo: ${REPO}.
|
||||
Transcript (JSONL, an agent's full tool trace): ${t.transcript.replace("WF/", WFDIR)}
|
||||
Transcript (JSONL, an agent's full tool trace): ${t.transcript}
|
||||
Function: ${t.fn} (arm ${t.arm}); the whole-binary byte-gate ACCEPTED the final draft, so the final body is ground truth. The agent's own closing note was: "${t.note}"
|
||||
|
||||
Read the transcript. Find the moment(s) the masked diff (match_one closeness) dropped to 0 and what C change caused it. Report ONE lesson — the most generalizable one — as:
|
||||
@@ -64,8 +65,8 @@ Read only; write nothing.`,
|
||||
refs the agent used: ${JSON.stringify(lesson.cookbook_refs_used)}
|
||||
Check docs/matching-cookbook.md (grep for the mechanism, the instruction pattern, the C construct; docs/cookbook-index.md is symptom-keyed) and decide:
|
||||
COVERED — an existing § already states this law (name it in covered_by)
|
||||
ADDENDUM — an existing § is close but this adds a byte-proven refinement (name it; entry_markdown = a 3-8 line addendum paragraph starting "**Addendum (P31 S62 T4, ${t.fn}):**")
|
||||
NEW — no § states it; entry_markdown = a full section in the book's style, header "## §NNN — <TITLE IN CAPS> (P31 S62 T4; byte-proven ${t.fn})" with NNN left literally as NNN, then the tell, the mechanism, the C shape, the evidence line
|
||||
ADDENDUM — an existing § is close but this adds a byte-proven refinement (name it; entry_markdown = a 3-8 line addendum paragraph starting "**Addendum (${LABEL}, ${t.fn}):**")
|
||||
NEW — no § states it; entry_markdown = a full section in the book's style, header "## §NNN — <TITLE IN CAPS> (${LABEL}; byte-proven ${t.fn})" with NNN left literally as NNN, then the tell, the mechanism, the C shape, the evidence line
|
||||
REFUTED — the claim contradicts a byte-proven § or the evidence does not support it (say why)
|
||||
Quote the § numbers you checked in why. Read only; write nothing.`,
|
||||
{ label: `verify:${t.fn}`, phase: 'Verify', model: 'opus', schema: VERDICT }).then(v => ({ ...lesson, ...(v || { verdict: 'UNVERIFIED', why: 'verifier returned null' }) }))
|
||||
@@ -75,4 +76,4 @@ const out = lessons.filter(Boolean)
|
||||
const tally = {}
|
||||
for (const l of out) tally[l.verdict] = (tally[l.verdict] || 0) + 1
|
||||
log('verdicts: ' + JSON.stringify(tally))
|
||||
return { lessons: out }
|
||||
return { label: LABEL, lessons: out }
|
||||
Reference in New Issue
Block a user