Drew T
881dd5a37c
The exact-hash twin tier found 22 of 352 reachable open stubs (6%). The edit-distance band added by `seed_ref --near` finds 75 of 352 (21%) — 3.4x — on a corpus we believed fully mined. 31 of the new rows were PURE reloc-only twins of already-banked bodies; 8 banked the same day at ~0 agent tokens, one 94-ins exemplar serving five open copies. * cookbook §389: the h_norm hole (norm_stream drops its pending lui-hi on an intervening R-type, so indexed-global reloc twins hash differently and vanish from seed_ref/twin_sweep/dedup/family-maps at once). Do NOT fix h_norm — every stored calibration keys on it; the near tier reads through it. * accelerators #17: the generalisable law. A similarity hash built for DEDUP under-matches by design, which is correct for dedup and silently lossy as a FRONTIER join — the two questions want opposite error directions, and the frontier failure looks exactly like "this function is unique". * generic-decomp-package §2b: build the near band at the same time as the exact tier, with the three verifications. It pays from the first bank for a new project, where we paid a session to recover the debt. * SETUP inventory row + playbook §2a (run it before believing any "no twin" verdict; never send a RELOC-ONLY row to a drafting agent).
6.3 KiB
The Generic Decomp Package — what a NEW decompilation should inherit on day one
Status: the thesis, recorded S68 (2026-08-31) by Drew, from hindsight over this project. Feeds the endgame deliverables (the retrospective + the public "how to AI-decomp" wiki). This is NOT a plan for BFM. It is what the NEXT project starts with instead of starting empty.
The claim
This project spent most of its life brute-forcing functions and then widening tooling whenever a new idiom revealed a population the tooling could not see. In hindsight that order is backwards. A new decomp should spend its FIRST phases building the wide tooling and seeding the knowledge base, and only then start cracking — because every tool built early pays on every function afterwards, while every function cracked early pays once.
The evidence is this project's own zero-token banks: whole classes (twins, families, siblings,
cousins, -O0 carves, propagation, stranded boundaries) that cost nothing per function ONCE the
tool existed — and that were invisible until an idiom taught us to look.
What the next project inherits, and does BEFORE cracking
1. The knowledge base, seeded from sources that exist before any match does.
- Mine the actual COMPILER SOURCE for the target triple. This project's highest-value late idioms
(§368 reload-remat, §372 copy-capture, §370's
schedule_selectbound, §373'spri(asm)=1) came from readinggcc-2.7.2's own passes —reload1.c,cse.c,local-alloc.c,sched.c,stmt.c. None of that required a single matched function. It could have been mined in week 1. - Mine SIBLING PROJECTS on the same compiler (this project used Vagrant Story / sotn-decomp).
- Carry
docs/matching-cookbook.md(399 sections) +cookbook-index.md(the symptom→section table) across as the starting corpus, adapted for the new triple rather than rebuilt.
2. The structural tooling, before the first crack.
The families/twins/dedup layer is what converts one crack into N banks. In BFM this arrived late and
retroactively harvested thousands of instructions. Port it first:
corpus (the coverage oracle) · seed_ref (exact-hash twin join AND its --near edit-distance
band — see 2b) · family_remap /
family_sweep · dedup_propagate (position-locked overlay sharing) · the -O0/opt-level carve
chain (o0_detect, o0_subsplit, o0_boundary) · wall_sweep (toolchain walls) · the draw
filter · the byte-gate + clean-fleet verifier.
2b. THE SIMILARITY JOIN MUST BE A BAND, NOT A POINT (P31 S69 — port this, it is cheap and it compounds). A twin oracle keyed on an exact signature hash answers only "is there a byte-identical copy?". That is the wrong question for a frontier. The right one is "is there anything CLOSE?", and the difference is not marginal:
| tier | reachable open stubs with a banked match |
|---|---|
| exact hash (d=0) — where this project sat for 60+ sessions | 22 of 352 (6%) |
edit-distance band to d<=25 (seed_ref --near) |
75 of 352 (21%) |
A 3.4x widening, found in one agent-run, on a corpus we thought was fully mined. 31 of the new rows were PURE reloc-only twins of already-banked bodies — free work that had been sitting invisible; 8 banked the same day at ~0 agent tokens. The root cause was a normalizer that under-matched by design (§389): safe for dedup, silently lossy as a frontier join.
For a NEW decomp this matters more than it did here, because the band pays from the very first banks: every function you crack immediately becomes a potential exemplar for everything within a few instructions of it, and you never accumulate the invisible-singleton debt this project spent a session recovering. Build the near tier at the same time as the exact tier — not sixty sessions later. Concretely: normalize relocations out of the instruction stream, prefilter soundly on length/opcode-histogram so no true pair can be lost, then edit-distance the survivors; assert the population (R32), cross-check that the band reproduces every exact-hash pair (R34), and control against random pairs for the base rate (R39: 1.17% here).
And audit every hash you own for BOTH questions. Dedup wants under-matching; a frontier join wants over-matching. One hash cannot serve both error directions, and the failure is silent.
3. The differential-oracle harness (accelerators #15) — the one that works at 0%. Two independent paths per question, disagreement fails loudly, on a schedule.
4. The periodic widening review (Drew's addition, and the part this project did only by accident). At every session/phase close: review the tooling against the idioms learned that phase and ask "which scanner's denominator just got wider?" New idioms do not only make the next crack easier — they retroactively convert already-open functions into free banks, but ONLY if a tool is widened to see them. S68's §332 sweep is the worked example.
The order this implies
phase 0 compiler-source + sibling-project idiom mining -> seed the cookbook
phase 1 structural tooling: corpus, families, twins, dedup, carves, walls, byte gate
phase 2 the differential-oracle harness + the draw filter
phase 3 FIRST cracks — and from here every crack feeds the widening review
... every phase close: idioms -> tooling widening -> free banks
The honest caveat
Tooling-first does not remove the hard tail. This project's remaining frontier at S68 was 418 functions / 62,717 instructions, of which only ~5% was mechanically free and 947 instructions were permanently unbankable from C (toolchain walls). The structural work — §366 case-label unstacking, §368's uncolorable local, §358's unreferenced aggregate — needed genuine reasoning and always will. Tooling-first makes the cheap half nearly free and stops the waste; it does not shrink the hard half. Sell it as that and it is true; sell it as "no hand-cracking" and it is not.
Where the pieces live today
docs/matching-cookbook.md + docs/cookbook-index.md (the knowledge) ·
docs/accelerators.md (hindsight tools, #15 is the day-one one) ·
docs/wave-playbook.md (the running procedure, each guard paired with the measurement that earned it) ·
docs/decision-log.md (WHY each pivot happened) · tools/ (the toolset) ·
phase-ends/ (the build history the retrospective is reconstructed from).