From a0d4ae8368bc51b22ef7d8c40468b6345db7f465 Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Mon, 7 Sep 2026 17:43:56 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-33.5):=20task=205=20=E2=80=94=20the?= =?UTF-8?q?=20six=20INTEGRATE=20documents=20folded=20into=20the=20wiki=20a?= =?UTF-8?q?nd=20how-to=20(three=20agent=20diffs):=20'Matching=20by=20hand'?= =?UTF-8?q?=20in=20The-matching-workflow=20(two=20decompiles=20+=20the=20a?= =?UTF-8?q?sm=20as=20arbiter,=20the=20loop,=20THE=20FIVE=20SIGNATURE=20MOV?= =?UTF-8?q?ES,=20choosing=20targets,=20running=20it=20as=20a=20wave);=20ch?= =?UTF-8?q?.07=20'Levers=20from=20the=20hand-matching=20era';=20ch.02=20co?= =?UTF-8?q?nflicts-before-the-gate=20+=20the=20red-gate=20diagnosis=20+=20?= =?UTF-8?q?propagation=20as=20a=20gate;=20ch.09=20'A=20wave,=20priced'=20/?= =?UTF-8?q?=20'Instruction=20weight,=20not=20head=20count'=20/=20'The=20ga?= =?UTF-8?q?te's=20cost=20is=20paid=20in=20wall-clock'=20+=20yield=20rows?= =?UTF-8?q?=20with=20denominators;=20ch.05=20the=20banked=20twin,=20face?= =?UTF-8?q?=20vs=20delivered=20mass,=20pipeline=20vs=20batch,=20gate=20the?= =?UTF-8?q?=20directory,=20step=206b=20Reconcile,=20the=20harvest=20yield?= =?UTF-8?q?=20series;=20the=20S37=20execution=20block=20into=20wave-playbo?= =?UTF-8?q?ok=20=C2=A75;=20ch.03=20the=20permuter-failures=20paragraph=20+?= =?UTF-8?q?=20the=20band=20attribution;=20ch.04=20cites=20tooling-audit;?= =?UTF-8?q?=2011=20referrers=20re-pointed=20(0=20links=20to=20the=20four?= =?UTF-8?q?=20leavers=20remain);=2012=20stale=20classes=20and=202=20supers?= =?UTF-8?q?eded=20figures=20deliberately=20not=20folded;=20log=20+=20check?= =?UTF-8?q?point?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/SETUP.md | 3 +- docs/accelerators.md | 3 +- docs/gen3-handoff.md | 2 +- docs/how-to-ai-decomp/00-README.md | 9 +- docs/how-to-ai-decomp/02-byte-gate.md | 19 ++- docs/how-to-ai-decomp/03-bootstrap-order.md | 10 +- .../04-oracles-and-instruments.md | 4 + docs/how-to-ai-decomp/05-cards-lanes-waves.md | 78 +++++++++-- docs/how-to-ai-decomp/07-compiler-source.md | 30 +++++ docs/how-to-ai-decomp/09-economics.md | 76 ++++++++++- docs/permuter-ils.md | 4 +- docs/retrospective.md | 15 ++- docs/wave-playbook.md | 67 +++++++++- docs/wiki/The-matching-workflow.md | 62 +++++++++ phase-ends/CURRENT_PHASE.md | 126 +++++++++++++----- 15 files changed, 440 insertions(+), 68 deletions(-) diff --git a/docs/SETUP.md b/docs/SETUP.md index 45d787604..18d572091 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -1309,7 +1309,8 @@ fills fast). Nothing is leaking — but the host does not get the memory back on renderer rewrites links rather than the authors avoiding them; the sync replaces the wiki's page set wholesale. Measured S88: 25 pages rendered, 264 relative links rewritten, 1,896 lines / 163 KB of source; `doc_links --strict`: 40 documents, 290 links, 0 broken; dry run before the wiki exists → "not clonable yet … dry run OK", exit 0. - **Sources the pages were distilled from** (no new facts were invented; every number is a recorded measurement): - `docs/portable-decomp-workflow.md`, `docs/hindsight-study.md`, `docs/generic-decomp-package.md`, `docs/accelerators.md` + `docs/portable-decomp-workflow.md` and `docs/hindsight-study.md` (both since archived under `docs/sunset/`, Phase 33.5 — the + chapters absorbed them), `docs/generic-decomp-package.md`, `docs/accelerators.md` (A1–A4, #15), `docs/effort-map.md`, `docs/retrospective.md`, `docs/story.md`, `docs/wave-playbook.md`, `docs/public-flip-runbook.md`, `docs/verification.md`, `README.md`, SETUP §4 / P33 B5 / P33 D1–D3, `docs/disc-completeness.md`, `docs/memory-map.md` (the load slots), the docstrings of `match_one` / `rtu_match` / `harvest_verify` / `gate_main` / diff --git a/docs/accelerators.md b/docs/accelerators.md index 1e7030699..f7aada039 100644 --- a/docs/accelerators.md +++ b/docs/accelerators.md @@ -4,7 +4,8 @@ PhaseEnds record *what happened*; `docs/decision-log.md` (R31) records *why we pivoted*. Neither answers the question this file exists for: **"what should phase 1 of the NEXT game do differently?"** -**Companion (added P31 S59):** `docs/portable-decomp-workflow.md` holds the part that TRANSFERS — +**Companion (added P31 S59; re-pointed at Phase 33.5):** the how-to chapters (`docs/how-to-ai-decomp/`, which absorbed the +earlier `portable-decomp-workflow.md`, now archived) hold the part that TRANSFERS — the system (gate discipline, the oracle ladder, lanes, cards, draw-time refusal, per-population clocks, model routing, the flywheel, the measured economics) written target- and compiler-agnostic, for the public "how to AI-decomp" deliverable. This file stays the BFM-specific ledger of *what we diff --git a/docs/gen3-handoff.md b/docs/gen3-handoff.md index 8e7680fe3..d6b16306b 100644 --- a/docs/gen3-handoff.md +++ b/docs/gen3-handoff.md @@ -207,7 +207,7 @@ The framework carries over unchanged: the constitution, the session protocol, th a time, the rules R1–R83 (the Phase-33 candidates were ratified as R74–R83 at the Phase-33.5 gate). A new generation starts with a fresh plan in plan mode at Max. Three things the record says to do first: -1. **Measure the shape before choosing** (`docs/hindsight-study.md` §2.1): which structures own most of the 61,898 +1. **Measure the shape before choosing** (the "characterize the corpus SHAPE first" principle — `docs/how-to-ai-decomp/03-bootstrap-order.md`, Phase 1): which structures own most of the 61,898 data symbols, and which families own most of the 43,925 pins — a census with a self-asserting scanner, checked against a case whose answer is known. 2. **Build the differential harness for the new question** before the campaign: "is this rename/type change diff --git a/docs/how-to-ai-decomp/00-README.md b/docs/how-to-ai-decomp/00-README.md index a7049f547..ca93b258d 100644 --- a/docs/how-to-ai-decomp/00-README.md +++ b/docs/how-to-ai-decomp/00-README.md @@ -46,6 +46,9 @@ against the wrong hash — was one of its own instruments. ## What transfers, and what does not +The test, for any claim in these chapters or in a project's own notes: *if it would still be true for a PS2 game built +with a different compiler, it transfers; if it is a fact about gcc 2.7.2 or about this game, it does not.* + **Transfers:** the gate discipline, the oracle ladder, lanes and cards, draw-time refusal, free-before-paid, per-population clocks, model routing, the flywheel and its inert-rider law, the governance cadence, the publishing discipline, and the *shape* of the economics (not its constants). @@ -56,8 +59,10 @@ the machine that builds it is universal.** That is the whole point of these chap ## The records these chapters are distilled from -- [`docs/portable-decomp-workflow.md`](../portable-decomp-workflow.md) — the system, written target-agnostic during the campaign -- [`docs/hindsight-study.md`](../hindsight-study.md) — the optimal *order*, with seven meta-principles, tagged UNIVERSAL / COMPILER-CLASS / BFM-SPECIFIC +- two earlier syntheses these chapters absorbed and superseded — the target-agnostic system written during the campaign + (`portable-decomp-workflow.md`) and the mid-campaign study of the optimal *order* with its seven meta-principles + (`hindsight-study.md`, written at 78%) — both archived at the Phase-33.5 consolidation; the wiki's Archive index records + what each became - [`docs/generic-decomp-package.md`](../generic-decomp-package.md) — what the next project should inherit on day one - [`docs/accelerators.md`](../accelerators.md) — what was learned late that day one should know, each entry dated with when it *could* have been found - [`docs/retrospective.md`](../retrospective.md) and [`docs/story.md`](../story.md) — what we believed, what failed, what it cost; and the narrative diff --git a/docs/how-to-ai-decomp/02-byte-gate.md b/docs/how-to-ai-decomp/02-byte-gate.md index 7e66108d2..ef835f48b 100644 --- a/docs/how-to-ai-decomp/02-byte-gate.md +++ b/docs/how-to-ai-decomp/02-byte-gate.md @@ -12,8 +12,8 @@ The first thing to establish — before writing a line of C — is a build that byte-identical binary, with the hash check wired into the build so there is **no third state** between "identical" and "failed". BFM had this on day five (an all-`INCLUDE_ASM` build of the executable, SHA1-identical); every claim of progress after that was measured against an incorruptible oracle. The two things that were correctly done first, in -this order: deterministic extraction with a committed manifest, then the all-assembly byte-identical baseline -([`docs/hindsight-study.md`](../hindsight-study.md) §1). +this order: deterministic extraction with a committed manifest, then the all-assembly byte-identical baseline — the +first two rungs of the order [chapter 03](03-bootstrap-order.md) argues for. The mechanics on BFM: one SHA1 contract file per binary (`config/check..sha`), a Makefile whose `check` target builds and compares and exits non-zero on any mismatch, a `NON_MATCHING` guard so that logically-correct-but-unmatched @@ -41,14 +41,25 @@ Populations differ. BFM had ~216 small overlay binaries (build ≈ seconds, one large executable (must rebuild wholly, ≈15 s clean when the batch is right, bisects when it is not). - **Parallel by default, never serial.** Overlays gate in git worktrees so a worker that carves (splits a translation - unit at a jump table) writes only inside its own tree — a carving worker corrupts a shared `asm/`. Sixteen binaries + unit at a jump table) writes only inside its own tree — a carving worker corrupts a shared `asm/` — and under a + per-binary lock, so two lanes never gate the same binary at once (the worktree gives isolation; the lock gives + mutual exclusion; a gate needs both to run as a lane). Sixteen binaries gated serially to protect one carving job cost an hour where the parallel form takes ten minutes; the project's standing bar became *a slow gate is a bug*. Pass `-j` to every build (a single-binary build without it is single-threaded: measured 7.2 s → 1.2 s). - **Batch the expensive one.** One clean rebuild verifies a whole batch of drafts; on failure, bisect so one bad draft cannot sink the rest. Two conditions make batching work: a pre-filter strong enough that most of the batch is right ([chapter 04](04-oracles-and-instruments.md)'s real-TU probe), and the bisect. Measure the gate before you fear it: - BFM's was believed to cost 40–60 minutes and cost ~15 seconds when clean; the stalls were bisects. + BFM's was believed to cost 40–60 minutes and cost ~15 seconds when clean; the stalls were bisects. One case where the + bisect blames the wrong draft: parallel drafters declare a shared callee two different ways, and the batch's failure + is the clash, not either draft — one early wave's whole gap between standalone matches and banks was such conflicts + (compile errors, zero codegen mismatches). Normalise the declarations before the gate, gate one draft at a time + when the failure class is conflicts, and keep a recovery pass that re-normalises the failures and re-gates. +- **A red gate is a hash, not a diagnosis.** Turn it into a named symbol with two commands: substitute the draft into + a copy of its destination file and read the compiler's first `conflicting`/`redefinition` line. Propagation is a + gate too — a fail-closed one that byte-gates every member, writes the shared body and registers the group — and a + propagated body ships its signature change to every member; its sweep mode also catches earlier matches never + propagated, which is the "banked in one binary only" drift the table above names. - **Give the expensive population its own lane** rather than excluding it. Excluded "for a good reason", the main executable got no attention at all for weeks while the fast loop ran at a quarter of capacity for lack of work items ([chapter 05](05-cards-lanes-waves.md)). diff --git a/docs/how-to-ai-decomp/03-bootstrap-order.md b/docs/how-to-ai-decomp/03-bootstrap-order.md index ea3f1b3fe..e74dd346f 100644 --- a/docs/how-to-ai-decomp/03-bootstrap-order.md +++ b/docs/how-to-ai-decomp/03-bootstrap-order.md @@ -58,7 +58,8 @@ ordering alone. What to build the moment you have two binaries ([chapter 10](10- - the corpus/coverage oracle; the signature layer (exact, relocation-normalized, mnemonic-sequence); - the twin join — **as a band, not a point**: an edit-distance band found 3.4× more reachable twins than the exact hash - on a corpus believed fully mined, and pays from the very first banks; + on a corpus believed fully mined (22 of 352 by exact hash, 75 of 352 by the band — [`docs/generic-decomp-package.md`](../generic-decomp-package.md) + §2b, which also says to rank the candidates by the *work* each needs, not by distance), and pays from the very first banks; - propagation (author once, instantiate at each member, byte-gate each, register fail-closed) and the family remap; - the **reconcile ladder** — declaration sync, callee casts, canonical-signature reconciliation, carve tools — because once cracking is systematic, banking a correct body past its translation unit is the bottleneck. This is the part @@ -77,6 +78,13 @@ Two compounding assets BFM built late ([chapter 07](07-compiler-source.md)): register-allocation and scheduling residuals. It runs on a different budget from everything else and should be grinding the near-miss backlog from the first close draft. (And verify that it actually runs on your drafts: BFM's had silently never run on a whole class — [chapter 12](12-failure-museum.md).) +- **Mine the permuter's failures, not just its wins.** A flywheel that only distils *successes* into the cookbook and the + permuter's weights leaves its largest signal unread: every near-miss the search plateaued on. BFM turned that log into + a corpus (`tools/autopsy.py`) and classified each plateau deterministically from the bytes (`tools/residual_class.py`) + into *missing transform* (the byte-match is reachable by a local mutation the search never tries — extend the + mutation set, a permanent offline win), *mis-classified* (fix the routing), *seed-structural* (redraft) or *genuine + wall*. The first bucket is the one that closes a whole class with no model at all. Build the classifier before the + backlog is large; 91% of BFM's open backlog carried no class label when the tool was written. Also seed the knowledge base from **sibling projects on the same compiler** — same-generation idioms transfer. diff --git a/docs/how-to-ai-decomp/04-oracles-and-instruments.md b/docs/how-to-ai-decomp/04-oracles-and-instruments.md index a92db7c25..043fdc965 100644 --- a/docs/how-to-ai-decomp/04-oracles-and-instruments.md +++ b/docs/how-to-ai-decomp/04-oracles-and-instruments.md @@ -23,6 +23,10 @@ nothing in either diff said why. ## The four instances, and the rules they produced +(These four are the ones with a rule attached. The full record of the class — the Phase-26 tooling-integrity audit, +finding by finding, in which several "compiler walls" turned out to be the instruments — is +[`docs/tooling-audit.md`](../tooling-audit.md).) + 1. Functions compiled at a different optimization level than their object was drawn as work **79 times across 19 waves**. None could ever have banked. Every failure looked like a model failure. → **R45 draw-time bankability:** never hand out work the pipeline cannot bank; filter at the source and *count what you filtered* in a visible census. diff --git a/docs/how-to-ai-decomp/05-cards-lanes-waves.md b/docs/how-to-ai-decomp/05-cards-lanes-waves.md index 400454552..6e8da564c 100644 --- a/docs/how-to-ai-decomp/05-cards-lanes-waves.md +++ b/docs/how-to-ai-decomp/05-cards-lanes-waves.md @@ -26,6 +26,21 @@ misplaced laws file silently degraded every wave's brief to "the pack alone" for the rest never mentioned it); and never assert "no banked twin" on a card unless the twin scan ran on the *world*, not on the card — an agent spent 102k tokens re-deriving a function banked verbatim at the same address in another binary. +Three refinements of the twin field, each measured: + +- **The banked twin beats the open one, and the card builder was discarding it.** The card's exemplar and sibling + pointers were drawn from the atlas's *open* set by construction (0 of 34 banked, at any maturity), while the atlas's + own matched-pool twin — computed all along — was being dropped; put on the card, it banked 4 of 4 on a control draw, + and the first wave to carry it cost 15% fewer tokens at 64% of the wall-clock. Reading a banked body beats deriving + from the assembly. +- **Step 0 of the drafter's search is a grep for a magic literal** from the target's disassembly across the whole + source tree, ahead of any same-file or shared-header search — those are scoped to the translation unit and + structurally cannot reach a banked twin in *another binary's* file, which is where the large template classes live. +- **Verify each claim in a source document separately.** A frontier report whose cluster pool checked out within 2–4% + on every size figure (and whose multiplier was better than claimed) was wrong about its other headline — the same + document, right here and wrong there. And a per-entry cache can lie: one prefetched decompile was a stale copy of the + wrong function. + ## Lanes: separate what has different clocks The naive design — pick, draft, verify, repeat — fails at scale because the steps have different natural rates and @@ -43,7 +58,9 @@ different failure modes. Split them into independent, restartable lanes: The single most expensive mistake in this area: stopping the drafting lane to ship a code change — 139 of 162 idle minutes in one session. Ship changes at a wave boundary, or to a lane that is not the clock-limited one. And an invisible trap: a shell parses a `while … done` body in full before running it, so editing the lane script does not -change the running loop, and restarting only the inner process re-runs the *old* arguments. Verify from the process, +change the running loop, and restarting only the inner process re-runs the *old* arguments. The general form: know, for +your runtime, which of the code, the arguments and the per-invocation defaults takes effect when — each can change +independently of the other two, and a restart at the wrong level picks up one and not the rest. Verify from the process, not the file. **An excluded population needs its own lane, not an exclusion.** BFM's main executable was left out of the main loop @@ -56,8 +73,11 @@ and let the lanes share the capacity ([chapter 02](02-byte-gate.md)). Every project has mechanical work that costs no model tokens: the same function in several binaries, a proven body remappable to a sibling, a deterministic transformation of an already-banked result. Run it in its own lane whenever the paid lane is idle. Corollary for the draw: if two work items are the *same* item, draft one and derive the other — -collapse same-skeleton members to one card and defer the siblings to a free remap. Rescan the twin graph after every -bank; it changes with every bank ([chapter 10](10-integration-and-propagation.md)). +collapse same-skeleton members to one card and defer the siblings to a free remap. Measured on one wave: the naive +fleet-wide draw had put 22 of 44 skeletons in the wave *twice* (sibling overlays at two addresses); drawn one-per-group +and ranked by total instructions, the same 71 agent slots carried 19,266 instructions of potential — 6,557 drafted plus +12,709 staged for the free remap — of which 4,954 sibling instructions banked mechanically the same session. Rescan the +twin graph after every bank; it changes with every bank ([chapter 10](10-integration-and-propagation.md)). ## The wave, step by step @@ -69,27 +89,63 @@ each guard with the measurement that produced it**. The spine: 1. **Draw — and exclude what cannot bank.** The exclude list is audited before every draw and a stale one is refused: measured the day after one was written, 88 of its 107 entries were stale — 28 already banked, 14 never targets, and 46 whose blocker had since been fixed: 12,750 instructions of drawable work a list would have hidden. An exclude list - records what the *tooling* could not do at the time, not a property of the functions. + records what the *tooling* could not do at the time, not a property of the functions. Rank the draw by *open + templatable instructions* derived from the corpus census, and know that **face mass and delivered mass are different + numbers**: the "unknown lever" lane maximises the first (its cards draft at 96–100%) at the expense of the second + (they are mostly singletons — zero free siblings and a harvest of 4 instead of 14 on the wave that ran it at scale); + pick per wave. And a default flag can override doctrine silently: one draw tool's default lever list excluded the + lane holding 607 of 1,179 in-band members while the prompt told every agent that lane drafts like any other — the + "thinning frontier" that followed was the flag. 2. **Cards — and make sure the twin is on them** (the twin scan over the world; the near band, not the exact hash). 3. **Packs.** One per target plus the shared laws file, at the paths the drafting prompt reads. 4. **Validate — never hand-type a target;** the launch asserts every target is still open at draw time (payloads go stale). Hand the near-misses to the permuter first — it costs no tokens. 5. **Draft.** One agent per target, model by size ([chapter 08](08-models-and-budgets.md)). Streaming (refill each slot on completion) beats batch (a batch cannot gate until its slowest agent lands: 18 of 20 drafts idle behind 2 - stragglers). Wave size: BFM's doctrine became **6,000+ instructions per wave** after a 40-card wave drafted its - "unknown lever" cards at 96% — "unknown" had meant "the atlas could not name a lever", not difficulty, and size was - nearly free mass. -6. **Gate — everything in parallel.** Overlays in worktrees; the executable by a clean rebuild of the whole batch; read - the verdicts (a "banked 1 / merged 0" is a failure and the tool now says so with exit 2). + stragglers). The measurement that settled it: two batched halves with a barrier between them ran a 14-target wave in + 136 minutes at 2.5× effective parallelism, with 37–50-minute dead gaps visible at every batch boundary in the launch + timestamps; a barrier-free pipeline ran 16 targets in 82 minutes at 3.8× — **40% faster on 14% more targets**. The + batching had existed to dodge a provider throttle the harness's own concurrency cap already dodged, so it bought + nothing and cost a barrier. Wave size: BFM's doctrine became **6,000+ instructions per wave** — six consecutive + mature waves drew 6,500–6,570 instructions on 65–73 cards — after a 40-card wave drafted its "unknown lever" cards at + 96%: "unknown" had meant "the atlas could not name a lever", not difficulty. (Whether such cards *deliver* mass is + the draw question above.) A wave's difficulty is a knob the draw sets — one wave offered 50,596 templatable + instructions at a median target of 438 against the previous wave's 16,884 at 143 — so bank rates across waves are + comparable only at the same setting. +6. **Gate — everything in parallel.** Overlays in worktrees; the executable by a clean rebuild of the whole batch. **Gate + the directory, never the verdict list:** a wave's verdicts are a claim about what the agents *returned*, not what is + on disk — an agent killed by a rate limit after writing a byte-perfect 362-instruction body returned "no draft", and + one gate driver that grepped its child's output for two line prefixes printed "banked 5 / failed 1" over sixteen + drafts, ten of them with no verdict at all because the child had crashed, one of those a byte-identical + 579-instruction match reported as nothing. Score every draft file the wave produced, and assert coverage before + recording a row: `banked + failed + no-verdict == drafts`. Gate-group concentration — how many whole-binary rebuilds + the wave needs — is a draw-time choice, and it is the wall-clock lever ([chapter 09](09-economics.md)). +6b. **Reconcile, before any rebuild.** The one change that made banking stable: after a gate group's drafts finish, one + agent per group runs the deterministic ladder over that group's own slate — the declaration reconciler, the refusal + playbook, the pre-gate check — re-verifying every edit against the standalone oracle *before* the rebuild. Two targets + sharing a translation unit can create each other's conflicts; the ladder sees both. The reconcile lane is the fallback, + not the plan: **when it goes quiet, the prompt is doing its job.** 7. **After ANY bank:** the twin sweep, then commit (R42). 8. **Harvest — before the next wave, a hard gate.** Every new idiom into the cookbook and its index; then ask of each idiom *is this mechanical?* — if yes, build the sweep and bank the free functions before drafting again. This is the - project thesis, not hygiene ([chapter 06](06-knowledge-base.md)). + project thesis, not hygiene ([chapter 06](06-knowledge-base.md)). The wave schema asks every agent what the index + *had* and what it *lacked*; harvesting those reports into the next wave's prompt is what compounds — the prompt block + that lifted the bank rate to 100% came from one agent's gap report. The mature harvest yield, per wave, of gap reports + → confirmed / rejected / already covered: 71 → 9 / 5 / 61, then 14 / 5 / 44, 14 / 9 / 76, 4 / 3 / 41, 7 / 2 / 56, + 5 / 8 / 53. The last column is the one to act on: **the knowledge base knew, and the agents did not find it** — + vocabulary and search order, not new idioms. Rising rejections with falling confirmations is convergence, not decay. 9. **Verify the fleet, not the binary.** A clean rebuild of everything; read the exit code. 10. **Checkpoint.** Refresh the replayable block; stale is worse than absent. Then **recover** before re-drawing: a gate's failures are triaged (body vs. plumbing) and most plumbing rejects bank -without a new draft ([chapter 10](10-integration-and-propagation.md)). +without a new draft ([chapter 10](10-integration-and-propagation.md)). **The gate number is not the close-rate.** One +hand session's whole-binary gate read 5 of 8 on the first pass while the standalone oracle read 8 of 8 — the gate +measured integration, not matching; after deterministic recovery, 8 of 8 banked for ≈0 tokens. Run the recovery ladder +before recording a wave's yield, or the table under-reports the drafters and sends the next wave hunting compiler walls +that are not there. The three first-pass failures of that session were three distinct plumbing levers and zero codegen +walls: a probe layer left in the draft (scaffolding is not part of the bank); a file-scope declaration contradicting a +block-scope one in the destination unit; and a definition whose parameter promotion made the no-prototype form illegal — +solved by a private C identifier bound to the link symbol. ## The one-agent-per-function shape diff --git a/docs/how-to-ai-decomp/07-compiler-source.md b/docs/how-to-ai-decomp/07-compiler-source.md index e5e5a836d..166945a13 100644 --- a/docs/how-to-ai-decomp/07-compiler-source.md +++ b/docs/how-to-ai-decomp/07-compiler-source.md @@ -21,6 +21,36 @@ compiler-internal levers without reading 80,000 lines of source. Whole classes long "confirmed unsteerable" fell to it: hoist-vs-rematerialize across calls, store-vs-load ordering, the delay-slot filler's choice, the spill-slot order (which is the *declaration* order, not first-assignment order). +**Levers from the hand-matching era**, each byte-proven before the map existed and each a one-line entry in it now — +the shape of what such a catalogue holds: + +- *The scaffold's arithmetic.* The decompiler writes `&D_X + (i * K)` with pointer arithmetic that scales by four; the + tell is a shift off by ×4 in the diff (`sll v0, 4` against `sll v0, 2`); the fix casts through a byte pointer. +- *Branch polarity.* Put the target's fall-through block in the `if` and the branched-to block in the `else`; the + scaffold's `== 0` form was right, and "improving" it to `!= 0` with swapped arms broke the match. +- *Rematerialize versus spill.* A cheap constant held live across a call is spilled to a callee-saved register and + grows the frame; set the constant on the exits instead (`if (cond && call() == 0) v = f(); else v = 1;`). +- *The cross-jump count law.* One call site and the compiler hoists (too few instructions); four or more and it + under-merges (too many); exactly the shared-block count matches — a shape choice turned into arithmetic. +- *Two base pointers, two callee-saved registers,* allocated in first-use order — the source-side prediction that the + allocation table (below) confirms. +- *The mask-local idiom.* `*(s16 *)f & (x & 0xFFFF)` written inline lets the compiler prove the result fits sixteen + bits and fold the load to `lhu`; hoisting the mask to a local (`s32 m = x & 0xFFFF;`) hides the range fact and keeps + the `lh`. Deliberately withholding a fact from the optimizer is a lever. +- *Shared returns cluster.* Write both early exits as `goto ret0;` to one trailing `ret0: return 0;`; a lone + `if (x) return 0;` inlines the return, inverts the polarity and moves the constant into the result register — one + shape, three effects. +- *Result-register coalescing (`v0`/`v1`)* was not reliably source-steerable — a variable at the top made it worse, an + early return broke polarity — and is a permuter target; recording the *failed* steers is what stops the next reader + repeating them. +- *The walking pointer.* The loop optimizer strength-reduces `*q++` to an indexed form where the original keeps the + walk; and the "phantom frame" class has two variants with different fixes, keyed to the optimization level. +- *When the permuter cannot help:* a residual on a call to an external callee is outside its search space entirely, + and it rejects register pins (a pinned draft is hand-tier work). +- *The triage routing that came out of it:* pure structure → reconstruct; a stack buffer passed to a callee → array + decay; a register swap across a call → the source *shape* (not a pin — R73); a last-instruction schedule → a + scheduling barrier. + **The source-version trap.** The "gcc 2.7.2" tree in community circulation is gcc **2.8.1** — a behavioural difference (a biv-elimination path disabled in 2.8.1 is live in the real 2.7.2 `cc1`) and a line-number drift large enough to land inside a different function. The map's citations were audited line by line against the vanilla 2.7.2 source by parallel diff --git a/docs/how-to-ai-decomp/09-economics.md b/docs/how-to-ai-decomp/09-economics.md index b14973912..76a733446 100644 --- a/docs/how-to-ai-decomp/09-economics.md +++ b/docs/how-to-ai-decomp/09-economics.md @@ -1,7 +1,9 @@ # 09 — Economics, as measured Every number here is BFM's, with its denominator (R41), stated so another project can *calibrate* rather than copy. -The shape transfers; the constants do not. +The shape transfers; the constants do not. Every number was derived from a log, never hand-transcribed (R33): drafts +from the wave's own verdict file, banked from the whole-binary gate log — the gate is the arbiter, never the agents' +claim — instructions from the corpus census, tokens and wall-clock from the harness's agent timestamps. ## The whole project @@ -17,8 +19,11 @@ The shape transfers; the constants do not. | Quantity | Measured | |---|---| | drafts → banked, fleet-wide | **34%** (8,885 drafts → 3,027 banked over 37 waves) | -| bank rate by function size | ~57% under 50 instructions · ~30% at 50–80 · ~22% at 80–120 · ~3% at 120–200 · ~6% above | +| bank rate by function size | ~57% under 50 instructions · ~30% at 50–80 · ~22% at 80–120 · ~3% at 120–200 · ~6% above — **a fleet-wide curve over 37 waves, most of them before the integration tooling existed**; one late wave banked 16 of 16 with targets up to 397 instructions, another 7 of 16 at a median of 438, a hand pass 8 of 8 including a 579-instruction function. Read the curve with its denominator, or 120-instruction functions look hopeless | | bank rate by population | the "hard" population banked 27–40% at small sizes — **the gap to the easy population was almost entirely size mix, not difficulty** | +| the prompt as the lever | over five consecutive waves with the models and the gate held constant, the bank rate rose from the mid-70s to 100%; the only variable was a block in the prompt harvested from the previous wave's agents' own "what the index lacked" reports. The targets also trended easier, so the claim that holds is the direction and the mechanism, not the percentage | +| standalone MATCH rate, the mature regime | 98–100% on six consecutive 65–73-card waves — and that variation is noise on a 70-card sample; **never report the 100% as a new floor** | +| claimed vs banked, per wave | the gate's number is below the agents' in every wave (15 claimed → 15, 13 → 10, 16 → 7 across three early waves); the gap is integration, not drafting, and the recovery ladder closes most of it (one wave: 18 of 45 banked on the first slate, and 26 of the 27 rejections were byte-perfect drafts; the reconcile lane's lifetime record 21 of 22) | | cheap vs. strong drafters, ≤ ~50 instructions (A/B, 100 workers) | the same match rate at ≈4.8× lower cost per match | | the cheap tier's honesty | 63 claimed matches → 63 real matches | | a local 7B model | saturated the ≤ 15-instruction population for $0 | @@ -37,6 +42,60 @@ The shape transfers; the constants do not. campaign. The project repeatedly reached for more workers when the answer was more work items; once the excluded population got its own lane, capacity stopped being the question. +## A wave, priced + +The per-wave ledger the campaign kept (one row per wave: agents, tokens, wall-clock, banked), condensed. The early +waves are the orchestrated shape; T–V are the mature 6,500-instruction waves. + +| Wave | Agents | Tokens | Wall | Effective parallelism | Banked | +|---|---|---|---|---|---| +| 1 | 25 | 6.87 M | 169 min | 4.8× | 15 heads + 65 free siblings | +| 2 | 19 | 4.51 M | 208 min | 2.7× | 10 + 18 | +| 3 | 14 | 2.73 M | 140 min | 2.1× | all + 21 | +| 4 | 17 | 2.99 M | 136 min | 2.5× | all + 26 | +| 5 | 19 | 2.71 M | 82 min | 3.8× | 16 of 16 + 26 | +| 6 | 24 | 5.20 M | 71 min | 7.4× | 7 of 16 + 25 (the hard knob: median target 438 instructions) | +| T | 87 | 8.71 M | 70 min | — | 70 heads + 49 mechanical siblings (4,954 instructions) | +| U | 86 | 7.42 M | 45 min | — | 73 heads + 34 siblings | +| V | 85 | 8.88 M | 71 min | — | 68 heads + 25 siblings | + +The units that came out of that ledger: **170k–300k tokens per banked head** in the stable early regime, **~124k** in +the mature one; a harvest attached to a wave costs about a quarter of the wave again (19 agents, 1.97 M tokens on T); +recovery of gate drops costs ≈0 agent tokens (8 of 8 in one session, +4 in another — "the lane cost nothing and +returned four functions"); putting the banked twin on the card was worth **15% fewer tokens and 64% of the wall-clock** +for two more cards (U against T). Outages are a casualty count, not a rate: twelve agents died to server overload in +one wave without starting, two to rate-limiting in another. + +## Instruction weight, not head count + +Two waves with the same card count and opposite bank rates: + +| | wave 5 | wave 6 | +|---|---|---| +| pool offered (templatable instructions) | 16,884 | 50,596 | +| heads banked | 16 of 16 | 7 of 16 | +| instructions in the banked heads | 2,753 | 2,204 | +| propagated members | 26 | 25 | +| total instructions banked | ≈7,200 (estimate) | 10,616 (exact; `make report` moved by 10,616) | +| pool realised | ≈43% | 21% | + +The wave with the worse bank rate banked ~47% more instructions. **Head count flatters small-target waves; instruction +weight is what moves the fleet, and the metric to beat is pool realisation, not bank rate** — a wave banking 16 of 16 +of a 17k pool is worth less than one banking 7 of 16 of a 50k pool. Wave T later beat both: 70 heads plus 49 siblings, +11,511 instructions for one wave's tokens. (Record the per-family member breakdown from the first wave; wave 5's total +is an estimate because it was not.) + +## The gate's cost is paid in wall-clock + +Tokens are one currency; a session that runs waves in series is bounded by the other. Each *gate group* is a +whole-binary rebuild, so the number to watch is groups per wave: **5 → 7 → 9 for the same ~6,500 instructions** across +three waves — the gate's cost per banked instruction rising while the bank rate held — then **3** once the draw +concentrated targets by binary (drafts per rebuild 7.8 → 24.3, roughly an hour back per wave). The concentration is a +*choice* at draw time, not a trend of the frontier: the "thinning" that looked like the frontier running out was a +default flag excluding one lever lane. And the floor under any wave's wall-clock is its slowest single chain (~50 +minutes at 200–300 turns; one agent tested 470 statement orderings on a 793-instruction function), so past that point +the lever is target *selection*, not concurrency. + ## Where the tokens went | Sink | Cost | Of | @@ -55,13 +114,24 @@ population got its own lane, capacity stopped being the question. The mechanical banks. Once a tool existed for a class — twins, families, siblings, opt-level carves, propagation, stranded boundaries — its members cost ≈0 tokens each. The zero-token pipeline of one phase (auto-drafts → symbol fixes → pre-checks → a lane gate) banked about 256 members for about no agent tokens, against an estimated ≈20 M as waves. +Per wave, the free siblings that followed the drafted heads ran 65 · 18 · 21 · 26 · 26 · 25 in the early waves and +49 · 34 · 25 · 0 · 1 · 4 in the mature ones — the zeros are the waves whose draw filled up on single-member functions. +The best single case: one 299-instruction head became 4,485 instructions across 15 siblings for ≈0 agent tokens. The +sweep yield is a property of the *family*, not the wave, and it is **bimodal, not average** — one wave swept 21 of 21, +the next 18 of 165, because the second wave's big families were per-location variants that do not template (settled by +probe: a remapped member that builds and is byte-different is genuine per-member codegen). **The economics of the project are the ratio between that row and the table above**, and the bootstrap order of [chapter 03](03-bootstrap-order.md) is about moving work from the second to the first. ## Budget for banking, not cracking Once cracking was systematic (11 of 12 first-pass MATCH with ordinary agents and the codegen map), every bank failure -was translation-unit plumbing. Budget the integration layer as the main cost centre, not the drafting. +was translation-unit plumbing. Budget the integration layer as the main cost centre, not the drafting. Read the +"recovery is free" rows above carefully before generalising them: recovery was cheap because the drops were +declaration refusals on already-byte-verified drafts — the work was done and only the plumbing failed, and five +refusal classes collapsed into one move (bind a private C identifier to the link symbol instead of negotiating over the +shared one). A recovery lane over genuine near-misses is a different economy. **Budget recovery against gate drops; +treat near-misses as permuter fuel** unless a specific new law applies. ## The cost asymmetry that makes the harness obvious diff --git a/docs/permuter-ils.md b/docs/permuter-ils.md index 004e96e7e..730818a8b 100644 --- a/docs/permuter-ils.md +++ b/docs/permuter-ils.md @@ -115,5 +115,7 @@ and let the residual class decide whose turn it is). And a masked "1" is not a c - Cookbook §3 (the harness), §42/§45 (the F-band cracks where pins beat the permuter), §66d–§66d-5 (the permuter⇄reader loop), §137 (REGALLOC-PERM as a two-compile arithmetic problem), §493–§495 (the S80 repairs). -- `docs/hindsight-study.md` §7, mine the permuter's failures, not just its wins (the offline-automatic endgame). +- Mine the permuter's failures, not just its wins (the offline-automatic endgame): `tools/autopsy.py` turns the plateau + log into a classified corpus and `tools/residual_class.py` decides each class from the bytes; the track is described in + `docs/how-to-ai-decomp/03-bootstrap-order.md`, Phase 2b (first proposed in the July hindsight study's §7, archived). - `docs/how-to-ai-decomp/07-compiler-source.md`, when the residual is the compiler's, read the pass instead. diff --git a/docs/retrospective.md b/docs/retrospective.md index 95aaa73a8..2837c91b3 100644 --- a/docs/retrospective.md +++ b/docs/retrospective.md @@ -6,9 +6,10 @@ > PhaseEnds that carry a "What we believed, what failed, and what we would do sooner" section (`PhaseEnd_Phase31.md`, > `PhaseEnd_Phase32.md`), and the Deviations table of every PhaseEnd (237 rows across 32 phases — what was planned > against what happened). `tools/mine_hindsight.py` gathers all three with `file:line` anchors; nothing here is -> backfilled into the log (R31). Two earlier syntheses are folded in by reference rather than rewritten: -> [`docs/hindsight-study.md`](hindsight-study.md) (the optimal ORDER for an AI-driven matching decompilation, with its -> seven meta-principles) and [`docs/generic-decomp-package.md`](generic-decomp-package.md) (what the next project should +> backfilled into the log (R31). Two earlier syntheses are folded in by reference rather than rewritten: the July +> hindsight study (the optimal ORDER for an AI-driven matching decompilation, with its seven meta-principles — written +> at 78%, superseded by the how-to chapters and archived as `docs/sunset/hindsight-study.md` at the Phase-33.5 +> consolidation) and [`docs/generic-decomp-package.md`](generic-decomp-package.md) (what the next project should > inherit on day one). Every cost carries its denominator (R41). ## 1. What we believed, turn by turn — and what was actually true @@ -93,7 +94,7 @@ claim"). The economics of the project are the ratio between those two rows. ## 4. What we would do sooner -*In the order `docs/hindsight-study.md` §0 argues for — byte-match baseline → pin the compiler by evidence → honest corpus +*In the order the July hindsight study first argued for and `docs/how-to-ai-decomp/03-bootstrap-order.md` now carries — byte-match baseline → pin the compiler by evidence → honest corpus census with self-asserting tools → propagation and reconcile tooling → the codegen map and the permuter → then crack, leverage-first — and with the specific lessons this project paid for:* @@ -127,8 +128,8 @@ leverage-first — and with the specific lessons this project paid for:* ## 5. What stayed genuinely hard -Not everything was tooling. `docs/hindsight-study.md` §5 names the classes that remain hard for any matching -decompilation of this compiler, and this project's last phases confirm them: +Not everything was tooling. The July hindsight study's §5 (archived as `docs/sunset/hindsight-study.md`) named the +classes that remain hard for any matching decompilation of this compiler, and this project's last phases confirm them: - **The novel unique tail** — functions with no duplicate, no structural sibling and a scaffold too far from byte-exact for the permuter. They need reasoning; they were the last and most expensive fraction (the 21 of Phase 32). @@ -140,7 +141,7 @@ decompilation of this compiler, and this project's last phases confirm them: - **The human gate.** ~99% of the work was the agent's, but every wrong *direction* was caught at a plan or milestone gate by a person — the re-charter of P31, the "everything short of Sony's objects and the original hand-asm must be C" directive of P32, the in-place-flip decision of P33. Automation filled between the gates; it did not replace them - (`docs/hindsight-study.md` §3, principle 7). + (the July hindsight study's principle 7; the chapters carry it as governance — `docs/how-to-ai-decomp/01-governance.md`). ## 6. For the wiki, in one paragraph diff --git a/docs/wave-playbook.md b/docs/wave-playbook.md index 55355ebed..f4e747aa8 100644 --- a/docs/wave-playbook.md +++ b/docs/wave-playbook.md @@ -443,6 +443,66 @@ be semantically divergent. banking starts immediately. `claude_wave_draft.js` with a single target IS a one-agent workflow; no new script needed. +**The execution block that settled batch vs stream (S37, `.run/s37w.js` — the file is untracked scratch; this is +the part worth keeping).** Two batched halves with a barrier ran 14 targets in 136 min at 2.5× effective +parallelism with 37–50-minute dead gaps at the boundary; this barrier-free `pipeline()` ran 16 in 82 min at 3.8×. +The escalation ladder in it (haiku → sonnet → opus) is the S37 one — the routing line below supersedes it; the +shape (draft → escalate per target, no barrier; the reduction that returns every agent's `index_hit`/`index_gap` +for the harvest) is what transfers: + +```js +phase('Draft') +// pipeline() has NO barrier: each target flows draft -> escalate independently, so wall-clock is +// the slowest SINGLE chain rather than the sum of two batch maxima. +const results = await pipeline( + T, + (t) => agent(prompt(t, false), { + label: `draft:${t.fn}(${t.n}i,x${t.m})`, + phase: 'Draft', + model: t.model, + schema: VERDICT, + }).then((v) => ({ t, v })), + + async ({ t, v }) => { + if (!v) return { t, v: { fn: t.fn, status: 'BLOCKED', summary: 'agent returned no verdict' }, tier: t.model } + if (v.status === 'MATCH' || t.model === 'opus') return { t, v, tier: t.model } + const nextTier = t.model === 'haiku' ? 'sonnet' : 'opus' + const v2 = await agent(prompt(t, true), { + label: `escalate:${t.fn}`, + phase: 'Escalate', + model: nextTier, + schema: VERDICT, + }) + return { t, v: v2 && v2.status === 'MATCH' ? v2 : (v2 || v), tier: nextTier + '-escalated' } + }, +) + +const ok = results.filter(Boolean) +const matched = ok.filter((r) => r.v && r.v.status === 'MATCH') +log(`wave: ${matched.length}/${T.length} claim MATCH (the gate is the arbiter)`) + +return { + claimed_match: matched.map((r) => r.t.fn), + verdicts: ok.map((r) => ({ + fn: r.t.fn, ov: r.t.ov, nins: r.t.n, members: r.t.m, tier: r.tier, + status: r.v ? r.v.status : 'NONE', + closeness: r.v ? r.v.closeness : null, + klass: r.v ? r.v.klass : null, + levers: r.v ? r.v.levers : null, + index_hit: r.v ? r.v.index_hit : null, + index_gap: r.v ? r.v.index_gap : null, + summary: r.v ? r.v.summary : null, + })), +} +``` + +The harness already caps workflow agents at `min(16, cores-2)`, so a batch that exists to dodge a provider throttle +buys nothing; the floor is the slowest single chain (~50 min at 200–300 turns), so past that point the lever is +target selection, not concurrency. The coverage assertion that once lived in the S6f gate script (every draft comes +back as exactly one of banked / failed / no-verdict — `banked + failed + no-verdict == drafts` — or the gate exits +non-zero) is §5b's law below; whichever driver batches the gate today must assert it, because a crashed child once +turned a byte-identical 579-instruction match into a silent zero. + Streaming **burns the 5-hour window faster** (it removes the idle gaps), so slots are the budget dial. Model routing (P31 S73): **≤120 ins Sonnet · >120 Opus**; Fable is the tier above Opus for >~340 ins BUT WAS EXHAUSTED account-wide in S73 (three agents died on "You've reached your Fable limit" after ~10 min / ~133k tokens each) — check `/usage-credits` before routing to it. Opus handled 424/459/464/663-ins targets fine; 1165 is beyond its measured band. Never Haiku→Opus directly. @@ -704,9 +764,10 @@ measurement that produced it. A generic decomp guide can tell you to verify your project log can tell you that a failed build leaves a stale binary whose hash reads green. Feeder documents for that template: `docs/decision-log.md` (R31 — the WHY behind every strategic -pivot), `docs/accelerators.md` (discoveries that would have sped up earlier work), -`docs/hindsight-study.md`, `docs/matching-cookbook.md` (the compiler-idiom knowledge base), and the -`phase-ends/` series (the build history). +pivot), `docs/accelerators.md` (discoveries that would have sped up earlier work), the how-to chapters +(`docs/how-to-ai-decomp/`, which absorbed the July hindsight study), `docs/matching-cookbook.md` (the +compiler-idiom knowledge base), and the `phase-ends/` series (the build history). The template itself +is the day-one decomp kit, `decomp-architect/` (Phase 33.5). ## S80 addendum — the one-agent-per-function shape (no wave) and drafters that outlive the session diff --git a/docs/wiki/The-matching-workflow.md b/docs/wiki/The-matching-workflow.md index cd866ea05..5f95a2f2b 100644 --- a/docs/wiki/The-matching-workflow.md +++ b/docs/wiki/The-matching-workflow.md @@ -63,6 +63,68 @@ intent to bank. pattern and the C-level lever, and the last functions of the project fell to five-line reproducers compiled with `-da` dumps rather than to more variants of the 500-instruction function ([chapter 07](../how-to-ai-decomp/07-compiler-source.md)). +## Matching by hand + +Most of the fleet was matched by agents under the wave machinery; the hand loop is what those agents ran inside, and it +is what a person runs on the residue. It was written down mid-project as a guide for the struct-heavy engine core +(cookbook §16 and §17, the compiler-quirk toolkit); this is the part of it that stayed true. + +**Two decompiles, and the assembly as arbiter.** Use both Ghidra's whole-program decompile and m2c's, not one chained +pipeline: Ghidra resolves locals against globals and callee identities from the whole binary, which m2c lacks; m2c +gives the better starting shape. They disagree, and the assembly decides — one made a struct of what the other made +separate globals, and the reuse of one `lui` base settled it. Two mechanics that cost time: the MCP server must be +serving the *overlay* (`tools/ghidra_mcp_start.sh `, then `/mcp`, R29) and its decompile is asynchronous (poll +the task); and the decompile wrapper hardcodes the main executable, so an overlay is decompiled by calling m2c +directly (`tools/m2c/m2c.py -t mipsel-gcc-c -f --context `). + +**The loop.** Standalone probe (`tools/match_one.py --c --asm-subdir asm//nonmatchings/`) +until the shape matches; the real-TU probe; the gate. A red gate is a hash, not a diagnosis — substitute the draft into +a copy of the destination file and read the compiler's first conflict: +`make build BINARY= 2>&1 | grep -iE 'conflicting|redefin|error'`. + +**The five signature moves.** The wall the project first called "fundamental" was a *signature-consistency* problem, +and it dissolved under byte-neutral canonical widening plus the codebase's own cast idioms, applied surgically — a +blanket declarations header broke loose matches (Phase 16); per-callee, byte-gated canonicalisation worked. +1. *Return widening.* A function with no explicit `return` compiles identically as `void` or `s32`, so widening the + canonical return type is free fleet-wide; the mechanizable predicate: a `void` function whose result some call site + uses → widen to `s32` everywhere. +2. *The sign cast at the use site.* Keep the canonical `s16`; write `(u16)D_X` where the assembly loads `lhu`. +3. *The arity cast.* `((ret (*)(argtypes))func_X)(args)` at a call whose arity the canonical signature does not carry. + Its boundary is default argument promotion: pointer arguments cast safely, narrow scalars did not — until the K&R + `s16`-parameter definition form dissolved that wall too (cookbook §43). +4. *Canonical-matching the target's own parameters.* If the consensus signature passes an address as `s32`, take it as + `s32` and access through `*(s32 *)(arg + 0xNN)` — the int-to-pointer cast is byte-neutral. +5. *Function-pointer-table typing.* `extern void (*D_x[])(argtypes);` turns an indirect-dispatch residual class from + "cannot express" into "compiles". +The layer lives in the overlay's own `.c` file, never in the shared header (a reach-1 name would collide across 134 +overlays). Two width facts: **types are byte-neutral for matching** — the compiler reads the access width off the +instruction (`lh`, `lbu`, `lw`), not off a struct, so recovered structures help comprehension and not the byte-close — +while **the widths of locals and parameters are not**: a value loaded with `lh` wants an `s32` local, because an `s16` +local adds a redundant `sll`/`sra` re-extension. + +**Choosing what to hand-match.** Sorting candidates by *ascending* scaffold mismatch selects for the compiler-quirk +tail (the scaffold already got the structure right, so only a codegen quirk remains); clean closes came from +function-pointer calls, relocation-free few-call bodies, and mis-structured-but-fixable scaffolds. The tractable shape +was ≤ 80 instructions and ≤ 4 calls. The progress metric was function-count-weighted, so a giant banks more bytes for +the same percentage — "fewest largest" has no edge — and declaration work removes friction without unlocking matches: +there was no "magic five that unlock hundreds". Match shared callees before their callers, so the definition fixes the +signature fleet-wide, and watch for the circular callee — a residual that is also a callee in other drafts broke three +of them at once. + +**Running it as a wave.** Parallel agents manufacture signature conflicts: one wave's whole gap between standalone +matches (60%) and whole-binary banks (33%) was compile errors — every one a callee declared differently by two drafts +in the same translation unit, and zero codegen mismatches. Hence the order *draft → normalise the declarations +(`tools/sig_unify.py`) → gate*, a gate that batches at chunk size 1 when the failure class is signature conflicts +(a batched bisect blames a good draft for another draft's clash), and a recovery pass that re-normalises the failures +and re-gates. Pre-decompile the targets headlessly rather than through live MCP contention (stop the server, then +`analyzeHeadless … -noanalysis -postScript DecompileFunctions.java`; rename Ghidra's lowercase `FUN_` outputs to the +project's `func_` form; Ghidra's program ids are opaque, so map the alias first). The harness's workflow `args` +channel does not transit arrays — embed the target list in the script. Filter agent scratch (`_try`, `_v2`) to the +canonical draft name before gating, and make "no draft" a valid result: agents told to write a file *only on a +standalone match* diagnosed twelve of thirty quirk walls precisely and correctly wrote nothing. Propagation is itself a +fail-closed byte gate — it authors the shared macro body, registers the group, and its `--auto-from` sweep catches any +earlier inline match never propagated — and a propagated body ships its signature change to every member. + ## The knowledge base, and feeding it Before each match, consult [`docs/matching-cookbook.md`](../matching-cookbook.md) (grep it by symptom through diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 9bb70136c..2b94491e8 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -43,7 +43,7 @@ in-tree links to `docs/wiki/.md`. 8. `.run/`: only what git tracks; no sca - [x] **2** Unblock, doc side: banners (`concurrency-design`, `gen2-mips-matching-model`, `psyq-worklist`, `distill/*`, `hindsight-study` HISTORICAL + supersession table); SETUP §1a rewritten; the pin count re-derived (one dated number + its command, both gen3 docs) — xHigh — see Log 2026-09-07 Task 2 - [x] **3** Wiki — conventions + firewall: `Docs-and-scratch-conventions.md`, `The-ROM-firewall.md` (the gitignore template block); sidebar rows; Repository-layout/Contributing/ch.11 links; forward links in `doc_links_pending.txt` — Max — see Log 2026-09-07 Task 3 - [x] **4** Wiki — reference + direction: `Reference-index.md` (list-driven), `Where-the-project-goes-next.md`, Contributing "AI use — conduct", Home rewritten, sidebar sections — xHigh — see Log 2026-09-07 Task 4 — **P6 rules check done after it** -- [ ] **5** Integrate the six (hand-matching-process; wave-metrics + the s37w block → wave-playbook; portable-decomp-workflow vs the 13 chapters; generic-decomp-package attributed; hindsight §2.1/§7 → ch.03 + permuter-ils; the gen3 docs → the direction page); re-point every referrer incl. `retrospective.md:10–11` — xHigh +- [x] **5** Integrate the six (hand-matching-process; wave-metrics + the s37w block → wave-playbook; portable-decomp-workflow vs the 13 chapters; generic-decomp-package attributed; hindsight §2.1/§7 → ch.03 + permuter-ils; the gen3 docs → the direction page); re-point every referrer incl. `retrospective.md:10–11` — xHigh — see Log 2026-09-07 Task 5 - [ ] **6** Sunset moves (~61 files incl. the 6 frozen snapshots) with the referrer census by command; `docs/sunset/README.md`; `Archive-index.md` rows (backticked paths + `v1.32.1`); DIGEST §4 + `docs/history/README.md` refreshed; ledger coverage: every `git ls-files docs` path outside wiki/how-to/sunset is KEEP in the ledger — xHigh - [ ] **7** Checkers: `doc_links.py` (sunset refusal, index-derived allowlist, wiki-first warnings, TRACKED/UNTRACKED citation classifier + `--disk`, coverage ⊆); reachability in `wiki_render --selftest`; `timeline.py` regen in `report BINARY=main` + `--check` in `audit-digest`; the 14 dangling cookbook cites; the gitignore-template diff; tools-health wiring; SETUP rows — xHigh - [ ] **8** Tracked `.run/` prune (the 172 + the session logs + the 2 firewall files; `untracked_after_rewrite.txt`; `audit_public` check 4 with both controls; runbook §11; the four tool notes) — xHigh — **then P6 rules check** @@ -283,47 +283,107 @@ per task after this log; never overwrite blind (H5 — every banner was added, n `fable_cd4` listings leave at task 8, before any push Drew makes of this phase would matter — they are already in the pushed history); never `git clean -x`; a match is byte-for-byte (no build input has changed). Continuing with task 5. -## 🛑 SESSION CHECKPOINT — Tasks 0–4 ✓ (P6 check done); NEXT = task 5 (integrate the six documents into the wiki; xHigh) +### 2026-09-07 — Task 5 — Integrate the six documents; re-point every referrer (xHigh; three read-only diff agents) +Three Explore agents diffed the three big INTEGRATE documents against their successors; I folded, re-pointed and verified. +**`portable-decomp-workflow.md` vs the 13 chapters:** a near-superset — 6 residual items. Folded: the per-binary gate LOCK (ch.02, +the parallel bullet: worktree = isolation, lock = mutual exclusion); the one-sentence portability TEST ("still true for a PS2 game +with a different compiler → transfers"; ch.00 "What transfers"); the three-part lane-restart trap (code / arguments / per-invocation +defaults; ch.05 "Lanes"). Reconciled, not folded: the ≤30-instruction cheap-tier boundary (the chapters' ~50 is corroborated by the +A/B and ch.09 — the ≤30 was the mid-campaign figure); "~780 sections" (measured: 490 `## §` headings over 442 numbers — the chapters' +"some 500" stands). Dropped: the automation-runbook sibling mention (it sunsets), the "any language" scope claim. +**`wave-metrics.md` vs ch.05/ch.09 + playbook §5:** the chapters priced drafting by token but never per wave, never the gate, never +in instructions, and carried two claims later waves refuted. Folded into ch.09: the sourcing rule (derived from logs, the gate is the +arbiter); a "A wave, priced" table (waves 1–6, T/U/V: agents / tokens / wall / parallelism / banked) + the units (170k–300k tokens +per banked head early, ~124k mature; harvest ≈ a quarter of a wave; recovery ≈0; the seed-twin payoff 15% tokens / 64% wall; +outages as casualty counts); "Instruction weight, not head count" (the wave-5/6 table, pool realisation 43% vs 21%, wave T's +11,511); "The gate's cost is paid in wall-clock" (gate groups 5→7→9→3, drafts per rebuild 7.8→24.3, concentration is a draw-time +CHOICE, the slowest-chain floor); Drafting-yield rows (the size curve's denominator caveat — a fleet-wide curve over 37 mostly +pre-tooling waves; the prompt as the lever, mid-70s→100% with its honesty caveat; 98–100% standalone is noise, never a new floor; +claimed-vs-banked per wave, 26 of 27 rejections byte-perfect, the reconcile lane 21/22); free siblings per wave (65·18·21·26·26·25 / +49·34·25·0·1·4; the 299→4,485 case; bimodal sweep, the family owns the rate); "budget recovery against gate drops, near-misses are +permuter fuel". Folded into ch.05: the banked twin beats the open one (0/34 vs 4/4; the card builder discarded it), STEP 0 = the +magic-literal grep, verify each claim separately + the stale prefetch; the dedup numbers (22 of 44 skeletons drawn twice; 19,266 +potential = 6,557 drafted + 12,709 staged; 4,954 banked free); draw by open templatable instructions, FACE vs DELIVERED mass (the +"nearly free mass" claim corrected: unknown-lever cards draft at 96–100% but yield zero siblings and a harvest of 4 vs 14), the +UNKNOWN default-flag trap (607 of 1,179); step 5: pipeline vs batched 136→82 min / 2.5×→3.8× / 37–50-min dead gaps, the six +6,500-ins waves, the difficulty knob; step 6 REWRITTEN: "gate the directory, never the verdict list" replaces "read the verdicts" +(the 362-ins NO-DRAFT and the 579-ins silent zero), the coverage assertion, gate-group concentration; NEW step 6b Reconcile before +any rebuild + "when the reconcile lane goes quiet the prompt is doing its job"; step 8: index_hit/index_gap harvest, the yield +series 9/14/14/4/7/5 vs already-covered 61/44/76/41/56/53 — "the knowledge base knew, the agents did not find it"; "Then recover": +the gate number is not the close-rate (5/8 gate vs 8/8 oracle → 8/8 after recovery), the three plumbing levers. `wave-playbook.md` +§5: the S37 execution block (the barrier-free `pipeline()` + the reduction, 46 lines, from `.run/s37w.js` — which is untracked +scratch and stays so; the ladder inside it noted as superseded by the routing line) and the coverage law (the s6f gate script's +assertion — NOT claimed to live in `gate_wave.py`: a grep found no such assertion there; worded as "whichever driver batches must +assert it"). Internal inconsistencies in wave-metrics itself (its Finding-1 series mixes numerators; "405 drafted, 405 banked" is +unsupported by its own table; wave W's draft count 69/70/71; "one more card" is two; ~45% is 47%) were NOT propagated — the table +values were used, as the agent showed the tables are the derivable ones. +**`hand-matching-process.md` vs The-matching-workflow / ch.07 / ch.02:** 48 unique items, 12 classes of STALE guidance excluded (the +55% / 136-binary numbers; pins as the lever and `func_8012B8E4` as their proof — contradicted by R73; the pre-masked-scorer +"permuter refuted" verdicts; `harvest_verify` as THE gate; the deleted signature tools' method; session scratch paths; forward plans; +historical close-rates; the "only genuine stub" narrow-param wall — dissolved by cookbook §43; the private-era NON_MATCHING line; +Ghidra program ids). Folded: a new "Matching by hand" section in `The-matching-workflow.md` (two decompiles + the asm as arbiter, +the MCP/overlay mechanics, the m2c direct invocation; the loop + the two-command red-gate diagnosis; THE FIVE SIGNATURE MOVES with +their byte-neutrality proofs and the surgical-not-blanket warning, the layer's placement, types byte-neutral vs local widths not; +choosing what to hand-match — the ascending-mismatch anti-signal, the tractable shape, the count-weighted metric, no magic five, +callees before callers, the circular callee; running it as a wave — conflicts are manufactured by parallelism (60%→33% = 100% +compile errors), draft → sig_unify → gate, chunk 1, the recovery pass, the headless pre-decompile + the FUN_ rename + opaque ids, +the Workflow args gotcha, scratch filtering, "no draft" as a valid result 12/30, propagation fail-closed + `--auto-from` + the +signature blast radius); a "Levers from the hand-matching era" list in ch.07 (the scaffold ×4 arithmetic tell, branch polarity, +rematerialize-vs-spill, the cross-jump count law, dual bases in first-use order, the mask-local idiom, `goto ret0` clustering, +v0/v1 coalescing's failed steers, the walking pointer, two phantom-frame variants, the permuter's two no-op modes, the triage +routing with the pin branch re-routed to shape per R73); two bullets in ch.02 (parallelism manufactures conflicts → normalise +before the gate, chunk 1, the recovery pass; a red gate is a hash — the two-command diagnosis; propagation as a fail-closed gate +with the blast radius). **The other three:** `generic-decomp-package.md` attributed in ch.03 (the band's numbers 22/352 → 75/352, +rank by work) — ch.06 already cited it; the hindsight §7 track → ch.03 Phase 2b (the autopsy paragraph) and `permuter-ils.md` §5; +the Gen3 pair are the direction page (task 4). ch.04 now cites `tooling-audit.md` as the evidence base. +**Referrers re-pointed (11 sites):** ch.00 :59–60 (both bullets → the archived-synthesis line), ch.02 :16 (→ ch.03), retrospective +:10 (link → archived citation), :96, :130, :143, gen3-handoff :210 (→ ch.03 Phase 1), permuter-ils :118, wave-playbook :708 (+ the +kit pointer), accelerators :7, SETUP :1312 (the S88 record line notes the archive). Census after: 0 LINKS to any of the four +leavers outside records; 6 backticked prose mentions remain, each naming the archive (`docs/sunset/…` or "archived") — the +sanctioned citation form. Verify: `doc_links` OK (1 pending), `wiki_render --selftest` 12/12. + +## 🛑 SESSION CHECKPOINT — Tasks 0–5 ✓; NEXT = task 6 (the sunset moves + the Archive index; xHigh) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase31/32/33.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest (R1–R83), then -WAIT for Drew. Rebuild the harness task list (16 rows, R28) marking tasks 0–4 completed and task 5 in progress. +WAIT for Drew. Rebuild the harness task list (16 rows, R28) marking tasks 0–5 completed and task 6 in progress. ### 1. Where we are **Phase 33.5** (sub-phase; v1.32.0 → v1.32.1), gate 1 approved 2026-09-07 by Drew in plan mode at Max; effort follows the plan's column (Max for tasks 10, 12, 15 — prompt at each transition, R27; xHigh now). Baseline HEAD `80d45b29b`; task 0 = `39d524991`; -task 1 = `a0cf302e5`; task 2 = `d06923a06`; task 3 = `5d10a0d12`; task 4 = the commit after it. Nothing under `src/`, `config/` or the -`Makefile` has changed; the fleet is 218/218 at the Phase-33 close. `docs/doc_links_pending.txt` carries ONE entry -(`docs/wiki/Archive-index.md` → task 6). Wiki pages now: 12 original + Docs-and-scratch-conventions, The-ROM-firewall, Reference-index, -Where-the-project-goes-next (16 + the 13 how-to chapters); the sidebar has sections Using / Working conventions / Reference / How to -AI-decomp / Where next / History (Archive index, pending). The coverage one-liner in the task-4 log entry is the prototype of task 7's -check: the 66 files it lists as uncovered are the task-6 leavers. +task 1 = `a0cf302e5`; task 2 = `d06923a06`; task 3 = `5d10a0d12`; task 4 = `9970f1e62`; task 5 = the commit after it. Nothing under +`src/`, `config/` or the `Makefile` has changed; the fleet is 218/218 at the Phase-33 close. `docs/doc_links_pending.txt` carries ONE +entry (`docs/wiki/Archive-index.md` → task 6). Wiki: 16 pages + 13 chapters; sidebar sections Using / Working conventions / Reference / +How to AI-decomp / Where next / History (Archive index, pending). The six INTEGRATE documents are folded; the four that leave +(`hand-matching-process.md`, `wave-metrics.md`, `portable-decomp-workflow.md`, `hindsight-study.md`) have 0 links pointing at them +outside the records and only "archived"-style prose mentions. -### 2. What NEXT does (task 5, xHigh) — exact steps -Fold the six INTEGRATE documents into the wiki/how-to and re-point every referrer, so task 6 can move the four that leave: -1. `docs/hand-matching-process.md` (32 KB, P17 Ghidra-driven hand-matching guide) → read it; fold what the wiki lacks into - `The-matching-workflow.md` (a "Matching by hand" section: the Ghidra-driven loop, the differ discipline, the register-pin toolkit - pointer §17, when to hand the near-miss to the permuter) and, for the transferable parts, how-to `07-compiler-source.md`; record - the "unique claims folded" list in this log. -2. `docs/wave-metrics.md` (20 KB, one row per crack wave) → the measurement source of how-to `09-economics.md` and `05-cards-lanes- - waves.md`: fold the per-wave table's headline numbers (agents, tokens, wall, banked per wave; the 76%→77%→100% bank-rate arc) where - the chapters cite economics without a table; move its two `.run/` prose citations FIRST: line 62's "Use `.run/s37w.js`'s execution - block for every future wave" → copy the execution block itself into `docs/wave-playbook.md` (a short fenced block in §5 Draft) and - line 153's `.run/s6f_gate.py` assertion → name it as archived. Then the file can leave. -3. `docs/portable-decomp-workflow.md` (14 KB, §0–§11) → diff each § against the 13 chapters (this is the breadth-shaped part — three - read-only agents may take §0–§4 / §5–§8 / §9–§11); fold any unique claim into ch.03 (bootstrap order), ch.05 (lanes/cards), ch.09 - (economics), ch.04 (refuse work the pipeline cannot bank — R43/R45 framing); re-point how-to `00-README.md:59`'s link to the chapters. -4. `docs/generic-decomp-package.md` (KEEP) → attribute it in ch.03 and ch.06 where its claims are used without citation (the twin BAND - 3.4× widening; "rank by work, not distance"; the widening review); it stays and is in the Reference index. -5. `docs/hindsight-study.md` → re-point how-to `00-README.md:60`, `02-byte-gate.md:16`, `docs/retrospective.md:10–11` and - `docs/gen3-handoff.md` §8 item 1 (which cites its §2.1: "measure the shape before choosing") to the chapters/the retrospective; - add to ch.03 Phase 2b the "mine the permuter's failures" paragraph (its §7 → `tools/autopsy.py` + `residual_class.py`; the - docstrings of both tools already cite ch.03 Phase 2b) and to `docs/permuter-ils.md` a one-line pointer to the autopsy. -6. `docs/gen3-standards.md` / `gen3-handoff.md` (KEEP) → already summarised in the direction page (task 4); nothing more. -7. ch.04 gains a citation of `docs/tooling-audit.md` as its evidence base (one sentence + a Reference-index-form link). -8. Referrer census for the four leavers (the D3 command) → must be 0 outside PhaseEnds/logs before task 6. Verify `doc_links` 0 broken; - log the folded-claims checklist; commit by explicit path. Task 6 is xHigh. +### 2. What NEXT does (task 6, xHigh) — exact steps +1. Build the move list from ledger L1 (this file): SUNSET (33 root files incl. the 12 per-binary snapshots) + FROZEN (6: + `frontier-atlas.md`, `family-hseq.md`, `family-cousins.md`, `worklist.md`, `autopsy.md`, `burndown.jsonl`) + INTEGRATE residue (4) + + `distill/` (2) + `history/` (the `.output` + the 3 `*link.txt`) + `tool-designs/` (11) = **61 files**. Cross-check against the + task-4 coverage one-liner's uncovered list (66 lines = 61 files + the 5 split by spaces in the history names) — they must agree. +2. Per file, the D3 census: `git grep -n -F '' -- '*.md' '*.py' '*.sh' Makefile ':!phase-ends/PhaseEnd_*' ':!phase-ends/logs' + ':!phase-ends/CURRENT_PHASE.md' ':!docs/matching-cookbook.md' ':!docs/cookbook-index.md' ':!docs/decision-log.md' ':!docs/commit-map.tsv'` + must return only the file itself, the two index pages, and "archived" prose mentions. Known referrers to fix BEFORE moving: + `resident-dossier.md` — check each lever it records has a cookbook § (grep its §-cites); `tool-designs/frontier-analysis-s60.md` + (docstrings already re-pointed, task 1); `frontier-p32.md` (DIGEST §4 names it — refresh §4 in this task); `roadmap-to-100.md`, + `automation-runbook.md` (DIGEST §4 + `wave-playbook.md:4` name them — the playbook line says "retired"; keep as an archived mention); + `gen2-roadmap.md` (DIGEST §1 names it — it is a record; add "(archived)" there). The cookbook/decision-log/PhaseEnds are records and + keep their citations untouched. +3. `mkdir -p docs/sunset/{generated-snapshots,tool-designs,distill,history}`; `git mv` each file to `docs/sunset/` + (root files → `docs/sunset/`; the 6 frozen + the 12 per-binary snapshots → `docs/sunset/generated-snapshots/`). +4. `docs/sunset/README.md`: the review checklist — one row per file: path · class · the Archive-index row title · ☐ reviewed; the + header says deletion is Drew's decision after review and that nothing links into this folder. +5. `docs/wiki/Archive-index.md`: one row per moved file — `` `docs/sunset/` `` (backticked, never a link) · archived at v1.32.1 · + what it was (one line) · what came of it · where the information lives now (a wiki page link, a PhaseEnd, a cookbook §). Group by + the ledger's classes. Then remove the `Archive-index.md` line from `docs/doc_links_pending.txt` (the last pending entry). +6. Refresh DIGEST §4 (the doc map: drop the sunset names, add the wiki index pages) and `docs/history/README.md` (what left, where). +7. Verify: `comm -3 <(paths from Archive-index) <(paths from sunset README) <(git ls-files docs/sunset ':!docs/sunset/README.md')` empty; + `git log --follow --oneline docs/sunset/hindsight-study.md | wc -l` > 1 (history kept) on 3 samples; the task-4 coverage one-liner + now lists 0 uncovered; `doc_links` 0 broken / 0 pending; `wiki_render --selftest`. Log + checkpoint; commit by explicit path + (`git add docs/sunset docs/wiki/Archive-index.md …` — `git mv` stages the renames). Task 7 is xHigh. ### 3. Standing facts for every task of this phase - One commit per task, after this file's log line (R8/R42 form); commit by explicit path; no trailers (R5); Drew pushes (R6).