From 98e5846662b3d3eee24281ba075fc1842d8a8d4d Mon Sep 17 00:00:00 2001 From: Drew T <50529377+Druthulu@users.noreply.github.com> Date: Mon, 7 Sep 2026 11:32:42 -0600 Subject: [PATCH] =?UTF-8?q?docs(phase-33):=20E6=20=E2=80=94=20docs/matchin?= =?UTF-8?q?g-drafter-pipeline.md:=20the=20fine-tuned=20local=20matching=20?= =?UTF-8?q?drafter=20written=20up=20from=20the=20Phase=2022=E2=80=9325=20r?= =?UTF-8?q?ecords=20(the=20toolchain-agnostic=20pipeline=20export=5Fpairs?= =?UTF-8?q?=20=E2=86=92=20format=5Ffinetune=20=E2=86=92=20train=5Flora=20?= =?UTF-8?q?=E2=86=92=20eval=5Flora=20=E2=86=92=20serve=5Flocal=20=E2=86=92?= =?UTF-8?q?=20api=5Fdraft/lora=5Fgrind/bulk=5Fharvest=20=E2=86=92=20the=20?= =?UTF-8?q?byte=20gate;=20the=20measured=20arc=20as=20one=20table=20?= =?UTF-8?q?=E2=80=94=20stock=20floor=20~0,=20corpus-v2=2085%=20on=206?= =?UTF-8?q?=E2=80=9315=20ins,=20the=200/222=20harness=20bugs,=20corpus-v3?= =?UTF-8?q?=2057.5%=20held-out=20+=20~352=20production=20banks=20at=20$0,?= =?UTF-8?q?=20bulk=5Fharvest=2065%=20at=200.4=20s/fn,=20GLM5.2=2010/18=20b?= =?UTF-8?q?odies=20but=203/18=20banks=20=3D=20the=20def-side=20wall,=20v4?= =?UTF-8?q?=20discarded;=20the=20portable=20lessons;=20hardware;=20a=20fiv?= =?UTF-8?q?e-step=20recipe;=20what=20is=20NOT=20published=20=E2=80=94=20th?= =?UTF-8?q?e=20ROM-derived=20pair=20dataset=20and=20the=20adapter=20weight?= =?UTF-8?q?s,=20both=20gitignored);=20SETUP=20row=20+=20P33=20E6=20section?= =?UTF-8?q?;=20doc=5Flinks=20default;=20log=20+=20checkpoint=20(Block=20E?= =?UTF-8?q?=20complete;=20NEXT=20=3D=20G1,=20pulled=20ahead=20of=20the=20p?= =?UTF-8?q?robe-gated=20items)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/SETUP.md | 15 ++++ docs/matching-drafter-pipeline.md | 118 ++++++++++++++++++++++++++++++ phase-ends/CURRENT_PHASE.md | 46 +++++++++--- tools/doc_links.py | 2 +- 4 files changed, 168 insertions(+), 13 deletions(-) create mode 100644 docs/matching-drafter-pipeline.md diff --git a/docs/SETUP.md b/docs/SETUP.md index cdb61e556..e86b6750e 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -767,6 +767,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/gccmap_cites.py [--dry-run \| --check \| --verify \| --controls \| --explain] [--retag]` | **(P33 E3)** Tag every `file.c:NNN` cite in `docs/gcc-2.7.2-map/*.md` with the source tree its line number belongs to — `[2.7.2]` (the vanilla subset), `[2.8.1 pm]` (gcc-papermario), `[repo]` — derived from the trees (quoted snippets, identifiers with function extents and nearest distance, the author's cues; a contradiction fails loudly; ties → `cite_overrides.tsv` → cue → 2.7.2); writes in place, idempotent, never silently changes an existing tag. `--check` is textual (every cite tagged, no stale override) and runs in `make tools-health` + CI; the others need both reference trees and refuse without them. | | | `tools/xsig/xsig.py sign-s \| sign-objdump \| cross \| verify \| selftest` | **(P33 E4; Phase-21 origin)** Relocation-masked per-function signatures for CROSS-PROJECT code identification: mask `j`/`jal` targets and HI16/LO16 immediates (from `%hi`/`%lo` operands or `objdump -dr` records), keep opcodes/registers/constants/branches; `cross` joins two JSONL sets on `sig` with a coverage line; `verify` prints the instruction-by-instruction diff (opcode/register/immediate/length). Self-contained (stdlib, MIT, own README + LICENSE + `tests/` from a game-free fixture compiled at two link addresses with `--emit-relocs`); `tests/test_xsig.py` (8) in `make tools-health` + CI. The standalone repo copy is prepared under `.run/P33/xsig-repo/` (Drew creates + pushes `Druthulu/xsig`). | | | `tools/permuter/upstream/0001-reloc-masked-scorer.patch` | **(P33 E5)** The upstream PR as a `git format-patch` (one commit against `simonlindholm/decomp-permuter` main `41bd0bfc`, 2026-09-05): `src/reloc_scorer.py` (`RelocMaskedScorer`, a `Scorer` subclass), `--score-mode {mnemonic,reloc-masked}` + the `score_mode` settings key, docs, `test/test_reloc_scorer.py` (10, no cross toolchain). Regenerate the branch: clone upstream, `git am` the patch (proven clean). The scratch clone `.run/P33/permuter-upstream/` holds branch `reloc-masked-scorer` under the noreply identity; **Drew pushes it to his fork and opens the PR** (`docs/permuter-ils.md` §2). | +| | `docs/matching-drafter-pipeline.md` | **(P33 E6)** The write-up of the fine-tuned local matching drafter (Phases 22–25): the pipeline (`export_pairs` → `format_finetune` → `train_lora` → `eval_lora` → `serve_local` → `api_draft` / `lora_grind` / `bulk_harvest` → the byte gate; `ab_score` + `workflows/ab_match.js`; `grinder`), every measurement in order (stock floor ~0 → v2 85% on 6–15 ins → v3 57.5% held-out and ~352 production banks → v4 negative → the GLM hard-band A/B and the def-side wall), the portable lessons, the hardware, a five-step recipe, and what is NOT published (the ROM-derived pair dataset `datasets/`, the adapter weights `models/` — both gitignored). | | | `tools/objdiff_report.py [--in docs/progress.json] [--out report.json]` | **(P33 D3)** progress.json → objdiff's report format (report.proto v2, snake_case — validated with `objdiff-cli` 3.8.1 `report changes`): one unit per binary (code = instructions × 4, functions byte-identical / matchable, metadata complete), categories `game-code` and `linked-sony-objects` (functions only). `.github/workflows/progress.yml` runs it on every push (no rebuild — the committed JSON) and uploads the artifact **`SLUS_007.26_report`** for decomp.dev (Drew registers at decomp.dev/manage/new after the flip). | | | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data///` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. | | | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. · `probe_github.sh [--after-flip]` (Drew's daily post-purge probe, C10: 33 sampled old shas via `gh api` + a fetch; **S88, R57:** the fetch runs in a throwaway bare repo under `.run/public_rewrite/` with `--filter=blob:none --depth=1`, never in the working repo — a successful fetch of an old sha imports its purged closure, which the S87/S88 runs did (5.97 GiB unreachable) — and it ends with a self-check naming any sampled old commit the working repo still holds + the gc recipe) | @@ -1146,6 +1147,20 @@ fills fast). Nothing is leaking — but the host does not get the memory back on `tools/ghidra_*.sh` are repo-relative (`BFM_GHIDRA_PROJ` overrides the project dir; `ghidra_mcp_verify.sh [PROG]`); Makefile `GHIDRA_PROJ := $(or $(BFM_GHIDRA_PROJ),$(CURDIR)/ghidra)`. +### P33 E6 (S88, 2026-09-07) — `docs/matching-drafter-pipeline.md`, the drafter write-up +- Distilled from `docs/gen2-mips-matching-model.md` (every measurement, 2026-06-29 → 07-08) and + `docs/community-matching-model-plan.md` (the parked community release) with the drafting loops of cookbook §12 / §500: + §1 why a specialist is safe under the byte gate, §2 the toolchain-agnostic pipeline (all 12 tools verified present), + §3 the measured arc as one table (stock floor ~0; v1 39/41 trivial / 0/34 non-trivial; corpus-v2 85% on 6–15 ins; + the 0/222 that was two harness bugs; 7/15 on open stubs; 0/15 on shared code → corpus-v3 with the 1,623 macro bodies → + 57.5% held-out, ~352 production banks, fleet +502 fns at $0; `bulk_harvest` 65% at 0.4 s/fn gate; GLM5.2 10/18 bodies + but 3/18 banks — the def-side wall; v4 discarded; the 4.8× Haiku A/B), §4 the portable lessons, §5 hardware, §6 a + five-step recipe, §7 what is published (the scripts, this doc) and what is NOT (the ROM-derived pair dataset + `datasets/match_pairs/` and the adapter weights `models/` — both gitignored, confirmed with `git check-ignore`; their + release is a Gen3 licensing decision). The 2026-07-08 decision-log entry ("the local-7B tier is capacity-bound and off + the endgame critical path") is the cited strategic conclusion. In `doc_links`' default set; this row. + + ### P33 E5 (S88, 2026-09-07) — the permuter upstream PR branch + `docs/permuter-ils.md` - **The PR (PR-1).** `RelocMaskedScorer` in `src/reloc_scorer.py` — the in-tree `masked_diff.diff_object_object` rule (mask from the TARGET's relocation records: `R_MIPS_26` → opcode only; the 16-bit-immediate relocs → opcode+rs+rt; a `j` diff --git a/docs/matching-drafter-pipeline.md b/docs/matching-drafter-pipeline.md new file mode 100644 index 000000000..c75e9e901 --- /dev/null +++ b/docs/matching-drafter-pipeline.md @@ -0,0 +1,118 @@ +# The matching-drafter pipeline — a fine-tuned local model that writes byte-exact C, and what it measured + +> **Status (P33 E6, 2026-09-07).** The write-up of BFM-decomp's *matching drafter*: a LoRA fine-tune of a small local +> code model that drafts C for one exact toolchain (gcc-2.7.2-psx `-O2 -G0`, maspsx 2.56), gated by the project's +> whole-binary byte gate so that a wrong draft can never land. Built and measured in Phases 22–25 (2026-06-29 → 07-08), +> frozen at its third version, and off the endgame's critical path from 2026-07-08. The pipeline scripts are in +> `tools/` and are published with the repository; **the paired dataset (`datasets/match_pairs/`, assembly ↔ matched C, +> ROM-derived) and the adapter weights (`models/`, a transform of that dataset) are NOT published** — both directories +> are gitignored, and their release is a separate licensing decision for Gen3. The design notes this distils are +> [`docs/gen2-mips-matching-model.md`](gen2-mips-matching-model.md) (how it was built, every measurement) and +> [`docs/community-matching-model-plan.md`](community-matching-model-plan.md) (what a community release would look +> like); the drafting loops it plugs into are cookbook §12 (the parallel-draft + byte-gate harvest) and §500 (the +> one-agent-per-function pass). + +## 1. The idea, and why it is safe + +There is no matching-decompilation model: the one reverse-engineering model family targets x86 and optimises for +readable, semantically equivalent C, not byte-identical output under a specific 1990s compiler. The reason is data — +the only place (target assembly ↔ byte-exact C) pairs exist is inside decompilation projects' histories. BFM had +~1,300 such pairs by Phase 22 (later ~2,900 with the shared macro bodies), each gate-verified, each specific to the +exact toolchain. That is the corpus a specialist needs and a generalist lacks. + +What makes a weak or wrong model *safe* to use at all is the project's byte gate: the model only affects +**throughput**, never correctness — a draft is substituted into the real translation unit, the binary is rebuilt, and +it is kept only if the SHA1 still matches ([the matching workflow](wiki/The-matching-workflow.md)). So the model can be +small, local, free and frequently wrong, and still bank thousands of functions. + +## 2. The pipeline (toolchain-agnostic; only the data and the compile command are project-specific) + +``` +tools/export_pairs.py mine (asm .s ↔ matched C) pairs from the project's banked functions — plain `src/` definitions + AND the shared `DEFINE_func_*` macro bodies — with the `extern` block each definition carries +tools/format_finetune.py chat-template examples; the project's shared types inlined; a compile filter (self-contained?) +tools/train_lora.py QLoRA on a code base model (Qwen2.5-Coder-7B: dense, permissive, fits a 12 GB card) +tools/eval_lora.py held-out, GATE-TRUE evaluation: the adapter drafts unseen banked functions; the gate decides +tools/serve_local.py serve base + adapter as an OpenAI-style endpoint (GPU; Unsloth/transformers) +tools/api_draft.py the drafter client (the LEAN prompt; the same prompt for every tier — local, cheap-cloud, frontier) +tools/lora_grind.py draft open stubs → gate → bank → propagate (interleaved; the first production loop) +tools/bulk_harvest.py the phase-separated loop: (A) bulk-draft on the GPU, (B) parallel per-binary byte gates on the + CPU, (C) dedupe once + one commit +tools/ab_score.py, tools/workflows/ab_match.js controlled A/B of two drafter arms on a fixed target set, gate-scored +tools/grinder.py the permuter over the near-misses the model leaves (CPU, concurrent with the GPU) +``` + +Two disciplines that the measurements below justify: **the prompt format at training time is byte-identical to the +prompt at inference** (any drift wastes the specialisation), and **every evaluation number is gate-true** — a +whole-binary match, never a standalone `match_one` proxy (which over-counts about 2×). + +## 3. What it measured, in order + +| Step (date) | Corpus | Result (gate-true unless stated) | What it taught | +|---|---|---|---| +| Stock local model floor (06-29) | — (Qwen3.6-35B-A3B, prompt only) | ~0 reliable banks: structure right, gcc-2.7.2 precision wrong (offset scaling, `lh`/`lhu`, an extra `move`, the frame size); a full-cookbook prompt was worse and 2.3× slower than a curated subset | more context is not the lever; precision must be in the weights | +| v1: 7B QLoRA, 3 epochs (06-29) | 638 compile-filtered pairs (297 trivial) | ≤5 ins **39/41**; ≥6 ins **0/34**; near-misses FAR (`near N ≈ nins`) | the compile filter had thrown out the 536 hard functions; a 7B memorised the leaf pattern | +| corpus-v2: the `extern` block captured (06-29) | 1,111 pairs, non-trivial 257 → 813; standalone compile 52% → 92% | 6–15 ins **85% (23/27)**; 16–40 ins 13% (3/22); >40 ins 0/10 (compile-fail: struct types) | **data quality was the bottleneck, not the model** | +| The 0/222 broad run (06-30) | v2 | banked **0/222** across 25 overlays | two harness bugs (a SHA string with its filename; hardcoded per-binary paths) — not the model; fixed with a loud negative-control guard (R40) | +| First trustworthy open-stub signal (06-30) | v2 | **7/15 (47%)** on the smallest open stubs of one overlay; 4 near-misses at closeness 1 | most small open stubs are overlay-unique (×1) — a bank-*rate* lever, not a fleet-% lever | +| reach ≥ 2 targeting (06-30) | v2 | **0/15** on shared functions; the model drafted `return 1` and `sw` setters as *empty* bodies | 96.6% of the corpus was overlay-unique inline definitions — the shared setters were never seen | +| corpus-v3: + 1,623 shared macro bodies; struct types inlined; the "never an empty body" prompt clause (06-30) | 2,891 pairs (2,534 train / 291 test), 97.8% compile | held-out **23/40 (57.5%)**; production: **~352 banks** + 45 new shared groups, fleet 63.67% → 63.82% (+502 functions), 136/136 clean, $0 | the previously impossible setter/leaf class banked; the flywheel (banks → corpus → retrain) closed | +| `bulk_harvest` phase separation (07-01) | v3 | **52/80 = 65%** on fresh ≤15-ins stubs across 50 overlays; gate 0.4 s/fn amortised (≈75× the interleaved loop); drafting 13.5 s/fn = 97% of wall clock | the bottleneck moved from gating to inference; batched serving became justified | +| GLM5.2 (cloud reasoning model) A/B on the hard band 16–22 ins (07-01) | v3 vs GLM | `match_one` 1/18 vs **10/18**; banks 1/18 vs **3/18** (+1 by reconciliation); ~$1.25 total | a better drafter's bodies were right and *still* did not bank: the **def-side declaration wall** caps every drafter (7 of 10 stranded); the reasoning model given the wall directly banked 1/7 | +| v4: retrain on the post-giant corpus (07-08) | 3,574 pairs (+994 medium, +597 large) | easy tie (5/5 both); medium **v3 better** (v4 farther, 4 compile-fails); hard 0/10 both | **discard v4** — the 7B is capacity-bound; over-length examples had trained on cut-off completions | +| Cheap-tier A/B (Phase 22) | Haiku drafters under an Opus orchestrator | same match rate as Opus on ≤ ~50-ins functions at ≈4.8× lower cost per match | the model ladder: Haiku → Sonnet → Opus → frontier only for a new wall class | + +**The strategic conclusion** (decision log, "2026-07-08 · Phase 25 — the local-7B tier is capacity-bound and off the +endgame critical path"): the engine that finished the project was *frontier crack → deterministic propagation +(families, dedup) → byte gate*, with the permuter softening the tail; the local model stayed a frozen $0 mop-up for +the ≤ ~15-instruction setter/leaf tail. A real capability jump needs a bigger base (14B four-bit on the card, 32B in +the cloud) or the frontier tier — not more data on a 7B. + +## 4. The lessons, stated portably + +- **Data is the bottleneck, not model size.** Every jump came from the corpus: the `extern` block (v2), the shared + macro bodies and inlined struct types (v3), one prompt clause. Retraining on more but harder data regressed (v4). +- **Completions must be self-contained** — the model emits the `extern`s and types it references, or it compiles in + isolation and fails in the real translation unit (the integration cap every drafter shares — see + [chapter 10](how-to-ai-decomp/10-integration-and-propagation.md)). +- **Corpus quality > size; weight by regime and residual class**, not by count — one overlay-unique pattern drowned + the shared setters until the shared bodies were mined explicitly. +- **Dense beats MoE for a small LoRA corpus; the model size must match the data size** (a 7B on 638 examples + memorised; on 2,891 it generalised; on 3,574 harder ones it saturated). +- **Drop over-length examples rather than train on truncated ones; train at the context you infer at** (v4's medium + regression traced to completions cut at `maxlen`). +- **The byte gate turns model quality into a throughput question** — iterate fearlessly, and never let an + intermediate metric stand in for the gate (R14, R63). +- **Exonerate the harness before the model** (R40): the 0/222 that nearly enshrined "the tail is exhausted" was two + bugs in the gate's caller. +- **Route by measured difficulty; run the permuter concurrently** (GPU drafts, CPU permutes) on the close near-misses + the model leaves; target shared functions deliberately (×reach), and expect the declaration wall to cap any drafter. + +## 5. Hardware, as used + +A single 12 GB consumer card (RTX 3080 Ti) trained the 7B QLoRA (≈2 h for 3 epochs at batch 1 / maxlen 2048, VRAM-tight) +and served it (Unsloth/transformers; the prebuilt llama.cpp CUDA wheels crashed on a CPU without AVX-512). Serving and +training contend for the one GPU — a retrain pauses the harvest; a second card breaks that. 14B and above: rent an +A100/H100 — the byte gate means a wrong big model is only a throughput cost. + +## 6. Train one for your own project + +1. Have the byte gate and at least a few hundred gate-verified pairs (the pipeline mines them from your banked sources). +2. `export_pairs` with the declarations each body needs; `format_finetune` with your shared types inlined; keep the + compile filter and *count what it drops* — the dropped set is where your hard functions are. +3. Fix the prompt format once; use it for training, evaluation and production alike. +4. QLoRA a dense 7B; evaluate **gate-true on held-out functions** before any production run; compare against the + stock model and against a cheap cloud tier on the same targets (`ab_score`). +5. Run it as the lowest rung of a model ladder, phase-separated (bulk draft, parallel gate, one commit), on the + population it wins (small, overlay-unique, leaf/setter); hand its close near-misses to the permuter; retrain only + when the corpus changes in *kind*, not size. + +## 7. What is and is not published + +Published with the repository: every script above, this write-up, the design notes. **Not published:** the pair +dataset (assembly from the game ↔ matched C — ROM-derived; `datasets/`, gitignored) and the adapter weights (`models/`, +gitignored; a learned transform of that dataset, treated conservatively). A community release — weights with a model +card naming the exact toolchain, a toolchain-agnostic training tutorial, and the ambitious *toolchain-conditioned* +universal model trained on pooled multi-project pairs (decomp.me is the ready labelled corpus) — is designed in +[`docs/community-matching-model-plan.md`](community-matching-model-plan.md) and parked for Gen3, gated on resolving +the licensing of pooled ROM-derived pairs per source project. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 585f9b806..64770cab8 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -82,7 +82,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo - [x] **E3** gcc-2.7.2 map README + `gccmap_cites.py` — xHigh — see Log 2026-09-07 E3 - [x] **E4** xsig packaging — xHigh — see Log 2026-09-07 E4 - [x] **E5** permuter upstream PR branch — Max (FULL, Drew 2026-09-07) — see Log 2026-09-07 E5 -- [ ] **E6** drafter write-up — Max +- [x] **E6** drafter write-up — Max — see Log 2026-09-07 E6 - [ ] **E1** decomp.me preset (after the flip) — xHigh - [ ] **E2** Archipelago outreach (after the flip) — xHigh - [ ] **C11** Aftercare (other clones, archive remote removed, `.run/` pruned, DIGEST/decision-log) — xHigh @@ -564,21 +564,41 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact flushed stdout R55; winner ≠ bank), §5 related. SETUP: row + the P33 E5 section (R21). **Drew:** fork upstream, `git -C .run/P33/permuter-upstream push -u fork reloc-masked-scorer`, open the PR (the commit message is the description), file the issue. Commit: see below. +- **2026-09-07 (S88, Max) — E6 `docs/matching-drafter-pipeline.md`, the drafter write-up.** Distilled from + `docs/gen2-mips-matching-model.md` (every measurement of the local-model tier, 2026-06-29 → 07-08) and + `docs/community-matching-model-plan.md` (the parked community release), with the drafting loops of cookbook §12 / §500: + §1 why a specialist is safe under the byte gate (the model affects throughput, never correctness); §2 the + toolchain-agnostic pipeline — `export_pairs` → `format_finetune` → `train_lora` → `eval_lora` → `serve_local` → + `api_draft` / `lora_grind` / `bulk_harvest` → the byte gate; `ab_score` + `workflows/ab_match.js`; `grinder` — every + tool verified present; §3 the measured arc as ONE table with dates (the stock 35B floor ~0; v1 39/41 trivial and 0/34 + non-trivial on 638 pairs; corpus-v2 with the `extern` block: 85% on 6–15 ins; the 0/222 broad run that was two harness + bugs, R40; 7/15 on open stubs; 0/15 on shared code → corpus-v3 with the 1,623 shared macro bodies + the "never an empty + body" clause: 57.5% held-out, ~352 production banks, fleet +502 fns at $0; `bulk_harvest` 65% with a 0.4 s/fn gate; + GLM5.2 on the hard band: 10/18 bodies right, 3/18 banked — the def-side declaration wall caps every drafter; v4 + discarded — the 7B is capacity-bound; the 4.8× Haiku A/B); §4 the portable lessons (data > size; self-contained + completions; weight by regime; dense > MoE; drop over-length examples; train at the inference context; the gate makes + quality a throughput question; exonerate the harness; route by measured difficulty); §5 hardware; §6 a five-step + recipe for another project; §7 **what is NOT published** — the ROM-derived pair dataset (`datasets/match_pairs/`) and + the adapter weights (`models/`), both gitignored (`git check-ignore` confirmed), a Gen3 licensing decision. Cites the + decision-log entry "2026-07-08 · Phase 25 — the local-7B tier is capacity-bound and off the endgame critical path". + Wiring: `doc_links` default set (`--strict` PASS), SETUP row + the P33 E6 section (R21). Commit: see below. + **Block E is complete on the still-private repo** (E3–E6 in full); E1/E2 and D3's outward actions wait for the flip. -## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3 ✓, E4 ✓, E5 ✓ (32 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = E6 (2026-09-07, written by session 4555f4e4 "S88" at the E5 close; SUPERSEDES the earlier blocks) +## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3–E6 ✓ (33 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = G1 (2026-09-07, written by session 4555f4e4 "S88" at the E6 close; SUPERSEDES the earlier blocks) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest -(R1–R73), then WAIT for Drew. **NEXT = E6** (Max, ≈0.5–1 session — the drafter write-up `docs/matching-drafter-pipeline.md`; -no cut candidates remain: F3, E3, E4, E5 all ran in full). Rebuild the harness task list (40 items, R28) marking A1–A5, -B1–B9/C3, C1–C9, D1–D5, F1–F3, E3, E4, E5 completed and C10 in progress. **Every commit cites NEW +(R1–R73), then WAIT for Drew. **NEXT = G1** (Max, ≈0.5 session — `docs/gen3-handoff.md`; it has NO dependency on the flip, +so it is pulled ahead of the probe-gated items — a sequencing deviation from the plan's C11 → G1 order, recorded here; the +probe-gated items C10 → E1 → E2 → D3 outward → C11 stay on Drew's side until `probe_github.sh` prints PASS). Rebuild the +harness task list (40 items, R28) marking A1–A5, B1–B9/C3, C1–C9, D1–D5, F1–F3, E3–E6 completed and C10 in progress. **Every commit cites NEW hashes only** (the history was rewritten; `docs/commit-map.tsv` maps ordinals → new hashes; the scratch `.run/public_rewrite/` holds the old ones and stays until the probe passes). **Never `git clean -x`** (CLAUDE.md fail-safe). ### 1. Where we are **Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). The approved plan -is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (32):** A1–A5, B1–B9/C3, +is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (33):** A1–A5, B1–B9/C3, C1–C9 (the rewrite, adopted, force-pushed by Drew, gc'd), D1–D5 (README, LICENSE/NOTICE/THIRD_PARTY, badges/objdiff/frogress, SETUP public-clean, governing docs + `doc_links`), F1 (timeline + story), F2 (retrospective), **F3 (S88: the wiki — 12 files under `docs/wiki/` + the 13 how-to chapters under `docs/how-to-ai-decomp/`, `tools/wiki_render.py` + `tools/wiki_sync.sh`, @@ -590,17 +610,19 @@ with one commit under the noreply identity — Drew creates `Druthulu/xsig` EMPT upstream PR branch `reloc-masked-scorer` in `.run/P33/permuter-upstream/` + its tracked copy `tools/permuter/upstream/0001-reloc-masked-scorer.patch` + `docs/permuter-ils.md`; 10/10 tests, black clean, mypy at upstream's baseline, the real permuter 4 → 0 in 256 iterations where the default scorer read 3,585 — Drew pushes the branch -to his fork and opens the PR + files the issue; the commit that carries this block).** **In progress (Drew, C10):** the GitHub +to his fork and opens the PR + files the issue; `bf002f84d2`), **E6 (S88: `docs/matching-drafter-pipeline.md` — the +local-model tier's pipeline, its measured arc, the portable lessons, what is NOT published; the commit that carries this +block).** **In progress (Drew, C10):** the GitHub Support ticket (text: `docs/public-flip-runbook.md` §11 — its filing was never confirmed to S88; ask) and the daily `tools/public_rewrite/probe_github.sh` until it prints PASS (S88's run: **31 of 33 old hashes still ALIVE = the S87 baseline; -no purge yet**). **Remaining (8):** E6 (drafter write-up) — on the still-private repo; then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset +no purge yet**). **Remaining (7):** nothing more on the still-private repo except G1 (pre-flip-able) and G2 (after everything); then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset — Drew), E2 (Archipelago — Drew), D3's outward actions (decomp.dev registration, frogress slug/key — Drew), **the wiki push (Drew: Wiki → "Create the first page" in the GitHub UI, then `tools/wiki_sync.sh --push`)**; then C11 (aftercare), G1 (`docs/gen3-handoff.md`), G2 (the PhaseEnd v2.0.0 + DIGEST + `v2.0.0` tag; Tier 1; WAIT for gate 2). ### 2. Facts the remaining tasks depend on (measured S88; verify if in doubt, R14) - **Repository state:** `main` = the rewritten history (4,031 commits) + the S87 tip commits (C7 → F2) + the S88 commits - (`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, `9c4d32d651` E4, the E5 commit = HEAD); `origin/main` == the F2 commit + (`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, `9c4d32d651` E4, `bf002f84d2` E5, the E6 commit = HEAD); `origin/main` == the F2 commit `5e57e88de2` — **Drew pushed the S87 tip on 2026-09-07 07:23Z; the S88 commits are NOT pushed** (a normal fast-forward push; R6). The first-ever GitHub runs of both workflows were GREEN on that push (`no-rom` 1 m 35 s, run 34095194524; `progress` 15 s, run 34095194479) — read the Actions tab again after the next push, fix red, never claim green unseen (P9). The repo is @@ -662,7 +684,7 @@ no purge yet**). **Remaining (8):** E6 (drafter write-up) — on the still-priva `git -C .run/P33/xsig-repo remote add origin https://github.com/Druthulu/xsig.git && git -C .run/P33/xsig-repo push -u origin main`; E5 — fork `simonlindholm/decomp-permuter`, `git -C .run/P33/permuter-upstream remote add fork && git -C .run/P33/permuter-upstream push -u fork reloc-masked-scorer`, open the PR (the commit message is the description) and - file the issue from `docs/permuter-ils.md` §3. **E6** (Max, 0.5–1): `docs/matching-drafter-pipeline.md` from `docs/community-matching-model-plan.md`, + file the issue from `docs/permuter-ils.md` §3. **E6 is DONE too** (`docs/matching-drafter-pipeline.md`). The old E6 spec, for the record: `docs/matching-drafter-pipeline.md` from `docs/community-matching-model-plan.md`, `docs/gen2-mips-matching-model.md`, cookbook §12/§500, `docs/wave-playbook.md`; the ROM-derived pair dataset is NOT published. One commit per task after this file is updated; log; refresh this block. 3. **After the probe PASSES (Drew):** C10 the flip (Settings → Change visibility → Public, only with D/E/F landed) → E1 @@ -679,13 +701,13 @@ no purge yet**). **Remaining (8):** E6 (drafter write-up) — on the still-priva the milestone evidence, WAIT for gate 2; then `PhaseEnd_Phase33.md` v2.0.0 with the rule candidates (a)–(h), `CURRENT_PHASE.md` → `phase-ends/logs/Phase33.md`, DIGEST §0/§2/§3 appended, the annotated `v2.0.0` tag; Drew pushes `main --tags`). -### 4. Files S88 touched (6 commits after the F2 tip) +### 4. Files S88 touched (7 commits after the F2 tip) Tools (new): `tools/wiki_render.py`, `tools/wiki_sync.sh`, `tools/gccmap_cites.py`, `tools/xsig/` (xsig.py, README, LICENSE, tests/: fixture.c, make_fixtures.sh, test_xsig.py, fixture_a.txt, fixture_b.txt, fixture_a.s), `tools/permuter/upstream/0001-reloc-masked-scorer.patch`; changed: `tools/public_rewrite/probe_github.sh` (scratch-repo fetch + self-check), `tools/doc_links.py` (DEFAULT_GLOBS + the map README), `Makefile` (`wiki_render --selftest` + `gccmap_cites --check` + the xsig tests in tools-health), `.github/workflows/no-rom.yml` (the gccmap_cites + xsig steps). Docs (new): `docs/wiki/*.md` (12), -`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`, `docs/permuter-ils.md`; changed: the five map files (135 +`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`, `docs/permuter-ils.md`, `docs/matching-drafter-pipeline.md`; changed: the five map files (135 cites tagged in place), `docs/SETUP.md` (the probe clause in the public_rewrite row; 2 wiki rows; the P33 F3 section), `docs/public-flip-runbook.md` (§11: the R57 probe paragraph; the wiki push step), `docs/doc_links_pending.txt` (10 entries mid-task → EMPTY), `phase-ends/CURRENT_PHASE.md` (F3 ticked; the S88 preflight + F3 log diff --git a/tools/doc_links.py b/tools/doc_links.py index 9e29bd679..88dd3db83 100644 --- a/tools/doc_links.py +++ b/tools/doc_links.py @@ -21,7 +21,7 @@ DEFAULT = ["README.md", "THIRD_PARTY.md", "CLAUDE.md", "src/NOTICE.md", "tools/R "docs/verification.md", "docs/public-flip-runbook.md", "docs/decision-log.md", "docs/accelerators.md", "docs/story.md", "docs/story-timeline.md", "docs/retrospective.md", "phase-ends/README.md", "phase-ends/DIGEST.md", "docs/gcc-2.7.2-map/README.md", "tools/xsig/README.md", - "docs/permuter-ils.md"] + "docs/permuter-ils.md", "docs/matching-drafter-pipeline.md"] # whole directories in the default set (P33 F3): the wiki pages and the how-to chapters — every file, so a new page is # checked the moment it exists (the glob is expanded at run time; the count is printed with the rest, R41) DEFAULT_GLOBS = ["docs/wiki/*.md", "docs/how-to-ai-decomp/*.md"]