mirror of
https://github.com/Druthulu/BFM-decomp
synced 2026-09-27 05:56:00 -04:00
docs(phase-33): E6 — docs/matching-drafter-pipeline.md: the fine-tuned local matching drafter written up from the Phase 22–25 records (the toolchain-agnostic pipeline export_pairs → format_finetune → train_lora → eval_lora → serve_local → api_draft/lora_grind/bulk_harvest → the byte gate; the measured arc as one table — stock floor ~0, corpus-v2 85% on 6–15 ins, the 0/222 harness bugs, corpus-v3 57.5% held-out + ~352 production banks at $0, bulk_harvest 65% at 0.4 s/fn, GLM5.2 10/18 bodies but 3/18 banks = the def-side wall, v4 discarded; the portable lessons; hardware; a five-step recipe; what is NOT published — the ROM-derived pair dataset and the adapter weights, both gitignored); SETUP row + P33 E6 section; doc_links default; log + checkpoint (Block E complete; NEXT = G1, pulled ahead of the probe-gated items)
This commit is contained in:
Drew T
@@ -767,6 +767,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo
|
||||
| | `tools/gccmap_cites.py [--dry-run \| --check \| --verify \| --controls \| --explain] [--retag]` | **(P33 E3)** Tag every `file.c:NNN` cite in `docs/gcc-2.7.2-map/*.md` with the source tree its line number belongs to — `[2.7.2]` (the vanilla subset), `[2.8.1 pm]` (gcc-papermario), `[repo]` — derived from the trees (quoted snippets, identifiers with function extents and nearest distance, the author's cues; a contradiction fails loudly; ties → `cite_overrides.tsv` → cue → 2.7.2); writes in place, idempotent, never silently changes an existing tag. `--check` is textual (every cite tagged, no stale override) and runs in `make tools-health` + CI; the others need both reference trees and refuse without them. |
|
||||
| | `tools/xsig/xsig.py sign-s \| sign-objdump \| cross \| verify \| selftest` | **(P33 E4; Phase-21 origin)** Relocation-masked per-function signatures for CROSS-PROJECT code identification: mask `j`/`jal` targets and HI16/LO16 immediates (from `%hi`/`%lo` operands or `objdump -dr` records), keep opcodes/registers/constants/branches; `cross` joins two JSONL sets on `sig` with a coverage line; `verify` prints the instruction-by-instruction diff (opcode/register/immediate/length). Self-contained (stdlib, MIT, own README + LICENSE + `tests/` from a game-free fixture compiled at two link addresses with `--emit-relocs`); `tests/test_xsig.py` (8) in `make tools-health` + CI. The standalone repo copy is prepared under `.run/P33/xsig-repo/` (Drew creates + pushes `Druthulu/xsig`). |
|
||||
| | `tools/permuter/upstream/0001-reloc-masked-scorer.patch` | **(P33 E5)** The upstream PR as a `git format-patch` (one commit against `simonlindholm/decomp-permuter` main `41bd0bfc`, 2026-09-05): `src/reloc_scorer.py` (`RelocMaskedScorer`, a `Scorer` subclass), `--score-mode {mnemonic,reloc-masked}` + the `score_mode` settings key, docs, `test/test_reloc_scorer.py` (10, no cross toolchain). Regenerate the branch: clone upstream, `git am` the patch (proven clean). The scratch clone `.run/P33/permuter-upstream/` holds branch `reloc-masked-scorer` under the noreply identity; **Drew pushes it to his fork and opens the PR** (`docs/permuter-ils.md` §2). |
|
||||
| | `docs/matching-drafter-pipeline.md` | **(P33 E6)** The write-up of the fine-tuned local matching drafter (Phases 22–25): the pipeline (`export_pairs` → `format_finetune` → `train_lora` → `eval_lora` → `serve_local` → `api_draft` / `lora_grind` / `bulk_harvest` → the byte gate; `ab_score` + `workflows/ab_match.js`; `grinder`), every measurement in order (stock floor ~0 → v2 85% on 6–15 ins → v3 57.5% held-out and ~352 production banks → v4 negative → the GLM hard-band A/B and the def-side wall), the portable lessons, the hardware, a five-step recipe, and what is NOT published (the ROM-derived pair dataset `datasets/`, the adapter weights `models/` — both gitignored). |
|
||||
| | `tools/objdiff_report.py [--in docs/progress.json] [--out report.json]` | **(P33 D3)** progress.json → objdiff's report format (report.proto v2, snake_case — validated with `objdiff-cli` 3.8.1 `report changes`): one unit per binary (code = instructions × 4, functions byte-identical / matchable, metadata complete), categories `game-code` and `linked-sony-objects` (functions only). `.github/workflows/progress.yml` runs it on every push (no rebuild — the committed JSON) and uploads the artifact **`SLUS_007.26_report`** for decomp.dev (Drew registers at decomp.dev/manage/new after the flip). |
|
||||
| | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data/<project>/<version>/` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. |
|
||||
| | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. · `probe_github.sh [--after-flip]` (Drew's daily post-purge probe, C10: 33 sampled old shas via `gh api` + a fetch; **S88, R57:** the fetch runs in a throwaway bare repo under `.run/public_rewrite/` with `--filter=blob:none --depth=1`, never in the working repo — a successful fetch of an old sha imports its purged closure, which the S87/S88 runs did (5.97 GiB unreachable) — and it ends with a self-check naming any sampled old commit the working repo still holds + the gc recipe) |
|
||||
@@ -1146,6 +1147,20 @@ fills fast). Nothing is leaking — but the host does not get the memory back on
|
||||
`tools/ghidra_*.sh` are repo-relative (`BFM_GHIDRA_PROJ` overrides the project dir; `ghidra_mcp_verify.sh <addr> <name>
|
||||
[PROG]`); Makefile `GHIDRA_PROJ := $(or $(BFM_GHIDRA_PROJ),$(CURDIR)/ghidra)`.
|
||||
|
||||
### P33 E6 (S88, 2026-09-07) — `docs/matching-drafter-pipeline.md`, the drafter write-up
|
||||
- Distilled from `docs/gen2-mips-matching-model.md` (every measurement, 2026-06-29 → 07-08) and
|
||||
`docs/community-matching-model-plan.md` (the parked community release) with the drafting loops of cookbook §12 / §500:
|
||||
§1 why a specialist is safe under the byte gate, §2 the toolchain-agnostic pipeline (all 12 tools verified present),
|
||||
§3 the measured arc as one table (stock floor ~0; v1 39/41 trivial / 0/34 non-trivial; corpus-v2 85% on 6–15 ins;
|
||||
the 0/222 that was two harness bugs; 7/15 on open stubs; 0/15 on shared code → corpus-v3 with the 1,623 macro bodies →
|
||||
57.5% held-out, ~352 production banks, fleet +502 fns at $0; `bulk_harvest` 65% at 0.4 s/fn gate; GLM5.2 10/18 bodies
|
||||
but 3/18 banks — the def-side wall; v4 discarded; the 4.8× Haiku A/B), §4 the portable lessons, §5 hardware, §6 a
|
||||
five-step recipe, §7 what is published (the scripts, this doc) and what is NOT (the ROM-derived pair dataset
|
||||
`datasets/match_pairs/` and the adapter weights `models/` — both gitignored, confirmed with `git check-ignore`; their
|
||||
release is a Gen3 licensing decision). The 2026-07-08 decision-log entry ("the local-7B tier is capacity-bound and off
|
||||
the endgame critical path") is the cited strategic conclusion. In `doc_links`' default set; this row.
|
||||
|
||||
|
||||
### P33 E5 (S88, 2026-09-07) — the permuter upstream PR branch + `docs/permuter-ils.md`
|
||||
- **The PR (PR-1).** `RelocMaskedScorer` in `src/reloc_scorer.py` — the in-tree `masked_diff.diff_object_object` rule
|
||||
(mask from the TARGET's relocation records: `R_MIPS_26` → opcode only; the 16-bit-immediate relocs → opcode+rs+rt; a `j`
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
# The matching-drafter pipeline — a fine-tuned local model that writes byte-exact C, and what it measured
|
||||
|
||||
> **Status (P33 E6, 2026-09-07).** The write-up of BFM-decomp's *matching drafter*: a LoRA fine-tune of a small local
|
||||
> code model that drafts C for one exact toolchain (gcc-2.7.2-psx `-O2 -G0`, maspsx 2.56), gated by the project's
|
||||
> whole-binary byte gate so that a wrong draft can never land. Built and measured in Phases 22–25 (2026-06-29 → 07-08),
|
||||
> frozen at its third version, and off the endgame's critical path from 2026-07-08. The pipeline scripts are in
|
||||
> `tools/` and are published with the repository; **the paired dataset (`datasets/match_pairs/`, assembly ↔ matched C,
|
||||
> ROM-derived) and the adapter weights (`models/`, a transform of that dataset) are NOT published** — both directories
|
||||
> are gitignored, and their release is a separate licensing decision for Gen3. The design notes this distils are
|
||||
> [`docs/gen2-mips-matching-model.md`](gen2-mips-matching-model.md) (how it was built, every measurement) and
|
||||
> [`docs/community-matching-model-plan.md`](community-matching-model-plan.md) (what a community release would look
|
||||
> like); the drafting loops it plugs into are cookbook §12 (the parallel-draft + byte-gate harvest) and §500 (the
|
||||
> one-agent-per-function pass).
|
||||
|
||||
## 1. The idea, and why it is safe
|
||||
|
||||
There is no matching-decompilation model: the one reverse-engineering model family targets x86 and optimises for
|
||||
readable, semantically equivalent C, not byte-identical output under a specific 1990s compiler. The reason is data —
|
||||
the only place (target assembly ↔ byte-exact C) pairs exist is inside decompilation projects' histories. BFM had
|
||||
~1,300 such pairs by Phase 22 (later ~2,900 with the shared macro bodies), each gate-verified, each specific to the
|
||||
exact toolchain. That is the corpus a specialist needs and a generalist lacks.
|
||||
|
||||
What makes a weak or wrong model *safe* to use at all is the project's byte gate: the model only affects
|
||||
**throughput**, never correctness — a draft is substituted into the real translation unit, the binary is rebuilt, and
|
||||
it is kept only if the SHA1 still matches ([the matching workflow](wiki/The-matching-workflow.md)). So the model can be
|
||||
small, local, free and frequently wrong, and still bank thousands of functions.
|
||||
|
||||
## 2. The pipeline (toolchain-agnostic; only the data and the compile command are project-specific)
|
||||
|
||||
```
|
||||
tools/export_pairs.py mine (asm .s ↔ matched C) pairs from the project's banked functions — plain `src/` definitions
|
||||
AND the shared `DEFINE_func_*` macro bodies — with the `extern` block each definition carries
|
||||
tools/format_finetune.py chat-template examples; the project's shared types inlined; a compile filter (self-contained?)
|
||||
tools/train_lora.py QLoRA on a code base model (Qwen2.5-Coder-7B: dense, permissive, fits a 12 GB card)
|
||||
tools/eval_lora.py held-out, GATE-TRUE evaluation: the adapter drafts unseen banked functions; the gate decides
|
||||
tools/serve_local.py serve base + adapter as an OpenAI-style endpoint (GPU; Unsloth/transformers)
|
||||
tools/api_draft.py the drafter client (the LEAN prompt; the same prompt for every tier — local, cheap-cloud, frontier)
|
||||
tools/lora_grind.py draft open stubs → gate → bank → propagate (interleaved; the first production loop)
|
||||
tools/bulk_harvest.py the phase-separated loop: (A) bulk-draft on the GPU, (B) parallel per-binary byte gates on the
|
||||
CPU, (C) dedupe once + one commit
|
||||
tools/ab_score.py, tools/workflows/ab_match.js controlled A/B of two drafter arms on a fixed target set, gate-scored
|
||||
tools/grinder.py the permuter over the near-misses the model leaves (CPU, concurrent with the GPU)
|
||||
```
|
||||
|
||||
Two disciplines that the measurements below justify: **the prompt format at training time is byte-identical to the
|
||||
prompt at inference** (any drift wastes the specialisation), and **every evaluation number is gate-true** — a
|
||||
whole-binary match, never a standalone `match_one` proxy (which over-counts about 2×).
|
||||
|
||||
## 3. What it measured, in order
|
||||
|
||||
| Step (date) | Corpus | Result (gate-true unless stated) | What it taught |
|
||||
|---|---|---|---|
|
||||
| Stock local model floor (06-29) | — (Qwen3.6-35B-A3B, prompt only) | ~0 reliable banks: structure right, gcc-2.7.2 precision wrong (offset scaling, `lh`/`lhu`, an extra `move`, the frame size); a full-cookbook prompt was worse and 2.3× slower than a curated subset | more context is not the lever; precision must be in the weights |
|
||||
| v1: 7B QLoRA, 3 epochs (06-29) | 638 compile-filtered pairs (297 trivial) | ≤5 ins **39/41**; ≥6 ins **0/34**; near-misses FAR (`near N ≈ nins`) | the compile filter had thrown out the 536 hard functions; a 7B memorised the leaf pattern |
|
||||
| corpus-v2: the `extern` block captured (06-29) | 1,111 pairs, non-trivial 257 → 813; standalone compile 52% → 92% | 6–15 ins **85% (23/27)**; 16–40 ins 13% (3/22); >40 ins 0/10 (compile-fail: struct types) | **data quality was the bottleneck, not the model** |
|
||||
| The 0/222 broad run (06-30) | v2 | banked **0/222** across 25 overlays | two harness bugs (a SHA string with its filename; hardcoded per-binary paths) — not the model; fixed with a loud negative-control guard (R40) |
|
||||
| First trustworthy open-stub signal (06-30) | v2 | **7/15 (47%)** on the smallest open stubs of one overlay; 4 near-misses at closeness 1 | most small open stubs are overlay-unique (×1) — a bank-*rate* lever, not a fleet-% lever |
|
||||
| reach ≥ 2 targeting (06-30) | v2 | **0/15** on shared functions; the model drafted `return 1` and `sw` setters as *empty* bodies | 96.6% of the corpus was overlay-unique inline definitions — the shared setters were never seen |
|
||||
| corpus-v3: + 1,623 shared macro bodies; struct types inlined; the "never an empty body" prompt clause (06-30) | 2,891 pairs (2,534 train / 291 test), 97.8% compile | held-out **23/40 (57.5%)**; production: **~352 banks** + 45 new shared groups, fleet 63.67% → 63.82% (+502 functions), 136/136 clean, $0 | the previously impossible setter/leaf class banked; the flywheel (banks → corpus → retrain) closed |
|
||||
| `bulk_harvest` phase separation (07-01) | v3 | **52/80 = 65%** on fresh ≤15-ins stubs across 50 overlays; gate 0.4 s/fn amortised (≈75× the interleaved loop); drafting 13.5 s/fn = 97% of wall clock | the bottleneck moved from gating to inference; batched serving became justified |
|
||||
| GLM5.2 (cloud reasoning model) A/B on the hard band 16–22 ins (07-01) | v3 vs GLM | `match_one` 1/18 vs **10/18**; banks 1/18 vs **3/18** (+1 by reconciliation); ~$1.25 total | a better drafter's bodies were right and *still* did not bank: the **def-side declaration wall** caps every drafter (7 of 10 stranded); the reasoning model given the wall directly banked 1/7 |
|
||||
| v4: retrain on the post-giant corpus (07-08) | 3,574 pairs (+994 medium, +597 large) | easy tie (5/5 both); medium **v3 better** (v4 farther, 4 compile-fails); hard 0/10 both | **discard v4** — the 7B is capacity-bound; over-length examples had trained on cut-off completions |
|
||||
| Cheap-tier A/B (Phase 22) | Haiku drafters under an Opus orchestrator | same match rate as Opus on ≤ ~50-ins functions at ≈4.8× lower cost per match | the model ladder: Haiku → Sonnet → Opus → frontier only for a new wall class |
|
||||
|
||||
**The strategic conclusion** (decision log, "2026-07-08 · Phase 25 — the local-7B tier is capacity-bound and off the
|
||||
endgame critical path"): the engine that finished the project was *frontier crack → deterministic propagation
|
||||
(families, dedup) → byte gate*, with the permuter softening the tail; the local model stayed a frozen $0 mop-up for
|
||||
the ≤ ~15-instruction setter/leaf tail. A real capability jump needs a bigger base (14B four-bit on the card, 32B in
|
||||
the cloud) or the frontier tier — not more data on a 7B.
|
||||
|
||||
## 4. The lessons, stated portably
|
||||
|
||||
- **Data is the bottleneck, not model size.** Every jump came from the corpus: the `extern` block (v2), the shared
|
||||
macro bodies and inlined struct types (v3), one prompt clause. Retraining on more but harder data regressed (v4).
|
||||
- **Completions must be self-contained** — the model emits the `extern`s and types it references, or it compiles in
|
||||
isolation and fails in the real translation unit (the integration cap every drafter shares — see
|
||||
[chapter 10](how-to-ai-decomp/10-integration-and-propagation.md)).
|
||||
- **Corpus quality > size; weight by regime and residual class**, not by count — one overlay-unique pattern drowned
|
||||
the shared setters until the shared bodies were mined explicitly.
|
||||
- **Dense beats MoE for a small LoRA corpus; the model size must match the data size** (a 7B on 638 examples
|
||||
memorised; on 2,891 it generalised; on 3,574 harder ones it saturated).
|
||||
- **Drop over-length examples rather than train on truncated ones; train at the context you infer at** (v4's medium
|
||||
regression traced to completions cut at `maxlen`).
|
||||
- **The byte gate turns model quality into a throughput question** — iterate fearlessly, and never let an
|
||||
intermediate metric stand in for the gate (R14, R63).
|
||||
- **Exonerate the harness before the model** (R40): the 0/222 that nearly enshrined "the tail is exhausted" was two
|
||||
bugs in the gate's caller.
|
||||
- **Route by measured difficulty; run the permuter concurrently** (GPU drafts, CPU permutes) on the close near-misses
|
||||
the model leaves; target shared functions deliberately (×reach), and expect the declaration wall to cap any drafter.
|
||||
|
||||
## 5. Hardware, as used
|
||||
|
||||
A single 12 GB consumer card (RTX 3080 Ti) trained the 7B QLoRA (≈2 h for 3 epochs at batch 1 / maxlen 2048, VRAM-tight)
|
||||
and served it (Unsloth/transformers; the prebuilt llama.cpp CUDA wheels crashed on a CPU without AVX-512). Serving and
|
||||
training contend for the one GPU — a retrain pauses the harvest; a second card breaks that. 14B and above: rent an
|
||||
A100/H100 — the byte gate means a wrong big model is only a throughput cost.
|
||||
|
||||
## 6. Train one for your own project
|
||||
|
||||
1. Have the byte gate and at least a few hundred gate-verified pairs (the pipeline mines them from your banked sources).
|
||||
2. `export_pairs` with the declarations each body needs; `format_finetune` with your shared types inlined; keep the
|
||||
compile filter and *count what it drops* — the dropped set is where your hard functions are.
|
||||
3. Fix the prompt format once; use it for training, evaluation and production alike.
|
||||
4. QLoRA a dense 7B; evaluate **gate-true on held-out functions** before any production run; compare against the
|
||||
stock model and against a cheap cloud tier on the same targets (`ab_score`).
|
||||
5. Run it as the lowest rung of a model ladder, phase-separated (bulk draft, parallel gate, one commit), on the
|
||||
population it wins (small, overlay-unique, leaf/setter); hand its close near-misses to the permuter; retrain only
|
||||
when the corpus changes in *kind*, not size.
|
||||
|
||||
## 7. What is and is not published
|
||||
|
||||
Published with the repository: every script above, this write-up, the design notes. **Not published:** the pair
|
||||
dataset (assembly from the game ↔ matched C — ROM-derived; `datasets/`, gitignored) and the adapter weights (`models/`,
|
||||
gitignored; a learned transform of that dataset, treated conservatively). A community release — weights with a model
|
||||
card naming the exact toolchain, a toolchain-agnostic training tutorial, and the ambitious *toolchain-conditioned*
|
||||
universal model trained on pooled multi-project pairs (decomp.me is the ready labelled corpus) — is designed in
|
||||
[`docs/community-matching-model-plan.md`](community-matching-model-plan.md) and parked for Gen3, gated on resolving
|
||||
the licensing of pooled ROM-derived pairs per source project.
|
||||
+34
-12
@@ -82,7 +82,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo
|
||||
- [x] **E3** gcc-2.7.2 map README + `gccmap_cites.py` — xHigh — see Log 2026-09-07 E3
|
||||
- [x] **E4** xsig packaging — xHigh — see Log 2026-09-07 E4
|
||||
- [x] **E5** permuter upstream PR branch — Max (FULL, Drew 2026-09-07) — see Log 2026-09-07 E5
|
||||
- [ ] **E6** drafter write-up — Max
|
||||
- [x] **E6** drafter write-up — Max — see Log 2026-09-07 E6
|
||||
- [ ] **E1** decomp.me preset (after the flip) — xHigh
|
||||
- [ ] **E2** Archipelago outreach (after the flip) — xHigh
|
||||
- [ ] **C11** Aftercare (other clones, archive remote removed, `.run/` pruned, DIGEST/decision-log) — xHigh
|
||||
@@ -564,21 +564,41 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact
|
||||
flushed stdout R55; winner ≠ bank), §5 related. SETUP: row + the P33 E5 section (R21). **Drew:** fork upstream,
|
||||
`git -C .run/P33/permuter-upstream push -u fork reloc-masked-scorer`, open the PR (the commit message is the
|
||||
description), file the issue. Commit: see below.
|
||||
- **2026-09-07 (S88, Max) — E6 `docs/matching-drafter-pipeline.md`, the drafter write-up.** Distilled from
|
||||
`docs/gen2-mips-matching-model.md` (every measurement of the local-model tier, 2026-06-29 → 07-08) and
|
||||
`docs/community-matching-model-plan.md` (the parked community release), with the drafting loops of cookbook §12 / §500:
|
||||
§1 why a specialist is safe under the byte gate (the model affects throughput, never correctness); §2 the
|
||||
toolchain-agnostic pipeline — `export_pairs` → `format_finetune` → `train_lora` → `eval_lora` → `serve_local` →
|
||||
`api_draft` / `lora_grind` / `bulk_harvest` → the byte gate; `ab_score` + `workflows/ab_match.js`; `grinder` — every
|
||||
tool verified present; §3 the measured arc as ONE table with dates (the stock 35B floor ~0; v1 39/41 trivial and 0/34
|
||||
non-trivial on 638 pairs; corpus-v2 with the `extern` block: 85% on 6–15 ins; the 0/222 broad run that was two harness
|
||||
bugs, R40; 7/15 on open stubs; 0/15 on shared code → corpus-v3 with the 1,623 shared macro bodies + the "never an empty
|
||||
body" clause: 57.5% held-out, ~352 production banks, fleet +502 fns at $0; `bulk_harvest` 65% with a 0.4 s/fn gate;
|
||||
GLM5.2 on the hard band: 10/18 bodies right, 3/18 banked — the def-side declaration wall caps every drafter; v4
|
||||
discarded — the 7B is capacity-bound; the 4.8× Haiku A/B); §4 the portable lessons (data > size; self-contained
|
||||
completions; weight by regime; dense > MoE; drop over-length examples; train at the inference context; the gate makes
|
||||
quality a throughput question; exonerate the harness; route by measured difficulty); §5 hardware; §6 a five-step
|
||||
recipe for another project; §7 **what is NOT published** — the ROM-derived pair dataset (`datasets/match_pairs/`) and
|
||||
the adapter weights (`models/`), both gitignored (`git check-ignore` confirmed), a Gen3 licensing decision. Cites the
|
||||
decision-log entry "2026-07-08 · Phase 25 — the local-7B tier is capacity-bound and off the endgame critical path".
|
||||
Wiring: `doc_links` default set (`--strict` PASS), SETUP row + the P33 E6 section (R21). Commit: see below.
|
||||
**Block E is complete on the still-private repo** (E3–E6 in full); E1/E2 and D3's outward actions wait for the flip.
|
||||
|
||||
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3 ✓, E4 ✓, E5 ✓ (32 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = E6 (2026-09-07, written by session 4555f4e4 "S88" at the E5 close; SUPERSEDES the earlier blocks)
|
||||
## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3–E6 ✓ (33 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = G1 (2026-09-07, written by session 4555f4e4 "S88" at the E6 close; SUPERSEDES the earlier blocks)
|
||||
|
||||
### 0. How to use this block
|
||||
You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file,
|
||||
and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest
|
||||
(R1–R73), then WAIT for Drew. **NEXT = E6** (Max, ≈0.5–1 session — the drafter write-up `docs/matching-drafter-pipeline.md`;
|
||||
no cut candidates remain: F3, E3, E4, E5 all ran in full). Rebuild the harness task list (40 items, R28) marking A1–A5,
|
||||
B1–B9/C3, C1–C9, D1–D5, F1–F3, E3, E4, E5 completed and C10 in progress. **Every commit cites NEW
|
||||
(R1–R73), then WAIT for Drew. **NEXT = G1** (Max, ≈0.5 session — `docs/gen3-handoff.md`; it has NO dependency on the flip,
|
||||
so it is pulled ahead of the probe-gated items — a sequencing deviation from the plan's C11 → G1 order, recorded here; the
|
||||
probe-gated items C10 → E1 → E2 → D3 outward → C11 stay on Drew's side until `probe_github.sh` prints PASS). Rebuild the
|
||||
harness task list (40 items, R28) marking A1–A5, B1–B9/C3, C1–C9, D1–D5, F1–F3, E3–E6 completed and C10 in progress. **Every commit cites NEW
|
||||
hashes only** (the history was rewritten; `docs/commit-map.tsv` maps ordinals → new hashes; the scratch `.run/public_rewrite/`
|
||||
holds the old ones and stays until the probe passes). **Never `git clean -x`** (CLAUDE.md fail-safe).
|
||||
|
||||
### 1. Where we are
|
||||
**Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). The approved plan
|
||||
is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (32):** A1–A5, B1–B9/C3,
|
||||
is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (33):** A1–A5, B1–B9/C3,
|
||||
C1–C9 (the rewrite, adopted, force-pushed by Drew, gc'd), D1–D5 (README, LICENSE/NOTICE/THIRD_PARTY, badges/objdiff/frogress,
|
||||
SETUP public-clean, governing docs + `doc_links`), F1 (timeline + story), F2 (retrospective), **F3 (S88: the wiki — 12 files
|
||||
under `docs/wiki/` + the 13 how-to chapters under `docs/how-to-ai-decomp/`, `tools/wiki_render.py` + `tools/wiki_sync.sh`,
|
||||
@@ -590,17 +610,19 @@ with one commit under the noreply identity — Drew creates `Druthulu/xsig` EMPT
|
||||
upstream PR branch `reloc-masked-scorer` in `.run/P33/permuter-upstream/` + its tracked copy
|
||||
`tools/permuter/upstream/0001-reloc-masked-scorer.patch` + `docs/permuter-ils.md`; 10/10 tests, black clean, mypy at
|
||||
upstream's baseline, the real permuter 4 → 0 in 256 iterations where the default scorer read 3,585 — Drew pushes the branch
|
||||
to his fork and opens the PR + files the issue; the commit that carries this block).** **In progress (Drew, C10):** the GitHub
|
||||
to his fork and opens the PR + files the issue; `bf002f84d2`), **E6 (S88: `docs/matching-drafter-pipeline.md` — the
|
||||
local-model tier's pipeline, its measured arc, the portable lessons, what is NOT published; the commit that carries this
|
||||
block).** **In progress (Drew, C10):** the GitHub
|
||||
Support ticket (text: `docs/public-flip-runbook.md` §11 — its filing was never confirmed to S88; ask) and the daily
|
||||
`tools/public_rewrite/probe_github.sh` until it prints PASS (S88's run: **31 of 33 old hashes still ALIVE = the S87 baseline;
|
||||
no purge yet**). **Remaining (8):** E6 (drafter write-up) — on the still-private repo; then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset
|
||||
no purge yet**). **Remaining (7):** nothing more on the still-private repo except G1 (pre-flip-able) and G2 (after everything); then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset
|
||||
— Drew), E2 (Archipelago — Drew), D3's outward actions (decomp.dev registration, frogress slug/key — Drew), **the wiki push
|
||||
(Drew: Wiki → "Create the first page" in the GitHub UI, then `tools/wiki_sync.sh --push`)**; then C11 (aftercare), G1
|
||||
(`docs/gen3-handoff.md`), G2 (the PhaseEnd v2.0.0 + DIGEST + `v2.0.0` tag; Tier 1; WAIT for gate 2).
|
||||
|
||||
### 2. Facts the remaining tasks depend on (measured S88; verify if in doubt, R14)
|
||||
- **Repository state:** `main` = the rewritten history (4,031 commits) + the S87 tip commits (C7 → F2) + the S88 commits
|
||||
(`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, `9c4d32d651` E4, the E5 commit = HEAD); `origin/main` == the F2 commit
|
||||
(`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, `9c4d32d651` E4, `bf002f84d2` E5, the E6 commit = HEAD); `origin/main` == the F2 commit
|
||||
`5e57e88de2` — **Drew pushed the S87 tip on 2026-09-07 07:23Z; the S88 commits are NOT pushed** (a normal fast-forward push;
|
||||
R6). The first-ever GitHub runs of both workflows were GREEN on that push (`no-rom` 1 m 35 s, run 34095194524; `progress`
|
||||
15 s, run 34095194479) — read the Actions tab again after the next push, fix red, never claim green unseen (P9). The repo is
|
||||
@@ -662,7 +684,7 @@ no purge yet**). **Remaining (8):** E6 (drafter write-up) — on the still-priva
|
||||
`git -C .run/P33/xsig-repo remote add origin https://github.com/Druthulu/xsig.git && git -C .run/P33/xsig-repo push -u origin
|
||||
main`; E5 — fork `simonlindholm/decomp-permuter`, `git -C .run/P33/permuter-upstream remote add fork <fork-url> && git -C
|
||||
.run/P33/permuter-upstream push -u fork reloc-masked-scorer`, open the PR (the commit message is the description) and
|
||||
file the issue from `docs/permuter-ils.md` §3. **E6** (Max, 0.5–1): `docs/matching-drafter-pipeline.md` from `docs/community-matching-model-plan.md`,
|
||||
file the issue from `docs/permuter-ils.md` §3. **E6 is DONE too** (`docs/matching-drafter-pipeline.md`). The old E6 spec, for the record: `docs/matching-drafter-pipeline.md` from `docs/community-matching-model-plan.md`,
|
||||
`docs/gen2-mips-matching-model.md`, cookbook §12/§500, `docs/wave-playbook.md`; the ROM-derived pair dataset is NOT published.
|
||||
One commit per task after this file is updated; log; refresh this block.
|
||||
3. **After the probe PASSES (Drew):** C10 the flip (Settings → Change visibility → Public, only with D/E/F landed) → E1
|
||||
@@ -679,13 +701,13 @@ no purge yet**). **Remaining (8):** E6 (drafter write-up) — on the still-priva
|
||||
the milestone evidence, WAIT for gate 2; then `PhaseEnd_Phase33.md` v2.0.0 with the rule candidates (a)–(h), `CURRENT_PHASE.md`
|
||||
→ `phase-ends/logs/Phase33.md`, DIGEST §0/§2/§3 appended, the annotated `v2.0.0` tag; Drew pushes `main --tags`).
|
||||
|
||||
### 4. Files S88 touched (6 commits after the F2 tip)
|
||||
### 4. Files S88 touched (7 commits after the F2 tip)
|
||||
Tools (new): `tools/wiki_render.py`, `tools/wiki_sync.sh`, `tools/gccmap_cites.py`, `tools/xsig/` (xsig.py, README, LICENSE,
|
||||
tests/: fixture.c, make_fixtures.sh, test_xsig.py, fixture_a.txt, fixture_b.txt, fixture_a.s),
|
||||
`tools/permuter/upstream/0001-reloc-masked-scorer.patch`; changed: `tools/public_rewrite/probe_github.sh` (scratch-repo fetch +
|
||||
self-check), `tools/doc_links.py` (DEFAULT_GLOBS + the map README), `Makefile` (`wiki_render --selftest` + `gccmap_cites --check` + the xsig tests in
|
||||
tools-health), `.github/workflows/no-rom.yml` (the gccmap_cites + xsig steps). Docs (new): `docs/wiki/*.md` (12),
|
||||
`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`, `docs/permuter-ils.md`; changed: the five map files (135
|
||||
`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`, `docs/permuter-ils.md`, `docs/matching-drafter-pipeline.md`; changed: the five map files (135
|
||||
cites tagged in place), `docs/SETUP.md` (the probe clause in the public_rewrite row;
|
||||
2 wiki rows; the P33 F3 section), `docs/public-flip-runbook.md` (§11: the R57 probe paragraph; the wiki push step),
|
||||
`docs/doc_links_pending.txt` (10 entries mid-task → EMPTY), `phase-ends/CURRENT_PHASE.md` (F3 ticked; the S88 preflight + F3 log
|
||||
|
||||
+1
-1
@@ -21,7 +21,7 @@ DEFAULT = ["README.md", "THIRD_PARTY.md", "CLAUDE.md", "src/NOTICE.md", "tools/R
|
||||
"docs/verification.md", "docs/public-flip-runbook.md", "docs/decision-log.md", "docs/accelerators.md",
|
||||
"docs/story.md", "docs/story-timeline.md", "docs/retrospective.md", "phase-ends/README.md", "phase-ends/DIGEST.md",
|
||||
"docs/gcc-2.7.2-map/README.md", "tools/xsig/README.md",
|
||||
"docs/permuter-ils.md"]
|
||||
"docs/permuter-ils.md", "docs/matching-drafter-pipeline.md"]
|
||||
# whole directories in the default set (P33 F3): the wiki pages and the how-to chapters — every file, so a new page is
|
||||
# checked the moment it exists (the glob is expanded at run time; the count is printed with the rest, R41)
|
||||
DEFAULT_GLOBS = ["docs/wiki/*.md", "docs/how-to-ai-decomp/*.md"]
|
||||
|
||||
Reference in New Issue
Block a user