diff --git a/docs/SETUP.md b/docs/SETUP.md index e86b6750e..b66244d9d 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -768,6 +768,7 @@ Every script under `tools/` (plus the two report make-targets), grouped by purpo | | `tools/xsig/xsig.py sign-s \| sign-objdump \| cross \| verify \| selftest` | **(P33 E4; Phase-21 origin)** Relocation-masked per-function signatures for CROSS-PROJECT code identification: mask `j`/`jal` targets and HI16/LO16 immediates (from `%hi`/`%lo` operands or `objdump -dr` records), keep opcodes/registers/constants/branches; `cross` joins two JSONL sets on `sig` with a coverage line; `verify` prints the instruction-by-instruction diff (opcode/register/immediate/length). Self-contained (stdlib, MIT, own README + LICENSE + `tests/` from a game-free fixture compiled at two link addresses with `--emit-relocs`); `tests/test_xsig.py` (8) in `make tools-health` + CI. The standalone repo copy is prepared under `.run/P33/xsig-repo/` (Drew creates + pushes `Druthulu/xsig`). | | | `tools/permuter/upstream/0001-reloc-masked-scorer.patch` | **(P33 E5)** The upstream PR as a `git format-patch` (one commit against `simonlindholm/decomp-permuter` main `41bd0bfc`, 2026-09-05): `src/reloc_scorer.py` (`RelocMaskedScorer`, a `Scorer` subclass), `--score-mode {mnemonic,reloc-masked}` + the `score_mode` settings key, docs, `test/test_reloc_scorer.py` (10, no cross toolchain). Regenerate the branch: clone upstream, `git am` the patch (proven clean). The scratch clone `.run/P33/permuter-upstream/` holds branch `reloc-masked-scorer` under the noreply identity; **Drew pushes it to his fork and opens the PR** (`docs/permuter-ils.md` §2). | | | `docs/matching-drafter-pipeline.md` | **(P33 E6)** The write-up of the fine-tuned local matching drafter (Phases 22–25): the pipeline (`export_pairs` → `format_finetune` → `train_lora` → `eval_lora` → `serve_local` → `api_draft` / `lora_grind` / `bulk_harvest` → the byte gate; `ab_score` + `workflows/ab_match.js`; `grinder`), every measurement in order (stock floor ~0 → v2 85% on 6–15 ins → v3 57.5% held-out and ~352 production banks → v4 negative → the GLM hard-band A/B and the def-side wall), the portable lessons, the hardware, a five-step recipe, and what is NOT published (the ROM-derived pair dataset `datasets/`, the adapter weights `models/` — both gitignored). | +| | `docs/gen3-handoff.md` | **(P33 G1)** The Gen3 seed: where Gen2 ends, the owner's next intent (casts → structs, pins off, names), the starter census with the commands that re-derive it (143 raw address casts · 61,898 `D_` · 16,335 `func_` · 44,243 register pins · 1,083 symbol-file entries · 1,232 struct defs · 2,220 dedup groups · 5 verbatim bodies), the one invariant (every edit byte-gated; shared bodies change every member; types are comprehension not bytes), the inherited levers, shiftability honestly scoped (position-locked slots, LZSS not byte-stable, the Sony regions), the parked ideas, and the governance for a new generation. | | | `tools/objdiff_report.py [--in docs/progress.json] [--out report.json]` | **(P33 D3)** progress.json → objdiff's report format (report.proto v2, snake_case — validated with `objdiff-cli` 3.8.1 `report changes`): one unit per binary (code = instructions × 4, functions byte-identical / matchable, metadata complete), categories `game-code` and `linked-sony-objects` (functions only). `.github/workflows/progress.yml` runs it on every push (no rebuild — the committed JSON) and uploads the artifact **`SLUS_007.26_report`** for decomp.dev (Drew registers at decomp.dev/manage/new after the flip). | | | `tools/frogress_upload.py [--push --project bfm --version us]` | **(P33 D3)** stdlib; `--dry-run` is the default (prints the payload); `--push` POSTs `{"api_key","entries":[{git_hash,timestamp,categories:{default:{measures…}}}]}` to `progress.deco.mp/data///` with `FROGRESS_API_SECRET` from the environment (never a file). frogress projects are admin-created — Drew requests the slug + key after the flip. | | | `tools/public_rewrite/` (P33 C1) | **The history-rewrite package** (`docs/public-flip-runbook.md` §3 is the operating table). `common.py` (shared: the purge rules, the DERIVED content-hash sets, identities from the log, the one hash regex, a persistent `cat-file --batch`) · `hash_dict.py [--write-mailmap]` (every commit OBJECT → `commit:NNNN` / twin / orphan; prefix index 7..40; asserts 0 ambiguous; records content-hash collisions as excluded; writes the scratch mailmap) · `scrub.py --test \| --sample \| --file` (THE scrub: hash tokens, addresses → noreply, trailer lines in messages; 12 known-true cases; the HEAD sample with git's own object lookup as the independent oracle) · `gate_scan.py --all\|--refs … [--worktree] [--expect-fail FIXTURE]` (paths ever touched × purge rules; every reachable blob's content sha1 × the ROM set; 5 byte signatures; 50 MiB; emits `rom_blob_ids.txt` = hits ∪ every blob ever under a purge path; the fixture `expected_offenders.txt` is the R39 negative control) · `run_filter.py [--sample]` (the git-filter-repo 2.47.0 module-API run inside the scratch bare clone; refuses elsewhere) · `verify_rewrite.py --old --new` (the pairwise proof) · `build_commit_map.py [--out]` (`docs/commit-map.tsv`, asserted free of old hashes) · `resolve_tokens.py [--check] [--map]` (tokens → shortest unique ≥9-char new abbreviations at the tip) · `absent_scan.py [--repo] [--tree]` (nothing old anywhere) · `probe_github.sh [--after-flip]` (Drew's purge probe). Scratch (`.run/public_rewrite/`, never committed): `dict.json`, `mailmap`, `rom_blob_ids.txt`, `old-to-new.tsv`, `repo.git`, the bundle. · `probe_github.sh [--after-flip]` (Drew's daily post-purge probe, C10: 33 sampled old shas via `gh api` + a fetch; **S88, R57:** the fetch runs in a throwaway bare repo under `.run/public_rewrite/` with `--filter=blob:none --depth=1`, never in the working repo — a successful fetch of an old sha imports its purged closure, which the S87/S88 runs did (5.97 GiB unreachable) — and it ends with a self-check naming any sampled old commit the working repo still holds + the gc recipe) | @@ -1147,6 +1148,15 @@ fills fast). Nothing is leaking — but the host does not get the memory back on `tools/ghidra_*.sh` are repo-relative (`BFM_GHIDRA_PROJ` overrides the project dir; `ghidra_mcp_verify.sh [PROG]`); Makefile `GHIDRA_PROJ := $(or $(BFM_GHIDRA_PROJ),$(CURDIR)/ghidra)`. +### P33 G1 (S88, 2026-09-07) — `docs/gen3-handoff.md` +- The census was DERIVED from the tree at writing (R33/R41; the commands are in the doc): 143 raw address casts, 61,898 + distinct `D_80xxxxxx`, 16,335 distinct `func_80xxxxxx` — the plan's three numbers reproduced exactly — plus 44,243 + register-pin declarations (the "pins off" campaign's true size; batch by family, a shared body's pins come off once + for every member), 1,258 `INCLUDE_ASM` tiles (all Sony regions), 1,083 symbol-file entries (1,081 curated), 1,232 + struct definitions in `engine_types.h` (many drafter-invented variants — `lift_types.py`'s collision landscape), 2,220 + dedup groups. Every lever/doc the handoff names was checked to exist. In `doc_links`' default set; this row. + + ### P33 E6 (S88, 2026-09-07) — `docs/matching-drafter-pipeline.md`, the drafter write-up - Distilled from `docs/gen2-mips-matching-model.md` (every measurement, 2026-06-29 → 07-08) and `docs/community-matching-model-plan.md` (the parked community release) with the drafting loops of cookbook §12 / §500: diff --git a/docs/gen3-handoff.md b/docs/gen3-handoff.md new file mode 100644 index 000000000..b5e5aedeb --- /dev/null +++ b/docs/gen3-handoff.md @@ -0,0 +1,136 @@ +# Gen3 handoff — where Gen2 ends, what comes next, and the levers already in hand + +> **Written at the Gen2 exit (Phase 33, 2026-09-07).** Gen2's contract is met and published: 218 binaries rebuild +> byte-identical from C ([`docs/verification.md`](verification.md)), every game-code function in every binary is C, the +> repository is public with its full rewritten history. This page is the seed for Gen3's first plan — it names the +> owner's stated next intent, gives the starter census derived from the tree (with the commands, so the next session +> re-derives rather than trusts), states the one invariant that must survive every Gen3 edit, and inventories the +> levers, documents and parked ideas Gen3 inherits. The constitution's Gen3 line ("shiftable build, asset repack, native +> recomp / PC port, randomizer-grade tooling") is unchanged; the order below is a recommendation for plan mode. + +## 1. Where Gen2 ends + +| | | +|---|---| +| Binaries byte-identical | 218 / 218 — the executable, the resident engine, 141 `ov_*` overlays, 75 `md_*` modules | +| Functions | 363,214 / 363,214; 360,737 in C (255,632 shared bodies via 2,220 dedup groups); 1,256 Sony PsyQ functions linked (or carried as 1,258 `INCLUDE_ASM` tiles without the SDK); 5 hand-written-assembly bodies kept verbatim | +| What is not C | Sony's objects (by design) and the 5 verbatim bodies (`config/verbatim_manifest.json`, PERMANENT) | +| The record | `phase-ends/` (33 PhaseEnds + the digest), `docs/decision-log.md`, `docs/matching-cookbook.md`, `docs/gcc-2.7.2-map/`, `docs/story.md`, `docs/retrospective.md`, the wiki and `docs/how-to-ai-decomp/` | + +The matching frontier is empty. Gen3's work is therefore of a different kind: **making the C legible and movable +without changing a byte**, and then the things a legible, movable source enables. + +## 2. The owner's next intent: readability and shiftability preparation + +Stated at the Phase-33 plan: *casts → structs, pins off, names.* Concretely: + +1. **Raw addresses → declared data.** Every `*(type *)0x80xxxxxx` cast becomes a reference to a declared symbol with + a type; every `D_80xxxxxx` that is a field of a known structure becomes `actor->field`. +2. **Register pins off.** Every `register … __asm__("$N")` that a banked body still carries is removed and the body + re-gated; Phase 32's finding is that the pins were symptoms — every one on the last four functions came off + byte-identical once the source *shape* was right (cookbook §501-E/P, R73). +3. **Names.** `func_80xxxxxx` / `D_80xxxxxx` → meaningful names, curated in the symbol files and mirrored into + Ghidra (G6, R15) — never edited in generated assembly. + +## 3. The starter census (derived 2026-09-07 — re-derive, do not trust) + +```bash +grep -rhoE '\*\)\s*0x80[0-9A-Fa-f]{6}' src --include=*.c --include=*.h | wc -l # raw address casts +grep -rhoE '\bD_80[0-9A-Fa-f]{6}\b' src --include=*.c --include=*.h | sort -u | wc -l # distinct data symbols +grep -rhoE '\bfunc_80[0-9A-Fa-f]{6}\b' src --include=*.c --include=*.h | sort -u | wc -l # distinct function names +grep -rhoE 'register\s+[^;]*__asm__' src --include=*.c --include=*.h | wc -l # register pins +grep -rhoE '^\s*INCLUDE_ASM\(' src --include=*.c | wc -l # assembly tiles +cat config/symbols.us*.txt | grep -cE '^[A-Za-z_]' # symbol-file entries +``` + +| Quantity | Value | Note | +|---|---|---| +| Raw address casts `*(T *)0x80…` | **143** | the low-hanging fruit; each becomes a declared symbol | +| Distinct `D_80xxxxxx` data symbols | **61,898** | most are fields of a handful of structures the engine indexes; `docs/actor-struct.md` recovered one of them (base `0x80078E00`, ~154 fields, live-verified) | +| Distinct `func_80xxxxxx` names | **16,335** | across 4,287 C files; the overlays' shared engine functions have one name each fleet-wide (position-locked) | +| Register-pin declarations | **44,243** | a campaign, not a chore — but batched by family, since a shared body's pins come off once for every member | +| `INCLUDE_ASM` tiles | 1,258 | all inside the executable's linked Sony regions; not Gen3 work | +| Symbol-file entries | 1,083 (1,081 curated names, 2 address-named) | `config/symbols.us*.txt` — the names the build already knows | +| Struct definitions in `src/shared/engine_types.h` | 1,232 | many are drafter-invented variants of the same type (`tools/lift_types.py` knows the collision landscape) | +| Dedup groups | 2,220 (255,708 instances) | a rename or a type change inside a shared body reaches every member | +| Verbatim `__asm__` bodies | 5 | PERMANENT; the manifest audits drift | + +## 4. The one invariant + +**Every Gen3 edit is gated exactly like a match was.** `make check BINARY=` for every binary a change touches, +and the clean fleet run (`make clean && make extract-all && make check-all` → `218 passed, 0 failed of 218`) after +anything that touches a shared body, a shared header or the executable — never an incremental check on the executable +(R22). Read the exit code (R53). Commit the moment a batch is green (R42). A rename is a symbol-file change mirrored +into Ghidra by the headless script, never a hand edit of assembly (G6, R15). Readability work is *safe* for precisely +the reason drafting was: a wrong edit cannot land — but it is also *slow* for the same reason, so batch by family and +gate in parallel worktrees ([the matching workflow](wiki/The-matching-workflow.md)). + +Two subtleties the census hides: + +- **A shared body is one source, many binaries.** Removing a pin or renaming a field inside a `DEFINE_func_…()` macro in + `src/shared/engine_core.h` changes up to 138 binaries at once; the gate must run on all of them, and a member whose + bytes happen to depend on that pin (they should not, but the gate decides) has to be split out of the group first. +- **Types are a comprehension lever, not a byte lever.** Phase 17 proved that recovering the actor structure and giving + it to the decompiler produced identical bytes (`docs/struct-core-pivot.md`): the compiler does not care what you call + a field. That is good news for Gen3 — struct-ification is byte-neutral by construction *as long as the layout and + the access widths are right* — and it is why every such edit still goes through the gate. + +## 5. Levers Gen3 inherits + +| Lever | What it does | Where | +|---|---|---| +| `tools/lift_types.py` | Lifts a named list of types fleet-wide into the shared header, picking the canonical (majority) definition among the drafter-invented variants and stripping the local copies; reports the variant users so the reconcile is targeted; dry-run by default | the struct-ification workhorse | +| `tools/cast_call_sites.py` | Per-site function-pointer / callee casts (cookbook §17a-1) so a call keeps its byte-exact argument codegen while the declaration becomes canonical | when a name/type change alters a call's conversions | +| `tools/canon_sig_reconcile.py` | Reconciles a definition to its canonical signature byte-neutrally (typedef strip, positional param types, casts at the uses — never through a fresh local, which shifts register allocation) | signature clean-ups | +| `tools/sync_tu_decls.py`, `fix_arity_callers.py`, `cast_self_callers.py`, `restore_dropped_decls.py`, `decl_from_use.py` | the declaration layer of the reconcile ladder ([chapter 10](how-to-ai-decomp/10-integration-and-propagation.md)) | any edit that touches declarations | +| `tools/alloc_table.py`, `tools/cc1_dumps_tu.sh` | the allocation order and the per-pass RTL dumps from the real translation unit — read before touching a pin (R73) | the pin-removal campaign | +| `tools/dedup_propagate.py`, `family_sweep.py`, `twin_rescan.py`, `config/dedup.us.yaml` | the propagation and registry machinery; `dedup_integrate --check` refuses a drifted share | keeping 2,220 groups consistent through renames | +| `tools/ghidra_apply_symbols.sh`, `tools/ghidra_scripts/ApplySymbols.java`, `config/ghidra/` + `tools/ghidra_rebuild.sh --proof` | names reach the Ghidra database only this way; the database is regenerable from text | the naming campaign | +| `tools/atlas.py` / `atlas_features.py` | per-function feature layer and similarity groups over the whole fleet | grouping candidates for a rename or a struct | +| `docs/actor-struct.md`, `docs/idxtab-map.md`, `docs/memory-map.md` | the recovered actor structure; the load/index-table map of the 218 payloads; every address with provenance | the ground truth for naming and typing | +| `docs/struct-core-pivot.md`, `docs/wall-taxonomy.md` | what struct recovery can and cannot do; the residual classes | expectations | + +## 6. Shiftability, honestly scoped + +A *shiftable* build is one whose addresses can move. Three facts frame it: + +1. **Overlays are position-locked** at one slot (`0x80128158`) by the loader, the resident at `0x800CEDF8`, the modules + at their slots (`docs/memory-map.md` §S44/§S45). Shifting code *within* a slot (a function grows) needs only that + every cross-reference be symbolic — which is what §2's cast → symbol work delivers — plus the data-side references + the splitter still emits as literal addresses in rodata/jump tables. +2. **LZSS recompression is not byte-stable** (`docs/formats.md` §4.6): a rebuilt overlay cannot be re-encoded to the + original compressed bytes. The verification layer is therefore the *decompressed* payload (as today); a rebuilt disc + is a Gen3 deliverable that verifies by booting, not by hash (the constitution's stage 5). +3. **The executable's Sony regions are linked objects**; shifting them means relinking with the real SDK (or the tiles). + +Recommended order: symbolic references first (byte-neutral, gated), then a `-Ttext`-shifted overlay that boots in +PCSX-Redux, then the disc rebuild with recompressed payloads. + +## 7. The parked Gen3 ideas, with their state + +- **Asset export / native rebuild** (`docs/gen3-parking-lot.md`, read-only survey 2026-07-01): rendering is stock + libgpu; model data goes through libgs `GsMapModelingData` (a documented TMD path — verified by real call sites); + animation is custom; textures are a VRAM rip away. Difficulty read and the decisive probes are in that file. +- **Native recompilation / PC port** — the original brief's ambition, properly sequenced after readability. +- **Randomizer-grade tooling** — the Archipelago world's RAM map is already cited in `docs/memory-map.md`; E2's + outreach note is the first contact. +- **The community matching model** — pipeline published ([`docs/matching-drafter-pipeline.md`](matching-drafter-pipeline.md)); + the dataset and weights wait on a licensing decision. +- **JP (SLPS-01490) and the prototypes as extra versions** — the two prototype executables are already imported + programs with tracked annotations (`config/ghidra/ROSTER.md`). +- **The decomp.me preset** (E1) and **libs-from-source for the Sony regions** — small and large stretches respectively. + +## 8. Governance for Gen3 + +The framework carries over unchanged: the constitution, the session protocol, the digest, the two gates, one task at +a time, the rules R1–R73 plus the Phase-33 candidates (a)–(h) once ratified. A new generation starts with a fresh +plan in plan mode at Max. Three things the record says to do first: + +1. **Measure the shape before choosing** (`docs/hindsight-study.md` §2.1): which structures own most of the 61,898 + data symbols, and which families own most of the 44,243 pins — a census with a self-asserting scanner, checked + against a case whose answer is known. +2. **Build the differential harness for the new question** before the campaign: "is this rename/type change + byte-neutral?" has two paths — the per-binary gate and the fleet run — and a scanner that asserts every reference + to a renamed symbol was rewritten (R32). +3. **Batch by leverage**: a pin in a shared body comes off for 138 binaries at once; a struct that explains a thousand + `D_` symbols is worth more than a hundred one-off names. Rank, then start. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index 64770cab8..1c1a78a0b 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -86,7 +86,7 @@ one-time snapshot, `CLAUDE.md` gains "never `git clean -x`" (R20 amendment propo - [ ] **E1** decomp.me preset (after the flip) — xHigh - [ ] **E2** Archipelago outreach (after the flip) — xHigh - [ ] **C11** Aftercare (other clones, archive remote removed, `.run/` pruned, DIGEST/decision-log) — xHigh -- [ ] **G1** `docs/gen3-handoff.md` — Max +- [x] **G1** `docs/gen3-handoff.md` — Max — see Log 2026-09-07 G1 - [ ] **G2** PhaseEnd v2.0.0 + DIGEST + archive this file + tag — Max, Tier 1, WAIT for gate 2 Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifacts immediately (R42); count from the SOURCE. @@ -583,22 +583,41 @@ Mid-phase rules check after every 4 completed tasks (P6). Commit banked artifact decision-log entry "2026-07-08 · Phase 25 — the local-7B tier is capacity-bound and off the endgame critical path". Wiring: `doc_links` default set (`--strict` PASS), SETUP row + the P33 E6 section (R21). Commit: see below. **Block E is complete on the still-private repo** (E3–E6 in full); E1/E2 and D3's outward actions wait for the flip. +- **2026-09-07 (S88, Max) — G1 `docs/gen3-handoff.md` (pulled ahead of the probe-gated C10 → E1 → E2 → D3-outward → C11 + chain: it depends on none of them — a sequencing deviation from the plan's C11 → G1 order, recorded).** §1 where Gen2 + ends (the contract, what is not C); §2 the owner's next intent (casts → structs, pins off, names); §3 the starter census + DERIVED from the tree with its commands (R33/R41): **143** raw address casts, **61,898** distinct `D_80xxxxxx`, + **16,335** distinct `func_80xxxxxx` — the plan's numbers reproduced exactly — and the numbers the plan did not have: + **44,243** register-pin declarations (the "pins off" campaign's true size — batch by family), 1,258 `INCLUDE_ASM` + tiles (all Sony regions), 1,083 symbol-file entries (1,081 curated), 1,232 struct definitions in `engine_types.h` + (drafter-invented variants included), 2,220 dedup groups, 5 verbatim bodies; §4 the one invariant (every edit + byte-gated like a match; a shared body changes every member; types are a comprehension lever, not a byte lever — + Phase 17's byte-neutral struct recovery); §5 the inherited levers (`lift_types`, `cast_call_sites`, + `canon_sig_reconcile`, the declaration ladder, `alloc_table` + `cc1_dumps_tu` for pins, the propagation/registry + machinery, the Ghidra text path, the atlas, the actor struct / idxtab map / memory map — every one checked to exist); + §6 shiftability honestly scoped (position-locked slots, LZSS recompression not byte-stable → a rebuilt disc verifies + by booting, the Sony regions relink); §7 the parked Gen3 ideas with their state (asset export survey, recomp, randomizer + tooling, the community model's licensing gate, JP/protos, the preset, libs-from-source); §8 governance for a new + generation and the three things to do first (measure the shape; a differential harness for "byte-neutral?"; batch by + leverage). Wiring: `doc_links` default (`--strict` PASS), SETUP row + the P33 G1 section (R21). Commit: see below. -## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3–E6 ✓ (33 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = G1 (2026-09-07, written by session 4555f4e4 "S88" at the E6 close; SUPERSEDES the earlier blocks) +## 🛑 SESSION CHECKPOINT — A1–A5 ✓, B1–B9/C3 ✓, C1–C9 ✓, D1–D5 ✓, F1–F3 ✓, E3–E6 ✓, G1 ✓ (34 of 41); C10 IN PROGRESS ON DREW'S SIDE; NEXT = THE PROBE-GATED CHAIN (Drew) then C11 then G2 (2026-09-07, written by session 4555f4e4 "S88" at the G1 close; SUPERSEDES the earlier blocks) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase30/31/32.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest -(R1–R73), then WAIT for Drew. **NEXT = G1** (Max, ≈0.5 session — `docs/gen3-handoff.md`; it has NO dependency on the flip, -so it is pulled ahead of the probe-gated items — a sequencing deviation from the plan's C11 → G1 order, recorded here; the -probe-gated items C10 → E1 → E2 → D3 outward → C11 stay on Drew's side until `probe_github.sh` prints PASS). Rebuild the -harness task list (40 items, R28) marking A1–A5, B1–B9/C3, C1–C9, D1–D5, F1–F3, E3–E6 completed and C10 in progress. **Every commit cites NEW +(R1–R73), then WAIT for Drew. **Everything Claude can do before the flip is DONE** (34 of 41; G1 was pulled ahead of the +plan's C11 → G1 order because it depends on nothing gated — recorded as a sequencing deviation). **NEXT = the probe-gated +chain on Drew's side**: `tools/public_rewrite/probe_github.sh` PASS → C10 the flip → E1 → E2 → D3 outward → the wiki push → +then Claude's **C11** (aftercare) → **G2** (the PhaseEnd, Tier 1, WAIT for gate 2). If the probe still fails, there is +nothing to draft: ask Drew about the Support ticket and stop. Rebuild the harness task list (40 items, R28) marking A1–A5, +B1–B9/C3, C1–C9, D1–D5, F1–F3, E3–E6, G1 completed and C10 in progress. **Every commit cites NEW hashes only** (the history was rewritten; `docs/commit-map.tsv` maps ordinals → new hashes; the scratch `.run/public_rewrite/` holds the old ones and stays until the probe passes). **Never `git clean -x`** (CLAUDE.md fail-safe). ### 1. Where we are **Phase 33 — 100% verification + the public flip + Gen2 exit.** Gate 1 approved 2026-09-06 (plan mode, Max). The approved plan -is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (33):** A1–A5, B1–B9/C3, +is VERBATIM at the end of this file — its Blocks E–G paragraphs are the specs for what remains. **Done (34):** A1–A5, B1–B9/C3, C1–C9 (the rewrite, adopted, force-pushed by Drew, gc'd), D1–D5 (README, LICENSE/NOTICE/THIRD_PARTY, badges/objdiff/frogress, SETUP public-clean, governing docs + `doc_links`), F1 (timeline + story), F2 (retrospective), **F3 (S88: the wiki — 12 files under `docs/wiki/` + the 13 how-to chapters under `docs/how-to-ai-decomp/`, `tools/wiki_render.py` + `tools/wiki_sync.sh`, @@ -611,18 +630,19 @@ upstream PR branch `reloc-masked-scorer` in `.run/P33/permuter-upstream/` + its `tools/permuter/upstream/0001-reloc-masked-scorer.patch` + `docs/permuter-ils.md`; 10/10 tests, black clean, mypy at upstream's baseline, the real permuter 4 → 0 in 256 iterations where the default scorer read 3,585 — Drew pushes the branch to his fork and opens the PR + files the issue; `bf002f84d2`), **E6 (S88: `docs/matching-drafter-pipeline.md` — the -local-model tier's pipeline, its measured arc, the portable lessons, what is NOT published; the commit that carries this -block).** **In progress (Drew, C10):** the GitHub +local-model tier's pipeline, its measured arc, the portable lessons, what is NOT published; `98e5846662`), **G1 (S88: +`docs/gen3-handoff.md` — the derived census incl. 44,243 register pins, the invariant, the levers, shiftability scoped, the +parked ideas, the governance for Gen3; the commit that carries this block).** **In progress (Drew, C10):** the GitHub Support ticket (text: `docs/public-flip-runbook.md` §11 — its filing was never confirmed to S88; ask) and the daily `tools/public_rewrite/probe_github.sh` until it prints PASS (S88's run: **31 of 33 old hashes still ALIVE = the S87 baseline; -no purge yet**). **Remaining (7):** nothing more on the still-private repo except G1 (pre-flip-able) and G2 (after everything); then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset +no purge yet**). **Remaining (6):** nothing more that Claude can do before the flip; then, gated on the probe PASS: C10 (the flip — Drew), E1 (decomp.me preset — Drew), E2 (Archipelago — Drew), D3's outward actions (decomp.dev registration, frogress slug/key — Drew), **the wiki push (Drew: Wiki → "Create the first page" in the GitHub UI, then `tools/wiki_sync.sh --push`)**; then C11 (aftercare), G1 (`docs/gen3-handoff.md`), G2 (the PhaseEnd v2.0.0 + DIGEST + `v2.0.0` tag; Tier 1; WAIT for gate 2). ### 2. Facts the remaining tasks depend on (measured S88; verify if in doubt, R14) - **Repository state:** `main` = the rewritten history (4,031 commits) + the S87 tip commits (C7 → F2) + the S88 commits - (`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, `9c4d32d651` E4, `bf002f84d2` E5, the E6 commit = HEAD); `origin/main` == the F2 commit + (`214d0dd15b` the probe fix, `954362c81e` F3 pages + tooling, `0cf971d1f4` the F3 wiring, `50c1b69e4d` E3, `9c4d32d651` E4, `bf002f84d2` E5, `98e5846662` E6, the G1 commit = HEAD); `origin/main` == the F2 commit `5e57e88de2` — **Drew pushed the S87 tip on 2026-09-07 07:23Z; the S88 commits are NOT pushed** (a normal fast-forward push; R6). The first-ever GitHub runs of both workflows were GREEN on that push (`no-rom` 1 m 35 s, run 34095194524; `progress` 15 s, run 34095194479) — read the Actions tab again after the next push, fix red, never claim green unseen (P9). The repo is @@ -701,13 +721,13 @@ no purge yet**). **Remaining (7):** nothing more on the still-private repo excep the milestone evidence, WAIT for gate 2; then `PhaseEnd_Phase33.md` v2.0.0 with the rule candidates (a)–(h), `CURRENT_PHASE.md` → `phase-ends/logs/Phase33.md`, DIGEST §0/§2/§3 appended, the annotated `v2.0.0` tag; Drew pushes `main --tags`). -### 4. Files S88 touched (7 commits after the F2 tip) +### 4. Files S88 touched (8 commits after the F2 tip) Tools (new): `tools/wiki_render.py`, `tools/wiki_sync.sh`, `tools/gccmap_cites.py`, `tools/xsig/` (xsig.py, README, LICENSE, tests/: fixture.c, make_fixtures.sh, test_xsig.py, fixture_a.txt, fixture_b.txt, fixture_a.s), `tools/permuter/upstream/0001-reloc-masked-scorer.patch`; changed: `tools/public_rewrite/probe_github.sh` (scratch-repo fetch + self-check), `tools/doc_links.py` (DEFAULT_GLOBS + the map README), `Makefile` (`wiki_render --selftest` + `gccmap_cites --check` + the xsig tests in tools-health), `.github/workflows/no-rom.yml` (the gccmap_cites + xsig steps). Docs (new): `docs/wiki/*.md` (12), -`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`, `docs/permuter-ils.md`, `docs/matching-drafter-pipeline.md`; changed: the five map files (135 +`docs/how-to-ai-decomp/*.md` (13), `docs/gcc-2.7.2-map/README.md` + `cite_overrides.tsv`, `docs/permuter-ils.md`, `docs/matching-drafter-pipeline.md`, `docs/gen3-handoff.md`; changed: the five map files (135 cites tagged in place), `docs/SETUP.md` (the probe clause in the public_rewrite row; 2 wiki rows; the P33 F3 section), `docs/public-flip-runbook.md` (§11: the R57 probe paragraph; the wiki push step), `docs/doc_links_pending.txt` (10 entries mid-task → EMPTY), `phase-ends/CURRENT_PHASE.md` (F3 ticked; the S88 preflight + F3 log diff --git a/tools/doc_links.py b/tools/doc_links.py index 88dd3db83..94edcf77b 100644 --- a/tools/doc_links.py +++ b/tools/doc_links.py @@ -21,7 +21,8 @@ DEFAULT = ["README.md", "THIRD_PARTY.md", "CLAUDE.md", "src/NOTICE.md", "tools/R "docs/verification.md", "docs/public-flip-runbook.md", "docs/decision-log.md", "docs/accelerators.md", "docs/story.md", "docs/story-timeline.md", "docs/retrospective.md", "phase-ends/README.md", "phase-ends/DIGEST.md", "docs/gcc-2.7.2-map/README.md", "tools/xsig/README.md", - "docs/permuter-ils.md", "docs/matching-drafter-pipeline.md"] + "docs/permuter-ils.md", "docs/matching-drafter-pipeline.md", + "docs/gen3-handoff.md"] # whole directories in the default set (P33 F3): the wiki pages and the how-to chapters — every file, so a new page is # checked the moment it exists (the glob is expanded at run time; the count is printed with the rest, R41) DEFAULT_GLOBS = ["docs/wiki/*.md", "docs/how-to-ai-decomp/*.md"]