diff --git a/CLAUDE.md b/CLAUDE.md index 3340e975e..b8f070874 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ Matching decompilation of **Brave Fencer Musashi** (PS1, SLUS-00726 USA). Goal: candidate). Read these files in this order — nothing else at session start: 1. Read `PROJECT_CONTEXT.md` in full (the permanent constitution: rules, protocols, roadmap). -2. Read `phase-ends/DIGEST.md` in full — every phase's synopsis, every rule's full text (R1–R64), the corrections that +2. Read `phase-ends/DIGEST.md` in full — every phase's synopsis, every rule's full text (R1–R83), the corrections that supersede parts of PROJECT_CONTEXT.md, and the doc map. It REPLACES reading every PhaseEnd (the PhaseEnds are still kept forever — read an older one on demand when a synopsis is not enough; **never** read `phase-ends/logs/` at session start, R19). @@ -58,7 +58,7 @@ These rules govern every session. They are not suggestions. ## Fail-safe hard rules (duplicated from PROJECT_CONTEXT.md in case the above is ever skipped) -- **Never commit ROM-derived content.** The game dump, `disks/`, `asm/`, `assets/`, `build/`, `expected/`, extracted `.CD` contents, the Ghidra project (`ghidra/`), the RAM dumps (`dumps/*.bin`), Sony's SDK (`tools/psyq/`) and the session archive stay out of git — in force again since the public flip (Phase 33); `tools/audit_public.py` and CI enforce it. Review `git status` before every commit. +- **Never commit ROM-derived content.** The game dump, `disks/`, `asm/`, `assets/`, `build/`, `expected/`, extracted `.CD` contents, the Ghidra project (`ghidra/`), the RAM dumps (`dumps/*.bin`), Sony's SDK (`tools/psyq/`) and the session archive stay out of git — in force again since Phase 33 C3 (2026-09-06, the history rewrite); `tools/audit_public.py` and CI enforce it. Review `git status` before every commit. - **Never edit `PROJECT_CONTEXT.md`.** It is permanent and static. Corrections and state go in `phase-ends/`. - **Never `git clean -x` / `git clean -fdx` in this tree.** Since Phase 33 C3 (2026-09-06) the Ghidra project (`ghidra/`), the RAM dumps (`dumps/*.bin`), the PsyQ SDK (`tools/psyq/`), the session archive and the extension zips are IGNORED BUT PRESENT on disk — a `-x` clean deletes the reverse-engineering database. `make clean` is the only clean. (Regenerable in principle from `config/ghidra/` + the disc via `tools/ghidra_rebuild.sh`, but the dumps are not.) - **A "match" is byte-for-byte assembly equality AND the full-binary SHA1 check staying green.** Never report a functionally-equivalent function, a passing-looking build, or any unverified outcome as done. @@ -76,7 +76,7 @@ These rules govern every session. They are not suggestions. ## Reasoning & Model Protocol -Effort is set with `/effort` on the ladder `low / medium / high / xHigh / Max` (this is *depth* — how hard one agent reasons). **Max** = deepest single-agent reasoning, **session-only** (re-apply each session; xHigh is the highest level that persists). **Ultracode is NOT a deeper level** — it runs at xHigh and adds multi-agent orchestration (breadth, not depth); also session-only. **The project default working level is Max** (usage is not a constraint; the only downside is mild overthinking/latency on trivial tasks); drop to xHigh only for routine work when faster turns are wanted, and **never globally enable Ultracode for the deep tasks** (it would cap them at xHigh). Breadth is applied surgically — Claude proposes a Workflow fan-out for genuinely parallel tasks. +Effort is set with `/effort` on the ladder `low / medium / high / xHigh / Max` (this is *depth* — how hard one agent reasons). **Max** = deepest single-agent reasoning, **session-only** (re-apply each session; xHigh is the highest level that persists). **Ultracode is NOT a deeper level** — it runs at xHigh and adds multi-agent orchestration (breadth, not depth); also session-only. **The doctrine (Drew, 2026-07-04; `docs/effort-map.md` governs): xHigh for most tasks, Max for the deep tasks (phase plans, PhaseEnds, architectural forks, non-obvious debugging — plan mode always Max), Ultracode for breadth**; Max and Ultracode are session-only, and **never globally enable Ultracode for the deep tasks** (it would cap them at xHigh). Breadth is applied surgically — Claude proposes a Workflow fan-out for genuinely parallel tasks. **Effort-map check (rule — formalized as R7 at PhaseEnd_Phase2):** the full task→effort map, the Max-vs-Ultracode decision rule, verified harness mechanics, and per-phase guidance live in `docs/effort-map.md` (evolvable; it governs where this section's tier language differs). Before each Phase Start plan, state the recommended effort for *planning that phase* per that file and confirm the developer has it set; annotate every task in the phase plan with its effort level; restate the recommended effort whenever you present the NEXT task. The complexity tiers below map onto the ladder. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/CURRENT_PHASE.md index fa688e2a0..155edf35b 100644 --- a/phase-ends/CURRENT_PHASE.md +++ b/phase-ends/CURRENT_PHASE.md @@ -39,7 +39,7 @@ in-tree links to `docs/wiki/.md`. 8. `.run/`: only what git tracks; no sca ## Tasks (strict order; one commit per task after this file is updated; harness task list = the same 16 rows, R28) - [x] **0** Gate 1 close-out: this file (ledgers + checkpoint), R74–R83 into DIGEST §3 (+ §0), `phase34-seed.md` amended, TaskCreate ×16 — Max -- [ ] **1** Unblock, code side: tool docstrings re-pointed (`decl_from_use.py`, `integration_resolver.py`, `jtbl_carve.py`, `autopsy.py`, `residual_class.py`); CLAUDE.md lines 11 / 61 / the effort-default sentence — xHigh +- [x] **1** Unblock, code side: tool docstrings re-pointed (`decl_from_use.py`, `integration_resolver.py`, `jtbl_carve.py`, `autopsy.py`, `residual_class.py`); CLAUDE.md lines 11 / 61 / the effort-default sentence — xHigh — see Log 2026-09-07 Task 1 - [ ] **2** Unblock, doc side: banners (`concurrency-design`, `gen2-mips-matching-model`, `psyq-worklist`, `distill/*`, `hindsight-study` HISTORICAL + supersession table); SETUP §1a rewritten; the pin count re-derived (one dated number + its command, both gen3 docs) — xHigh - [ ] **3** Wiki — conventions + firewall: `Docs-and-scratch-conventions.md`, `The-ROM-firewall.md` (the gitignore template block); sidebar rows; Repository-layout/Contributing/ch.11 links; forward links in `doc_links_pending.txt` — Max - [ ] **4** Wiki — reference + direction: `Reference-index.md` (list-driven), `Where-the-project-goes-next.md`, Contributing "AI use — conduct", Home rewritten, sidebar sections — xHigh — **then P6 rules check** @@ -212,31 +212,54 @@ check whose shapes come from the offenders; disk-independent citation checks; th document for PA's own interview; dry-run write guardrails; a cut line). Spot-checked the claims the plan leans on (172 tracked-but- ignored; the two disassembly files; §265–§269 landed; the tools-health chain). R74–R83 written into DIGEST §3 from the Phase-33 candidates' text; DIGEST §0 notes the sub-phase; `phase34-seed.md` amended. Harness task list built (16 rows). Effort for task 1 is -xHigh — the transition is prompted (R27). +xHigh — the transition is prompted (R27). Commit `39d524991`. -## 🛑 SESSION CHECKPOINT — Task 0 ✓ (gate 1 closed, R74–R83 ratified); NEXT = task 1 (unblock, code side; xHigh) +### 2026-09-07 — Task 1 — Unblock, code side (xHigh; Drew set `/effort xhigh`) +Seven docstring/comment cites re-pointed to durable records: `decl_from_use.py` + `integration_resolver.py` → the decision log's +2026-08-25 (P31 S61) "the wall is an integration wall" entry (the S60 frontier analysis's 178/292 = 61% and ~571 finished-draft +measurements live there); `jtbl_carve.py` ×3 → cookbook §260 (the leading-island split; the review's correction #7); `autopsy.py` + +`residual_class.py` → `docs/how-to-ai-decomp/03-bootstrap-order.md` Phase 2b (the permuter track — task 5 adds the "mine the permuter's +failures" paragraph there; "originally hindsight-study §7, archived" kept as provenance). `CLAUDE.md`: R-range R1–R83; the H1 fail-safe +dates the re-tightening to Phase 33 C3 (2026-09-06, the history rewrite) — the flip has not happened; the Reasoning section's "default +working level is Max" replaced by the DIGEST §1 doctrine (xHigh for most tasks, Max for deep tasks, Ultracode for breadth; plan mode +always Max). Verify: `py_compile` on the five tools OK; `git grep -E 'tool-designs|hindsight-study' tools/ Makefile` = 0; `doc_links` +OK. **Deviation from the plan's verify column:** `make tools-health` was NOT run for docstring-only edits (proportionality — nothing it +measures changed); the first tools-health of the phase runs at task 7, where the Makefile and the checkers change. + +## 🛑 SESSION CHECKPOINT — Tasks 0–1 ✓; NEXT = task 2 (unblock, doc side: banners, SETUP §1a, the pin count; xHigh) ### 0. How to use this block You are a FRESH SESSION that has read `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, `PhaseEnd_Phase31/32/33.md` and this file, and nothing else (R64). Replay this block verbatim, state phase / done / NEXT / effort, list the rules from the digest (R1–R83), then -WAIT for Drew. Rebuild the harness task list (16 rows, R28) marking task 0 completed and task 1 in progress. +WAIT for Drew. Rebuild the harness task list (16 rows, R28) marking tasks 0–1 completed and task 2 in progress. ### 1. Where we are -**Phase 33.5** (sub-phase; v1.32.0 → v1.32.1), gate 1 approved 2026-09-07 by Drew in plan mode at Max. Baseline HEAD `80d45b29b`. -Task 0 is committed (this file, DIGEST §0/§3, `docs/phase34-seed.md`). Nothing under `src/`, `config/`, `Makefile` or `tools/` has -changed yet; the fleet is 218/218 at the Phase-33 close. +**Phase 33.5** (sub-phase; v1.32.0 → v1.32.1), gate 1 approved 2026-09-07 by Drew in plan mode at Max; Drew set `/effort xhigh` after +task 0 (the Max tasks are 3, 10, 12, 15 — prompt at each, R27). Baseline HEAD `80d45b29b`; task 0 = `39d524991`; task 1 = the commit +after it (docstrings + CLAUDE.md only). Nothing under `src/`, `config/` or the `Makefile` has changed; the fleet is 218/218 at the +Phase-33 close. `docs/doc_links_pending.txt` is empty (comments only). -### 2. What NEXT does (task 1, xHigh) — exact steps -1. `git grep -n -F 'tool-designs' tools/` and `git grep -n -F 'hindsight-study' tools/` → the docstring cites in `tools/decl_from_use.py`, - `tools/integration_resolver.py`, `tools/jtbl_carve.py` (→ the cookbook § that carries each finding: §8/§8a/§8b/§8e/§61b–d/§62 for the - jtbl lane, §236 for main-lane-fix, the S60 "571 already-drafted" finding → PhaseEnd_Phase31 / cookbook) and in `tools/autopsy.py`, - `tools/residual_class.py` (→ `docs/how-to-ai-decomp/03-bootstrap-order.md`'s "mine the permuter's failures" anchor, which task 5 - creates — cite the chapter file, the anchor lands in task 5). -2. `CLAUDE.md`: line 11 "R1–R64" → "R1–R83"; line 61 "since the public flip (Phase 33)" → "since Phase 33 C3 (2026-09-06)"; the - Reasoning section's "The project default working level is Max" sentence → the DIGEST §1 doctrine (xHigh for most tasks, Max for - deep tasks, Ultracode for breadth, plan mode always Max; Max/Ultracode session-only). Keep everything else verbatim (H5). -3. `make tools-health` in the FOREGROUND (~15 min; never background it) → OK; then this file's log + checkpoint; commit - `docs(phase-33.5): task 1 — …` by explicit path. Then prompt: task 2 is xHigh (no change). +### 2. What NEXT does (task 2, xHigh) — exact steps +1. Banners/status lines (H5: add, never drop): `docs/concurrency-design.md` — replace "**Not yet implemented.**" with "implemented in + `tools/shared_lock.py` + the gate snapshot layer (P31); cited by `gate_stage.py`, `bulk_harvest.py`, `dedup_propagate.py`, + `ox_campaign.py`, `verify_worktree.py`"; `docs/gen2-mips-matching-model.md` — "**Status:** design / not started" → built (P22–P23; + `train_lora.py`/`export_pairs.py`/`format_finetune.py`; distilled by `docs/matching-drafter-pipeline.md`); `docs/psyq-worklist.md` — a + top banner "COMPLETE at P31 S79 #5 — the residue (SSGM.o 8 ins inside matched C; the sound region's two cross-object-common walls) is + named at the tail"; `docs/distill/atbhbkbl.md` + `axbm.md` — replace the "PROPOSED ONLY" banner with "LANDED as cookbook §265, §266, + §267 (this file) / §268, §269 (axbm) on P31 S59b/c; kept as the A/B evidence ledger those sections cite"; `docs/hindsight-study.md` — + a top banner "HISTORICAL — written 2026-07-20 mid-P29 at ~78%; superseded part by part" + the supersession table (§0–§2 → how-to + ch.03 + `generic-decomp-package.md`; §3 → `portable-decomp-workflow.md`/the chapters; §4 → `gen3-handoff.md` §5 + wiki + Tools-from-this-project; §5 → `retrospective.md` §5; §6 → fulfilled by `retrospective.md` + the how-to + `tools/mine_hindsight.py`; + §7 → `tools/autopsy.py` — live pointer stays in ch.03). +2. `docs/SETUP.md` §1a (line ~68): rewrite — `.run/` is gitignored by the contents-exclude rule `/.run/*` with 22 dated `!` allowlist + blocks (R20: "commit what a rerun CANNOT reproduce"); what IS tracked (P32/P33 evidence, the giants recon, the R20 ledgers) and what + never is (the 29 GB of regenerable scratch); point at the wiki conventions page (task 3 — register the forward link in + `docs/doc_links_pending.txt` if you link it; or just name it in prose). +3. The pin count: run ONE grep over `src/` (e.g. `grep -rhoE 'register [^;]*__asm__\("\$[a-z0-9]+"\)' src | wc -l` — check what each doc's + number was derived from first: `gen3-handoff.md` §3 shows its six commands) and write the same dated number + the exact command into + `docs/gen3-standards.md` §4 and `docs/gen3-handoff.md` §3 (R75: a number in prose is a dated snapshot with its command). +4. Verify: `.venv/bin/python tools/doc_links.py` OK; the pin-count command reproduces the number; then this file's log + checkpoint; + commit by explicit path (`docs(phase-33.5): task 2 — …`). Task 3 is **Max** — prompt Drew and WAIT (R27). ### 3. Standing facts for every task of this phase - One commit per task, after this file's log line (R8/R42 form); commit by explicit path; no trailers (R5); Drew pushes (R6). diff --git a/tools/autopsy.py b/tools/autopsy.py index 914c3cc2f..0307cdce3 100644 --- a/tools/autopsy.py +++ b/tools/autopsy.py @@ -1,7 +1,8 @@ #!/usr/bin/env python3 """autopsy.py — the permuter-failure autopsy: mine the search-closer's FAILURES (Phase-29 Task-13). -docs/hindsight-study.md §7. Our flywheel (R16) has always evolved the offline tooling REACTIVELY +The permuter track of docs/how-to-ai-decomp/03-bootstrap-order.md (Phase 2b; originally §7 of the July-2026 +hindsight study, since archived). Our flywheel (R16) has always evolved the offline tooling REACTIVELY FROM SUCCESSES: a human or an LLM cracks a function, we distil the idiom into the cookbook and the permuter weights. The permuter's FAILURES — every near-miss it plateaued on — are a larger and completely unused signal. This tool turns that log into a corpus, classifies it deterministically, diff --git a/tools/decl_from_use.py b/tools/decl_from_use.py index 651430b8a..c00fc4f0d 100644 --- a/tools/decl_from_use.py +++ b/tools/decl_from_use.py @@ -1,7 +1,8 @@ #!/usr/bin/env python3 """decl_from_use.py — infer a draft's MISSING data extern from the target .s + the draft's own use. -WHY THIS EXISTS (P31 S61; docs/tool-designs/frontier-analysis-s60.md §5 step 4). Two refusal sites +WHY THIS EXISTS (P31 S61; the S60 frontier analysis — recorded in docs/decision-log.md, the 2026-08-25 +"the wall is an integration wall" entry). Two refusal sites throw away drafts whose only defect is a data symbol the destination TU never declares: * aprop_autodraft.py (~:522) refuses `no seed decl for D_x` — the member body is correct, but no seed/destination/fleet TU spells the extern, so the A-prop lane never emits the draft; diff --git a/tools/integration_resolver.py b/tools/integration_resolver.py index 984650f28..def1e1263 100644 --- a/tools/integration_resolver.py +++ b/tools/integration_resolver.py @@ -1,7 +1,8 @@ #!/usr/bin/env python3 """integration_resolver.py — the zero-token lane over drafts that are ALREADY CORRECT (P31 S61). -WHY THIS EXISTS. The S60 frontier analysis (docs/tool-designs/frontier-analysis-s60.md §4) measured +WHY THIS EXISTS. The S60 frontier analysis (recorded in docs/decision-log.md, the 2026-08-25 "the wall is +an integration wall" entry) measured that the gen6+ "wall" is an INTEGRATION wall, not a codegen wall: of the 292 functions the gate had refused six or more times, 178 (61%) had already produced a closeness-0 draft — match_one byte- equality at the object level, whole-binary gate rejection — and across the whole open pool ~571 diff --git a/tools/jtbl_carve.py b/tools/jtbl_carve.py index dc75522c4..cea80af8e 100644 --- a/tools/jtbl_carve.py +++ b/tools/jtbl_carve.py @@ -743,7 +743,7 @@ def island_split(ov, func): Writes ONLY config/splat..yaml (one inserted line). No overlays.mk var (a single-table piece needs none — §8e), no extract (the caller extracts). The island piece at 0x0 is left untouched: its extent shrinks automatically (end = next piece's offset) — the review-proven - correction to the `_pre` design (docs/tool-designs/jtbl-island-split-review.md #7).""" + correction to the `_pre` design (cookbook §260, the review's correction #7).""" kind, detail = island_probe(ov, func) if kind == "covered": print(f"jtbl_carve --island-split {ov} {func}: already split — no-op ({detail})") @@ -891,7 +891,7 @@ def build_carve(ov, funcs): # rewrites [tail_start, region_end), so the piece line would land out of address order # and splat would mis-slice the module. That is the island-split lane's job — ONE # inserted `.rodata` line at the table's own offset plus jr_isolate_all.py --only - # (docs/tool-designs/jtbl-island-split-review.md) — so name it and refuse (R43). + # (cookbook §260) — so name it and refuse (R43). isl_off = int(jh, 16) - base if isl_off < tail_start: sys.exit( @@ -900,7 +900,7 @@ def build_carve(ov, funcs): f"island, which a tail carve cannot reach. That is the island split (§260): " f"jr_isolate_all.py --only {f}, then `jtbl_carve.py {ov} --island-split " f"--func {f}` — automated at gate time by harvest_verify._jtbl_prep_one " - f"(docs/tool-designs/jtbl-island-split-review.md). " + f"(cookbook §260). " f"FOR AN md_* MODULE, §303 SUPERSEDES THIS: no isolation, no carve state — splice and " f"`make build`; the Makefile's jtbl_rodata_pads --derive stage reproduces the island pads at " f"build time (P32 S83: md_SC03_054/func_801EF6D8, 7 tables, 0,0t1,0t1,0t1,0t1,0t1,0).") diff --git a/tools/residual_class.py b/tools/residual_class.py index 2fe71006e..ec541411c 100644 --- a/tools/residual_class.py +++ b/tools/residual_class.py @@ -1,7 +1,8 @@ #!/usr/bin/env python3 """residual_class.py — the DETERMINISTIC residual→class classifier (Phase-29 Task-13A). -The permuter-failure autopsy (docs/hindsight-study.md §7) needs to route every plateau to one of +The permuter-failure autopsy (the permuter track of docs/how-to-ai-decomp/03-bootstrap-order.md, Phase 2b; +originally §7 of the July-2026 hindsight study, since archived) needs to route every plateau to one of missing-transform | mis-classified | seed-structural | genuine-wall and today that routing is done by an LLM reading a scalar (`closeness`) plus a free-text `where_stuck` label a wave agent typed months ago. Both are weak: 91% of the open backlog has NO