diff --git a/phase-ends/Phase8_PLAN.md b/phase-ends/Phase8_PLAN.md new file mode 100644 index 0000000..dd4d5d5 --- /dev/null +++ b/phase-ends/Phase8_PLAN.md @@ -0,0 +1,254 @@ +# Phase 8 Plan — Coordinated Multi-Session Matching + +**Status:** DRAFT — requires explicit developer approval before activation. Do not begin any task until the developer approves this plan. +**Planning effort:** Max +**Prepared:** 2026-09-23, immediately after Phase 7 closure. + +## Session start (for the coordinating session) + +Read in order before doing anything: `AGENTS.md`, `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`, +`phase-ends/PhaseEnd_Phase7.md`, this plan, then `docs/PHASE7_EXTENTS.md`, `docs/PHASE7_DUPES.md`, +`docs/PHASE7_TRIAGE.md`, `docs/MATCHING_CONVENTIONS.md`, `docs/MATCHING_COOKBOOK.md`. There is no +`CURRENT_PHASE.md` at the start of this phase; the coordinating session must create one when activated. + +## Goal + +Phase 7 made matching *possible* at scale. Phase 8 makes it **parallel**: three pi sessions on one +machine, coordinated over pi-intercom, with one session acting as **coordinator** — it dispatches +charters, answers questions, merges what comes back, and independently verifies every claim — and two +**worker** sessions matching functions concurrently. The coordinator does not compete for matching work; +its job is throughput through supervision plus the verification the byte gate demands. + +Target: at least **70 distinct matched bodies** (from 34), with every one of them verified by the +coordinator's own clean full-binary gate rather than by a worker's report. + +## The difference from every previous phase + +| | Phases 0–7 | Phase 8 | +|---|---|---| +| Sessions | one | three (1 coordinator, 2 workers) | +| Work discovery | by the session doing the work | coordinator generates disjoint partitions | +| Trust | the session's own claim | **no claim is trusted**; the coordinator re-verifies | +| Shared files | edited freely | workers may not touch them at all | +| Commits | by the working session | coordinator only | +| Verification | at the end of a task | every merge cycle, plus the phase gate | + +## Concurrency safety (the part that decides whether this works) + +All three sessions share **one worktree**. Two sessions editing `config/regions.tsv` or running `make` +concurrently would corrupt each other's work. The protocol therefore fixes the following as hard rules, +and the coordinator enforces them: + +| Resource | Rule | Why | +|---|---|---| +| `src/*.c` | Each worker may **create** files only for addresses in its own partition | Partitions are disjoint by construction, so no two sessions can write the same file | +| `config/regions.tsv`, `config/symbols.tsv`, `config/match_worklist.tsv` | **Coordinator only.** Workers never edit them | Single-writer rule for the tracked registries | +| `build/`, `.run/*` (except a worker's own staging dir) | **Coordinator only** for `make`; workers use `tools/sf3_match range --work .run/p8//work/` | `make` targets share `build/`; `range` writes only to the directory it is given | +| `git` | **Coordinator only** for `add`/`commit`. Workers never commit | Concurrent index writes in one repo | +| Worker deliverables | `.run/p8//claims.tsv`, `symbols.tsv`, `report.tsv` (ignored paths) | Staging is ignored, so nothing ROM-derived or half-finished can be committed by accident | +| Any other tracked file | Workers must not modify them; if one needs to, it asks the coordinator | Prevents silent drift in docs/config | + +The coordinator's pre-merge check is mechanical: `git status --short` must show only new `src/*.c` files +from workers, and any unexpected modification is reverted and raised as a protocol violation. + +### The safety property + +A worker cannot fake a match. Its claims are only *inputs* to the tracked registries; the authority is +the coordinator's own `make gate` over the merged registry, which compares the whole 1,886,208-byte +executable and requires SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`. A fabricated claim breaks the +hash and fails the gate. `make extents-verify` additionally re-derives every registered region's extent +from control flow, so a claim cannot smuggle in a hand-chosen end address either. + +## Intercom protocol + +Addressing uses intercom **short session ids**, because runtime `subagent-chat-…` aliases are not +reconnect identities. The two peers observed at planning time are `01a0d143` and `01a0d145`, both idle +in this repository. Each worker is asked to set an alias (`/alias sf3-worker-a`, `/alias sf3-worker-b`) +as its first act; if it cannot, short ids remain the handle. + +| Message | Direction | Shape | +|---|---|---| +| `CHARTER` | coordinator → worker | Task id, its partition file, the protocol, the hard rules, the claim format, the target | +| `REPORT` | worker → coordinator | Machine-parseable: `claims.tsv` path, counts, negatives, blockers, files created | +| `CHECKIN` | coordinator → worker | Non-blocking status request when a worker has been quiet | +| `DECISION` | coordinator → worker | Reply to a worker's `ask` | +| `STOP` | coordinator → worker | Stop matching, flush claims, report; used at cycle end or when a partition is exhausted | + +Workers use `ask` **only** for blocking decisions (a shape that will not match, a symbol they cannot +resolve, a protocol ambiguity). Everything else is `send`, so a worker never stalls waiting on the +coordinator. The coordinator answers asks promptly and otherwise works on merges and records between +messages; incoming messages start its turns, so it never busy-waits. + +### Claim format + +A claim is a row in the same shape as `config/regions.tsv`, so merging is a mechanical operation: + +``` +# .run/p8//claims.tsv +0x80012345 0x80012360 src/func_80012345.c +0x80012360 0x80012370 src/func_80012345.c cc1=-O1 +``` + +A duplicate group is claimed by every address it registers, all pointing at the one source file. +Symbols the worker needs that are not address-shaped go in `.run/p8//symbols.tsv` as +`NAMEaddress[gp]`. Negatives and blockers go in `.run/p8//report.tsv` as +`addressstatusevidence`, so a failed candidate is recorded rather than quietly dropped. + +## Preconditions + +- The clean Phase 7 baseline reproduces: `make check` green at `c_regions=39`, 168 tests, extents + agreement 39/39. +- Two peer pi sessions are live in this repository and can run shell commands (verified by P8-T1's + capability probe, not assumed). +- The peers have read the same session-start files; the coordinator's `CHARTER` restates the essential + constraints so a worker cannot proceed on a partial reading. +- All ROM-derived and proprietary material stays in ignored paths. + +## Scope and safeguards + +- **ROM/SDK firewall:** no disc dump, extracted content, disassembly, generated assembly, build output, + expected binary, Ghidra database, RAM dump, or proprietary SDK material may be tracked. Worker staging + is ignored; the coordinator reviews `git status` before every commit. +- One task at a time for the coordinator. A worker's cycle is a task; the coordinator verifies before + moving on. +- Workers may not weaken a gate, widen a region to silence a `LENGTH-MISMATCH`, or register a region + whose extent is not graded `exact`. +- Every match claim must have `sf3_match range` `differing_bytes=0` from the worker **and** the + coordinator's own green `make gate`. +- A failed candidate is a recorded negative. Stop after two distinct unexplained attempts on one item. +- Never use `git clean -x` or `git clean -fdx`. +- If a worker session dies, stalls, or turns out to lack the tools, the coordinator records it, takes + over the remaining partition itself, and the milestone is re-stated with the evidence — not silently + reduced. + +## Tasks + +- [ ] **P8-T1 — Control records, baseline revalidation, and the intercom channel handshake** *(xHigh)* + - Create `phase-ends/CURRENT_PHASE.md`; re-run the clean baseline (`make clean`, `make all`, `cmp`, + SHA-1, `make test`, `make gate`, `make extents-verify`) and the firewall/Git review. + - `intercom list`; send each peer a **capability probe** (a trivial, verifiable request: report cwd, + `git rev-parse HEAD`, the row count of `config/match_worklist.tsv`, and whether it can run shell + commands). Record what each peer actually is. + - Generate the two disjoint partitions `.run/p8/worklist-a.tsv` and `.run/p8/worklist-b.tsv` from + `config/match_worklist.tsv`, and **prove disjointness** (empty address intersection) and coverage. + - **Verify:** all baseline gates pass; both peers answered the probe or are recorded as unusable with + the failure evidence; partitions are disjoint and their union is the worklist; no ROM-derived + material tracked. + +- [ ] **P8-T2 — Claim-merge tooling and the protocol record** *(xHigh)* + - Add tracked `tools/sf3_merge` (standard library only, synthetic-tested): merge worker claim files + into `config/regions.tsv` and `config/symbols.tsv`, validating that every claimed extent exists and + is graded `exact`, that regions stay ordered, non-overlapping and inside the payload, that sources + exist, and that symbols are unique and well-formed. Rejected claims are reported with reasons and + the tool exits non-zero rather than writing a partial merge. + - Write `docs/PHASE8_PROTOCOL.md`: the message types, the hard rules, the claim format, the partition + scheme, and the verification the coordinator runs per cycle. + - **Verify:** synthetic tests cover every rejection reason (missing extent, non-`exact` grade, + overlap with the registry, overlap between two workers' claims, out-of-payload, missing source, + duplicate symbol, malformed row); the tool is deterministic; a merge of the current registry plus + the current 39 regions is a no-op. + +- [ ] **P8-T3 — Dispatch both charters and verify cycle 1** *(xHigh)* + - Send `CHARTER` to both workers: partition file, protocol, hard rules, claim format, target of at + least **8 new bodies each** for the cycle, and the instruction to report every 4 claims. + - As reports arrive: check the protocol (no shared-file edits, no commits, no `make`), run + `sf3_merge`, then `make extents-verify`, `make test` and `make gate`; reply to questions; re-dispatch + to a worker whose partition is exhausted. + - **Verify:** cycle-1 target met or a blocker recorded; every merged region covered by a green + `make gate` (0 differing bytes, SHA-1 `e173426c…`) and `regions=N disagreements=0`; worker claims + that failed verification are reported back with the exact evidence. + +- [ ] **P8-T4 — Verification cycle 2 with re-partitioning** *(xHigh)* + - Regenerate `config/match_worklist.tsv` (now excluding everything registered in cycle 1), re-partition + disjointly, and dispatch cycle 2 with a cumulative target of at least **22 new bodies**. + - Merge and verify exactly as in P8-T3; record any candidate a worker reported as a near-match. + - **Verify:** cumulative ≥22 new bodies or a recorded blocker; green gate and extents agreement after + the final merge of the cycle; partitions re-proved disjoint. + +- [ ] **Rules check** + - Re-read `AGENTS.md` mandatory behavior after P8-T4 and state: `Rules check — re-read complete. + Continuing with [next task].` + +- [ ] **P8-T5 — The SDK-shape investigation, bounded** *(High)* + - Assign one worker the class Phase 7 deferred (`docs/MATCHING_COOKBOOK.md` finding 15): can the + identified toolchain reproduce the BIOS `syscall` wrappers (`0x80103FCC`, `0x80103FEC`), the GTE + (COP2) body (`0x80010810`), and the stack-pointer accessor (`0x800FB5D4`) — from re-derived macro + definitions, a compiler builtin, or inline assembly? + - Bounded: two distinct approaches, then stop. Either those candidates become registered matches, or + the exact evidence for why they cannot is recorded. + - **Verify:** every claim verified by the coordinator's own gate; if the class stays closed, the + bounded negative names the approach, the commands and the observed bytes; the finding is folded into + the cookbook either way. + +- [ ] **P8-T6 — Verification cycle 3 and consolidation** *(xHigh)* + - Third dispatch to reach the phase target of at least **36 new bodies** (70 distinct total), then + consolidate: any orphaned claims, any candidate left in a partial state, and the recorded negatives. + - Run the clean-state gate: `make clean && make all`, `cmp`, SHA-1, `make test`, `make gate`, + `make extents-verify`, firewall and Git review. + - **Verify:** the milestone number is met or each shortfall has an evidence-backed blocker; every + registered region is covered by the clean full-binary gate; the registry audit shows ordered, + non-overlapping regions all agreeing with derived extents; worker staging holds no unmerged claim. + +- [ ] **P8-T7 — Cookbook, conventions, protocol record, and phase gate** *(xHigh)* + - Fold new byte-proven findings into `docs/MATCHING_COOKBOOK.md` and new conventions into + `docs/MATCHING_CONVENTIONS.md`; finish `docs/PHASE8_PROTOCOL.md` with what actually happened (what + the workers did well, where the protocol needed a decision, which rules were load-bearing); + write `docs/PHASE8_VERIFICATION.md`; update `README.md`. + - Complete every clean gate, then request milestone confirmation. + - **Verify:** every match claim has direct instruction and clean full-binary evidence; no prohibited + artifact is tracked; the protocol record is honest about failures and about any peer that was + unusable; developer confirmation is requested before any PhaseEnd. + +## Milestone + +At least **70 distinct instruction-identical C function bodies** (34 carried in, ≥36 new) across the +corresponding registered regions, every one verified by the coordinator's own clean full-binary `cmp` + +SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9` and by `make extents-verify`; the multi-session protocol +tracked, documented and reproducible; the claim-merge tool tracked and synthetic-tested; and the +SDK-shape investigation resolved either way. + +**Or** Phase 8 records the evidence-backed blocker for each unmet part — including the honest statement +of what a peer session turned out to be capable of — and leaves all unmatched content as the data +fallback. + +Explicitly **out of scope** this phase, recorded as such: `0x8005DEF8`, `0x800F3160`, `0x800F7FB4`, +`0x800F8AEC`, the numeric `-G`, the CRT entry, and library-versus-game-code — which stays **unresolved** +unless cycle evidence settles it, in which case the finding is recorded rather than acted on silently. +Also out of scope: the 73 `indirect` (jump-table) candidates and the 15 `escape` candidates, which need +tooling the phase does not build. + +Phase 8 closes only after the developer confirms the milestone; then write a PhaseEnd record, update +`phase-ends/DIGEST.md`, archive `CURRENT_PHASE.md` to `phase-ends/logs/Phase8.md`, commit, and stop +without beginning Phase 9. + +## Verification criteria (phase gate) + +| Gate | Command | Requirement | +|---|---|---| +| Synthetic suite | `make test` | exit 0; 168 tests plus this phase's additions | +| Clean baseline | `make clean && make all` | exit 0 | +| Full-binary comparison | `cmp build/scus_946_40.rebuilt 'extracted/SCUS_946.40;1'` | exit 0 | +| SHA-1 | `sha1sum` on both files | `e173426c157384ebf1b6caf8c6fea18a85a14af9` | +| Ordered C gate | `make gate` | 0 differing bytes, `result=MATCH`, exit 0 | +| Extents agreement | `make extents-verify` | `disagreements=0`, `result=AGREE` | +| Registry audit | every region ordered, non-overlapping, inside the payload | no violation | +| Protocol audit | `git log` authorship and `git status` | coordinator-only commits; no worker-authored commit; only new `src/*.c` from workers | +| Firewall | `git ls-files` under prohibited roots | 0 tracked | +| Git hygiene | `git status --short`, `git diff --check` | clean, exit 0 | + +## Decisions taken in this draft (developer may redirect) + +1. **The coordinating session.** The developer's instruction is that the session which kicks off the two + agents supervises them, and this session holds the planning context, so the plan assumes **this + session is the coordinator**. `AGENTS.md` otherwise expects a phase to begin in a fresh session; + naming the coordinator explicitly resolves the tension. If the developer prefers a fresh session to + coordinate, the plan is unchanged apart from who runs P8-T1. +2. **Workers match; the coordinator does not.** The coordinator's own work is the merge tooling, the + verification cycles, the records, and answering questions — matching only as a fallback if a peer + turns out to be unusable. +3. **The two peers are used as they are.** `01a0d143` and `01a0d145` are already live in this repository. + P8-T1 probes what they can actually do rather than assuming; a peer that cannot run shell commands + changes the plan's execution but not its verification. +4. **Target size.** ≥70 distinct bodies (≥36 new) is set from Phase 7's measured rate (23 bodies in a + session that also built three tools) and is staged across three cycles (≥8, ≥22, ≥36 new) so a stall + is visible early rather than at the end.