phase8: draft the coordinated multi-session plan
Phase 7 made matching possible at scale; Phase 8 makes it parallel. Three pi sessions in one worktree, coordinated over pi-intercom: one coordinator that dispatches charters, merges claims and independently verifies them, and two workers that match concurrently. The plan's centre of gravity is concurrency safety, because three sessions share one worktree: src/*.c is partitioned disjointly so no two sessions can write the same file, the tracked registries and build/ and git are coordinator-only, and worker deliverables stage in ignored paths. The safety property is that a worker's claim is never trusted -- the coordinator's own clean make gate over the whole 1,886,208-byte executable is the only authority, so a fabricated match breaks the SHA-1, and make extents-verify stops a hand-chosen end address. Target: at least 70 distinct bodies (from 34), staged across three cycles so a stall shows up early, plus one bounded investigation into the SDK-header shapes Phase 7 deferred. Draft, awaiting approval. Decisions flagged for the developer: this session coordinates, workers match while the coordinator verifies, the two already-live peers (01a0d143, 01a0d145) are probed rather than assumed, and the target size.
This commit is contained in:
Christopher Williams
@@ -0,0 +1,254 @@
|
||||
# Phase 8 Plan — Coordinated Multi-Session Matching
|
||||
|
||||
**Status:** DRAFT — requires explicit developer approval before activation. Do not begin any task until the developer approves this plan.
|
||||
**Planning effort:** Max
|
||||
**Prepared:** 2026-09-23, immediately after Phase 7 closure.
|
||||
|
||||
## Session start (for the coordinating session)
|
||||
|
||||
Read in order before doing anything: `AGENTS.md`, `PROJECT_CONTEXT.md`, `phase-ends/DIGEST.md`,
|
||||
`phase-ends/PhaseEnd_Phase7.md`, this plan, then `docs/PHASE7_EXTENTS.md`, `docs/PHASE7_DUPES.md`,
|
||||
`docs/PHASE7_TRIAGE.md`, `docs/MATCHING_CONVENTIONS.md`, `docs/MATCHING_COOKBOOK.md`. There is no
|
||||
`CURRENT_PHASE.md` at the start of this phase; the coordinating session must create one when activated.
|
||||
|
||||
## Goal
|
||||
|
||||
Phase 7 made matching *possible* at scale. Phase 8 makes it **parallel**: three pi sessions on one
|
||||
machine, coordinated over pi-intercom, with one session acting as **coordinator** — it dispatches
|
||||
charters, answers questions, merges what comes back, and independently verifies every claim — and two
|
||||
**worker** sessions matching functions concurrently. The coordinator does not compete for matching work;
|
||||
its job is throughput through supervision plus the verification the byte gate demands.
|
||||
|
||||
Target: at least **70 distinct matched bodies** (from 34), with every one of them verified by the
|
||||
coordinator's own clean full-binary gate rather than by a worker's report.
|
||||
|
||||
## The difference from every previous phase
|
||||
|
||||
| | Phases 0–7 | Phase 8 |
|
||||
|---|---|---|
|
||||
| Sessions | one | three (1 coordinator, 2 workers) |
|
||||
| Work discovery | by the session doing the work | coordinator generates disjoint partitions |
|
||||
| Trust | the session's own claim | **no claim is trusted**; the coordinator re-verifies |
|
||||
| Shared files | edited freely | workers may not touch them at all |
|
||||
| Commits | by the working session | coordinator only |
|
||||
| Verification | at the end of a task | every merge cycle, plus the phase gate |
|
||||
|
||||
## Concurrency safety (the part that decides whether this works)
|
||||
|
||||
All three sessions share **one worktree**. Two sessions editing `config/regions.tsv` or running `make`
|
||||
concurrently would corrupt each other's work. The protocol therefore fixes the following as hard rules,
|
||||
and the coordinator enforces them:
|
||||
|
||||
| Resource | Rule | Why |
|
||||
|---|---|---|
|
||||
| `src/*.c` | Each worker may **create** files only for addresses in its own partition | Partitions are disjoint by construction, so no two sessions can write the same file |
|
||||
| `config/regions.tsv`, `config/symbols.tsv`, `config/match_worklist.tsv` | **Coordinator only.** Workers never edit them | Single-writer rule for the tracked registries |
|
||||
| `build/`, `.run/*` (except a worker's own staging dir) | **Coordinator only** for `make`; workers use `tools/sf3_match range --work .run/p8/<worker>/work/<addr>` | `make` targets share `build/`; `range` writes only to the directory it is given |
|
||||
| `git` | **Coordinator only** for `add`/`commit`. Workers never commit | Concurrent index writes in one repo |
|
||||
| Worker deliverables | `.run/p8/<worker>/claims.tsv`, `symbols.tsv`, `report.tsv` (ignored paths) | Staging is ignored, so nothing ROM-derived or half-finished can be committed by accident |
|
||||
| Any other tracked file | Workers must not modify them; if one needs to, it asks the coordinator | Prevents silent drift in docs/config |
|
||||
|
||||
The coordinator's pre-merge check is mechanical: `git status --short` must show only new `src/*.c` files
|
||||
from workers, and any unexpected modification is reverted and raised as a protocol violation.
|
||||
|
||||
### The safety property
|
||||
|
||||
A worker cannot fake a match. Its claims are only *inputs* to the tracked registries; the authority is
|
||||
the coordinator's own `make gate` over the merged registry, which compares the whole 1,886,208-byte
|
||||
executable and requires SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`. A fabricated claim breaks the
|
||||
hash and fails the gate. `make extents-verify` additionally re-derives every registered region's extent
|
||||
from control flow, so a claim cannot smuggle in a hand-chosen end address either.
|
||||
|
||||
## Intercom protocol
|
||||
|
||||
Addressing uses intercom **short session ids**, because runtime `subagent-chat-…` aliases are not
|
||||
reconnect identities. The two peers observed at planning time are `01a0d143` and `01a0d145`, both idle
|
||||
in this repository. Each worker is asked to set an alias (`/alias sf3-worker-a`, `/alias sf3-worker-b`)
|
||||
as its first act; if it cannot, short ids remain the handle.
|
||||
|
||||
| Message | Direction | Shape |
|
||||
|---|---|---|
|
||||
| `CHARTER` | coordinator → worker | Task id, its partition file, the protocol, the hard rules, the claim format, the target |
|
||||
| `REPORT` | worker → coordinator | Machine-parseable: `claims.tsv` path, counts, negatives, blockers, files created |
|
||||
| `CHECKIN` | coordinator → worker | Non-blocking status request when a worker has been quiet |
|
||||
| `DECISION` | coordinator → worker | Reply to a worker's `ask` |
|
||||
| `STOP` | coordinator → worker | Stop matching, flush claims, report; used at cycle end or when a partition is exhausted |
|
||||
|
||||
Workers use `ask` **only** for blocking decisions (a shape that will not match, a symbol they cannot
|
||||
resolve, a protocol ambiguity). Everything else is `send`, so a worker never stalls waiting on the
|
||||
coordinator. The coordinator answers asks promptly and otherwise works on merges and records between
|
||||
messages; incoming messages start its turns, so it never busy-waits.
|
||||
|
||||
### Claim format
|
||||
|
||||
A claim is a row in the same shape as `config/regions.tsv`, so merging is a mechanical operation:
|
||||
|
||||
```
|
||||
# .run/p8/<worker>/claims.tsv
|
||||
0x80012345 0x80012360 src/func_80012345.c
|
||||
0x80012360 0x80012370 src/func_80012345.c cc1=-O1
|
||||
```
|
||||
|
||||
A duplicate group is claimed by every address it registers, all pointing at the one source file.
|
||||
Symbols the worker needs that are not address-shaped go in `.run/p8/<worker>/symbols.tsv` as
|
||||
`NAME<TAB>address[<TAB>gp]`. Negatives and blockers go in `.run/p8/<worker>/report.tsv` as
|
||||
`address<TAB>status<TAB>evidence`, so a failed candidate is recorded rather than quietly dropped.
|
||||
|
||||
## Preconditions
|
||||
|
||||
- The clean Phase 7 baseline reproduces: `make check` green at `c_regions=39`, 168 tests, extents
|
||||
agreement 39/39.
|
||||
- Two peer pi sessions are live in this repository and can run shell commands (verified by P8-T1's
|
||||
capability probe, not assumed).
|
||||
- The peers have read the same session-start files; the coordinator's `CHARTER` restates the essential
|
||||
constraints so a worker cannot proceed on a partial reading.
|
||||
- All ROM-derived and proprietary material stays in ignored paths.
|
||||
|
||||
## Scope and safeguards
|
||||
|
||||
- **ROM/SDK firewall:** no disc dump, extracted content, disassembly, generated assembly, build output,
|
||||
expected binary, Ghidra database, RAM dump, or proprietary SDK material may be tracked. Worker staging
|
||||
is ignored; the coordinator reviews `git status` before every commit.
|
||||
- One task at a time for the coordinator. A worker's cycle is a task; the coordinator verifies before
|
||||
moving on.
|
||||
- Workers may not weaken a gate, widen a region to silence a `LENGTH-MISMATCH`, or register a region
|
||||
whose extent is not graded `exact`.
|
||||
- Every match claim must have `sf3_match range` `differing_bytes=0` from the worker **and** the
|
||||
coordinator's own green `make gate`.
|
||||
- A failed candidate is a recorded negative. Stop after two distinct unexplained attempts on one item.
|
||||
- Never use `git clean -x` or `git clean -fdx`.
|
||||
- If a worker session dies, stalls, or turns out to lack the tools, the coordinator records it, takes
|
||||
over the remaining partition itself, and the milestone is re-stated with the evidence — not silently
|
||||
reduced.
|
||||
|
||||
## Tasks
|
||||
|
||||
- [ ] **P8-T1 — Control records, baseline revalidation, and the intercom channel handshake** *(xHigh)*
|
||||
- Create `phase-ends/CURRENT_PHASE.md`; re-run the clean baseline (`make clean`, `make all`, `cmp`,
|
||||
SHA-1, `make test`, `make gate`, `make extents-verify`) and the firewall/Git review.
|
||||
- `intercom list`; send each peer a **capability probe** (a trivial, verifiable request: report cwd,
|
||||
`git rev-parse HEAD`, the row count of `config/match_worklist.tsv`, and whether it can run shell
|
||||
commands). Record what each peer actually is.
|
||||
- Generate the two disjoint partitions `.run/p8/worklist-a.tsv` and `.run/p8/worklist-b.tsv` from
|
||||
`config/match_worklist.tsv`, and **prove disjointness** (empty address intersection) and coverage.
|
||||
- **Verify:** all baseline gates pass; both peers answered the probe or are recorded as unusable with
|
||||
the failure evidence; partitions are disjoint and their union is the worklist; no ROM-derived
|
||||
material tracked.
|
||||
|
||||
- [ ] **P8-T2 — Claim-merge tooling and the protocol record** *(xHigh)*
|
||||
- Add tracked `tools/sf3_merge` (standard library only, synthetic-tested): merge worker claim files
|
||||
into `config/regions.tsv` and `config/symbols.tsv`, validating that every claimed extent exists and
|
||||
is graded `exact`, that regions stay ordered, non-overlapping and inside the payload, that sources
|
||||
exist, and that symbols are unique and well-formed. Rejected claims are reported with reasons and
|
||||
the tool exits non-zero rather than writing a partial merge.
|
||||
- Write `docs/PHASE8_PROTOCOL.md`: the message types, the hard rules, the claim format, the partition
|
||||
scheme, and the verification the coordinator runs per cycle.
|
||||
- **Verify:** synthetic tests cover every rejection reason (missing extent, non-`exact` grade,
|
||||
overlap with the registry, overlap between two workers' claims, out-of-payload, missing source,
|
||||
duplicate symbol, malformed row); the tool is deterministic; a merge of the current registry plus
|
||||
the current 39 regions is a no-op.
|
||||
|
||||
- [ ] **P8-T3 — Dispatch both charters and verify cycle 1** *(xHigh)*
|
||||
- Send `CHARTER` to both workers: partition file, protocol, hard rules, claim format, target of at
|
||||
least **8 new bodies each** for the cycle, and the instruction to report every 4 claims.
|
||||
- As reports arrive: check the protocol (no shared-file edits, no commits, no `make`), run
|
||||
`sf3_merge`, then `make extents-verify`, `make test` and `make gate`; reply to questions; re-dispatch
|
||||
to a worker whose partition is exhausted.
|
||||
- **Verify:** cycle-1 target met or a blocker recorded; every merged region covered by a green
|
||||
`make gate` (0 differing bytes, SHA-1 `e173426c…`) and `regions=N disagreements=0`; worker claims
|
||||
that failed verification are reported back with the exact evidence.
|
||||
|
||||
- [ ] **P8-T4 — Verification cycle 2 with re-partitioning** *(xHigh)*
|
||||
- Regenerate `config/match_worklist.tsv` (now excluding everything registered in cycle 1), re-partition
|
||||
disjointly, and dispatch cycle 2 with a cumulative target of at least **22 new bodies**.
|
||||
- Merge and verify exactly as in P8-T3; record any candidate a worker reported as a near-match.
|
||||
- **Verify:** cumulative ≥22 new bodies or a recorded blocker; green gate and extents agreement after
|
||||
the final merge of the cycle; partitions re-proved disjoint.
|
||||
|
||||
- [ ] **Rules check**
|
||||
- Re-read `AGENTS.md` mandatory behavior after P8-T4 and state: `Rules check — re-read complete.
|
||||
Continuing with [next task].`
|
||||
|
||||
- [ ] **P8-T5 — The SDK-shape investigation, bounded** *(High)*
|
||||
- Assign one worker the class Phase 7 deferred (`docs/MATCHING_COOKBOOK.md` finding 15): can the
|
||||
identified toolchain reproduce the BIOS `syscall` wrappers (`0x80103FCC`, `0x80103FEC`), the GTE
|
||||
(COP2) body (`0x80010810`), and the stack-pointer accessor (`0x800FB5D4`) — from re-derived macro
|
||||
definitions, a compiler builtin, or inline assembly?
|
||||
- Bounded: two distinct approaches, then stop. Either those candidates become registered matches, or
|
||||
the exact evidence for why they cannot is recorded.
|
||||
- **Verify:** every claim verified by the coordinator's own gate; if the class stays closed, the
|
||||
bounded negative names the approach, the commands and the observed bytes; the finding is folded into
|
||||
the cookbook either way.
|
||||
|
||||
- [ ] **P8-T6 — Verification cycle 3 and consolidation** *(xHigh)*
|
||||
- Third dispatch to reach the phase target of at least **36 new bodies** (70 distinct total), then
|
||||
consolidate: any orphaned claims, any candidate left in a partial state, and the recorded negatives.
|
||||
- Run the clean-state gate: `make clean && make all`, `cmp`, SHA-1, `make test`, `make gate`,
|
||||
`make extents-verify`, firewall and Git review.
|
||||
- **Verify:** the milestone number is met or each shortfall has an evidence-backed blocker; every
|
||||
registered region is covered by the clean full-binary gate; the registry audit shows ordered,
|
||||
non-overlapping regions all agreeing with derived extents; worker staging holds no unmerged claim.
|
||||
|
||||
- [ ] **P8-T7 — Cookbook, conventions, protocol record, and phase gate** *(xHigh)*
|
||||
- Fold new byte-proven findings into `docs/MATCHING_COOKBOOK.md` and new conventions into
|
||||
`docs/MATCHING_CONVENTIONS.md`; finish `docs/PHASE8_PROTOCOL.md` with what actually happened (what
|
||||
the workers did well, where the protocol needed a decision, which rules were load-bearing);
|
||||
write `docs/PHASE8_VERIFICATION.md`; update `README.md`.
|
||||
- Complete every clean gate, then request milestone confirmation.
|
||||
- **Verify:** every match claim has direct instruction and clean full-binary evidence; no prohibited
|
||||
artifact is tracked; the protocol record is honest about failures and about any peer that was
|
||||
unusable; developer confirmation is requested before any PhaseEnd.
|
||||
|
||||
## Milestone
|
||||
|
||||
At least **70 distinct instruction-identical C function bodies** (34 carried in, ≥36 new) across the
|
||||
corresponding registered regions, every one verified by the coordinator's own clean full-binary `cmp` +
|
||||
SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9` and by `make extents-verify`; the multi-session protocol
|
||||
tracked, documented and reproducible; the claim-merge tool tracked and synthetic-tested; and the
|
||||
SDK-shape investigation resolved either way.
|
||||
|
||||
**Or** Phase 8 records the evidence-backed blocker for each unmet part — including the honest statement
|
||||
of what a peer session turned out to be capable of — and leaves all unmatched content as the data
|
||||
fallback.
|
||||
|
||||
Explicitly **out of scope** this phase, recorded as such: `0x8005DEF8`, `0x800F3160`, `0x800F7FB4`,
|
||||
`0x800F8AEC`, the numeric `-G`, the CRT entry, and library-versus-game-code — which stays **unresolved**
|
||||
unless cycle evidence settles it, in which case the finding is recorded rather than acted on silently.
|
||||
Also out of scope: the 73 `indirect` (jump-table) candidates and the 15 `escape` candidates, which need
|
||||
tooling the phase does not build.
|
||||
|
||||
Phase 8 closes only after the developer confirms the milestone; then write a PhaseEnd record, update
|
||||
`phase-ends/DIGEST.md`, archive `CURRENT_PHASE.md` to `phase-ends/logs/Phase8.md`, commit, and stop
|
||||
without beginning Phase 9.
|
||||
|
||||
## Verification criteria (phase gate)
|
||||
|
||||
| Gate | Command | Requirement |
|
||||
|---|---|---|
|
||||
| Synthetic suite | `make test` | exit 0; 168 tests plus this phase's additions |
|
||||
| Clean baseline | `make clean && make all` | exit 0 |
|
||||
| Full-binary comparison | `cmp build/scus_946_40.rebuilt 'extracted/SCUS_946.40;1'` | exit 0 |
|
||||
| SHA-1 | `sha1sum` on both files | `e173426c157384ebf1b6caf8c6fea18a85a14af9` |
|
||||
| Ordered C gate | `make gate` | 0 differing bytes, `result=MATCH`, exit 0 |
|
||||
| Extents agreement | `make extents-verify` | `disagreements=0`, `result=AGREE` |
|
||||
| Registry audit | every region ordered, non-overlapping, inside the payload | no violation |
|
||||
| Protocol audit | `git log` authorship and `git status` | coordinator-only commits; no worker-authored commit; only new `src/*.c` from workers |
|
||||
| Firewall | `git ls-files` under prohibited roots | 0 tracked |
|
||||
| Git hygiene | `git status --short`, `git diff --check` | clean, exit 0 |
|
||||
|
||||
## Decisions taken in this draft (developer may redirect)
|
||||
|
||||
1. **The coordinating session.** The developer's instruction is that the session which kicks off the two
|
||||
agents supervises them, and this session holds the planning context, so the plan assumes **this
|
||||
session is the coordinator**. `AGENTS.md` otherwise expects a phase to begin in a fresh session;
|
||||
naming the coordinator explicitly resolves the tension. If the developer prefers a fresh session to
|
||||
coordinate, the plan is unchanged apart from who runs P8-T1.
|
||||
2. **Workers match; the coordinator does not.** The coordinator's own work is the merge tooling, the
|
||||
verification cycles, the records, and answering questions — matching only as a fallback if a peer
|
||||
turns out to be unusable.
|
||||
3. **The two peers are used as they are.** `01a0d143` and `01a0d145` are already live in this repository.
|
||||
P8-T1 probes what they can actually do rather than assuming; a peer that cannot run shell commands
|
||||
changes the plan's execution but not its verification.
|
||||
4. **Target size.** ≥70 distinct bodies (≥36 new) is set from Phase 7's measured rate (23 bodies in a
|
||||
session that also built three tools) and is staged across three cycles (≥8, ≥22, ≥36 new) so a stall
|
||||
is visible early rather than at the end.
|
||||
Reference in New Issue
Block a user