diff --git a/docs/PHASE8_PROTOCOL.md b/docs/PHASE8_PROTOCOL.md index a1b36fa..cce02b6 100644 --- a/docs/PHASE8_PROTOCOL.md +++ b/docs/PHASE8_PROTOCOL.md @@ -129,7 +129,117 @@ tracked registry changes. | Partition disjointness | set intersection of the two partition files | empty | | Firewall | `git ls-files` under prohibited roots | 0 tracked | -## Lessons from cycle 1 +## Lessons from the whole phase + +Phase 8 ran three sessions in one worktree for a full phase and closed **115 new bodies** (34 → 149). What +worked, what needed a decision, and what to change next time: + +### The rules that were load-bearing + +| Rule | What it actually prevented | +|---|---| +| Workers never touch the tracked registries, `build/`, or writing git commands | Zero conflicts across three sessions in one worktree for a whole phase. Every merge was a coordinator action. | +| Merge to a **candidate**, gate it, then promote | The first cycle-2 candidate failed the gate (`differing_bytes=1117442`, binary 4 bytes short) and the tracked registry was never touched. This is the single most valuable rule in the document. | +| Regenerate the worklist and re-partition at the start of every cycle | The tracked worklist was **stale** before the first dispatch — generated in P7-T4 with only 12 regions registered, it still listed 23 candidates matched later in Phase 7. Dispatching on it would have sent both workers at already-matched addresses. | +| A worker's `range` result is necessary but not sufficient | Every claim was re-verified by the coordinator's own whole-binary gate. Nothing was tracked on a worker's word. | +| Self-contained charters | Neither of the first two workers had read `PROJECT_CONTEXT.md`, the digest or the plan. A charter that assumes reading is a charter that gets ignored. | +| Two attempts on a candidate, then record | Kept the phase out of rabbit holes; the recorded negatives are now the most useful planning data for the next phase. | + +### Where the protocol needed a decision + +- **Worker findings can be harness changes.** Four of the phase's most valuable outputs came from workers and + were implemented by the coordinator: `maspsx=off`, `gp=-NAME`, the trapping-class exclusion, and the + `syscall`-is-not-a-terminator fix. The protocol should say explicitly that a worker which finds a *class* + problem reports it rather than grinding, and that the coordinator owns the harness fix. +- **A worker's own tooling may be worth promoting.** One worker built an instruction-level diff tool and a + payload resolver/classifier as scratch, and every diagnosis in the phase depended on them. It also correctly + argued that two of its own artifacts should **not** be tracked (stale snapshots of a regenerated worklist). + Ask a worker for a promotion proposal rather than assuming. +- **Workers catch each other's over-generalised rules.** One worker's rule was scoped correctly by a second + worker testing it against its own open rows and reporting a negative result. Negative results must be reported + as first-class outcomes, with the scope they bound. +- **A worker may need to retract.** One worker reported a tool defect, then partially retracted it when the + effect did not reproduce after another fix. The record says "observed, impact not demonstrated". + +### Failure modes seen, worth pre-empting in a charter + +| Hazard | Mitigation | +|---|---| +| An address-named symbol **resolves silently**, so a transcribed typo produces a plausible wrong candidate instead of an error | Have the resolver tool print the ready-to-paste name (`D_80121BFC`) rather than bare hex; keep `range` as the oracle | +| Unmatched drafts accumulate in `src/` | State the rule in the charter ("a tracked `src/` file is a claim") and check `git status --short src/` at every merge; the coordinator clears strays | +| A worker spends budget on a class that cannot be matched | The worklist now excludes two such classes with counted reasons; charters carry the worker's own known-negatives list | +| A worker stalls silently waiting for the coordinator | Workers `ask` only when blocked; everything else is `send`, and the coordinator answers asks first | + +### Reusable charter template + +This is the structure that worked. Fill the placeholders; do not shorten section 0 or 6. + +```text +COORDINATOR HANDSHAKE — Phase N, Syphon Filter 3 matching decompilation +(repo: ) + +You are being recruited as a WORKER session for Phase N. Reply with +`intercom({action:"ask", to:"", message:""})` so the +coordinator's session wakes, containing: PROBE REPORT / cwd / head / +worklist_rows / shell yes|no / tools / read_status (which docs you have read). + +HARD RULES (three sessions share this worktree): +1. Do NOT edit config/*.tsv, build/, docs, or any tracked file other than NEW + src/*.c files you create inside your own partition. The registries are + coordinator-only. +2. Do NOT run `make` (it writes shared build/ state) and do NOT run any git + command that writes. The coordinator is the only committer. +3. Never commit or push. Never `git clean -x`. + +0. PROJECT IN ONE PARAGRAPH — what the project is, the toolchain, that a match + means instruction-identical bytes AND a green whole-binary gate, and the + current counts. (Do not omit this; most workers have not read the docs.) +1. YOUR PARTITION — the file path, row count, first/last address, the column + meaning, "claim ONLY addresses in this file", and the cycle target. +2. THE PER-FUNCTION LOOP — disassemble (Ghidra MCP, synchronous), write + src/func_XXXXXXXX.c with a header stating the observed instructions and LIMITS, + then the exact `range` command with a FRESH --work dir, and the required + output (`differing_bytes=0`, `result=MATCH`). "NEVER run make." +3. SYMBOLS — address-named symbols resolve implicitly; anything else fails + loudly. gp-relative needs a registry row: ask, do not edit. +4. DUPLICATES — if the row's duplicate column is not `-`, claim EVERY address of + the group, all pointing at one source. +5. STAGING AND REPORTING — .run/pN//claims.tsv (registry format), + symbols-request.tsv (append-only), report.tsv + (addressstatusevidence). Report every 4 claims. `ask` only when + blocked. "Your range result is necessary but not sufficient." +6. KNOWN NEGATIVES — the addresses this worker must not spend budget on, with + the reason. Check the worklist for rows that are already registered. +7. THE FACTS YOU WILL NEED — the cookbook entries that apply, copied in, with + their scope limits (a rule quoted without its limit causes wasted attempts). +8. NEXT — what to do when the cycle target is met, and the standing instruction + to keep the claims file cumulative. +``` + +### Worker report template + +```text +W- CYCLE: claims= new files= negatives= needs= + + + +``` + +A report that omits the per-claim evidence or the negative detail is not actionable, because the coordinator +re-verifies everything and needs to know what was already ruled out. + +### Closing checklist for a coordinated phase + +1. Every worker sends a final report: verified-but-unclaimed work, unmatched drafts removed from `src/`, and a + complete negatives list. +2. `git status --short src/` shows **no** untracked file — a tracked `src/` file is a claim. +3. Registry audit: ordered, non-overlapping, inside the payload, every extent `exact`, every source present. +4. `make clean && make all`, `cmp`, SHA-1, `make test`, `make gate`, `make extents-verify` — all from clean. +5. Worklist regenerates with `excluded_already_registered` equal to the registry size. +6. Extract the workers' negative metadata into the tracked index before the ignored staging is lost. +7. Firewall and Git review; then request milestone confirmation before writing any PhaseEnd. + +### Early-cycle lessons (kept for the record) - **A stale tracked input is the first thing to check.** The worklist was generated during P7-T4 with only 12 regions registered and still listed 23 candidates matched later in Phase 7. Dispatching on it diff --git a/phase-ends/DIGEST.md b/phase-ends/DIGEST.md index f2eeaf2..cad0086 100644 --- a/phase-ends/DIGEST.md +++ b/phase-ends/DIGEST.md @@ -64,3 +64,7 @@ deviation: the duplicate census is tracked rather than ignored, because it holds grades only and the worklist must be reproducible from tracked inputs. Unresolved: `0x8005DEF8`, `0x800F3160`, `0x800F7FB4`, `0x800F8AEC`, the SDK-header shapes, the numeric `-G`, the CRT entry, and library-versus-game-code. No rules were added. + +## Phase 8 — Coordinated Multi-Session Matching (2026-09-23) + +Phase 8 ran the decompilation as a team: three pi sessions in one worktree under a written protocol, with one coordinator that dispatched charters, merged claims and independently verified them, and two workers that matched concurrently (a third worker was recruited mid-phase for the class the others kept deferring). **149 distinct matched bodies / 158 registered regions**, from 34 / 39, against a milestone of 70 — 115 new bodies. All gates exit 0 from a clean state: `make clean`, `make all`, `cmp`, both files SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, `make check` at 203 tests with `regions=158 disagreements=0` and `c_regions=158`, 0 differing bytes. The registry audit reports 0 violations across 158 ordered, non-overlapping regions whose every extent is `exact` and whose every source exists, and `git status --short src/` shows no untracked file: a tracked `src/` file is a claim. Every near-miss recorded by Phases 5-7 in the reachable set is closed: `0x800F3160` (Phase 5's store-in-delay-slot anomaly, unresolved for three phases), `0x800F8AEC`, `0x800F8B58/6C`, `0x800F8FE4`, `0x80102B10` via a new per-region `maspsx=off` override; `0x800F7FB4` ×3 via a cc1 loop-shape rewrite (explicit guard + do/while removes an unused frame); `0x8009E8D0` ×2; and the handoff rows `0x8002DEB4` and `0x80019B6C`. Two corrections to the evidence model: the gp marker is per **symbol** but the original's access form is per **site** (`0x80121F84` is read gp-relative at `0x800A80BC` and written absolutely at `0x8002D288`, so a new per-region `gp=-NAME` override expresses both — cookbook finding 10 was incomplete), and `syscall` **returns**, so treating it as a walk terminator had truncated two BIOS stubs to 8 bytes when their bodies are 16. The SDK-shape investigation **refuted its own premise** — the missing PsyQ headers were never the blocker — and registered 7 COP2/BIOS regions under a developer decision (inline assembly accepted for coprocessor and kernel instructions, documented per file, integer logic in C, no `.word` lumps); `include/gtemac.h` records the project's own re-derived COP2 macros and the non-textbook control-register map. It also produced the phase's strongest structural signal: **50 of the 1,937 exact extents contain trapping `add`/`sub` that no available compiler emits** (ten `cc1` builds plus the real `CC1PSX` 4.0-4.6, ~30 C shapes, 15 flags, `-ftrapv` rejected everywhere) while **0 of the 158 registered regions does** — now excluded by `sf3_triage` with a counted reason and the best current candidate for part of the unresolved library-versus-game-code boundary, recorded as a hypothesis with the explicit limit that the detector is sufficient but not complete. Worker B tested worker A's mirrored-layout lever against its own bounded negatives and produced a decision rule instead of a match (the lever applies when the instruction count is right and the block order is wrong); worker A partially retracted its own maspsx load-delay defect claim when the effect did not reproduce. New tracked artifacts: `tools/sf3_merge` (28 tests), `docs/PHASE8_PROTOCOL.md` (hard rules, message types, staging formats, the merge workflow, per-cycle verification, a retrospective and reusable charter templates), `docs/PHASE8_VERIFICATION.md`, `config/near_match_negatives.tsv` (38 open negatives as addresses, sizes, statuses and class labels only), `include/gtemac.h`, and cookbook findings 16-28. 244 tracked files, none under a prohibited root. One convention decided by the developer: inline assembly accepted for coprocessor and kernel instructions. Unresolved and recorded: the trapping-arithmetic class (highest value), the maspsx load-delay observation, the reorg thread-fill and two-epilogue classes, the numeric `-G`, the CRT entry, and library-versus-game-code. No rules were added. diff --git a/phase-ends/PhaseEnd_Phase8.md b/phase-ends/PhaseEnd_Phase8.md new file mode 100644 index 0000000..7446b00 --- /dev/null +++ b/phase-ends/PhaseEnd_Phase8.md @@ -0,0 +1,128 @@ +# PhaseEnd — Phase 8: Coordinated Multi-Session Matching + +**Date:** 2026-09-23 +**Phase Status:** Complete +**Milestone confirmed by developer:** yes + +## Completed Checklist + +- P8-T1 — Created the phase control record, revalidated the clean baseline, probed both peer sessions with a + capability probe, and generated disjoint partitions with the disjointness proved. +- P8-T2 — Built `tools/sf3_merge` (validated claim merge, 28 synthetic tests) and wrote + `docs/PHASE8_PROTOCOL.md`. +- P8-T3 — Dispatched both charters and verified cycle 1: 18 claims, 57 regions. +- P8-T4 — Verified cycle 2 with re-partitioning: 58 claims accepted, 115 regions, **milestone met**. +- Rules check — Re-read `AGENTS.md` mandatory behavior after P8-T4 and stated the required notice. +- P8-T5 — The bounded SDK-shape investigation: the class was **refuted** (7 regions registered under the + developer's convention), and it produced the trapping-arithmetic census that was adopted into `sf3_triage`. +- P8-T6 — Cycle 3 and consolidation: 158 regions / 149 bodies, registry audit clean. +- P8-T7 — Cookbook findings 16–28, conventions, protocol retrospective with reusable charter templates, the + verification record, and the phase gate; developer confirmation obtained. + +## Verified Results + +**149 distinct matched bodies / 158 registered regions** (from 34 / 39), against a milestone of 70. All gates +exit 0 from a clean state: `make clean`, `make all`, `cmp`, both files SHA-1 +`e173426c157384ebf1b6caf8c6fea18a85a14af9`, and `make check` (203 tests, `regions=158 disagreements=0`, +`c_regions=158`, 0 differing bytes). + +| Measure | Phase 7 close | Phase 8 close | +|---|---|---| +| Registered regions | 39 | **158** | +| Distinct matched bodies | 34 | **149** | +| Tracked symbols | 23 | 367 | +| Synthetic tests | 168 | **203** | +| Duplicate groups fully matched | 2 of 10 | **6 of 10** | + +### Every near-miss recorded by Phases 5–7 in the reachable set is now closed + +| Was | Now | Mechanism | +|---|---|---| +| `0x800F3160` — "no tested compiler + assembler pair reproduces its store-in-delay-slot scheduling" (Phase 5) | matched | `maspsx=off` | +| `0x800F8AEC`, `0x800F8B58`, `0x800F8B6C`, `0x800F8FE4`, `0x80102B10` | matched | `maspsx=off` | +| `0x800F7FB4` ×3 addresses — "loop body exact, unexplained 8-byte frame" (Phase 7) | matched | explicit guard + do/while (a cc1 loop-shape artifact) | +| `0x8009E8D0` ×2 (g0030) — a near-match for two workers | matched | ordinary matching | +| `0x8002DEB4`, `0x80019B6C` | matched | handoff rows closed by a third worker | + +### Harness mechanisms added, all from worker findings + +| Mechanism | What it fixes | +|---|---| +| `maspsx=off` (per-region) | maspsx's unconditional `nop` for a jump destroys the delay-slot fill GNU `as` reorder mode performs on an expanded symbol store — 6 regions | +| `gp=-NAME` (per-region) | the gp marker is per symbol but the original's access form is per **site** (`0x80121F84` is read gp-relative at `0x800A80BC` and written absolutely at `0x8002D288`) | +| `syscall` is not a walk terminator | it returns to `EPC+4`; treating it as a terminator truncated two BIOS stubs to 8 bytes when their bodies are 16 | +| trapping-arithmetic exclusion in `sf3_triage` | 50 of 1,937 exact extents contain `add`/`sub` that no available compiler emits; 0 of the 158 registered regions does | +| `--skip-registered` in `sf3_merge` | a worker's claims file is naturally cumulative | + +### New tracked artifacts + +`tools/sf3_merge` (28 tests), `docs/PHASE8_PROTOCOL.md` (roles, hard rules, message types, staging formats, the +merge workflow, per-cycle verification, the retrospective and reusable charter templates), +`docs/PHASE8_VERIFICATION.md`, `config/near_match_negatives.tsv` (38 open negatives: addresses, sizes, statuses +and class labels only), `include/gtemac.h` (the project's own re-derived COP2 macros and control-register map), +and cookbook findings 16–28. + +## Deviations and Bounded Results + +| Item | Plan | Actual | Reason / limit | +|---|---|---|---| +| Merge workflow | "merge, then gate" | merge to a **candidate**, gate, then promote | Strengthens the plan's intent: the tracked registry never contains an unverified claim. It saved a cycle — the first cycle-2 candidate failed the gate with the binary 4 bytes short. | +| Census artifact (Phase 7) | — | tracked | Carried over: addresses, sizes and grades only, and the worklist must be reproducible from tracked inputs. | +| The GTE/SDK class | investigate whether the toolchain can reach it | **refuted the premise**; 7 regions registered; the class is no longer a blocker | The missing SDK headers were never the issue. The real remaining wall is the trapping-arithmetic class, which is a compiler difference. | +| `g0013` (`0x80010810`) | match it | **near-match**, deliberately unclaimed | ~30 source shapes, 13 flag sets and 6 compilers all give the same body one instruction shorter. A pinned register would distort allocation rather than reproduce the source. | +| `0x800FF47C`, `0x800FFBBC` | reopen if a lever appeared | **bounded negatives**, lever tested | The mirrored-layout lever applies when the instruction count is right and the block order is wrong; here the order was already right and the count wrong, so the lever only traded one defect for another. | +| Inline assembly | a policy question for the developer | **Option A accepted**: documented per file | Integer logic in C, one `__asm__ volatile` for the coprocessor/kernel instruction, no `.word` lumps, and a register-name binding is not inline assembly. | +| maspsx load-delay defect | reported by a worker as a defect | **observation, impact not demonstrated** | The worker partially retracted it after the effect did not reproduce once another fix landed. Recorded with that limit. | + +## Verification and Firewall + +- Registry audit: 158 regions, 149 bodies, 8 with overrides — ordered, non-overlapping, inside the payload, + every extent graded `exact`, every source present. **0 violations.** +- Worklist regenerates with `excluded_already_registered=158` (equal to the registry size) and + `excluded_trapping_arith=50`. +- `git status --short src/` → 0 untracked files: every tracked `src/` file is a registered match. +- 244 tracked files, **0** under any prohibited root; `git diff --check` exit 0. +- No game bytes, disassembly, generated assembly, build outputs, expected binaries, Ghidra material, RAM dumps + or proprietary SDK material are tracked. The new tables hold addresses, sizes, statuses and class labels + only; `include/gtemac.h` is the project's own derivation with no SDK text. + +## Rules Added This Phase + +None. One convention was decided by the developer: inline assembly is accepted for coprocessor and kernel +instructions, documented per file. + +## Next + +Phase 8 is closed. Phase 9 is planned as a second coordinated phase (one coordinator, three workers), with the +infrastructure, protocol and charter templates now in place. A future phase must preserve the all-payload data +baseline, the ordered-layout harness, the corrected toolchain, the evidence tables +(`function_inventory` / `function_extents` / `duplicate_bodies` / `match_worklist` / `near_match_negatives`), +the per-region overrides, and the byte gate, and must treat the following as unresolved: the trapping-arithmetic +class (the highest-value open item), the maspsx load-delay observation, the reorg thread-fill and two-epilogue +classes, the `-G` threshold, the CRT entry, and library-versus-game-code. + +## Plain-English Recap + +Phase 8 ran the decompilation as a team. Three pi sessions worked in one directory at once under a written +protocol: two matched functions while a third attacked the class the other two kept deferring, and one +coordinator session did nothing but hand out work, answer questions, merge what came back, and verify it +independently. The verification model was the important part — a worker's claim was never trusted, only the +coordinator's own whole-binary gate — and it paid for itself immediately when a merge failed the gate and the +tracked registry was still untouched. + +The phase went from 34 matched functions to 149, against a target of 70. Along the way it closed every +near-miss the previous three phases had recorded as unreachable, including a scheduling anomaly that had stood +since Phase 5. It found two real mistakes in the project's own evidence model (a gp marker that assumed one +access form per symbol when the executable uses two, and a `syscall` treated as a function terminator when it +returns). It also found the largest remaining obstacle and measured it precisely: fifty function bodies +containing arithmetic instructions that no compiler in the project can produce, disjoint from everything +matched so far — which is now the best lead on the still-unresolved question of which code is library and which +is game. + +Three workers stopped cleanly, one of them after removing its own convention violation. Nothing was committed +by a worker; every claim was re-verified; and the executable still rebuilds byte for byte. + +## 🛑 Stop Here + +`CURRENT_PHASE.md` is archived as `phase-ends/logs/Phase8.md`; this PhaseEnd and the digest update belong in the +closure commit. Do not begin Phase 9 execution in this session. diff --git a/phase-ends/CURRENT_PHASE.md b/phase-ends/logs/Phase8.md similarity index 100% rename from phase-ends/CURRENT_PHASE.md rename to phase-ends/logs/Phase8.md