phase8: close the phase with the verification milestone
Developer-confirmed milestone: 149 distinct byte-identical bodies across 158 registered regions, up from 34 / 39 at Phase 7 close - 115 new bodies against a target of 70, produced by three sessions working one worktree under a written protocol. Adds PhaseEnd_Phase8.md and the digest entry, archives CURRENT_PHASE.md as logs/Phase8.md, and completes docs/PHASE8_PROTOCOL.md with the retrospective the developer asked for: which rules were load-bearing (worker/coordinator file ownership, candidate-first merging, worklist regeneration, self-contained charters), where the protocol needed a decision, the four failure modes seen with their mitigations, and reusable CHARTER and worker-report templates so the next coordinated phase does not pay the cost of writing them again. Final gates: 203 tests; make clean, make all, cmp and SHA-1 e173426c...; make check green at c_regions=158 with 0 differing bytes and regions=158 disagreements=0; registry audit 0 violations; no untracked file in src/; 244 tracked files, none under a prohibited root.
This commit is contained in:
Christopher Williams
+111
-1
@@ -129,7 +129,117 @@ tracked registry changes.
|
||||
| Partition disjointness | set intersection of the two partition files | empty |
|
||||
| Firewall | `git ls-files` under prohibited roots | 0 tracked |
|
||||
|
||||
## Lessons from cycle 1
|
||||
## Lessons from the whole phase
|
||||
|
||||
Phase 8 ran three sessions in one worktree for a full phase and closed **115 new bodies** (34 → 149). What
|
||||
worked, what needed a decision, and what to change next time:
|
||||
|
||||
### The rules that were load-bearing
|
||||
|
||||
| Rule | What it actually prevented |
|
||||
|---|---|
|
||||
| Workers never touch the tracked registries, `build/`, or writing git commands | Zero conflicts across three sessions in one worktree for a whole phase. Every merge was a coordinator action. |
|
||||
| Merge to a **candidate**, gate it, then promote | The first cycle-2 candidate failed the gate (`differing_bytes=1117442`, binary 4 bytes short) and the tracked registry was never touched. This is the single most valuable rule in the document. |
|
||||
| Regenerate the worklist and re-partition at the start of every cycle | The tracked worklist was **stale** before the first dispatch — generated in P7-T4 with only 12 regions registered, it still listed 23 candidates matched later in Phase 7. Dispatching on it would have sent both workers at already-matched addresses. |
|
||||
| A worker's `range` result is necessary but not sufficient | Every claim was re-verified by the coordinator's own whole-binary gate. Nothing was tracked on a worker's word. |
|
||||
| Self-contained charters | Neither of the first two workers had read `PROJECT_CONTEXT.md`, the digest or the plan. A charter that assumes reading is a charter that gets ignored. |
|
||||
| Two attempts on a candidate, then record | Kept the phase out of rabbit holes; the recorded negatives are now the most useful planning data for the next phase. |
|
||||
|
||||
### Where the protocol needed a decision
|
||||
|
||||
- **Worker findings can be harness changes.** Four of the phase's most valuable outputs came from workers and
|
||||
were implemented by the coordinator: `maspsx=off`, `gp=-NAME`, the trapping-class exclusion, and the
|
||||
`syscall`-is-not-a-terminator fix. The protocol should say explicitly that a worker which finds a *class*
|
||||
problem reports it rather than grinding, and that the coordinator owns the harness fix.
|
||||
- **A worker's own tooling may be worth promoting.** One worker built an instruction-level diff tool and a
|
||||
payload resolver/classifier as scratch, and every diagnosis in the phase depended on them. It also correctly
|
||||
argued that two of its own artifacts should **not** be tracked (stale snapshots of a regenerated worklist).
|
||||
Ask a worker for a promotion proposal rather than assuming.
|
||||
- **Workers catch each other's over-generalised rules.** One worker's rule was scoped correctly by a second
|
||||
worker testing it against its own open rows and reporting a negative result. Negative results must be reported
|
||||
as first-class outcomes, with the scope they bound.
|
||||
- **A worker may need to retract.** One worker reported a tool defect, then partially retracted it when the
|
||||
effect did not reproduce after another fix. The record says "observed, impact not demonstrated".
|
||||
|
||||
### Failure modes seen, worth pre-empting in a charter
|
||||
|
||||
| Hazard | Mitigation |
|
||||
|---|---|
|
||||
| An address-named symbol **resolves silently**, so a transcribed typo produces a plausible wrong candidate instead of an error | Have the resolver tool print the ready-to-paste name (`D_80121BFC`) rather than bare hex; keep `range` as the oracle |
|
||||
| Unmatched drafts accumulate in `src/` | State the rule in the charter ("a tracked `src/` file is a claim") and check `git status --short src/` at every merge; the coordinator clears strays |
|
||||
| A worker spends budget on a class that cannot be matched | The worklist now excludes two such classes with counted reasons; charters carry the worker's own known-negatives list |
|
||||
| A worker stalls silently waiting for the coordinator | Workers `ask` only when blocked; everything else is `send`, and the coordinator answers asks first |
|
||||
|
||||
### Reusable charter template
|
||||
|
||||
This is the structure that worked. Fill the placeholders; do not shorten section 0 or 6.
|
||||
|
||||
```text
|
||||
COORDINATOR HANDSHAKE — Phase N, Syphon Filter 3 matching decompilation
|
||||
(repo: <abs path>)
|
||||
|
||||
You are being recruited as a WORKER session for Phase N. Reply with
|
||||
`intercom({action:"ask", to:"<coordinator-id>", message:"<report>"})` so the
|
||||
coordinator's session wakes, containing: PROBE REPORT / cwd / head /
|
||||
worklist_rows / shell yes|no / tools / read_status (which docs you have read).
|
||||
|
||||
HARD RULES (three sessions share this worktree):
|
||||
1. Do NOT edit config/*.tsv, build/, docs, or any tracked file other than NEW
|
||||
src/*.c files you create inside your own partition. The registries are
|
||||
coordinator-only.
|
||||
2. Do NOT run `make` (it writes shared build/ state) and do NOT run any git
|
||||
command that writes. The coordinator is the only committer.
|
||||
3. Never commit or push. Never `git clean -x`.
|
||||
|
||||
0. PROJECT IN ONE PARAGRAPH — what the project is, the toolchain, that a match
|
||||
means instruction-identical bytes AND a green whole-binary gate, and the
|
||||
current counts. (Do not omit this; most workers have not read the docs.)
|
||||
1. YOUR PARTITION — the file path, row count, first/last address, the column
|
||||
meaning, "claim ONLY addresses in this file", and the cycle target.
|
||||
2. THE PER-FUNCTION LOOP — disassemble (Ghidra MCP, synchronous), write
|
||||
src/func_XXXXXXXX.c with a header stating the observed instructions and LIMITS,
|
||||
then the exact `range` command with a FRESH --work dir, and the required
|
||||
output (`differing_bytes=0`, `result=MATCH`). "NEVER run make."
|
||||
3. SYMBOLS — address-named symbols resolve implicitly; anything else fails
|
||||
loudly. gp-relative needs a registry row: ask, do not edit.
|
||||
4. DUPLICATES — if the row's duplicate column is not `-`, claim EVERY address of
|
||||
the group, all pointing at one source.
|
||||
5. STAGING AND REPORTING — .run/pN/<worker>/claims.tsv (registry format),
|
||||
symbols-request.tsv (append-only), report.tsv
|
||||
(address<TAB>status<TAB>evidence). Report every 4 claims. `ask` only when
|
||||
blocked. "Your range result is necessary but not sufficient."
|
||||
6. KNOWN NEGATIVES — the addresses this worker must not spend budget on, with
|
||||
the reason. Check the worklist for rows that are already registered.
|
||||
7. THE FACTS YOU WILL NEED — the cookbook entries that apply, copied in, with
|
||||
their scope limits (a rule quoted without its limit causes wasted attempts).
|
||||
8. NEXT — what to do when the cycle target is met, and the standing instruction
|
||||
to keep the claims file cumulative.
|
||||
```
|
||||
|
||||
### Worker report template
|
||||
|
||||
```text
|
||||
W-<X> CYCLE<n>: claims=<N> new files=<list> negatives=<N> needs=<list>
|
||||
<per-claim range evidence: differing_bytes / result>
|
||||
<negatives with their exact residual and the hypothesis tried>
|
||||
<needs: symbols, decisions, or nothing>
|
||||
```
|
||||
|
||||
A report that omits the per-claim evidence or the negative detail is not actionable, because the coordinator
|
||||
re-verifies everything and needs to know what was already ruled out.
|
||||
|
||||
### Closing checklist for a coordinated phase
|
||||
|
||||
1. Every worker sends a final report: verified-but-unclaimed work, unmatched drafts removed from `src/`, and a
|
||||
complete negatives list.
|
||||
2. `git status --short src/` shows **no** untracked file — a tracked `src/` file is a claim.
|
||||
3. Registry audit: ordered, non-overlapping, inside the payload, every extent `exact`, every source present.
|
||||
4. `make clean && make all`, `cmp`, SHA-1, `make test`, `make gate`, `make extents-verify` — all from clean.
|
||||
5. Worklist regenerates with `excluded_already_registered` equal to the registry size.
|
||||
6. Extract the workers' negative metadata into the tracked index before the ignored staging is lost.
|
||||
7. Firewall and Git review; then request milestone confirmation before writing any PhaseEnd.
|
||||
|
||||
### Early-cycle lessons (kept for the record)
|
||||
|
||||
- **A stale tracked input is the first thing to check.** The worklist was generated during P7-T4 with
|
||||
only 12 regions registered and still listed 23 candidates matched later in Phase 7. Dispatching on it
|
||||
|
||||
@@ -64,3 +64,7 @@ deviation: the duplicate census is tracked rather than ignored, because it holds
|
||||
grades only and the worklist must be reproducible from tracked inputs. Unresolved: `0x8005DEF8`,
|
||||
`0x800F3160`, `0x800F7FB4`, `0x800F8AEC`, the SDK-header shapes, the numeric `-G`, the CRT entry, and
|
||||
library-versus-game-code. No rules were added.
|
||||
|
||||
## Phase 8 — Coordinated Multi-Session Matching (2026-09-23)
|
||||
|
||||
Phase 8 ran the decompilation as a team: three pi sessions in one worktree under a written protocol, with one coordinator that dispatched charters, merged claims and independently verified them, and two workers that matched concurrently (a third worker was recruited mid-phase for the class the others kept deferring). **149 distinct matched bodies / 158 registered regions**, from 34 / 39, against a milestone of 70 — 115 new bodies. All gates exit 0 from a clean state: `make clean`, `make all`, `cmp`, both files SHA-1 `e173426c157384ebf1b6caf8c6fea18a85a14af9`, `make check` at 203 tests with `regions=158 disagreements=0` and `c_regions=158`, 0 differing bytes. The registry audit reports 0 violations across 158 ordered, non-overlapping regions whose every extent is `exact` and whose every source exists, and `git status --short src/` shows no untracked file: a tracked `src/` file is a claim. Every near-miss recorded by Phases 5-7 in the reachable set is closed: `0x800F3160` (Phase 5's store-in-delay-slot anomaly, unresolved for three phases), `0x800F8AEC`, `0x800F8B58/6C`, `0x800F8FE4`, `0x80102B10` via a new per-region `maspsx=off` override; `0x800F7FB4` ×3 via a cc1 loop-shape rewrite (explicit guard + do/while removes an unused frame); `0x8009E8D0` ×2; and the handoff rows `0x8002DEB4` and `0x80019B6C`. Two corrections to the evidence model: the gp marker is per **symbol** but the original's access form is per **site** (`0x80121F84` is read gp-relative at `0x800A80BC` and written absolutely at `0x8002D288`, so a new per-region `gp=-NAME` override expresses both — cookbook finding 10 was incomplete), and `syscall` **returns**, so treating it as a walk terminator had truncated two BIOS stubs to 8 bytes when their bodies are 16. The SDK-shape investigation **refuted its own premise** — the missing PsyQ headers were never the blocker — and registered 7 COP2/BIOS regions under a developer decision (inline assembly accepted for coprocessor and kernel instructions, documented per file, integer logic in C, no `.word` lumps); `include/gtemac.h` records the project's own re-derived COP2 macros and the non-textbook control-register map. It also produced the phase's strongest structural signal: **50 of the 1,937 exact extents contain trapping `add`/`sub` that no available compiler emits** (ten `cc1` builds plus the real `CC1PSX` 4.0-4.6, ~30 C shapes, 15 flags, `-ftrapv` rejected everywhere) while **0 of the 158 registered regions does** — now excluded by `sf3_triage` with a counted reason and the best current candidate for part of the unresolved library-versus-game-code boundary, recorded as a hypothesis with the explicit limit that the detector is sufficient but not complete. Worker B tested worker A's mirrored-layout lever against its own bounded negatives and produced a decision rule instead of a match (the lever applies when the instruction count is right and the block order is wrong); worker A partially retracted its own maspsx load-delay defect claim when the effect did not reproduce. New tracked artifacts: `tools/sf3_merge` (28 tests), `docs/PHASE8_PROTOCOL.md` (hard rules, message types, staging formats, the merge workflow, per-cycle verification, a retrospective and reusable charter templates), `docs/PHASE8_VERIFICATION.md`, `config/near_match_negatives.tsv` (38 open negatives as addresses, sizes, statuses and class labels only), `include/gtemac.h`, and cookbook findings 16-28. 244 tracked files, none under a prohibited root. One convention decided by the developer: inline assembly accepted for coprocessor and kernel instructions. Unresolved and recorded: the trapping-arithmetic class (highest value), the maspsx load-delay observation, the reorg thread-fill and two-epilogue classes, the numeric `-G`, the CRT entry, and library-versus-game-code. No rules were added.
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
# PhaseEnd — Phase 8: Coordinated Multi-Session Matching
|
||||
|
||||
**Date:** 2026-09-23
|
||||
**Phase Status:** Complete
|
||||
**Milestone confirmed by developer:** yes
|
||||
|
||||
## Completed Checklist
|
||||
|
||||
- P8-T1 — Created the phase control record, revalidated the clean baseline, probed both peer sessions with a
|
||||
capability probe, and generated disjoint partitions with the disjointness proved.
|
||||
- P8-T2 — Built `tools/sf3_merge` (validated claim merge, 28 synthetic tests) and wrote
|
||||
`docs/PHASE8_PROTOCOL.md`.
|
||||
- P8-T3 — Dispatched both charters and verified cycle 1: 18 claims, 57 regions.
|
||||
- P8-T4 — Verified cycle 2 with re-partitioning: 58 claims accepted, 115 regions, **milestone met**.
|
||||
- Rules check — Re-read `AGENTS.md` mandatory behavior after P8-T4 and stated the required notice.
|
||||
- P8-T5 — The bounded SDK-shape investigation: the class was **refuted** (7 regions registered under the
|
||||
developer's convention), and it produced the trapping-arithmetic census that was adopted into `sf3_triage`.
|
||||
- P8-T6 — Cycle 3 and consolidation: 158 regions / 149 bodies, registry audit clean.
|
||||
- P8-T7 — Cookbook findings 16–28, conventions, protocol retrospective with reusable charter templates, the
|
||||
verification record, and the phase gate; developer confirmation obtained.
|
||||
|
||||
## Verified Results
|
||||
|
||||
**149 distinct matched bodies / 158 registered regions** (from 34 / 39), against a milestone of 70. All gates
|
||||
exit 0 from a clean state: `make clean`, `make all`, `cmp`, both files SHA-1
|
||||
`e173426c157384ebf1b6caf8c6fea18a85a14af9`, and `make check` (203 tests, `regions=158 disagreements=0`,
|
||||
`c_regions=158`, 0 differing bytes).
|
||||
|
||||
| Measure | Phase 7 close | Phase 8 close |
|
||||
|---|---|---|
|
||||
| Registered regions | 39 | **158** |
|
||||
| Distinct matched bodies | 34 | **149** |
|
||||
| Tracked symbols | 23 | 367 |
|
||||
| Synthetic tests | 168 | **203** |
|
||||
| Duplicate groups fully matched | 2 of 10 | **6 of 10** |
|
||||
|
||||
### Every near-miss recorded by Phases 5–7 in the reachable set is now closed
|
||||
|
||||
| Was | Now | Mechanism |
|
||||
|---|---|---|
|
||||
| `0x800F3160` — "no tested compiler + assembler pair reproduces its store-in-delay-slot scheduling" (Phase 5) | matched | `maspsx=off` |
|
||||
| `0x800F8AEC`, `0x800F8B58`, `0x800F8B6C`, `0x800F8FE4`, `0x80102B10` | matched | `maspsx=off` |
|
||||
| `0x800F7FB4` ×3 addresses — "loop body exact, unexplained 8-byte frame" (Phase 7) | matched | explicit guard + do/while (a cc1 loop-shape artifact) |
|
||||
| `0x8009E8D0` ×2 (g0030) — a near-match for two workers | matched | ordinary matching |
|
||||
| `0x8002DEB4`, `0x80019B6C` | matched | handoff rows closed by a third worker |
|
||||
|
||||
### Harness mechanisms added, all from worker findings
|
||||
|
||||
| Mechanism | What it fixes |
|
||||
|---|---|
|
||||
| `maspsx=off` (per-region) | maspsx's unconditional `nop` for a jump destroys the delay-slot fill GNU `as` reorder mode performs on an expanded symbol store — 6 regions |
|
||||
| `gp=-NAME` (per-region) | the gp marker is per symbol but the original's access form is per **site** (`0x80121F84` is read gp-relative at `0x800A80BC` and written absolutely at `0x8002D288`) |
|
||||
| `syscall` is not a walk terminator | it returns to `EPC+4`; treating it as a terminator truncated two BIOS stubs to 8 bytes when their bodies are 16 |
|
||||
| trapping-arithmetic exclusion in `sf3_triage` | 50 of 1,937 exact extents contain `add`/`sub` that no available compiler emits; 0 of the 158 registered regions does |
|
||||
| `--skip-registered` in `sf3_merge` | a worker's claims file is naturally cumulative |
|
||||
|
||||
### New tracked artifacts
|
||||
|
||||
`tools/sf3_merge` (28 tests), `docs/PHASE8_PROTOCOL.md` (roles, hard rules, message types, staging formats, the
|
||||
merge workflow, per-cycle verification, the retrospective and reusable charter templates),
|
||||
`docs/PHASE8_VERIFICATION.md`, `config/near_match_negatives.tsv` (38 open negatives: addresses, sizes, statuses
|
||||
and class labels only), `include/gtemac.h` (the project's own re-derived COP2 macros and control-register map),
|
||||
and cookbook findings 16–28.
|
||||
|
||||
## Deviations and Bounded Results
|
||||
|
||||
| Item | Plan | Actual | Reason / limit |
|
||||
|---|---|---|---|
|
||||
| Merge workflow | "merge, then gate" | merge to a **candidate**, gate, then promote | Strengthens the plan's intent: the tracked registry never contains an unverified claim. It saved a cycle — the first cycle-2 candidate failed the gate with the binary 4 bytes short. |
|
||||
| Census artifact (Phase 7) | — | tracked | Carried over: addresses, sizes and grades only, and the worklist must be reproducible from tracked inputs. |
|
||||
| The GTE/SDK class | investigate whether the toolchain can reach it | **refuted the premise**; 7 regions registered; the class is no longer a blocker | The missing SDK headers were never the issue. The real remaining wall is the trapping-arithmetic class, which is a compiler difference. |
|
||||
| `g0013` (`0x80010810`) | match it | **near-match**, deliberately unclaimed | ~30 source shapes, 13 flag sets and 6 compilers all give the same body one instruction shorter. A pinned register would distort allocation rather than reproduce the source. |
|
||||
| `0x800FF47C`, `0x800FFBBC` | reopen if a lever appeared | **bounded negatives**, lever tested | The mirrored-layout lever applies when the instruction count is right and the block order is wrong; here the order was already right and the count wrong, so the lever only traded one defect for another. |
|
||||
| Inline assembly | a policy question for the developer | **Option A accepted**: documented per file | Integer logic in C, one `__asm__ volatile` for the coprocessor/kernel instruction, no `.word` lumps, and a register-name binding is not inline assembly. |
|
||||
| maspsx load-delay defect | reported by a worker as a defect | **observation, impact not demonstrated** | The worker partially retracted it after the effect did not reproduce once another fix landed. Recorded with that limit. |
|
||||
|
||||
## Verification and Firewall
|
||||
|
||||
- Registry audit: 158 regions, 149 bodies, 8 with overrides — ordered, non-overlapping, inside the payload,
|
||||
every extent graded `exact`, every source present. **0 violations.**
|
||||
- Worklist regenerates with `excluded_already_registered=158` (equal to the registry size) and
|
||||
`excluded_trapping_arith=50`.
|
||||
- `git status --short src/` → 0 untracked files: every tracked `src/` file is a registered match.
|
||||
- 244 tracked files, **0** under any prohibited root; `git diff --check` exit 0.
|
||||
- No game bytes, disassembly, generated assembly, build outputs, expected binaries, Ghidra material, RAM dumps
|
||||
or proprietary SDK material are tracked. The new tables hold addresses, sizes, statuses and class labels
|
||||
only; `include/gtemac.h` is the project's own derivation with no SDK text.
|
||||
|
||||
## Rules Added This Phase
|
||||
|
||||
None. One convention was decided by the developer: inline assembly is accepted for coprocessor and kernel
|
||||
instructions, documented per file.
|
||||
|
||||
## Next
|
||||
|
||||
Phase 8 is closed. Phase 9 is planned as a second coordinated phase (one coordinator, three workers), with the
|
||||
infrastructure, protocol and charter templates now in place. A future phase must preserve the all-payload data
|
||||
baseline, the ordered-layout harness, the corrected toolchain, the evidence tables
|
||||
(`function_inventory` / `function_extents` / `duplicate_bodies` / `match_worklist` / `near_match_negatives`),
|
||||
the per-region overrides, and the byte gate, and must treat the following as unresolved: the trapping-arithmetic
|
||||
class (the highest-value open item), the maspsx load-delay observation, the reorg thread-fill and two-epilogue
|
||||
classes, the `-G` threshold, the CRT entry, and library-versus-game-code.
|
||||
|
||||
## Plain-English Recap
|
||||
|
||||
Phase 8 ran the decompilation as a team. Three pi sessions worked in one directory at once under a written
|
||||
protocol: two matched functions while a third attacked the class the other two kept deferring, and one
|
||||
coordinator session did nothing but hand out work, answer questions, merge what came back, and verify it
|
||||
independently. The verification model was the important part — a worker's claim was never trusted, only the
|
||||
coordinator's own whole-binary gate — and it paid for itself immediately when a merge failed the gate and the
|
||||
tracked registry was still untouched.
|
||||
|
||||
The phase went from 34 matched functions to 149, against a target of 70. Along the way it closed every
|
||||
near-miss the previous three phases had recorded as unreachable, including a scheduling anomaly that had stood
|
||||
since Phase 5. It found two real mistakes in the project's own evidence model (a gp marker that assumed one
|
||||
access form per symbol when the executable uses two, and a `syscall` treated as a function terminator when it
|
||||
returns). It also found the largest remaining obstacle and measured it precisely: fifty function bodies
|
||||
containing arithmetic instructions that no compiler in the project can produce, disjoint from everything
|
||||
matched so far — which is now the best lead on the still-unresolved question of which code is library and which
|
||||
is game.
|
||||
|
||||
Three workers stopped cleanly, one of them after removing its own convention violation. Nothing was committed
|
||||
by a worker; every claim was re-verified; and the executable still rebuilds byte for byte.
|
||||
|
||||
## 🛑 Stop Here
|
||||
|
||||
`CURRENT_PHASE.md` is archived as `phase-ends/logs/Phase8.md`; this PhaseEnd and the digest update belong in the
|
||||
closure commit. Do not begin Phase 9 execution in this session.
|
||||
Reference in New Issue
Block a user