diff --git a/phase-ends/Phase11_PLAN.md b/phase-ends/Phase11_PLAN.md new file mode 100644 index 0000000..301c3e3 --- /dev/null +++ b/phase-ends/Phase11_PLAN.md @@ -0,0 +1,180 @@ +# Phase 11 Plan — Breaking the 244-Byte Ceiling + +**Status:** DRAFT — requires explicit developer approval before activation. +**Prepared:** 2026-09-24, immediately after Phase 10 closure (484 bodies / 493 regions). +**Milestone:** **600 distinct matched bodies** from 484 (i.e. **+116**), with **700 as a stretch**. +Measured on the orchestrator's clean whole-binary gate, never as "keep matching". + +## 0. The honest starting point — and the thing this phase is actually about + +Phase 10 closed at 484 bodies with the size-band law as its headline result. Measuring the corpus +against the remaining pool produces two facts that define this phase: + +| | n | median | p90 | **max** | +|---|---|---|---|---| +| **matched corpus** | 493 regions | 52 B | 108 B | **244 B** | +| remaining worklist | 1193 rows | — | — | — | + +**Not one body larger than 244 bytes has ever been matched.** Every one of the 493 matches is ≤244 B. + +The remaining pool by size: + +| band | rows | status | +|---|---|---| +| ≤120 B | 115 | proven-matchable band | +| 121–200 B | 284 | proven-matchable band | +| 201–400 B | 367 | **partially proven** (up to 244 B) | +| 401–800 B | 245 | **unproven — nothing has ever matched here** | +| >800 B | 182 | **unproven** | + +So the ≈400–450 rows at ≤244 B are the finite, proven band, and **+116 bodies means matching roughly a +quarter to a third of it.** That is achievable at Phase 10's measured rates with a fourth worker and +better context management — but it is the *end* of that band, and the 244-byte ceiling is the real +subject of this phase. + +**Therefore Phase 11 has two goals, not one:** + +- **A — consume the remaining proven band** (the milestone path: +116 from ≤244 B). +- **B — break the 244-byte ceiling** (the strategic path: find out whether the 401–800 B and >800 B + bands are tractable at all, and if so what it takes). + +Goal B is what makes 700 reachable and what a Phase 12 would build on. It is a **bounded +investigation**, not open-ended matching, and it is stated as such below. + +## 1. Goal + +**Goal A (the milestone): 600 distinct bodies.** Consume the remaining ≤244 B rows, size-band-first, +using the accumulated 57-finding lever set. + +**Goal B (the strategic result): a measured answer to why nothing above 244 B has matched.** The +deliverable is *evidence*, and any of these outcomes is a success: (a) the band is tractable and here is +the lever; (b) the band needs a specific harness change, named and demonstrated on at least one row; +(c) the band is dominated by a class that cannot be reproduced with this toolchain, with the class +characterised and counted. A goal-B result that adds zero bodies is still a met goal B. + +**Goal C (carry-forward, small): close the three named loose ends** — document `0x80107C5C` (which +already matches but has no documented source), settle the two unproven opt-in maspsx modes, and either +demonstrate or retire the scheduler class. Each is bounded and each is worth bodies. + +## 2. Structure — orchestrator + 4 workers, spawned by the orchestrator + +**This phase changes who runs the show.** The orchestrator spawns, names, monitors, and retires its own +workers; the developer spawns only the orchestrator. + +- **Topology: one tab, four agent panes** (developer's recommendation), plus the orchestrator's own pane. + Layout: split right for the first worker, then split down for the rest, so no column becomes unusably + narrow. All panes share the project cwd. +- **Spawn:** `herdr pane split --current --direction --cwd "$PWD" --no-focus` → read the new + pane id from `.result.pane.pane_id` → `herdr agent start --kind pi --pane `. The default + model is already configured, so no model flag is needed. +- **Communicate:** `herdr agent prompt ""` to charter, `herdr agent read ` to + inspect, `herdr agent wait --until idle --timeout N` to wait. **Intercom remains the reporting + channel** — it is already proven and the workers' reports arrive as messages — but herdr is the + control channel for spawn/retire/read. +- **Retire:** when a worker has compacted to exhaustion or a band is done, read its final state, verify + its staging, then close its pane. Never close a pane the orchestrator did not create. +- **Scaling:** the roster is **4 by default**. A fifth worker is added only if the measured + attempts-per-match says the pool supports it; workers are retired rather than left idle. + +## 3. Context management — the phase's biggest mechanical change + +`pi-context-tools` is installed **globally**, so every session (orchestrator and workers) has +`context_info` and `compact_context`. This changes the rotation calculus that governed Phases 8–10: + +- **Measurement is now exact and self-service.** Every worker reports its context by calling + `context_info` rather than estimating, and the orchestrator quotes that figure. Phase 10 had a worker + estimate 87% when it was at 50%; that class of error disappears. +- **`compact_context` replaces rotation as the first response to a full context.** A worker at 70–80% + compacts and continues, keeping its partition, its staging, and its learned levers. Rotation becomes + the *second* response — for a worker whose band is done or whose staging has gone stale. +- **This is only safe because the state lives in files.** The ledger, `CURRENT_PHASE.md`, the tracked + negatives index and each worker's staging are current at every merge, so a compaction that summarises + away the conversation loses nothing that matters. **The ledger discipline is what makes compaction + safe**, and keeping it current becomes a hard requirement rather than good practice. +- **The orchestrator compacts too** rather than stopping at the cap. Its gate/merge role is the one that + cannot be delegated, so it must be able to run the whole phase. + +## 4. The carried machinery (do not re-derive; verify before relying) + +- Toolchain: PsyQ 4.0 `CC1PSX` (GCC 2.7.2.SN32.3.7.0002); flags `-quiet -O2 -G0`; maspsx (ASPSX 2.56 + emulator) then GNU `as`. +- Harness: `tools/sf3_match` (per-region `cc1=`, `as=`, `gp=-NAME`, `maspsx=off|noreordernop|regread`), + `tools/sf3_diff`, `tools/sf3_merge`, `tools/sf3_triage` (with `restores_unsaved`). +- **The tracked maspsx patch** (`tools/patches/maspsx-phase10-r1r2.patch`) — `tools/maspsx/` is + git-ignored, so the patch is the only reproducible carrier. Verify it is applied before any build. +- Evidence tables: `function_inventory`, `function_extents`, `duplicate_bodies`, `match_worklist`, + `near_match_negatives` (194 rows at close). +- **The hardened merge flow**: verify → merge to CANDIDATE → gate whole-binary → promote only on MATCH → + reconcile negatives → regenerate worklist + refilter partitions → commit → push. The gate rejected + three batches in Phase 10 and the registry was never corrupted. +- **The size-band law** (cookbook 41) as the dispatch rule. +- **The md5-in-claim-row guard** and the four process rules in cookbook 57. + +## 5. Rules already in force (carry into every charter) + +`git ls-files --error-unmatch` before any `rm` under `src/`; the dependent-claim rule; exact symbol-name +spelling (the gp marker is name-keyed); compute-don't-eyeball; the leading-indicator rule; the +three-encoding-trap habit; the amended register-field rule (**verify the dataflow before calling it a +tie-break**); the two-step two-arm-branch rule (invert-and-swap first, then `goto`); the paired +`&&`/named-boolean rule; **verify the file you are about to stage, at the path you are about to claim**; +**run the `src/` cleanup audit before the final report**; **read start and end from the worklist row, +never from a hand dump**; **one attempt on a named lever, then classify**; **fold region options into the +claim row**; **record the source md5 per claim**. + +## 6. Verification ladder + +| Frequency | Checks | +|---|---| +| Every merge | candidate gate (whole-binary, SHA-1 fixed) → promote → `make check` | +| Every 3rd merge | `make clean && make all`, `cmp`, SHA-1, registry audit, `git status --short src/`, firewall | +| Every merge | worklist regen (`excluded_already_registered` = registry size) + partition refilter with the disjointness/union proof | +| Every worker retire | staging verified, `src/` cleanup audit, negatives extracted into the tracked index | +| Phase close | full clean gates, negatives extraction, milestone confirmation requested, PhaseEnd, digest, ledger, stop | + +**Push after every verified merge** (the developer's standing preference). + +## 7. Scope and safeguards + +- ROM/SDK firewall unchanged; worker staging ignored; `git clean -x`/`-fdx` forbidden. +- Workers may not weaken a gate, widen regions, or register a non-`exact` extent. +- **No scheduler-changing flag** (`-fno-schedule-insns` etc.) may be granted for a match. It is a + diagnosis-only tool; taking it would mean the corpus is no longer all built by one compiler + configuration. Workers may run it for diagnosis and must record the row as a negative. +- Inline asm remains limited to shapes plain C **provably cannot express**, per-file documented. +- The blocked classes (trapping arithmetic, the `0x80012xxx` primitive-init family, the maspsx + rare-epilogue mutual exclusion) stay excluded unless Goal B produces evidence that reopens them. +- **Goal B is bounded**: two distinct root-cause attempts per hypothesis, then record. It is an + investigation with a deliverable, not an open-ended grind. + +## 8. Tasks + +- **P11-T1** — control records, clean-baseline revalidation, herdr roster spawn (4 workers), capability + probes including a `context_info` check, size-band-first partitions. +- **P11-T2** — Goal A cycle 1 on the ≤244 B band: first checkpoint (+30 bodies). +- **P11-T3** — Goal B opening: the 244-byte ceiling investigation. Characterise what the 401–800 B band + actually contains (tie-break classes, scheduling share, harness-blocked share) with counted evidence + from the payload and the extents table, and name the candidate levers. +- **P11-T4 … T-n** — the autonomy loop: per-cycle regeneration, full audits every 3rd merge, workers + compacted rather than rotated, Goal B probed in bounded slices alongside Goal A. +- **P11-Tn** — phase close: cookbook/conventions/verification records, negatives extraction, milestone + confirmation requested, PhaseEnd, digest, ledger archive, stop. + +## 9. Decisions requested from the developer + +1. **Milestone:** 600 with 700 as a stretch, or a different number. +2. **Goal B scope:** is the bounded 244-byte-ceiling investigation in scope for this phase (it is the + route to 700), or should Phase 11 be Goal A only? +3. **Roster:** 4 workers as the default, with the orchestrator free to spawn a fifth if the measured rate + supports it. +4. **Compaction:** confirm that workers compact and continue rather than rotating at the cap, and that + the orchestrator does the same. + +## 10. Preconditions + +- `make check` green at 493 regions / 484 bodies (Phase 10 close state), with the tracked maspsx patch + applied. +- The Phase 10 closure commit on `main` and pushed. +- herdr roster spawned and probed; `context_info` confirmed working in every worker. + +Milestone: **600 distinct matched bodies on the clean whole-binary gate, plus a measured Goal B answer** — +or the evidence-backed blocker recorded.