Files
Drew T 7622ea7609 feat(phase-13): T7 — effort experiment (Max vs Ultracode yield = noise) + bank 14
Controlled: 80 fixed SC01/077 fns (nins 8-15), identical blind-draft Workflow,
only session effort varied. Arm A (Ultracode/xHigh) 13/80, Arm B (Max) 14/80;
A∩B=12. 13-vs-14 = noise -> per-agent depth does NOT change blind-draft harvest
yield; the 'pass 1 Max did better' observation was difficulty-confounded.
DECISION: prefer Ultracode for breadth harvests (same yield + orchestration),
Max for deep synthesis. Banked Arm B's 14 -> ov_SC01_077 REAL 704 (byte-identical).
(scriptPath re-invoke didn't forward args -> Arm B overwrote Arm A's drafts dir;
Arm A was already gated/recorded, so the comparison stands; 1 A-only fn unbanked.)
finding -> docs/effort-map.md (Evidence section)
2026-06-16 17:50:31 -06:00

13 KiB
Raw Permalink Blame History

Effort Map — Claude Code reasoning settings for BFM-decomp

Evolvable reference (docs/ layer). Created 2026-06-13. Update freely as the harness changes — this is not the static constitution. Companion to CLAUDE.md → "Reasoning & Model Protocol." CLAUDE.md points here; the per-phase guidance, the Max-vs-Ultracode rule, and the verified harness mechanics live here. Where CLAUDE.md's tier language and this file differ, this file governs.


How the controls actually work (verified 2026-06-13)

  • Effort ladder: low / medium / high / xHigh / Max, set with /effort (slider) or /effort <level>. This is depth — how hard a single agent reasons. /effort auto resets to model default.
  • Persistence: low / medium / high / xHigh persist across sessions (saved as the baseline). Max and Ultracode are session-only — they revert to the persistent baseline (currently xHigh) on a new session. xHigh is the highest level you can make sticky. To run Max, re-apply /effort max at the start of each session.
  • Ultracode is NOT a deeper effort level. /effort ultracode runs the model at xHigh and additionally turns on multi-agent orchestration (Claude fans out parallel subagents via Workflow). Same as putting the keyword ultracode in a prompt. Session-only.
  • Why "Ultracode looks like xHigh": because it is xHigh per agent — the indicator correctly shows xHigh. Ultracode adds breadth (parallel agents), not depth.
  • Confirm what's active: the status line next to the spinner (e.g. "with max effort"), /status, or /effort (slider position).

The two axes (the key mental model)

Depth — how hard one agent thinks Breadth — how many angles at once
Control /effort level (… xHigh < Max) Ultracode, or Claude-driven Workflow fan-out
Best for one hard, irreversible reasoning problem surveying / auditing / matching many independent items
BFM examples compiler fingerprint, US overlay-address derivation, phase planning, PhaseEnd synthesis EXE-wide duplicate report, difficulty inventory, bulk matching of independent leaves

They are not combinable via /effort (it holds one value). And global Ultracode caps every task at xHigh depth — exactly wrong for the fingerprint/address work. So breadth is applied surgically: Claude proposes a Workflow fan-out for genuinely breadth-shaped tasks (you opt in) while single-thread depth stays at Max.


Policy for this project

  1. Default working level = Max. Usage is not a constraint (Max20 plan). Max's only cost is mild overthinking / latency on trivial tasks. Re-apply /effort max each session (it does not persist).
  2. Drop to xHigh only for speed on purely mechanical Tier-3 stretches (installs, parser boilerplate, report scripts) when slow turns get annoying. Correctness is unaffected.
  3. Do NOT globally enable Ultracode. It would run the deep, crown-jewel tasks at only xHigh depth. Instead, parallelism is surgical: Claude invokes Workflow/subagent fan-out for breadth-shaped tasks (you opt in). You may flip Ultracode on for an explicitly breadth-only stretch — accepting xHigh per-agent depth there.

Proactively prompt for Ultracode on breadth-heavy stretches (proven, Phase 12)

Standing instruction (Drew, 2026-06-16): during a Max session, the moment a breadth-shaped, parallelizable sub-task appears, PROMPT Drew to enable /effort ultracode — do not silently grind it serially at Max, and do not settle for a single one-off Workflow when a sustained fan-out would compound. Breadth-shaped = the same analysis across many independent items: bulk function matching / a whole-binary harvest, an EXE- or fleet-wide audit / survey / dedup, drafting C for dozens of functions, etc. (The litmus, encoded so I apply it: "would 10+ agents each doing one item beat me doing them in sequence?" — if yes, prompt.)

Why (the evidence): Phase 12 ran the resident harvest under Ultracode and took it 1.4% → 71.7% byte-identical in one session via 5 parallel-draft + byte-gate workflow passes (cookbook §12). Max-serial would have been an order of magnitude slower. Ultracode-on makes Workflow fan-out the default for every substantive task, which is what sustained the gains across passes — that compounding is the point, and it only happens if Ultracode is actually enabled, not improvised per task.

The prompt to use: "🟦 This stretch is breadth-shaped (~N independent items) — Ultracode (xHigh + multi-agent fan-out) would parallelize it; it took the resident 1.4%→72% in one Phase-12 session. Enable /effort ultracode? (I'll flip back to Max for the deep single-thread parts.)"

Caveat — keep the deep work at Max. Ultracode caps per-agent depth at xHigh, so it is wrong for the deep single-thread tasks (Phase-Start planning, the compiler fingerprint, US-address derivation, non-obvious debugging, PhaseEnd synthesis). The prompt is for the breadth stretch only; flip back to Max for those. And the judgment "is this actually breadth-shaped, or is it mine-to-author with full context?" is itself a Max call — writing a doc from this session's context is NOT breadth (subagents would have less context); fanning out a 134-overlay harvest IS.

Effort is mapped per-task AND re-evaluated continuously — pause+prompt at EVERY transition (Drew, 2026-06-16)

Effort management is a per-task, continuously-re-evaluated discipline, not a set-once choice. Claude cannot toggle effort itself, so it must surface every transition and wait for Drew to toggle:

  1. Plan-time mapping. Every task in a phase plan is annotated with its recommended effort (R7) — the baseline map.
  2. Mid-task re-evaluation. The right effort can be revealed during the work — a task planned at Max may surface a breadth-shaped parallel sub-task (bulk matching, a fleet survey). When it does: pause → prompt /effort ultracode → wait for the toggle (a system-reminder confirms) → then run the Workflow. Never launch multi-agent fan-out on a verbal "yes" before the toggle lands.
  3. Sub-task-done switch-back. The moment the breadth stretch completes and work returns to deep single-thread (synthesis, design, the PhaseEnd): pause → prompt to switch back to /effort max (Ultracode caps depth at xHigh, wrong for synthesis).
  4. Task-completion hand-off. At a task boundary, if Ultracode was used on the task just finished and the next task doesn't need it, stop and prompt to switch before starting the next task — do not roll into the next task at the wrong effort.

Failure modes that motivated this (both 2026-06-16): (a) launched a harvest Workflow on Drew's verbal "Enable Ultracode" before he toggled it; (b) finished the 5 Ultracode harvest passes and continued straight into the deep T2–T6 work without prompting the switch back to Max — Drew had to ask. Memory: effort-prompt-ultracode-on-breadth.

Evidence — Max vs Ultracode Workflow yield is the SAME for breadth harvests (controlled, Phase 13 T7, 2026-06-16)

A controlled experiment (Drew's, prompted by "did the Max workflow do more than the Ultracode one?"): 80 fixed unmatched SC01/077 functions (nins 8–15), the identical blind-draft Workflow + byte-gate, run once with Ultracode (xHigh agents, arm A) and once with Max agents (arm B) from the same committed baseline — the ONLY variable is the session effort the agents inherit.

arm session effort → agent depth byte-matched (of 80)
A Ultracode → xHigh 13
B Max → Max 14

Overlap: A∩B = 12, xHigh-only = 1, Max-only = 2 (union 15). 13 vs 14 is noise (a 1-function delta, ~80% of the union shared) — per-agent effort depth does not meaningfully change blind-draft harvest yield. The pre-experiment observation that "pass 1 at Max matched 92% vs later Ultracode passes' 54%" was confounded by function difficulty (pass 1 = the easiest functions), not effort.

Decision this settles: for breadth harvests, prefer Ultracode — it delivers the same per-agent yield as Max AND the multi-agent orchestration, so there is no yield cost to using it. Keep Max for deep single-thread synthesis (planning, fingerprinting, PhaseEnd) where reasoning depth is the bottleneck, not parallel coverage. (Caveat: one trial per arm — directional, not definitive; rules out a LARGE effect, not a small one. Re-test with multiple trials if a future task hinges on a subtle difference.)


The decision rule — Max vs xHigh vs Ultracode

  • Max — a wrong answer poisons everything downstream and is hard to reverse, or root cause is unclear (non-obvious debugging), or it is deep synthesis (phase plan / PhaseEnd). Depth matters.
  • xHigh — the design is already settled in the approved plan and this is execution; single-file, clear-requirements, mechanical. (Also the highest persistent baseline.)
  • Ultracode / Workflow fan-out — the task is "run the same analysis across many independent items" (survey, audit, dedup, bulk match). Breadth matters more than per-item depth.

Live heuristic, in order:

  1. Is the design already settled in the approved plan, and this is just execution? → xHigh.
  2. Is this a phase plan, a PhaseEnd, a genuine architectural fork, or non-obvious debugging? → Max.
  3. Would a wrong answer systematically poison everything downstream and be hard to reverse? → Max (the "use sparingly" tier).
  4. Is it the same analysis across many items? → Ultracode / Workflow.

Per-phase map (Gen1)

Constant for every phase: the Phase Start plan and the PhaseEnd file → Ultracode minimum; Max only for the heavy phases (3, 6) and the Gen1-exit PhaseEnd (7). Mechanical-phase PhaseEnds (2, 4, 5) are fine at xHigh/Ultracode — the synthesis is bookkeeping, not novel reasoning. (Confirmed 2026-06-13: Drew called Max unnecessary for the Phase-2 PhaseEnd.)

Phase Max — mandatory (never drop) xHigh — fine (drop for speed) Breadth — Ultracode/Workflow helps
1 — RE stack + import ✅ — (ran at ~xHigh; correct — headless scripts) —
2 — Extraction pipeline none on first pass; escalate only if LZSS cross-validation vs CUE fails (non-obvious debugging) ISO9660 walker, .CD/PAC splitter, LZSS decoder (first pass), SQV awareness, sha1 manifest, format docs —
3 — File-loader & overlay map ⭐ US address derivation (resident blob + ≥1 overlay) + PCSX-Redux RAM-dump proof; the deep RE call-chain tracing ledger seeding, blob import once proven, map write-up (optional) surveying many candidate loader functions
4 — Build toolchain none everything — apt installs, sha-verified compiler fetch, make check-env, version recording —
5 — splat config + byte-match splat .yaml section-boundary authoring/debugging (esp. if the clean link fights back) Makefile pipeline, check.us.sha, make expected —
6 — Compiler fingerprint + first matches ⭐⭐ compiler-fingerprint ladder + arbitration (the single highest-stakes task in Gen1); first matches incl. LZSS (proving loop + pin) recording the pinned triple, decompile.py/asm-differ wiring, INCLUDE_ASM headers (later) bulk matching of independent leaf functions
7 — Loop at scale + Gen1 close Gen1-exit synthesis (in the PhaseEnd); first jump-table match; loader-cluster matching (deep) report scripts (difficulty/dup/progress), README ⭐ EXE-wide duplicate report + difficulty inventory + bulk matching — real Workflow fan-out wins

The Max shortlist (reach for the ceiling only here): (1) Phase 6 fingerprint ladder, (2) Phase 3 address derivation + RAM proof. Conditional escalations: Phase 5 splat config if the link won't go clean; the Phase 7 Gen1-exit PhaseEnd.


Plan-mode (Phase Start) effort

Phase Start planning is deep single-thread synthesis → Max. Mandatory-Max for Phases 3 and 6; xHigh would suffice for the mechanical phases (2, 4, 5, 7) but since Max is the default, just leave it. Use Ultracode during planning only when the plan needs broad reconnaissance first (surveying many candidate functions) — realistically only Phase 3 and Phase 7.


The Effort-map check (Rule R7 — formalizes at PhaseEnd_Phase2)

  • Before each Phase Start plan: state the recommended effort for planning that phase (per this file) and confirm the developer has it set.
  • In the phase plan: annotate every task with its recommended effort level.
  • At every task handoff and in the Session Start "next task" statement: restate the recommended effort.
  • This file is evolvable — correct it as the harness changes or as the per-phase reality is learned.