diff --git a/experiments/atomic-codemode/README.md b/experiments/atomic-codemode/README.md index 7e488f31de..0c0589923c 100644 --- a/experiments/atomic-codemode/README.md +++ b/experiments/atomic-codemode/README.md @@ -23,7 +23,10 @@ The [control-query study](control-query-study.md) verifies every derived control triple on the real SQLite VM, fixes unknown CFG seeds and reuses a lazy postdominator index. Its [two-level investigation](controller-oracle.md) is frozen before model runs and separates alternative path predicates, SSA identity -and finite recovered-edge witnesses from runtime claims. +and finite recovered-edge witnesses from runtime claims. Both real-model answers +now pass: Code Mode reduces cumulative input but increases uncached input, time, +CPU and memory in this pair. Correct facts support both calling strategies; +the query-cache improvement is separate from those mixed whole-task costs. The [exact-value query study](value-query-study.md) adds location-to-value and value-to-operand-use queries, including inputs/constants without defining operations. diff --git a/experiments/atomic-codemode/capability-map.md b/experiments/atomic-codemode/capability-map.md index 08542cc051..becbb3b370 100644 --- a/experiments/atomic-codemode/capability-map.md +++ b/experiments/atomic-codemode/capability-map.md @@ -35,6 +35,14 @@ combines them. A query can be fine-grained even when its fresh native capture still computes a whole function. Same-lease reuse is measured separately from that work and does not establish a cross-session cache or native acceleration. +The [control-query study](control-query-study.md) independently verifies all +2,845 control-dependence triples in the real SQLite VM through both MCP adapters. +Two real models also complete a two-level controller/SSA/path-witness task and +distinguish alternative predicates from conjunction. Unknown CFG seeds are +explicit, and the snapshot owner reuses a bounded lazy postdominator index. +Code Mode has lower cumulative input but higher uncached input and runtime costs +in that pair; precise derived facts do not imply a preferred calling adapter. + The [OpenSSH value-query study](value-query-study.md) adds exact location lookup and inverse operand occurrences for value identities, including inputs/constants without definitions. It checks every retained selected version/use against the diff --git a/experiments/atomic-codemode/control-query-study.md b/experiments/atomic-codemode/control-query-study.md index 24cabc3f51..b09e28e625 100644 --- a/experiments/atomic-codemode/control-query-study.md +++ b/experiments/atomic-codemode/control-query-study.md @@ -81,6 +81,72 @@ excludes fresh native work, MCP discovery/serialization and model tokens; it does not establish a whole-agent speedup or fewer reads/tokens. Source/module hashes and facts are retained in `controls-profile-v1.json`. -The [two-level question and oracle](controller-oracle.md) require actual model -composition of control, values, operands, PHIs, CFG witnesses and machine facts. -Real-agent results must remain separate from these scripted checks. +## Real-agent two-level investigation + +The [question and oracle](controller-oracle.md) and runtime were frozen at +`8d13c67e7` before either model launch, Code Mode first and direct second. +Both complete answers pass the frozen oracle/source review: format ownership, +PHI/call binding, both complete controller levels, all four Boolean guards and +operand widths/LOADs, exact shared/distinct SSA identities, alternative predicates, +valid recovered witnesses, machine corroboration and explicit limits. Both close +the lease. The relation list is correctly distinguished from a conjunction. + +Direct chooses longer bypass witnesses than the oracle examples. Every reported +edge is valid; both paths end at zero-successor block 1207 and avoid block 1159. +Including the initial controller edge, direct uses nine/twelve edges, Code Mode +six/six; shortest paths were not required. Both full native function records equal +the earlier golden. All thirteen Code Mode and ten direct cited source IDs exist +with matching artifact/engine identities. Code Mode's extra pointer/spill facts +also match the retained IR and selected GNU decoding. The cost summarizer does +not automatically grade natural language. + +| Whole-task measure | Direct atomic | Code Mode | +| ------------------------------ | ------------: | --------: | +| Cumulative input tokens | 935,185 | 503,113 | +| Included cached input | 878,592 | 436,736 | +| Uncached input | 56,593 | 66,377 | +| Output tokens | 7,320 | 7,450 | +| Included reasoning output | 757 | 835 | +| Observed model responses | 24 | 17 | +| Client executions / waits | 22 / 1 | 15 / 1 | +| MCP attempts | 36 | 15 | +| Rejected MCP / inner API reads | 0 / 0 | 0 / 0 | +| Successful atomic fact reads | 32 | 33 | +| Model wall seconds | 230.280 | 246.479 | +| Family wall seconds | 230.755 | 248.500 | +| Family CPU seconds | 52.458 | 72.837 | +| Kernel process-family peak MiB | 876.5 | 944.3 | +| Native source-capture wall s | 37.852 | 52.471 | + +Both request `gpt-6.1-sol/high` via Codex 0.161.0, with complete 18/4-tool catalogs +and common exact-name/envelope/stopping/long-wait guidance. Actual client calls +use JavaScript `exec`/`wait` in both lanes despite false client Code Mode flags. +This compares server Code Mode with direct atomic MCP on that host, not +composition against its absence. Resolved service build and remote compute are +unknown. Per-response usage sums match the ledger; cached input is included in +input, reasoning in output. Discovery, schemas, waits and all calls are charged. +Runtime, target and engine remain unchanged; both profiles/payloads have verified +cleanup, OOM/kill 0 and no resource abort under the same one-CPU / 4 GiB / 3 GiB +bounds. No new warm real-agent task was run. + +Both actually query `controllers` at both levels. Neither uses the C tool or +fetches the complete raw IR report. Code Mode reads the complete block-successor +inventory internally for path search and does extra pointer/spill analysis: +fourteen block, six operation and three instruction API reads. Direct uses ten +block, five operation, two instruction reads, two bounded slices and more selected +CFG reads. Different strategies and native capture times remain confounds; +source-capture time is not an isolated native-drive interval and must not be +subtracted to manufacture a dispatch-only effect. + +In this single pair, Code Mode reduces cumulative input by about 46% and uses +fewer model rounds, but increases uncached input, output, elapsed time, CPU and +peak memory. The index's core-query improvement does not establish a causal +whole-agent benefit. Correct composable facts support both strategies; there is +still no stable overall Code Mode advantage or broad complex-software superiority +claim. No traditional-provider cell is invented for this native-identity question. + +Local records: `sqlite-controller-{direct,codemode}-v1*`, +`controller-matrix-costs.json` and `controller-answer-checks.json` under +`.cache/atomic-codemode/`. `summarize-trials.py` reconstructs costs. Fresh leases +and private native caches do not control OS/model-service caches; kernel peaks +include charged file cache, not RSS. Scripted warm/core results stay separate.