docs(experiment): report two-level control model trials

Both real-model answers satisfy the frozen controller, SSA identity, path-witness and machine-evidence oracle. Code Mode reduces cumulative input while increasing uncached input and runtime costs in this sample; preserve the complete mixed result and native-work confounds.

Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com>
Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
N0zoM1z0
2026-10-16 18:40:12 +00:00
parent 8d13c67e7e
commit 3e4c0d8b59
3 changed files with 81 additions and 4 deletions
+4 -1
View File
@@ -23,7 +23,10 @@ The [control-query study](control-query-study.md) verifies every derived control
triple on the real SQLite VM, fixes unknown CFG seeds and reuses a lazy
postdominator index. Its [two-level investigation](controller-oracle.md) is
frozen before model runs and separates alternative path predicates, SSA identity
and finite recovered-edge witnesses from runtime claims.
and finite recovered-edge witnesses from runtime claims. Both real-model answers
now pass: Code Mode reduces cumulative input but increases uncached input, time,
CPU and memory in this pair. Correct facts support both calling strategies;
the query-cache improvement is separate from those mixed whole-task costs.
The [exact-value query study](value-query-study.md) adds location-to-value and
value-to-operand-use queries, including inputs/constants without defining operations.
@@ -35,6 +35,14 @@ combines them. A query can be fine-grained even when its fresh native capture
still computes a whole function. Same-lease reuse is measured separately from
that work and does not establish a cross-session cache or native acceleration.
The [control-query study](control-query-study.md) independently verifies all
2,845 control-dependence triples in the real SQLite VM through both MCP adapters.
Two real models also complete a two-level controller/SSA/path-witness task and
distinguish alternative predicates from conjunction. Unknown CFG seeds are
explicit, and the snapshot owner reuses a bounded lazy postdominator index.
Code Mode has lower cumulative input but higher uncached input and runtime costs
in that pair; precise derived facts do not imply a preferred calling adapter.
The [OpenSSH value-query study](value-query-study.md) adds exact location lookup
and inverse operand occurrences for value identities, including inputs/constants
without definitions. It checks every retained selected version/use against the
@@ -81,6 +81,72 @@ excludes fresh native work, MCP discovery/serialization and model tokens;
it does not establish a whole-agent speedup or fewer reads/tokens. Source/module
hashes and facts are retained in `controls-profile-v1.json`.
The [two-level question and oracle](controller-oracle.md) require actual model
composition of control, values, operands, PHIs, CFG witnesses and machine facts.
Real-agent results must remain separate from these scripted checks.
## Real-agent two-level investigation
The [question and oracle](controller-oracle.md) and runtime were frozen at
`8d13c67e7` before either model launch, Code Mode first and direct second.
Both complete answers pass the frozen oracle/source review: format ownership,
PHI/call binding, both complete controller levels, all four Boolean guards and
operand widths/LOADs, exact shared/distinct SSA identities, alternative predicates,
valid recovered witnesses, machine corroboration and explicit limits. Both close
the lease. The relation list is correctly distinguished from a conjunction.
Direct chooses longer bypass witnesses than the oracle examples. Every reported
edge is valid; both paths end at zero-successor block 1207 and avoid block 1159.
Including the initial controller edge, direct uses nine/twelve edges, Code Mode
six/six; shortest paths were not required. Both full native function records equal
the earlier golden. All thirteen Code Mode and ten direct cited source IDs exist
with matching artifact/engine identities. Code Mode's extra pointer/spill facts
also match the retained IR and selected GNU decoding. The cost summarizer does
not automatically grade natural language.
| Whole-task measure | Direct atomic | Code Mode |
| ------------------------------ | ------------: | --------: |
| Cumulative input tokens | 935,185 | 503,113 |
| Included cached input | 878,592 | 436,736 |
| Uncached input | 56,593 | 66,377 |
| Output tokens | 7,320 | 7,450 |
| Included reasoning output | 757 | 835 |
| Observed model responses | 24 | 17 |
| Client executions / waits | 22 / 1 | 15 / 1 |
| MCP attempts | 36 | 15 |
| Rejected MCP / inner API reads | 0 / 0 | 0 / 0 |
| Successful atomic fact reads | 32 | 33 |
| Model wall seconds | 230.280 | 246.479 |
| Family wall seconds | 230.755 | 248.500 |
| Family CPU seconds | 52.458 | 72.837 |
| Kernel process-family peak MiB | 876.5 | 944.3 |
| Native source-capture wall s | 37.852 | 52.471 |
Both request `gpt-6.1-sol/high` via Codex 0.161.0, with complete 18/4-tool catalogs
and common exact-name/envelope/stopping/long-wait guidance. Actual client calls
use JavaScript `exec`/`wait` in both lanes despite false client Code Mode flags.
This compares server Code Mode with direct atomic MCP on that host, not
composition against its absence. Resolved service build and remote compute are
unknown. Per-response usage sums match the ledger; cached input is included in
input, reasoning in output. Discovery, schemas, waits and all calls are charged.
Runtime, target and engine remain unchanged; both profiles/payloads have verified
cleanup, OOM/kill 0 and no resource abort under the same one-CPU / 4 GiB / 3 GiB
bounds. No new warm real-agent task was run.
Both actually query `controllers` at both levels. Neither uses the C tool or
fetches the complete raw IR report. Code Mode reads the complete block-successor
inventory internally for path search and does extra pointer/spill analysis:
fourteen block, six operation and three instruction API reads. Direct uses ten
block, five operation, two instruction reads, two bounded slices and more selected
CFG reads. Different strategies and native capture times remain confounds;
source-capture time is not an isolated native-drive interval and must not be
subtracted to manufacture a dispatch-only effect.
In this single pair, Code Mode reduces cumulative input by about 46% and uses
fewer model rounds, but increases uncached input, output, elapsed time, CPU and
peak memory. The index's core-query improvement does not establish a causal
whole-agent benefit. Correct composable facts support both strategies; there is
still no stable overall Code Mode advantage or broad complex-software superiority
claim. No traditional-provider cell is invented for this native-identity question.
Local records: `sqlite-controller-{direct,codemode}-v1*`,
`controller-matrix-costs.json` and `controller-answer-checks.json` under
`.cache/atomic-codemode/`. `summarize-trials.py` reconstructs costs. Fresh leases
and private native caches do not control OS/model-service caches; kernel peaks
include charged file cache, not RSS. Scripted warm/core results stay separate.