mirror of
https://github.com/morluto/rea
synced 2026-10-28 06:46:47 -04:00
docs(experiment): report two-level control model trials
Both real-model answers satisfy the frozen controller, SSA identity, path-witness and machine-evidence oracle. Code Mode reduces cumulative input while increasing uncached input and runtime costs in this sample; preserve the complete mixed result and native-work confounds. Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com>
This commit is contained in:
@@ -23,7 +23,10 @@ The [control-query study](control-query-study.md) verifies every derived control
|
||||
triple on the real SQLite VM, fixes unknown CFG seeds and reuses a lazy
|
||||
postdominator index. Its [two-level investigation](controller-oracle.md) is
|
||||
frozen before model runs and separates alternative path predicates, SSA identity
|
||||
and finite recovered-edge witnesses from runtime claims.
|
||||
and finite recovered-edge witnesses from runtime claims. Both real-model answers
|
||||
now pass: Code Mode reduces cumulative input but increases uncached input, time,
|
||||
CPU and memory in this pair. Correct facts support both calling strategies;
|
||||
the query-cache improvement is separate from those mixed whole-task costs.
|
||||
|
||||
The [exact-value query study](value-query-study.md) adds location-to-value and
|
||||
value-to-operand-use queries, including inputs/constants without defining operations.
|
||||
|
||||
@@ -35,6 +35,14 @@ combines them. A query can be fine-grained even when its fresh native capture
|
||||
still computes a whole function. Same-lease reuse is measured separately from
|
||||
that work and does not establish a cross-session cache or native acceleration.
|
||||
|
||||
The [control-query study](control-query-study.md) independently verifies all
|
||||
2,845 control-dependence triples in the real SQLite VM through both MCP adapters.
|
||||
Two real models also complete a two-level controller/SSA/path-witness task and
|
||||
distinguish alternative predicates from conjunction. Unknown CFG seeds are
|
||||
explicit, and the snapshot owner reuses a bounded lazy postdominator index.
|
||||
Code Mode has lower cumulative input but higher uncached input and runtime costs
|
||||
in that pair; precise derived facts do not imply a preferred calling adapter.
|
||||
|
||||
The [OpenSSH value-query study](value-query-study.md) adds exact location lookup
|
||||
and inverse operand occurrences for value identities, including inputs/constants
|
||||
without definitions. It checks every retained selected version/use against the
|
||||
|
||||
@@ -81,6 +81,72 @@ excludes fresh native work, MCP discovery/serialization and model tokens;
|
||||
it does not establish a whole-agent speedup or fewer reads/tokens. Source/module
|
||||
hashes and facts are retained in `controls-profile-v1.json`.
|
||||
|
||||
The [two-level question and oracle](controller-oracle.md) require actual model
|
||||
composition of control, values, operands, PHIs, CFG witnesses and machine facts.
|
||||
Real-agent results must remain separate from these scripted checks.
|
||||
## Real-agent two-level investigation
|
||||
|
||||
The [question and oracle](controller-oracle.md) and runtime were frozen at
|
||||
`8d13c67e7` before either model launch, Code Mode first and direct second.
|
||||
Both complete answers pass the frozen oracle/source review: format ownership,
|
||||
PHI/call binding, both complete controller levels, all four Boolean guards and
|
||||
operand widths/LOADs, exact shared/distinct SSA identities, alternative predicates,
|
||||
valid recovered witnesses, machine corroboration and explicit limits. Both close
|
||||
the lease. The relation list is correctly distinguished from a conjunction.
|
||||
|
||||
Direct chooses longer bypass witnesses than the oracle examples. Every reported
|
||||
edge is valid; both paths end at zero-successor block 1207 and avoid block 1159.
|
||||
Including the initial controller edge, direct uses nine/twelve edges, Code Mode
|
||||
six/six; shortest paths were not required. Both full native function records equal
|
||||
the earlier golden. All thirteen Code Mode and ten direct cited source IDs exist
|
||||
with matching artifact/engine identities. Code Mode's extra pointer/spill facts
|
||||
also match the retained IR and selected GNU decoding. The cost summarizer does
|
||||
not automatically grade natural language.
|
||||
|
||||
| Whole-task measure | Direct atomic | Code Mode |
|
||||
| ------------------------------ | ------------: | --------: |
|
||||
| Cumulative input tokens | 935,185 | 503,113 |
|
||||
| Included cached input | 878,592 | 436,736 |
|
||||
| Uncached input | 56,593 | 66,377 |
|
||||
| Output tokens | 7,320 | 7,450 |
|
||||
| Included reasoning output | 757 | 835 |
|
||||
| Observed model responses | 24 | 17 |
|
||||
| Client executions / waits | 22 / 1 | 15 / 1 |
|
||||
| MCP attempts | 36 | 15 |
|
||||
| Rejected MCP / inner API reads | 0 / 0 | 0 / 0 |
|
||||
| Successful atomic fact reads | 32 | 33 |
|
||||
| Model wall seconds | 230.280 | 246.479 |
|
||||
| Family wall seconds | 230.755 | 248.500 |
|
||||
| Family CPU seconds | 52.458 | 72.837 |
|
||||
| Kernel process-family peak MiB | 876.5 | 944.3 |
|
||||
| Native source-capture wall s | 37.852 | 52.471 |
|
||||
|
||||
Both request `gpt-6.1-sol/high` via Codex 0.161.0, with complete 18/4-tool catalogs
|
||||
and common exact-name/envelope/stopping/long-wait guidance. Actual client calls
|
||||
use JavaScript `exec`/`wait` in both lanes despite false client Code Mode flags.
|
||||
This compares server Code Mode with direct atomic MCP on that host, not
|
||||
composition against its absence. Resolved service build and remote compute are
|
||||
unknown. Per-response usage sums match the ledger; cached input is included in
|
||||
input, reasoning in output. Discovery, schemas, waits and all calls are charged.
|
||||
Runtime, target and engine remain unchanged; both profiles/payloads have verified
|
||||
cleanup, OOM/kill 0 and no resource abort under the same one-CPU / 4 GiB / 3 GiB
|
||||
bounds. No new warm real-agent task was run.
|
||||
|
||||
Both actually query `controllers` at both levels. Neither uses the C tool or
|
||||
fetches the complete raw IR report. Code Mode reads the complete block-successor
|
||||
inventory internally for path search and does extra pointer/spill analysis:
|
||||
fourteen block, six operation and three instruction API reads. Direct uses ten
|
||||
block, five operation, two instruction reads, two bounded slices and more selected
|
||||
CFG reads. Different strategies and native capture times remain confounds;
|
||||
source-capture time is not an isolated native-drive interval and must not be
|
||||
subtracted to manufacture a dispatch-only effect.
|
||||
|
||||
In this single pair, Code Mode reduces cumulative input by about 46% and uses
|
||||
fewer model rounds, but increases uncached input, output, elapsed time, CPU and
|
||||
peak memory. The index's core-query improvement does not establish a causal
|
||||
whole-agent benefit. Correct composable facts support both strategies; there is
|
||||
still no stable overall Code Mode advantage or broad complex-software superiority
|
||||
claim. No traditional-provider cell is invented for this native-identity question.
|
||||
|
||||
Local records: `sqlite-controller-{direct,codemode}-v1*`,
|
||||
`controller-matrix-costs.json` and `controller-answer-checks.json` under
|
||||
`.cache/atomic-codemode/`. `summarize-trials.py` reconstructs costs. Fresh leases
|
||||
and private native caches do not control OS/model-service caches; kernel peaks
|
||||
include charged file cache, not RSS. Scripted warm/core results stay separate.
|
||||
|
||||
Reference in New Issue
Block a user