diff --git a/experiments/atomic-codemode/call-oracle.md b/experiments/atomic-codemode/call-oracle.md index 65668d8103..1dbaa6fd32 100644 --- a/experiments/atomic-codemode/call-oracle.md +++ b/experiments/atomic-codemode/call-oracle.md @@ -54,3 +54,11 @@ added. Older sources used pretty JSON; their smaller/newer encoded sizes must not be attributed to the call exporter. The earlier compact-writer change is a separate experiment. Unsupported old schema graphs remain unavailable with valid C retained, rather than acquiring guessed call records. + +The first paired question runs at `01e5c5955` both explain the primary state flow +correctly but omit the oracle's separate return-1/return-18 distinctions. Because +the initial question did not explicitly enumerate those codes, the next round +clarifies that requirement in `sqlite-call-question.txt` before either lane +starts. The required facts and independent oracle stay unchanged. Earlier +questions and costs remain pinned to their original commits; neither incomplete +sample is counted as a full-oracle success. diff --git a/experiments/atomic-codemode/sqlite-call-question.txt b/experiments/atomic-codemode/sqlite-call-question.txt index d6b7440fe1..c57b6090d4 100644 --- a/experiments/atomic-codemode/sqlite-call-question.txt +++ b/experiments/atomic-codemode/sqlite-call-question.txt @@ -6,6 +6,8 @@ the corresponding recovered function input. Explain which context field the help updates for opening/closing parentheses, where the caller checks that same field, and how the successful and malformed paths differ. Verify the essential pointer, field and control-transfer claims against selected machine instructions. +In the final answer, explicitly distinguish the wrapper's routing for helper +return codes 0, 1 and 18, including the malformed and depth-limit diagnostics. Keep the answer focused on this cross-function state flow. Give function and instruction addresses and real Evidence references. Distinguish recovered call diff --git a/experiments/atomic-codemode/trials.md b/experiments/atomic-codemode/trials.md index 7393e5577c..99ec9cde50 100644 --- a/experiments/atomic-codemode/trials.md +++ b/experiments/atomic-codemode/trials.md @@ -467,3 +467,51 @@ family CPU 3.619 seconds and memory peak 436.6 MiB. Cleanup is verified, OOM 0. The failed sample and raw receipts remain under `.cache/atomic-codemode/` and are excluded from correctness-equal paired costs. Subsequent attempts use an independent preserved executable and explicit pre/post engine validation. + +## Cross-function call task: initial question pair + +`sqlite-call-codemode-v2` and `sqlite-call-atomic-direct-v2` both run at +`01e5c5955`, with engine `64c6b3f...` from an independent preserved executable. +The fourteen fact methods, native profile, artifact, requested model/effort and +common guidance match; Code Mode runs first. Runtime, engine and target bytes +remain unchanged, per-response usage matches runner totals, cleanup is verified +and OOM kills are zero. Both fresh target leases use uncontrolled warm OS caches. + +| Measurement | Code Mode call v2 | Atomic direct call v2 | +| ------------------------------------ | ----------------: | --------------------: | +| MCP requests / transport failures | 10 / 0 | 27 / 0 | +| Observed model responses | 13 | 16 | +| Client exec calls | 12 | 15 | +| Model wall seconds | 168.622 | 156.033 | +| Family wall seconds | 169.190 | 156.570 | +| Family CPU seconds | 16.096 | 15.854 | +| Family memory peak MiB | 514.0 | 608.1 | +| Cumulative input tokens | 454,199 | 564,158 | +| Cached input tokens | 377,216 | 497,408 | +| Uncached input tokens | 76,983 | 66,750 | +| Output tokens, including reasoning | 4,987 | 4,838 | +| Reasoning tokens, included in output | 1,301 | 935 | + +Both answers correctly connect call `0x65211 -> 0x64060`, caller SSA +`v100000118`, passing storage `register:0x38/8`, and callee input `v300000042`. +They verify caller LEA, helper pointer preservation, character dispatch, +field increment/decrement and the caller's post-success field check against +selected machine bytes. They preserve storage-match/alias and runtime +limitations. Code Mode actually composes `calls`, `values`, operations and machine +reads; direct uses the same primitives, including a bounded slice. All ten/nine +cited Evidence sources respectively have matching retained-source SHA-256 and +artifact identity. + +Neither final answer includes the original oracle's required separate routing +for helper return 1 and 18. The question only explicitly asked to distinguish +successful/malformed paths; the oracle was stricter than that wording. Do not +retroactively delete this requirement or mark either sample a full-oracle pass. +The task wording is now clarified before the next paired round, retaining the +same oracle and adding no solution addresses. The table is an observed partial +coverage result, not a completed-task efficiency claim. Its mixed costs also +do not show a general Code Mode advantage: cumulative input is lower while +uncached input and wall time are higher in this sample. + +Raw ledger: `.cache/atomic-codemode/ledger-call-round-01e5c59.json`. Each cell is +one stochastic sample. The complete client already provides JavaScript dispatch +in both lanes. No new traditional-provider cell is being claimed.