From e9348df0c3932c7a85eecb8d4a7751397d46eb80 Mon Sep 17 00:00:00 2001 From: N0zoM1z0 <161784452+N0zoM1z0@users.noreply.github.com> Date: Fri, 16 Oct 2026 08:10:41 +0000 Subject: [PATCH] experiment(re): align cross-function question with pinned return-path oracle Retain both initial real-model answers and whole-task costs. Both prove the core context connection but omit the oracle-required return-code distinctions; clarify that requirement before the next pair without changing the oracle or supplying addresses. Co-authored-by: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Co-authored-by: Kaoru0822 <208281322+kaoru0822-kitauji@users.noreply.github.com> --- experiments/atomic-codemode/call-oracle.md | 8 ++++ .../atomic-codemode/sqlite-call-question.txt | 2 + experiments/atomic-codemode/trials.md | 48 +++++++++++++++++++ 3 files changed, 58 insertions(+) diff --git a/experiments/atomic-codemode/call-oracle.md b/experiments/atomic-codemode/call-oracle.md index 65668d8103..1dbaa6fd32 100644 --- a/experiments/atomic-codemode/call-oracle.md +++ b/experiments/atomic-codemode/call-oracle.md @@ -54,3 +54,11 @@ added. Older sources used pretty JSON; their smaller/newer encoded sizes must not be attributed to the call exporter. The earlier compact-writer change is a separate experiment. Unsupported old schema graphs remain unavailable with valid C retained, rather than acquiring guessed call records. + +The first paired question runs at `01e5c5955` both explain the primary state flow +correctly but omit the oracle's separate return-1/return-18 distinctions. Because +the initial question did not explicitly enumerate those codes, the next round +clarifies that requirement in `sqlite-call-question.txt` before either lane +starts. The required facts and independent oracle stay unchanged. Earlier +questions and costs remain pinned to their original commits; neither incomplete +sample is counted as a full-oracle success. diff --git a/experiments/atomic-codemode/sqlite-call-question.txt b/experiments/atomic-codemode/sqlite-call-question.txt index d6b7440fe1..c57b6090d4 100644 --- a/experiments/atomic-codemode/sqlite-call-question.txt +++ b/experiments/atomic-codemode/sqlite-call-question.txt @@ -6,6 +6,8 @@ the corresponding recovered function input. Explain which context field the help updates for opening/closing parentheses, where the caller checks that same field, and how the successful and malformed paths differ. Verify the essential pointer, field and control-transfer claims against selected machine instructions. +In the final answer, explicitly distinguish the wrapper's routing for helper +return codes 0, 1 and 18, including the malformed and depth-limit diagnostics. Keep the answer focused on this cross-function state flow. Give function and instruction addresses and real Evidence references. Distinguish recovered call diff --git a/experiments/atomic-codemode/trials.md b/experiments/atomic-codemode/trials.md index 7393e5577c..99ec9cde50 100644 --- a/experiments/atomic-codemode/trials.md +++ b/experiments/atomic-codemode/trials.md @@ -467,3 +467,51 @@ family CPU 3.619 seconds and memory peak 436.6 MiB. Cleanup is verified, OOM 0. The failed sample and raw receipts remain under `.cache/atomic-codemode/` and are excluded from correctness-equal paired costs. Subsequent attempts use an independent preserved executable and explicit pre/post engine validation. + +## Cross-function call task: initial question pair + +`sqlite-call-codemode-v2` and `sqlite-call-atomic-direct-v2` both run at +`01e5c5955`, with engine `64c6b3f...` from an independent preserved executable. +The fourteen fact methods, native profile, artifact, requested model/effort and +common guidance match; Code Mode runs first. Runtime, engine and target bytes +remain unchanged, per-response usage matches runner totals, cleanup is verified +and OOM kills are zero. Both fresh target leases use uncontrolled warm OS caches. + +| Measurement | Code Mode call v2 | Atomic direct call v2 | +| ------------------------------------ | ----------------: | --------------------: | +| MCP requests / transport failures | 10 / 0 | 27 / 0 | +| Observed model responses | 13 | 16 | +| Client exec calls | 12 | 15 | +| Model wall seconds | 168.622 | 156.033 | +| Family wall seconds | 169.190 | 156.570 | +| Family CPU seconds | 16.096 | 15.854 | +| Family memory peak MiB | 514.0 | 608.1 | +| Cumulative input tokens | 454,199 | 564,158 | +| Cached input tokens | 377,216 | 497,408 | +| Uncached input tokens | 76,983 | 66,750 | +| Output tokens, including reasoning | 4,987 | 4,838 | +| Reasoning tokens, included in output | 1,301 | 935 | + +Both answers correctly connect call `0x65211 -> 0x64060`, caller SSA +`v100000118`, passing storage `register:0x38/8`, and callee input `v300000042`. +They verify caller LEA, helper pointer preservation, character dispatch, +field increment/decrement and the caller's post-success field check against +selected machine bytes. They preserve storage-match/alias and runtime +limitations. Code Mode actually composes `calls`, `values`, operations and machine +reads; direct uses the same primitives, including a bounded slice. All ten/nine +cited Evidence sources respectively have matching retained-source SHA-256 and +artifact identity. + +Neither final answer includes the original oracle's required separate routing +for helper return 1 and 18. The question only explicitly asked to distinguish +successful/malformed paths; the oracle was stricter than that wording. Do not +retroactively delete this requirement or mark either sample a full-oracle pass. +The task wording is now clarified before the next paired round, retaining the +same oracle and adding no solution addresses. The table is an observed partial +coverage result, not a completed-task efficiency claim. Its mixed costs also +do not show a general Code Mode advantage: cumulative input is lower while +uncached input and wall time are higher in this sample. + +Raw ledger: `.cache/atomic-codemode/ledger-call-round-01e5c59.json`. Each cell is +one stochastic sample. The complete client already provides JavaScript dispatch +in both lanes. No new traditional-provider cell is being claimed.