From 2817d30ab2fc41ae08a7c5f49a6647b2eda39d64 Mon Sep 17 00:00:00 2001 From: gensokyo-dev <342580529+gensokyo-dev@users.noreply.github.com> Date: Tue, 27 Oct 2026 15:10:48 +0800 Subject: [PATCH] fix(docs): preserve verification reports after experiment archival (#7462) Move the retained allocation report and unchanged receipt JSON into verification documentation. Keep one report producer, repair current and archived links, document test-support ownership, and remove retired lint inputs. Fixes #7458. Validated focused/Python checks, report consistency, docs rendering, and real Bash CLI/MCP facts against an independent oracle. Greptile reviewed all seven files without comments. CI source/Windows/shard-8 failures match clean-main or existing-main controls; the requester authorized merging with those established failures. Codex review quota was unavailable. --- docs/testing.md | 5 ++-- .../mcp-integer-allocation.json | 0 .../mcp-integer-allocation.md | 15 ++++++++---- tests/conformance/readmeConsistency.test.ts | 2 +- tests/support/analysis-efficiency/README.md | 23 +++++++++++++++++++ .../mcp-integer-report.mjs | 8 +++++-- turbo.json | 2 -- 7 files changed, 44 insertions(+), 11 deletions(-) rename {tests/support => docs/verification}/analysis-efficiency/mcp-integer-allocation.json (100%) rename {tests/support => docs/verification}/analysis-efficiency/mcp-integer-allocation.md (88%) create mode 100644 tests/support/analysis-efficiency/README.md diff --git a/docs/testing.md b/docs/testing.md index b01aef3d58..7dab7ead0b 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -7,8 +7,9 @@ installed-client and agent runs. Historical verification records may refer to `experiments/analysis-efficiency`. Those experiment scripts are available in the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency); -they are not included in current checkouts. Test support remains under -`tests/support/analysis-efficiency`. +they are not included in current checkouts. The retained [test support](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/README.md) +contains deterministic checkers and fixtures. The [allocation report](verification/analysis-efficiency/mcp-integer-allocation.md) +and its original receipt JSON remain documentation, checked by the report producer. ## Developer commands diff --git a/tests/support/analysis-efficiency/mcp-integer-allocation.json b/docs/verification/analysis-efficiency/mcp-integer-allocation.json similarity index 100% rename from tests/support/analysis-efficiency/mcp-integer-allocation.json rename to docs/verification/analysis-efficiency/mcp-integer-allocation.json diff --git a/tests/support/analysis-efficiency/mcp-integer-allocation.md b/docs/verification/analysis-efficiency/mcp-integer-allocation.md similarity index 88% rename from tests/support/analysis-efficiency/mcp-integer-allocation.md rename to docs/verification/analysis-efficiency/mcp-integer-allocation.md index 64a85ac79e..7b404afdeb 100644 --- a/tests/support/analysis-efficiency/mcp-integer-allocation.md +++ b/docs/verification/analysis-efficiency/mcp-integer-allocation.md @@ -22,7 +22,7 @@ Seven dependency-boundary variants match parsed values and complete error issues ## Tasks, artifacts and bounds -Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the catalog study](mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded. +Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the archived catalog study](https://github.com/morluto/rea/blob/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency/mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](https://github.com/morluto/rea/blob/main/docs/verification/analysis-efficiency/mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded. Normal cohort: three alternating baseline/candidate pairs for each task × cold/warm server compile cache × default compact/full inputs: 72 trials, plus four excluded full-task warm priming runs. Separate diagnostic cohort: three alternating pairs per profile, cold Zellij tasks, 12 trials. All pass, with identical complete parent IDs/hashes and answers. @@ -115,6 +115,12 @@ Question, explicit recipe, server instructions, one complete name/description/in - Both runtime builds and all six static checks plus architecture guards pass. Before the final main synchronization, 5,972 source/contract/public CLI/MCP tests pass across 323 files; four skip for unavailable/guarded coverage. After synchronization, both builds/static checks repeat successfully, 77 affected producer and CLI/MCP tests pass, and all 974 registered numeric-parameter parser cases match again. Includes real SQLite CLI/MCP record/column selection and JavaScript heap/deadline preservation. A first direct schema-validity invocation lacked generated catalog metadata; the supported focused entry generates it and all 15 cases pass. A later affected-test invocation selected two nonexistent paths; the corrected three-file producer/consumer run is the 77-case result above. Neither setup failure was counted as a passing test, and no production or timeout change was needed. - All 84 measured tasks and four warm priming tasks pass independent oracles, canonical refusal, parent/directory identity and owned cleanup checks. This does not verify external Hopper/Ghidra/IDA, Android devices, all providers or foreign OS behavior. The earlier macOS numbers in #4360 describe a different revision/platform and are not combined with these results. +The historical cohort commands below use the retired experiment layout. Run them +from the matching recorded producer checkout; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency) +provides the former files for reference. Current checkouts retain the report and +its conformance producer, rather than the full measurement suite. Moving these +files does not requalify the historical measurements against current REA. + Build the pinned baseline/candidate in separate clean worktrees with the locked toolchain. Use the candidate’s producer scripts for both roles. `measurement_inputs` in the JSON records the complete matrix, questions/recipes, runtime bindings and producer hashes; extract it unchanged as the bindings file. Historical receipts require their matching producer revision. One normal task: ```bash @@ -133,11 +139,12 @@ python experiments/analysis-efficiency/mcp-startup-costs.py "$RECEIPT_ROOT" \ The aggregator requires the recorded tiktoken version, exact catalog/parent/runtime/producer/guard identities, clean source, successful payload cleanup and matching matrix coverage. Default driver behavior continues without an inspector; the heap flag is confined to the experiment. -Regenerate or verify every numerical section directly from the committed JSON: +In a current checkout, regenerate or verify every numerical section directly +from the committed JSON: ```bash -node experiments/analysis-efficiency/mcp-integer-report.mjs -node experiments/analysis-efficiency/mcp-integer-report.mjs --check +node tests/support/analysis-efficiency/mcp-integer-report.mjs +node tests/support/analysis-efficiency/mcp-integer-report.mjs --check ``` The documentation conformance gate checks current receipts, rejects stale normal/heap tables, and refuses missing producer markers without writing. diff --git a/tests/conformance/readmeConsistency.test.ts b/tests/conformance/readmeConsistency.test.ts index e0660600cc..ee984a100e 100644 --- a/tests/conformance/readmeConsistency.test.ts +++ b/tests/conformance/readmeConsistency.test.ts @@ -89,7 +89,7 @@ describe("published performance evidence", () => { "tests/support/analysis-efficiency/mcp-integer-report.mjs", ); const report = resolve( - "tests/support/analysis-efficiency/mcp-integer-allocation.md", + "docs/verification/analysis-efficiency/mcp-integer-allocation.md", ); processTest( "keeps generated report facts aligned with committed receipts", diff --git a/tests/support/analysis-efficiency/README.md b/tests/support/analysis-efficiency/README.md new file mode 100644 index 0000000000..fd504226d4 --- /dev/null +++ b/tests/support/analysis-efficiency/README.md @@ -0,0 +1,23 @@ +# Analysis verification test support + +These files support the existing comparison, independent-oracle, receipt and +cleanup regressions. The TypeScript consumers live in `tests/boundary/`, +`tests/evaluation/` and `tests/conformance/`; `npm run test:python` discovers +the Python checks here. Build first for checks that use the compiled runtime. + +The agent-harness tests inject command outcomes and own temporary workspaces. +They do not invoke a real model or establish real-provider behavior. The +retained resource observer keeps its process-family CPU, memory and cleanup +guards. Public corpus identities and original receipt provenance stay intact. + +The historical [allocation report and receipt JSON](../../../docs/verification/analysis-efficiency/mcp-integer-allocation.md) +belong to documentation. Their numerical sections are produced and checked by: + +```sh +node tests/support/analysis-efficiency/mcp-integer-report.mjs --check +``` + +The full experiment suite was retired. For historical reproduction, use the +recorded runtime and producer revisions; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency) +preserves the former layout. Current deterministic checks do not requalify +those historical measurements. diff --git a/tests/support/analysis-efficiency/mcp-integer-report.mjs b/tests/support/analysis-efficiency/mcp-integer-report.mjs index 839102e50f..bb897d9b5a 100644 --- a/tests/support/analysis-efficiency/mcp-integer-report.mjs +++ b/tests/support/analysis-efficiency/mcp-integer-report.mjs @@ -10,10 +10,14 @@ const { values } = parseArgs({ check: { type: "boolean", default: false }, }, }); +const reportDirectory = new URL( + "../../../docs/verification/analysis-efficiency/", + import.meta.url, +); const dataPath = - values.data ?? new URL("mcp-integer-allocation.json", import.meta.url); + values.data ?? new URL("mcp-integer-allocation.json", reportDirectory); const markdownPath = - values.markdown ?? new URL("mcp-integer-allocation.md", import.meta.url); + values.markdown ?? new URL("mcp-integer-allocation.md", reportDirectory); const data = JSON.parse(await readFile(dataPath, "utf8")); assert.equal(data.status, "passed"); const original = await readFile(markdownPath, "utf8"); diff --git a/turbo.json b/turbo.json index e2b94188f9..653a253846 100644 --- a/turbo.json +++ b/turbo.json @@ -84,11 +84,9 @@ "docs/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}", ".github/actions/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}", "website/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}", - "experiments/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}", "!docs/.vitepress/cache/**", "!docs/.vitepress/dist/**", "!website/_site/**", - "!experiments/analysis-efficiency/results/**", "*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}" ] },