fix(docs): preserve verification reports after experiment archival (#7462)

Move the retained allocation report and unchanged receipt JSON into verification documentation. Keep one report producer, repair current and archived links, document test-support ownership, and remove retired lint inputs.

Fixes #7458.

Validated focused/Python checks, report consistency, docs rendering, and real Bash CLI/MCP facts against an independent oracle. Greptile reviewed all seven files without comments. CI source/Windows/shard-8 failures match clean-main or existing-main controls; the requester authorized merging with those established failures. Codex review quota was unavailable.
This commit is contained in:
gensokyo-dev
2026-10-27 15:10:48 +08:00
committed by GitHub
parent 0676ad254c
commit 2817d30ab2
7 changed files with 44 additions and 11 deletions
+3 -2
View File
@@ -7,8 +7,9 @@ installed-client and agent runs.
Historical verification records may refer to `experiments/analysis-efficiency`.
Those experiment scripts are available in the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency);
they are not included in current checkouts. Test support remains under
`tests/support/analysis-efficiency`.
they are not included in current checkouts. The retained [test support](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/README.md)
contains deterministic checkers and fixtures. The [allocation report](verification/analysis-efficiency/mcp-integer-allocation.md)
and its original receipt JSON remain documentation, checked by the report producer.
## Developer commands
@@ -22,7 +22,7 @@ Seven dependency-boundary variants match parsed values and complete error issues
## Tasks, artifacts and bounds
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the catalog study](mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the archived catalog study](https://github.com/morluto/rea/blob/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency/mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](https://github.com/morluto/rea/blob/main/docs/verification/analysis-efficiency/mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
Normal cohort: three alternating baseline/candidate pairs for each task × cold/warm server compile cache × default compact/full inputs: 72 trials, plus four excluded full-task warm priming runs. Separate diagnostic cohort: three alternating pairs per profile, cold Zellij tasks, 12 trials. All pass, with identical complete parent IDs/hashes and answers.
@@ -115,6 +115,12 @@ Question, explicit recipe, server instructions, one complete name/description/in
- Both runtime builds and all six static checks plus architecture guards pass. Before the final main synchronization, 5,972 source/contract/public CLI/MCP tests pass across 323 files; four skip for unavailable/guarded coverage. After synchronization, both builds/static checks repeat successfully, 77 affected producer and CLI/MCP tests pass, and all 974 registered numeric-parameter parser cases match again. Includes real SQLite CLI/MCP record/column selection and JavaScript heap/deadline preservation. A first direct schema-validity invocation lacked generated catalog metadata; the supported focused entry generates it and all 15 cases pass. A later affected-test invocation selected two nonexistent paths; the corrected three-file producer/consumer run is the 77-case result above. Neither setup failure was counted as a passing test, and no production or timeout change was needed.
- All 84 measured tasks and four warm priming tasks pass independent oracles, canonical refusal, parent/directory identity and owned cleanup checks. This does not verify external Hopper/Ghidra/IDA, Android devices, all providers or foreign OS behavior. The earlier macOS numbers in #4360 describe a different revision/platform and are not combined with these results.
The historical cohort commands below use the retired experiment layout. Run them
from the matching recorded producer checkout; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
provides the former files for reference. Current checkouts retain the report and
its conformance producer, rather than the full measurement suite. Moving these
files does not requalify the historical measurements against current REA.
Build the pinned baseline/candidate in separate clean worktrees with the locked toolchain. Use the candidate’s producer scripts for both roles. `measurement_inputs` in the JSON records the complete matrix, questions/recipes, runtime bindings and producer hashes; extract it unchanged as the bindings file. Historical receipts require their matching producer revision. One normal task:
```bash
@@ -133,11 +139,12 @@ python experiments/analysis-efficiency/mcp-startup-costs.py "$RECEIPT_ROOT" \
The aggregator requires the recorded tiktoken version, exact catalog/parent/runtime/producer/guard identities, clean source, successful payload cleanup and matching matrix coverage. Default driver behavior continues without an inspector; the heap flag is confined to the experiment.
Regenerate or verify every numerical section directly from the committed JSON:
In a current checkout, regenerate or verify every numerical section directly
from the committed JSON:
```bash
node experiments/analysis-efficiency/mcp-integer-report.mjs
node experiments/analysis-efficiency/mcp-integer-report.mjs --check
node tests/support/analysis-efficiency/mcp-integer-report.mjs
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
```
The documentation conformance gate checks current receipts, rejects stale normal/heap tables, and refuses missing producer markers without writing.
+1 -1
View File
@@ -89,7 +89,7 @@ describe("published performance evidence", () => {
"tests/support/analysis-efficiency/mcp-integer-report.mjs",
);
const report = resolve(
"tests/support/analysis-efficiency/mcp-integer-allocation.md",
"docs/verification/analysis-efficiency/mcp-integer-allocation.md",
);
processTest(
"keeps generated report facts aligned with committed receipts",
@@ -0,0 +1,23 @@
# Analysis verification test support
These files support the existing comparison, independent-oracle, receipt and
cleanup regressions. The TypeScript consumers live in `tests/boundary/`,
`tests/evaluation/` and `tests/conformance/`; `npm run test:python` discovers
the Python checks here. Build first for checks that use the compiled runtime.
The agent-harness tests inject command outcomes and own temporary workspaces.
They do not invoke a real model or establish real-provider behavior. The
retained resource observer keeps its process-family CPU, memory and cleanup
guards. Public corpus identities and original receipt provenance stay intact.
The historical [allocation report and receipt JSON](../../../docs/verification/analysis-efficiency/mcp-integer-allocation.md)
belong to documentation. Their numerical sections are produced and checked by:
```sh
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
```
The full experiment suite was retired. For historical reproduction, use the
recorded runtime and producer revisions; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
preserves the former layout. Current deterministic checks do not requalify
those historical measurements.
@@ -10,10 +10,14 @@ const { values } = parseArgs({
check: { type: "boolean", default: false },
},
});
const reportDirectory = new URL(
"../../../docs/verification/analysis-efficiency/",
import.meta.url,
);
const dataPath =
values.data ?? new URL("mcp-integer-allocation.json", import.meta.url);
values.data ?? new URL("mcp-integer-allocation.json", reportDirectory);
const markdownPath =
values.markdown ?? new URL("mcp-integer-allocation.md", import.meta.url);
values.markdown ?? new URL("mcp-integer-allocation.md", reportDirectory);
const data = JSON.parse(await readFile(dataPath, "utf8"));
assert.equal(data.status, "passed");
const original = await readFile(markdownPath, "utf8");
-2
View File
@@ -84,11 +84,9 @@
"docs/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
".github/actions/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
"website/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
"experiments/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
"!docs/.vitepress/cache/**",
"!docs/.vitepress/dist/**",
"!website/_site/**",
"!experiments/analysis-efficiency/results/**",
"*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}"
]
},