mirror of
https://github.com/morluto/rea
synced 2026-10-27 22:45:19 -04:00
fix(docs): preserve verification reports after experiment archival (#7462)
Move the retained allocation report and unchanged receipt JSON into verification documentation. Keep one report producer, repair current and archived links, document test-support ownership, and remove retired lint inputs. Fixes #7458. Validated focused/Python checks, report consistency, docs rendering, and real Bash CLI/MCP facts against an independent oracle. Greptile reviewed all seven files without comments. CI source/Windows/shard-8 failures match clean-main or existing-main controls; the requester authorized merging with those established failures. Codex review quota was unavailable.
This commit is contained in:
+3
-2
@@ -7,8 +7,9 @@ installed-client and agent runs.
|
||||
|
||||
Historical verification records may refer to `experiments/analysis-efficiency`.
|
||||
Those experiment scripts are available in the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency);
|
||||
they are not included in current checkouts. Test support remains under
|
||||
`tests/support/analysis-efficiency`.
|
||||
they are not included in current checkouts. The retained [test support](https://github.com/morluto/rea/blob/main/tests/support/analysis-efficiency/README.md)
|
||||
contains deterministic checkers and fixtures. The [allocation report](verification/analysis-efficiency/mcp-integer-allocation.md)
|
||||
and its original receipt JSON remain documentation, checked by the report producer.
|
||||
|
||||
## Developer commands
|
||||
|
||||
|
||||
+11
-4
@@ -22,7 +22,7 @@ Seven dependency-boundary variants match parsed values and complete error issues
|
||||
|
||||
## Tasks, artifacts and bounds
|
||||
|
||||
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the catalog study](mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
|
||||
Reuse the shared SDK task driver and independent ELF/section/zlib/JSON oracles from [the archived catalog study](https://github.com/morluto/rea/blob/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency/mcp-input-catalog.md): official Alpine v3.21 x86_64 ripgrep 14.1.1-r0 and Zellij 0.40.0-r0. All 42/422 packages and 88/1,063 edges match the independent oracles. Select serde name/version/kind, repeat exact retained-parent reads twice, verify a canonical negative-offset refusal, export/check complete Evidence and confirm the unchanged directory. `kind: null` stays unknown. Targets are not executed; dependency presence does not establish runtime use. Acquisition URLs, archive/executable/oracle hashes and complete answers are in [the receipts](https://github.com/morluto/rea/blob/main/docs/verification/analysis-efficiency/mcp-integer-allocation.json). Previously acquired public bytes are rehashed, not redownloaded.
|
||||
|
||||
Normal cohort: three alternating baseline/candidate pairs for each task × cold/warm server compile cache × default compact/full inputs: 72 trials, plus four excluded full-task warm priming runs. Separate diagnostic cohort: three alternating pairs per profile, cold Zellij tasks, 12 trials. All pass, with identical complete parent IDs/hashes and answers.
|
||||
|
||||
@@ -115,6 +115,12 @@ Question, explicit recipe, server instructions, one complete name/description/in
|
||||
- Both runtime builds and all six static checks plus architecture guards pass. Before the final main synchronization, 5,972 source/contract/public CLI/MCP tests pass across 323 files; four skip for unavailable/guarded coverage. After synchronization, both builds/static checks repeat successfully, 77 affected producer and CLI/MCP tests pass, and all 974 registered numeric-parameter parser cases match again. Includes real SQLite CLI/MCP record/column selection and JavaScript heap/deadline preservation. A first direct schema-validity invocation lacked generated catalog metadata; the supported focused entry generates it and all 15 cases pass. A later affected-test invocation selected two nonexistent paths; the corrected three-file producer/consumer run is the 77-case result above. Neither setup failure was counted as a passing test, and no production or timeout change was needed.
|
||||
- All 84 measured tasks and four warm priming tasks pass independent oracles, canonical refusal, parent/directory identity and owned cleanup checks. This does not verify external Hopper/Ghidra/IDA, Android devices, all providers or foreign OS behavior. The earlier macOS numbers in #4360 describe a different revision/platform and are not combined with these results.
|
||||
|
||||
The historical cohort commands below use the retired experiment layout. Run them
|
||||
from the matching recorded producer checkout; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
|
||||
provides the former files for reference. Current checkouts retain the report and
|
||||
its conformance producer, rather than the full measurement suite. Moving these
|
||||
files does not requalify the historical measurements against current REA.
|
||||
|
||||
Build the pinned baseline/candidate in separate clean worktrees with the locked toolchain. Use the candidate’s producer scripts for both roles. `measurement_inputs` in the JSON records the complete matrix, questions/recipes, runtime bindings and producer hashes; extract it unchanged as the bindings file. Historical receipts require their matching producer revision. One normal task:
|
||||
|
||||
```bash
|
||||
@@ -133,11 +139,12 @@ python experiments/analysis-efficiency/mcp-startup-costs.py "$RECEIPT_ROOT" \
|
||||
|
||||
The aggregator requires the recorded tiktoken version, exact catalog/parent/runtime/producer/guard identities, clean source, successful payload cleanup and matching matrix coverage. Default driver behavior continues without an inspector; the heap flag is confined to the experiment.
|
||||
|
||||
Regenerate or verify every numerical section directly from the committed JSON:
|
||||
In a current checkout, regenerate or verify every numerical section directly
|
||||
from the committed JSON:
|
||||
|
||||
```bash
|
||||
node experiments/analysis-efficiency/mcp-integer-report.mjs
|
||||
node experiments/analysis-efficiency/mcp-integer-report.mjs --check
|
||||
node tests/support/analysis-efficiency/mcp-integer-report.mjs
|
||||
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
|
||||
```
|
||||
|
||||
The documentation conformance gate checks current receipts, rejects stale normal/heap tables, and refuses missing producer markers without writing.
|
||||
@@ -89,7 +89,7 @@ describe("published performance evidence", () => {
|
||||
"tests/support/analysis-efficiency/mcp-integer-report.mjs",
|
||||
);
|
||||
const report = resolve(
|
||||
"tests/support/analysis-efficiency/mcp-integer-allocation.md",
|
||||
"docs/verification/analysis-efficiency/mcp-integer-allocation.md",
|
||||
);
|
||||
processTest(
|
||||
"keeps generated report facts aligned with committed receipts",
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
# Analysis verification test support
|
||||
|
||||
These files support the existing comparison, independent-oracle, receipt and
|
||||
cleanup regressions. The TypeScript consumers live in `tests/boundary/`,
|
||||
`tests/evaluation/` and `tests/conformance/`; `npm run test:python` discovers
|
||||
the Python checks here. Build first for checks that use the compiled runtime.
|
||||
|
||||
The agent-harness tests inject command outcomes and own temporary workspaces.
|
||||
They do not invoke a real model or establish real-provider behavior. The
|
||||
retained resource observer keeps its process-family CPU, memory and cleanup
|
||||
guards. Public corpus identities and original receipt provenance stay intact.
|
||||
|
||||
The historical [allocation report and receipt JSON](../../../docs/verification/analysis-efficiency/mcp-integer-allocation.md)
|
||||
belong to documentation. Their numerical sections are produced and checked by:
|
||||
|
||||
```sh
|
||||
node tests/support/analysis-efficiency/mcp-integer-report.mjs --check
|
||||
```
|
||||
|
||||
The full experiment suite was retired. For historical reproduction, use the
|
||||
recorded runtime and producer revisions; the [archived source tree](https://github.com/morluto/rea/tree/a2e2c29b59e8775b0e6aedb09a9eb16adf32646b/experiments/analysis-efficiency)
|
||||
preserves the former layout. Current deterministic checks do not requalify
|
||||
those historical measurements.
|
||||
@@ -10,10 +10,14 @@ const { values } = parseArgs({
|
||||
check: { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
const reportDirectory = new URL(
|
||||
"../../../docs/verification/analysis-efficiency/",
|
||||
import.meta.url,
|
||||
);
|
||||
const dataPath =
|
||||
values.data ?? new URL("mcp-integer-allocation.json", import.meta.url);
|
||||
values.data ?? new URL("mcp-integer-allocation.json", reportDirectory);
|
||||
const markdownPath =
|
||||
values.markdown ?? new URL("mcp-integer-allocation.md", import.meta.url);
|
||||
values.markdown ?? new URL("mcp-integer-allocation.md", reportDirectory);
|
||||
const data = JSON.parse(await readFile(dataPath, "utf8"));
|
||||
assert.equal(data.status, "passed");
|
||||
const original = await readFile(markdownPath, "utf8");
|
||||
|
||||
@@ -84,11 +84,9 @@
|
||||
"docs/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
||||
".github/actions/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
||||
"website/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
||||
"experiments/**/*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}",
|
||||
"!docs/.vitepress/cache/**",
|
||||
"!docs/.vitepress/dist/**",
|
||||
"!website/_site/**",
|
||||
"!experiments/analysis-efficiency/results/**",
|
||||
"*.{cjs,cts,js,jsx,mjs,mts,ts,tsx}"
|
||||
]
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user