test(environment): verify real bundles and cover shared input ownership

This commit is contained in:
gensokyo-dev
2026-10-19 20:22:26 +00:00
parent 026b29ab8d
commit 62e8ce3897
4 changed files with 75 additions and 7 deletions
@@ -1,5 +1,9 @@
# Verification
The initial author record below describes the original macOS tree. The Linux
followup record at the end covers the integrated serve changes and subsequent
repairs in this PR. The two records have different toolchains and cache states.
Verified base: 24574576e4fa74ff56eda965ae287555b7552c03, the tip of the environment-serve increment on fetched main b83b9caf (this increment builds on it).
The worktree carries the manifest-view, streamable-HTTP and serve increments
@@ -242,3 +246,50 @@ dispatched from this tree. The built projects ran on this host's Node 22.23.1
rather than the pinned 24.18.0. The `/usr/bin/time -l` figures account for
each command and the children it waited for; Docker's own work runs in the
daemon and is outside them. Windows was not run.
## Linux followup verification (2026-10-19)
The original PR was updated from main after #4955 and #4975 merged. The
followups retain the author's commits and portable-name checks, seal private
JSON parser output once for shared immutable authentication/replay, reserve
sealing work before reading, preflight the served catalog before exporting,
translate changed-file failures by input role, defer export imports during CLI
registration, and verify the supplied framework executable before attributing
its results. The default `rea mcp` transport remains stdio.
Executed on Linux x64 with Node 24.18.0, npm 11.16.0, Docker 20.10.24,
Python 3.12.13 and an isolated `agentenv-framework==0.9.1334` environment:
- `turbo run typecheck lint:built format:check docs:check:built knip metadata:check artifacts:skills --concurrency=1`: ten tasks passed, one cached, 289.315 s elapsed, 284.552 s CPU, 4450.18 MiB peak process-family RSS.
- The focused source/compiled CLI/SDK/lifecycle/configuration run passed 399 tests in 24 files. A later run passed the updated 231 CI-selection cases, three executable-identity cases and ten product-catalog cases. The latter needed its generated catalog first; its initial missing-file failure was preparation, not a passing check.
- Four previously recorded, unauthenticated real GitHub API responses were exported with the checkout tarball. The generated Dockerfile built unchanged. A fresh non-root container replayed every response twice through the official MCP SDK, matching body, status, Evidence ordinal/id and untrusted-content provenance. Discovery/card parity and tarball removal passed. The standalone test supplies `ENVIRONMENT_NAME` as the framework does. Runtime verification with the built image took 12.393 s / 6.002 s CPU / 313.11 MiB process-family RSS; image-build cost is excluded. A Docker stats sample was 250.5 MiB, not a measured container peak.
- The production `verify-agentenv-environment.mjs` observed `agent-env 0.9.1334`, exported its 28-record source-owned Grafana-shaped fixture, deployed it with the real framework and passed `tasks/smoke.json`. The receipt reported five public cases, one sandbox torn down, no new task containers and no supervised descendants. This successful run used already prepared infrastructure: 141.697 s / 101.971 s CPU / 501.61 MiB process-family RSS. Earlier attempts failed because this host's proxy environment required optional SOCKS support and its default home sandbox directory was unwritable. Only the test process's proxy variables were cleared; supported `XDG_STATE_HOME` and `AGENT_ENV_LOCAL_SANDBOX_DIR` pointed into temporary storage. The shared Docker daemon was not reconfigured or restarted. The framework's persistent registry helper and images remain as its documented behavior.
A matched public `environment-serve`/SDK task compared the pre-followup serve
build with the final implementation. Both used the same GitHub capture with
1,000 synthetic HAR-extension items (each containing additional containers),
one first call and twenty warm identical replays. Fixture generation was
excluded. This is controlled stress data, not newly observed application data.
Larger 25,000/75,000-item projection attempts returned the existing typed
representation-budget refusal; that budget was not relaxed.
| Measurement | Before | After |
| ----------------------- | ----------: | ----------: |
| Cold listener startup | 2189.064 ms | 2862.860 ms |
| First call | 76.950 ms | 45.265 ms |
| Twenty warm calls | 324.119 ms | 267.132 ms |
| Completed task elapsed | 3.229 s | 3.885 s |
| Completed task CPU | 3.196 s | 3.827 s |
| Peak process-family RSS | 466.43 MiB | 470.21 MiB |
These single completed samples do not establish an overall speed or memory
improvement. Separate identity regressions establish that large reported JSON
trees are shared rather than copied and that caller-owned mutable trees are
not frozen. Token usage was not measured. CPU is child user+system time;
process-family memory sums sampled descendant RSS every 50 ms and may count
shared pages twice. Docker daemon/container CPU and peaks are outside that
process-family measurement and remain unmeasured. Builds used one task at a
time, one CPU and a soft 3 GiB Go limit; Docker builds were serialized and
limited to one CPU/4 GiB, and the standalone replay container to one CPU/2 GiB.
Windows execution and the published registry package were not tested in this
followup; the checkout tarball proves the code before its release.
+2 -1
View File
@@ -441,8 +441,9 @@ export const SCOPES = {
agentenv: [
".github/workflows/real-agentenv.yml",
"scripts/verify-agentenv-environment.mjs",
"tests/boundary/process/agentenvVerifierIdentity.test.ts",
"src/{application,server}/environment/**",
"src/application/{JsonInputFile,OwnedJsonInput,RegularFileRead}.ts",
"src/cliJsonInput.ts",
"src/server/http/**",
"src/{cli,cliCommandNames,cli/environmentCommands}.ts",
"src/config/{environmentName,mcpHttpEndpoint}.ts",
@@ -806,6 +806,18 @@ it.each([
"scripts/verify-agentenv-environment.mjs",
{ code: true, agentenv: true, runtime: false, package: false },
],
[
"src/application/OwnedJsonInput.ts",
{ code: true, agentenv: true, runtime: true, package: false },
],
[
"src/cliJsonInput.ts",
{ code: true, agentenv: true, runtime: true, package: false },
],
[
"tests/boundary/process/agentenvVerifierIdentity.test.ts",
{ code: true, agentenv: false, runtime: false, package: false },
],
[
".github/workflows/real-agentenv.yml",
{ code: false, agentenv: true, workflows: true, package: false },
@@ -8,11 +8,13 @@ import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
// The local framework and these executable shebang fixtures target POSIX hosts.
it
.skipIf(process.platform === "win32")
.each(["agent-env 0.9.9999", "other-runner 0.9.1334"])(
"refuses the observed executable identity %s before touching Docker",
async (version) => {
it.skipIf(process.platform === "win32").each([
["agent-env 0.9.9999", false],
["other-runner 0.9.1334", false],
["agent-env 0.9.1334", true],
] as const)(
"admits the observed executable identity %s before Docker: %s",
async (version, accepted) => {
const root = await createTestTempDirectory("rea-agentenv-identity-");
const runner = join(root, "agent-env.mjs");
const docker = join(root, "docker.mjs");
@@ -55,6 +57,8 @@ it
);
expect(result.stderr).toContain('"public_cases": 0');
expect(result.stderr).not.toContain('"status": "passed"');
await expect(readFile(marker)).rejects.toMatchObject({ code: "ENOENT" });
if (accepted) expect(await readFile(marker, "utf8")).toBe("called");
else
await expect(readFile(marker)).rejects.toMatchObject({ code: "ENOENT" });
},
);