chore: merge current main after environment export

This commit is contained in:
gensokyo-dev
2026-10-19 21:30:53 +00:00
21 changed files with 1116 additions and 245 deletions
+8
View File
@@ -334,6 +334,14 @@ remove it in the same layer, which is how a checkout rather than a release is
run. The command prints the
written files with their sizes and the pinned package source.
Export uses the shared output-tree owner to check directory identities around
exclusive creates, verify written bytes, and reject unowned entries before
reporting success. It does not delete an inspectable partial folder on failure.
Node's portable pathname APIs leave a syscall-boundary race, so the output's
parent directories must be trusted. On POSIX output files use mode `0600`;
Windows inherits ACLs and REA does not establish owner-only access there. Choose
an already private Windows destination when captured content needs it.
The Dockerfile sets `MCP_HOST=0.0.0.0` and `MCP_PORT=18765`, the port the
framework maps for an MCP server env; the framework's own deploy supplies
`MCP_HOST`, `MCP_PORT` and `ENVIRONMENT_NAME`, so the served card carries the
+1 -1
View File
@@ -378,7 +378,7 @@ rea environment-export manifest.json --evidence capture.json ./grafana-example -
The folder holds a `README.md` with the bundle's provenance, an env folder
`envs/<name>/` with a `Dockerfile` that installs this exact `rea-agents`
version and serves the copied `manifest.json` and `evidence.json` (written
readable by their owner only, since they hold captured content) with
with mode `0600` on POSIX; Windows retains inherited ACLs) with
`environment-serve`, a Python verifier under `artifacts/check_<name>/` and the
smoke task `tasks/smoke.json`. `--package-tarball` copies a packed `rea-agents`
tarball into the env folder and installs it instead of the registry package,
@@ -30,8 +30,8 @@ previous increment.
nothing is ever overwritten.
- Write `README.md`, `envs/<name>/Dockerfile`, `envs/<name>/env.toml`,
`envs/<name>/manifest.json` and `envs/<name>/evidence.json` as byte-exact
copies of the inputs (mode `0600`, so captured content stays readable by its
owner only whatever the umask), `artifacts/check_<name>/verify.py` and
copies of the inputs (mode `0600` on POSIX; Windows retains inherited ACLs
and does not provide an owner-only guarantee), `artifacts/check_<name>/verify.py` and
`tasks/smoke.json`, where `<name>` defaults to the manifest's kebab-case name
and `--name` must be a kebab-case label of at most 64 characters that is not a
device name Windows reserves (`con`, `prn`, `aux`, `nul`, `com1`–`com9`,
@@ -293,3 +293,70 @@ time, one CPU and a soft 3 GiB Go limit; Docker builds were serialized and
limited to one CPU/4 GiB, and the standalone replay container to one CPU/2 GiB.
Windows execution and the published registry package were not tested in this
followup; the checkout tarball proves the code before its release.
### Output ownership and final review followups
The output boundary now uses the existing `SafeOutputTree` owner instead of
separate directory admission and pathname writes. It checks the admitted root,
parents, file identities and complete entry inventory around publication, keeps
failed-close authority, and preserves an existing empty caller root on rollback.
The optional package is held as one regular-file descriptor before output
writes, streamed with hash/readback verification, and checked for changes before
release. POSIX file modes are set and verified as `0600` independently of umask;
newly owned directories are `0700`. Existing root permissions remain caller owned.
Windows privacy depends on the selected destination ACL; no owner-only Windows
ACL guarantee or Windows runtime validation is claimed. Portable pathname
operations still require a trusted destination and parents against concurrent
replacement.
A compiled CLI counterexample under umask `0400` failed before the mode fix and
succeeded afterward with every generated file readable at `0600` and exact
Evidence bytes. Public CLI coverage also verifies that `symlink/..` retains its
filesystem meaning for the destination and package path. Real-filesystem tests
exercise destination replacement, foreign entries, parent symlinks, read-only
permissions and admitted-tarball replacement. Controlled close failures exercise
post-publication recovery and persistent retained ownership.
The inventory owner accepts canonically equivalent Unicode directory listings
without collapsing distinct filenames: an NFC multiset comparison is followed
by the original path identity checks. Tests simulate decomposed `readdir`
spelling over a real owned file and create an extra distinct Linux decomposed
foreign file. These establish the boundary behavior, not actual HFS+ execution.
Manifest, Evidence and tarball descriptors use the shared resource scope's close
retry. Persistent failures retain the canonical cleanup lease and produce a
tagged `environment-export` error. The compiled JSON CLI regression injects a
persistent manifest close failure before any destination write.
The framework verifier reuses `execNpm` and honors selected `npm_execpath`.
Four process-boundary attribution cases pass, including a conflicting PATH npm.
Those controlled executables verify attribution only.
After merging main `b4938aed5`, the generated Dockerfile built unchanged with the
checkout package and replayed all four captured public GitHub API responses
twice through the official MCP SDK in a fresh non-root container. Input bytes,
status/body/Evidence/provenance, discovery/card parity and package removal passed.
Warm base-image build plus pack/export/runtime verification: **40.699 s elapsed,
8.274 s process CPU, 421.04 MiB peak process-family RSS**. The real pinned
`agent-env 0.9.1334` verifier then passed all five public cases and the smoke task,
tore down its sandbox, and reported no new task containers or supervised
descendants: **156.895 s / 102.871 s CPU / 500.34 MiB**. Framework infrastructure
was already prepared. The later Unicode, close-failure and npm-selection repairs
have separate boundary regressions; they do not establish new real-platform
coverage.
A matched export comparison before the main merge used the same manifest
(6,416 bytes), capture (98,519 bytes) and checkout tarball (6,221,957 bytes), with
all copied inputs byte-exact. Before/after the output-owner repair: **1.827/2.604
s elapsed, 1.807/2.545 s CPU, 316.43/334.11 MiB** peak process-family RSS. These are
single process samples with uncontrolled host filesystem cache; the additional
ownership checks increased measured cost. They do not establish a speed or
memory improvement.
All followup heavy work uses one CPU and one test/build task at a time. Docker
builds are limited to one CPU/4 GiB, standalone runtime containers one CPU/2 GiB.
Process-family RSS is sampled every 50 ms and may double-count shared pages.
Docker daemon/container CPU and peak memory, agent tokens and fresh-framework
infrastructure cost are unmeasured. The previously documented early catalog
heap refusal is not represented as a passing run.
Final focused verification on the reviewed source passed **452 tests in 33 files** (including 232 CI-selection cases and the compiled CLI regressions): **161.276 s elapsed / 159.900 s CPU / 1242.43 MiB** peak process-family RSS. The final typecheck/build/lint gate passed: **218.427 s / 218.130 s CPU / 4449.45 MiB**.
+5 -2
View File
@@ -1571,8 +1571,11 @@ the host's Docker CLI named by absolute `REA_DOCKER_COMMAND`, Python 3.12 with
`agentenv-framework==0.9.1334` installed in a virtual environment whose
`agent-env` binary is named by absolute `REA_AGENT_ENV_COMMAND`, and outbound
network for `pip` on the runner and for `npm` inside the image build. Both
variables are required; the verifier resolves nothing from `PATH` and installs
nothing. It checks the supplied runner's `--version` against the pin before
variables are required; neither upstream executable is resolved from `PATH`,
and the verifier installs nothing. Packing uses the shared npm helper, honoring
`npm_execpath` from the caller's selected npm. Direct Node invocations without
that value use the helper's platform fallback, so configure the pinned toolchain
on `PATH` for those runs. It checks the supplied runner's `--version` against the pin before
using Docker and retains the observed version in both passing and failing receipts.
Its disposable CI runner creates the virtual environment in runner
storage, exports both paths and uses the runner's own Docker; no developer host
+1
View File
@@ -443,6 +443,7 @@ export const SCOPES = {
"scripts/verify-agentenv-environment.mjs",
"src/{application,server}/environment/**",
"src/application/{JsonInputFile,OwnedJsonInput,RegularFileRead}.ts",
"src/artifacts/{SafeOutputTree*,ArtifactResourceScope,StreamBytes}.ts",
"src/cliJsonInput.ts",
"src/server/http/**",
"src/{cli,cliCommandNames,cli/environmentCommands}.ts",
+2 -2
View File
@@ -13,6 +13,7 @@ import {
runOwnedCommand,
} from "../dist/process/OwnedCommand.js";
import { createVerifierRun, completeVerifierRun } from "./lib/verifier-run.mjs";
import { execNpm } from "./lib/verify-package-core.mjs";
const AGENTENV_VERSION = "0.9.1334";
const AGENTENV_PIN = `agentenv-framework ${AGENTENV_VERSION}`;
@@ -123,8 +124,7 @@ try {
await writeFile(capturePath, JSON.stringify(capture));
await writeFile(manifestPath, JSON.stringify(manifest));
// The lane proves this checkout, not the registry package: pack it and install the tarball in the image.
const { stdout: packed } = await execFileAsync(
"npm",
const { stdout: packed } = await execNpm(
["pack", "--ignore-scripts", "--json", "--pack-destination", runtime.path],
{ cwd: checkout, env: environment, maxBuffer: 16 * 1024 * 1024 },
);
@@ -243,6 +243,10 @@ An agent environment exported by REA from one retained ${source.captureKind ===
Every tool answers from a recorded exchange; nothing here runs the captured
application.
The bundle contains captured application content. Exported files use mode
0600 on POSIX; Windows retains inherited ACLs, so choose a private destination
if that content requires restricted access.
## Provenance
- ${source.captureKind === "browser-scenario" ? "Scenario definition" : "Capture artifact"} SHA-256: \`${source.captureSha256 ?? "excluded from Evidence"}\`
@@ -0,0 +1,203 @@
import {
chmod,
mkdir,
readFile,
readdir,
rename,
symlink,
writeFile,
} from "node:fs/promises";
import { join } from "node:path";
import { expect, it, vi } from "vitest";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import {
environmentServeSource,
writeEnvironmentServeFiles,
} from "../../../tests/fixtures/environmentServe.js";
import { SafeOutputTree } from "../../artifacts/SafeOutputTree.js";
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import {
exportEnvironmentBundle,
readEnvironmentExportInputs,
} from "./environmentExport.js";
it("retains the admitted tarball descriptor and refuses a pathname replacement during output", async () => {
const root = await createTestTempDirectory("rea-tarball-instance-");
const files = await writeEnvironmentServeFiles(root);
const inputs = await readEnvironmentExportInputs(
files.manifestPath,
files.capturePath,
);
if (!inputs.ok) throw inputs.error;
const tarball = join(root, "package.tgz");
const originalBytes = Buffer.from("original admitted package");
await writeFile(tarball, originalBytes);
const directory = join(root, "bundle");
const original = SafeOutputTree.prototype.write;
let replaced = false;
const writer = vi
.spyOn(SafeOutputTree.prototype, "write")
.mockImplementation(async function (
this: SafeOutputTree,
...args: Parameters<SafeOutputTree["write"]>
) {
const result = await original.apply(this, args);
if (!replaced) {
replaced = true;
await rename(tarball, join(root, "admitted.tgz"));
await writeFile(tarball, "replacement package");
}
return result;
});
try {
const result = await exportEnvironmentBundle({
...inputs.value,
source: environmentServeSource(),
directory,
packageTarball: tarball,
});
expect(result).toMatchObject({
ok: false,
error: { _tag: "AnalysisArtifactChangedError", path: tarball },
});
expect(
await readFile(join(directory, "envs/grafana-example/rea-agents.tgz")),
).toEqual(originalBytes);
} finally {
writer.mockRestore();
}
});
it("uses confirmed publication after the output owner's descriptor cleanup completes", async () => {
const root = await createTestTempDirectory("rea-bundle-published-");
const files = await writeEnvironmentServeFiles(root);
const inputs = await readEnvironmentExportInputs(
files.manifestPath,
files.capturePath,
);
if (!inputs.ok) throw inputs.error;
const directory = join(root, "bundle");
const original = SafeOutputTree.prototype.commit;
const commit = vi
.spyOn(SafeOutputTree.prototype, "commit")
.mockImplementation(async function (this: SafeOutputTree, ...args) {
await original.apply(this, args);
throw new ArtifactReaderFailure(
"unavailable",
"controlled post-publication close failure",
{
cleanup: {
reason: "controlled directory close failure",
resources: [directory],
},
},
);
});
try {
const result = await exportEnvironmentBundle({
...inputs.value,
source: environmentServeSource(),
directory,
});
expect(result).toMatchObject({ ok: true, value: { status: "exported" } });
expect(
await readFile(join(directory, "envs/grafana-example/evidence.json")),
).toEqual(inputs.value.evidence.bytes);
} finally {
commit.mockRestore();
}
});
it.skipIf(process.platform === "win32" || process.getuid?.() === 0)(
"returns a tagged permission refusal for an existing read-only empty directory",
async () => {
const root = await createTestTempDirectory("rea-bundle-readonly-");
const files = await writeEnvironmentServeFiles(root);
const inputs = await readEnvironmentExportInputs(
files.manifestPath,
files.capturePath,
);
if (!inputs.ok) throw inputs.error;
const directory = join(root, "bundle");
await mkdir(directory, { mode: 0o500 });
try {
const result = await exportEnvironmentBundle({
...inputs.value,
source: environmentServeSource(),
directory,
});
expect(result).toMatchObject({
ok: false,
error: {
_tag: "AnalysisAccessDeniedError",
operation: "environment-export",
},
});
expect(await readdir(directory)).toEqual([]);
} finally {
await chmod(directory, 0o700);
}
},
);
it
.skipIf(process.platform === "win32")
.each(["replace-root", "foreign-entry", "symlink-parent"] as const)(
"refuses publication after an actual output transition: %s",
async (transition) => {
const root = await createTestTempDirectory("rea-bundle-output-transition-");
const files = await writeEnvironmentServeFiles(root);
const inputs = await readEnvironmentExportInputs(
files.manifestPath,
files.capturePath,
);
if (!inputs.ok) throw inputs.error;
const directory = join(root, "bundle");
const foreign = join(root, "foreign");
await mkdir(foreign);
const original = SafeOutputTree.prototype.write;
let changed = false;
const writer = vi
.spyOn(SafeOutputTree.prototype, "write")
.mockImplementation(async function (
this: SafeOutputTree,
...args: Parameters<SafeOutputTree["write"]>
) {
const result = await original.apply(this, args);
if (!changed) {
changed = true;
if (transition === "replace-root") {
await rename(directory, join(root, "moved"));
await mkdir(directory);
await writeFile(join(directory, "foreign.txt"), "keep");
} else if (transition === "foreign-entry") {
await writeFile(join(directory, "foreign.txt"), "keep");
} else {
await symlink(foreign, join(directory, "envs"), "dir");
}
}
return result;
});
try {
const result = await exportEnvironmentBundle({
...inputs.value,
source: environmentServeSource(),
directory,
});
expect(changed).toBe(true);
expect(result).toMatchObject({
ok: false,
error: { operation: "environment-export" },
});
expect(await readdir(foreign)).toEqual([]);
if (transition !== "symlink-parent")
expect(await readFile(join(directory, "foreign.txt"), "utf8")).toBe(
"keep",
);
} finally {
writer.mockRestore();
}
},
);
@@ -0,0 +1,194 @@
import { createHash } from "node:crypto";
import { mkdir, type FileHandle } from "node:fs/promises";
import { dirname } from "node:path";
import { Readable } from "node:stream";
import {
AnalysisInputError,
AnalysisAccessDeniedError,
} from "../../domain/analysisErrorCore.js";
import { AnalysisCleanupLease } from "../../artifacts/AnalysisCleanupLease.js";
import { ArtifactFileCleanupFailure } from "../../artifacts/ArtifactFileOperation.js";
import { RegularFileCleanupFailure } from "../../filesystem/OwnedRegularFile.js";
import { streamChunkToBuffer } from "../../artifacts/StreamBytes.js";
import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
import { SafeOutputTree } from "../../artifacts/SafeOutputTree.js";
import { SafeOutputTreeCreationFailure } from "../../artifacts/SafeOutputTreeCreationFailure.js";
import { RegularFileChangedError } from "../../filesystem/RegularFile.js";
import { withRegularFile } from "../RegularFileRead.js";
import {
ENVIRONMENT_EXPORT_OPERATION,
type EnvironmentBundlePlan,
} from "./environmentBundle.js";
import type {
EnvironmentExportInput,
WrittenBundleFile,
} from "./environmentExport.js";
const writeBytes = async (
tree: SafeOutputTree,
relative: string,
bytes: Buffer,
): Promise<WrittenBundleFile> => {
const written = await tree.write(relative, Readable.from([bytes]), {
bytes: bytes.length,
sha256: createHash("sha256").update(bytes).digest("hex"),
});
return { path: written.relativePath, bytes: written.bytesWritten };
};
interface BundleTarball {
readonly path: string;
readonly handle: FileHandle;
readonly size: number;
}
/** Hash and copy only the instance retained before destination admission. */
const writeTarball = async (
tree: SafeOutputTree,
relative: string,
source: BundleTarball,
): Promise<WrittenBundleFile> => {
const hash = createHash("sha256");
let observed = 0;
for await (const chunk of source.handle.createReadStream({
start: 0,
autoClose: false,
})) {
const bytes = streamChunkToBuffer(chunk);
observed += bytes.length;
if (observed > source.size) throw new RegularFileChangedError(source.path);
hash.update(bytes);
}
if (observed !== source.size) throw new RegularFileChangedError(source.path);
const written = await tree.write(
relative,
source.handle.createReadStream({ start: 0, autoClose: false }),
{
bytes: source.size,
sha256: hash.digest("hex"),
},
);
return { path: written.relativePath, bytes: written.bytesWritten };
};
/** The output owner anchors every create and verifies the complete tree before publication. */
const writeBundleFiles = async (
input: EnvironmentExportInput,
plan: EnvironmentBundlePlan,
tarball?: BundleTarball,
): Promise<readonly WrittenBundleFile[]> => {
const resources = new ArtifactResourceScope();
let tree: SafeOutputTree | undefined;
let written: WrittenBundleFile[] | undefined;
let outcome:
| {
readonly kind: "completed";
readonly files: readonly WrittenBundleFile[];
}
| { readonly kind: "failed"; readonly cause: unknown };
try {
await mkdir(dirname(input.directory), { recursive: true, mode: 0o700 });
try {
tree = await SafeOutputTree.create(input.directory, process.platform, {
acceptExistingEmpty: true,
});
} catch (cause: unknown) {
if (cause instanceof SafeOutputTreeCreationFailure) tree = cause.tree;
throw cause;
}
written = [];
for (const file of plan.files)
written.push(
await writeBytes(tree, file.path, Buffer.from(file.content)),
);
written.push(
await writeBytes(tree, plan.copies.manifest, input.manifest.bytes),
);
written.push(
await writeBytes(tree, plan.copies.evidence, input.evidence.bytes),
);
if (plan.copies.tarball !== null && tarball !== undefined)
written.push(await writeTarball(tree, plan.copies.tarball, tarball));
await tree.commit();
outcome = { kind: "completed", files: written };
} catch (cause: unknown) {
outcome = { kind: "failed", cause };
}
const owner = tree;
if (owner !== undefined) {
const closed = await resources.release({
kind: "lease",
resource: input.directory,
lease: { release: () => owner.closeHandles() },
});
if (closed.kind === "failed")
throw resources.cleanupFailure(outcome, [closed]);
}
if (outcome.kind === "failed") {
if (owner?.published === true && written !== undefined) return written;
throw outcome.cause;
}
return outcome.files;
};
/** Retain one tarball instance across planning output writes and final stability verification. */
export const writeEnvironmentBundle = async (
input: EnvironmentExportInput,
plan: EnvironmentBundlePlan,
): Promise<readonly WrittenBundleFile[]> => {
if (plan.packageSource.kind !== "tarball")
return writeBundleFiles(input, plan);
const path = plan.packageSource.path;
const resources = new ArtifactResourceScope();
let admitted = false;
try {
return await withRegularFile(
path,
async (handle, state) => {
admitted = true;
return writeBundleFiles(input, plan, {
path,
handle,
size: state.size,
});
},
{ resources },
);
} catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure)
throw new ArtifactFileCleanupFailure(
cause,
AnalysisCleanupLease.forCloseOwner(cause.owner, path),
);
if (!admitted && cause instanceof Error && "code" in cause) {
if (cause.code === "EACCES" || cause.code === "EPERM")
throw new AnalysisAccessDeniedError(
ENVIRONMENT_EXPORT_OPERATION,
path,
cause.code,
{ cause },
);
if (
["ENOENT", "ENOTDIR", "ENAMETOOLONG", "EISDIR", "ENOTFILE"].includes(
String(cause.code),
)
)
throw new AnalysisInputError(
ENVIRONMENT_EXPORT_OPERATION,
{
cause,
userMessage: `${path} must name a readable regular package tarball.`,
},
[
{
path: ["package_tarball"],
reason: "invalid_value",
message: "The package tarball must be an existing regular file.",
},
],
);
}
throw cause;
}
};
+123 -168
View File
@@ -1,18 +1,27 @@
import { constants } from "node:fs";
import { copyFile, mkdir, readdir, stat, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { getHeapStatistics } from "node:v8";
import type { AnalysisError } from "../../domain/analysisErrorBase.js";
import { AnalysisError } from "../../domain/analysisErrorBase.js";
import { analysisErrorWithCleanupFailure } from "../../domain/analysisErrorCleanup.js";
import {
AnalysisAccessDeniedError,
AnalysisArtifactChangedError,
AnalysisInputError,
AnalysisOutputError,
AnalysisResourceConstraintError,
type AnalysisInputIssue,
} from "../../domain/analysisErrorCore.js";
import type { JsonValue } from "../../domain/jsonValue.js";
import { err, ok, type Result } from "../../domain/result.js";
import { RegularFileChangedError } from "../../filesystem/RegularFile.js";
import {
bindArtifactFileCleanup,
ArtifactFileCleanupFailure,
} from "../../artifacts/ArtifactFileOperation.js";
import { AnalysisCleanupLease } from "../../artifacts/AnalysisCleanupLease.js";
import { ArtifactResourceScope } from "../../artifacts/ArtifactResourceScope.js";
import { RegularFileCleanupFailure } from "../../filesystem/OwnedRegularFile.js";
import { ArtifactReaderFailure } from "../../artifacts/ArtifactReader.js";
import { writeEnvironmentBundle } from "./environmentBundleWrite.js";
import { sealOwnedJsonInput } from "../OwnedJsonInput.js";
import { nativeJsonParseRequiredBytes } from "../JsonInputFile.js";
import { readRegularFile } from "../RegularFileRead.js";
@@ -24,7 +33,6 @@ import {
ENVIRONMENT_EXPORT_OPERATION,
planEnvironmentBundle,
registryPackageSource,
type EnvironmentBundlePlan,
type EnvironmentPackageSource,
} from "./environmentBundle.js";
import type { EnvironmentSource } from "./environmentSource.js";
@@ -82,92 +90,39 @@ const systemCode = (cause: unknown): string | undefined =>
? cause.code
: undefined;
/** Host permission failures are typed; any other filesystem failure is unexpected and propagates. */
const rethrowUnlessDenied = (cause: unknown, path: string): never => {
const code = systemCode(cause);
if (code === "EACCES" || code === "EPERM")
throw new AnalysisAccessDeniedError(
ENVIRONMENT_EXPORT_OPERATION,
path,
code,
{
cause,
},
);
throw cause;
};
/** The directory may be absent or an empty directory; anything else is refused before any write. */
const checkDirectory = async (
directory: string,
): Promise<Result<undefined, AnalysisError>> => {
let entries: readonly string[];
try {
entries = await readdir(directory);
} catch (cause: unknown) {
const code = systemCode(cause);
if (code === "ENOENT") return ok(undefined);
if (code === "ENOTDIR")
return err(
inputError(
{
path: ["directory"],
reason: "invalid_value",
message: "The bundle directory path names a file.",
},
`${directory} exists and is not a directory.`,
),
);
return rethrowUnlessDenied(cause, directory);
}
if (entries.length > 0)
return err(
inputError(
{
path: ["directory"],
reason: "invalid_value",
message:
"The bundle directory must be absent or empty; nothing is overwritten.",
},
`${directory} is not empty (${String(entries.length)} entries); choose an absent or empty directory.`,
),
);
return ok(undefined);
};
/** The tarball must be a regular file; its bytes are copied, never inspected. */
const checkTarball = async (
const accessDeniedError = (
cause: unknown,
path: string,
): Promise<Result<EnvironmentPackageSource, AnalysisError>> => {
try {
const info = await stat(path);
if (!info.isFile())
return err(
inputError(
{
path: ["package_tarball"],
reason: "invalid_value",
message: "The package tarball must be a regular file.",
},
`${path} is not a regular file.`,
),
): AnalysisAccessDeniedError | undefined => {
const seen = new Set<Error>();
let current = cause;
while (current instanceof Error && !seen.has(current)) {
seen.add(current);
const code = systemCode(current);
if (code === "EACCES" || code === "EPERM") {
const error = new AnalysisAccessDeniedError(
ENVIRONMENT_EXPORT_OPERATION,
path,
code,
{
cause,
...(cause instanceof ArtifactReaderFailure &&
cause.cleanup !== undefined
? { cleanup: cause.cleanup }
: {}),
},
);
} catch (cause: unknown) {
const code = systemCode(cause);
if (code === "ENOENT" || code === "ENOTDIR" || code === "ENAMETOOLONG")
return err(
inputError(
{
path: ["package_tarball"],
reason: "invalid_value",
message: "The package tarball path must name an existing file.",
},
`${path} does not exist.`,
),
);
return rethrowUnlessDenied(cause, path);
bindArtifactFileCleanup(cause, error);
return error;
}
current = current.cause;
}
return ok({ kind: "tarball", path });
return undefined;
};
/** Host permission failures retain their typed cause; unexpected failures propagate. */
const rethrowUnlessDenied = (cause: unknown, path: string): never => {
throw accessDeniedError(cause, path) ?? cause;
};
/** A path that is not a readable regular file, including inline JSON text offered as one. */
@@ -208,17 +163,46 @@ const readInput = async (
role: "manifest" | "evidence",
path: string,
): Promise<Result<EnvironmentExportFile, AnalysisError>> => {
const resources = new ArtifactResourceScope();
let bytes: Buffer;
try {
// Admit the whole read against the opened file's own size, before materializing the
// exact bytes the bundle copies.
bytes = await readRegularFile(path, {
resources,
admit: (state) => {
const refused = headroomRefusal(path, state.size);
if (refused !== undefined) throw refused;
},
});
} catch (cause: unknown) {
if (cause instanceof RegularFileCleanupFailure) {
const failure = new ArtifactFileCleanupFailure(
cause,
AnalysisCleanupLease.forCloseOwner(cause.owner, path),
);
const cleanup = new AnalysisOutputError(
ENVIRONMENT_EXPORT_OPERATION,
`Could not confirm closing the ${role} input file.`,
{
cause: failure,
cleanup: failure.cleanup,
diagnostics: { input_path: path, input_role: role },
},
);
bindArtifactFileCleanup(failure, cleanup);
const primary =
cause.outcome.kind === "failed" ? cause.outcome.cause : undefined;
return err(
primary instanceof AnalysisError
? analysisErrorWithCleanupFailure(
primary,
cleanup,
ENVIRONMENT_EXPORT_OPERATION,
).bindCleanupDependencies([primary, cleanup])
: cleanup,
);
}
if (cause instanceof AnalysisResourceConstraintError) return err(cause);
if (cause instanceof RegularFileChangedError)
return err(
@@ -284,100 +268,71 @@ export const readEnvironmentExportInputs = async (
return ok({ manifest: manifest.value, evidence: evidence.value });
};
/** Captured content (the manifest and Evidence copies) stays readable by its owner only, whatever the umask. */
const CAPTURED_FILE_MODE = 0o600;
const writeNew = async (
root: string,
relative: string,
content: string | Uint8Array,
mode?: number,
): Promise<WrittenBundleFile> => {
const path = join(root, relative);
try {
await mkdir(dirname(path), { recursive: true });
await writeFile(path, content, {
flag: "wx",
...(mode === undefined ? {} : { mode }),
});
} catch (cause: unknown) {
rethrowUnlessDenied(cause, path);
}
return { path: relative, bytes: Buffer.byteLength(content) };
};
const copyNew = async (
root: string,
relative: string,
from: string,
): Promise<WrittenBundleFile> => {
const path = join(root, relative);
try {
await mkdir(dirname(path), { recursive: true });
await copyFile(from, path, constants.COPYFILE_EXCL);
return { path: relative, bytes: (await stat(path)).size };
} catch (cause: unknown) {
return rethrowUnlessDenied(cause, path);
}
};
const packageSourceValue = (source: EnvironmentPackageSource): JsonValue =>
source.kind === "registry"
? { kind: "registry", specifier: source.specifier }
: { kind: "tarball", path: source.path };
/** Write every planned file in plan order; manifest and evidence are the authenticated bytes unchanged. */
const writeBundle = async (
input: EnvironmentExportInput,
plan: EnvironmentBundlePlan,
): Promise<readonly WrittenBundleFile[]> => {
const written: WrittenBundleFile[] = [];
for (const file of plan.files)
written.push(await writeNew(input.directory, file.path, file.content));
written.push(
await writeNew(
input.directory,
plan.copies.manifest,
input.manifest.bytes,
CAPTURED_FILE_MODE,
),
);
written.push(
await writeNew(
input.directory,
plan.copies.evidence,
input.evidence.bytes,
CAPTURED_FILE_MODE,
),
);
if (plan.copies.tarball !== null && plan.packageSource.kind === "tarball")
written.push(
await copyNew(
input.directory,
plan.copies.tarball,
plan.packageSource.path,
),
);
return written;
};
/** Export the authenticated pair as an agent environment bundle folder. */
export const exportEnvironmentBundle = async (
input: EnvironmentExportInput,
): Promise<Result<EnvironmentExportResult, AnalysisError>> => {
const directory = await checkDirectory(input.directory);
if (!directory.ok) return directory;
const packageSource =
const packageSource: EnvironmentPackageSource =
input.packageTarball === undefined
? ok<EnvironmentPackageSource>(registryPackageSource())
: await checkTarball(input.packageTarball);
if (!packageSource.ok) return packageSource;
? registryPackageSource()
: { kind: "tarball", path: input.packageTarball };
const plan = planEnvironmentBundle(input.source, {
name: input.name,
packageSource: packageSource.value,
packageSource,
});
if (!plan.ok) return plan;
const files = await writeBundle(input, plan.value);
let files: readonly WrittenBundleFile[];
try {
files = await writeEnvironmentBundle(input, plan.value);
} catch (cause: unknown) {
if (cause instanceof AnalysisError) return err(cause);
const denied = accessDeniedError(cause, input.directory);
if (denied !== undefined) return err(denied);
if (cause instanceof RegularFileChangedError)
return err(
new AnalysisArtifactChangedError(
ENVIRONMENT_EXPORT_OPERATION,
cause.path,
cause.message,
{ cause },
),
);
if (cause instanceof ArtifactReaderFailure) {
if (cause.reason === "path" && cause.cleanup === undefined)
return err(
inputError(
{
path: ["directory"],
reason: "invalid_value",
message: cause.message,
},
cause.message,
),
);
const error = new AnalysisOutputError(
ENVIRONMENT_EXPORT_OPERATION,
cause.message,
{
cause,
...(cause.cleanup === undefined ? {} : { cleanup: cause.cleanup }),
},
);
bindArtifactFileCleanup(cause, error);
return err(error);
}
if (cause instanceof Error && systemCode(cause) !== undefined)
return err(
new AnalysisOutputError(ENVIRONMENT_EXPORT_OPERATION, cause.message, {
cause,
}),
);
throw cause;
}
return ok({
status: "exported",
directory: input.directory,
@@ -0,0 +1,65 @@
import { expect, it, vi } from "vitest";
import { writeEnvironmentServeFiles } from "../../../tests/fixtures/environmentServe.js";
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
import { ArtifactFileCleanupFailure } from "../../artifacts/ArtifactFileOperation.js";
import { OwnedFileHandle } from "../../filesystem/OwnedFileHandle.js";
import { readEnvironmentExportInputs } from "./environmentExport.js";
it.each([false, true])(
"retries a failed input close and retains persistent cleanup authority: %s",
async (persistent) => {
const root = await createTestTempDirectory("rea-environment-input-close-");
const files = await writeEnvironmentServeFiles(root);
const original = OwnedFileHandle.prototype.close;
let owner: OwnedFileHandle | undefined;
let attempts = 0;
let released = false;
const firstOwner = (value: OwnedFileHandle): OwnedFileHandle => {
owner ??= value;
return owner;
};
const close = vi
.spyOn(OwnedFileHandle.prototype, "close")
.mockImplementation(function (this: OwnedFileHandle) {
if (this === firstOwner(this)) {
attempts += 1;
if (!released && (persistent || attempts === 1))
return Promise.reject(
Object.assign(new Error("controlled input close failure"), {
code: "EIO",
}),
);
}
return original.call(this);
});
try {
const result = await readEnvironmentExportInputs(
files.manifestPath,
files.capturePath,
);
if (!persistent) {
expect(result.ok).toBe(true);
expect(attempts).toBe(2);
} else {
if (result.ok) throw new Error("Expected cleanup refusal");
expect(result.error).toMatchObject({
_tag: "AnalysisOutputError",
operation: "environment-export",
cleanupIncomplete: true,
});
const failure = result.error.cause;
expect(failure).toBeInstanceOf(ArtifactFileCleanupFailure);
if (!(failure instanceof ArtifactFileCleanupFailure))
throw new Error("Missing exact cleanup owner");
released = true;
await failure.cleanupLease.release();
expect(result.error.cleanupIncomplete).toBe(false);
}
} finally {
released = true;
close.mockRestore();
if (owner !== undefined) await original.call(owner);
}
},
);
+122 -63
View File
@@ -1,6 +1,7 @@
import { createHash } from "node:crypto";
import { constants } from "node:fs";
import {
chmod,
mkdir,
open,
readdir,
@@ -23,6 +24,7 @@ import { SafeOutputTreeCreationFailure } from "./SafeOutputTreeCreationFailure.j
import { hashOutputFile } from "./SafeOutputTreeReadback.js";
import { removeOwnedTree } from "./SafeOutputTreeCleanup.js";
import {
assertExactTreeEntries,
assertHandleIdentity,
assertFilePathIdentity,
assertPathIdentity,
@@ -66,6 +68,7 @@ export class SafeOutputTree {
readonly #openFiles = new Map<string, OwnedFileHandle>();
readonly #openHandles = new Set<OwnedFileHandle>();
#published = false;
#createdRoot = true;
#cleanup: SafeOutputCleanup = {
status: "not-required",
};
@@ -73,20 +76,23 @@ export class SafeOutputTree {
private constructor(
outputRoot: string,
private readonly platform: NodeJS.Platform,
private readonly selectedRoot: string,
) {
this.#outputRoot = outputRoot;
}
/**
* Exclusively create the absent destination as this operation's owned tree.
* Exclusively create the destination, or explicitly admit a caller-owned empty
* directory. Admission never grants authority to remove that existing root.
*
* POSIX mode bits and directory `fchmod`/`fsync` have no Windows equivalent:
* `mkdir` already applies the requested mode, so the redundant handle
* `chmod` is skipped there to avoid an `EPERM` on the directory descriptor.
* Windows inherits ACLs: mode bits do not establish owner-only access there.
* Directory `fchmod`/`fsync` are skipped on Windows because those operations
* have no equivalent on the directory descriptor.
*/
static async create(
outputRoot: string,
platform: NodeJS.Platform = process.platform,
options: { readonly acceptExistingEmpty?: boolean } = {},
): Promise<SafeOutputTree> {
if (!isAbsolute(outputRoot))
throw new ArtifactReaderFailure(
@@ -108,57 +114,31 @@ export class SafeOutputTree {
},
);
const canonicalOutput = join(parent, name);
let created = true;
await mkdir(canonicalOutput, { mode: 0o700 }).catch((cause: unknown) => {
if (isAlreadyExists(cause))
throw new ArtifactReaderFailure(
"path",
"Extraction output root already exists",
{ cause },
);
throw cause;
if (!isAlreadyExists(cause)) throw cause;
if (options.acceptExistingEmpty) {
created = false;
return;
}
throw new ArtifactReaderFailure(
"path",
"Extraction output root already exists",
{ cause },
);
});
const tree = new SafeOutputTree(canonicalOutput, platform);
const tree = new SafeOutputTree(canonicalOutput, platform, requested);
tree.#createdRoot = created;
try {
const rootIdentity = await readDirectoryIdentity(canonicalOutput);
tree.#rootIdentity = rootIdentity;
if (platform !== "win32") {
const stagingHandle = tree.#trackHandle(
await open(
canonicalOutput,
constants.O_RDONLY | constants.O_DIRECTORY,
),
if (!created && (await readdir(canonicalOutput)).length > 0)
throw new ArtifactReaderFailure(
"path",
"Output directory is not empty",
);
let setup:
| { readonly kind: "ready" }
| {
readonly kind: "failed";
readonly cause: unknown;
} = { kind: "ready" };
try {
await assertHandleIdentity(
stagingHandle.handle,
rootIdentity,
canonicalOutput,
);
await stagingHandle.handle.chmod(0o700);
} catch (cause: unknown) {
setup = { kind: "failed", cause };
}
try {
await tree.#closeHandle(stagingHandle);
} catch (closeCause: unknown) {
if (setup.kind === "failed")
throw ArtifactReaderFailure.withCleanup(
setup.cause,
ArtifactReaderFailure.cleanupObservation(
closeCause,
canonicalOutput,
),
);
throw closeCause;
}
if (setup.kind === "failed") throw setup.cause;
}
if (created)
await tree.#secureCreatedDirectory(canonicalOutput, rootIdentity);
await assertPathIdentity(canonicalOutput, rootIdentity);
return tree;
} catch (cause: unknown) {
@@ -223,6 +203,14 @@ export class SafeOutputTree {
);
await assertFilePathIdentity(destination, fileIdentity);
this.#ownedFiles.set(destination, fileIdentity);
if (this.platform !== "win32") {
await ownedHandle.handle.chmod(0o600);
if (((await ownedHandle.handle.stat()).mode & 0o777) !== 0o600)
throw new ArtifactReaderFailure(
"path",
"Could not establish private output file permissions",
);
}
const hash = createHash("sha256");
let bytes = 0;
await this.#assertLineage(lineage);
@@ -286,6 +274,7 @@ export class SafeOutputTree {
this.#assertWritable();
abortIfNeeded(signal);
await this.#assertOwnedDirectories();
await this.#assertExactEntries();
abortIfNeeded(signal);
// Windows has no directory fsync; file contents are already synced in write().
if (this.platform === "win32") {
@@ -319,6 +308,7 @@ export class SafeOutputTree {
);
await output.handle.sync();
await parent.handle.sync();
await this.#assertExactEntries();
abortIfNeeded(signal);
this.#published = true;
} catch (cause: unknown) {
@@ -366,7 +356,7 @@ export class SafeOutputTree {
/** Remove only this operation's unsealed tree and verify absence. */
async rollback(): Promise<SafeOutputCleanup> {
if (this.#published) {
await this.#closePendingHandles();
await this.closeHandles();
return structuredClone(this.#cleanup);
}
if (this.#cleanup.status === "complete")
@@ -378,26 +368,30 @@ export class SafeOutputTree {
let removalFailure: unknown;
try {
await this.#admitPendingFiles();
await this.#closePendingHandles();
await this.closeHandles();
await this.#assertOwnedDirectories();
await removeOwnedTree({
outputRoot: this.#outputRoot,
rootIdentity: this.#requireRootIdentity(),
directories: this.#nestedDirectories,
files: this.#ownedFiles,
preserveRoot: !this.#createdRoot,
});
if (!this.#createdRoot)
await assertPathIdentity(this.#outputRoot, this.#requireRootIdentity());
} catch (cause: unknown) {
removalFailure = cause;
}
const absent =
removalFailure === undefined && (await isAbsent(this.#outputRoot));
this.#cleanup = absent
const complete =
removalFailure === undefined &&
(!this.#createdRoot || (await isAbsent(this.#outputRoot)));
this.#cleanup = complete
? { status: "complete", residualPaths: [] }
: {
status: "incomplete",
residualPaths: [basename(this.#outputRoot)],
};
if (!absent)
if (!complete)
throw new ArtifactReaderFailure(
"integrity",
`Extraction output cleanup could not be verified${removalFailure === undefined ? "" : `: ${removalFailure instanceof Error ? removalFailure.message : String(removalFailure)}`}`,
@@ -415,6 +409,59 @@ export class SafeOutputTree {
return structuredClone(this.#cleanup);
}
async #secureCreatedDirectory(
path: string,
identity: DirectoryIdentity,
): Promise<void> {
if (this.platform === "win32") return;
const stagingHandle = this.#trackHandle(
await open(path, constants.O_RDONLY | constants.O_DIRECTORY).catch(
async (cause: unknown) => {
if (
!(cause instanceof Error) ||
!("code" in cause) ||
cause.code !== "EACCES"
)
throw cause;
// Owner-read can be masked at mkdir; repair only the proven created
// directory under the same portable pathname-race boundary.
await assertPathIdentity(path, identity);
await chmod(path, 0o700);
await assertPathIdentity(path, identity);
return open(path, constants.O_RDONLY | constants.O_DIRECTORY);
},
),
);
let setup:
| { readonly kind: "ready" }
| {
readonly kind: "failed";
readonly cause: unknown;
} = { kind: "ready" };
try {
await assertHandleIdentity(stagingHandle.handle, identity, path);
await stagingHandle.handle.chmod(0o700);
if (((await stagingHandle.handle.stat()).mode & 0o777) !== 0o700)
throw new ArtifactReaderFailure(
"path",
"Could not establish owner access to output directory",
);
} catch (cause: unknown) {
setup = { kind: "failed", cause };
}
try {
await this.#closeHandle(stagingHandle);
} catch (closeCause: unknown) {
if (setup.kind === "failed")
throw ArtifactReaderFailure.withCleanup(
setup.cause,
ArtifactReaderFailure.cleanupObservation(closeCause, path),
);
throw closeCause;
}
if (setup.kind === "failed") throw setup.cause;
}
async #prepareParent(
relativePath: string,
): Promise<readonly DirectoryLineageEntry[]> {
@@ -446,13 +493,14 @@ export class SafeOutputTree {
if (identity === undefined) throw replacedDirectory(current);
await assertPathIdentity(current, identity);
} else {
try {
const identity = await readDirectoryIdentity(current);
this.#nestedDirectories.set(current, identity);
} catch (cause: unknown) {
this.#unknownDirectories.add(current);
throw cause;
}
const identity = await readDirectoryIdentity(current).catch(
(cause: unknown) => {
this.#unknownDirectories.add(current);
throw cause;
},
);
this.#nestedDirectories.set(current, identity);
await this.#secureCreatedDirectory(current, identity);
}
await assertPathIdentity(parent.path, parent.identity);
const identity = this.#nestedDirectories.get(current);
@@ -474,6 +522,7 @@ export class SafeOutputTree {
async #assertOwnedDirectories(): Promise<void> {
const rootIdentity = this.#requireRootIdentity();
await assertPathIdentity(this.#outputRoot, rootIdentity);
await assertPathIdentity(this.selectedRoot, rootIdentity);
for (const [path, identity] of this.#nestedDirectories)
await assertPathIdentity(path, identity);
if (this.#unknownDirectories.size > 0)
@@ -483,6 +532,15 @@ export class SafeOutputTree {
);
}
async #assertExactEntries(): Promise<void> {
await assertExactTreeEntries(
this.#outputRoot,
this.#nestedDirectories,
this.#ownedFiles,
);
await this.#assertOwnedDirectories();
}
#requireRootIdentity(): DirectoryIdentity {
if (this.#rootIdentity === undefined)
throw new ArtifactReaderFailure(
@@ -516,7 +574,8 @@ export class SafeOutputTree {
this.#openHandles.delete(owner);
}
async #closePendingHandles(): Promise<void> {
/** Release descriptors without deleting an inspectable partial output tree. */
async closeHandles(): Promise<void> {
const failures: unknown[] = [];
for (const owner of this.#openHandles) {
try {
+2 -1
View File
@@ -16,6 +16,7 @@ export const removeOwnedTree = async (input: {
readonly rootIdentity: DirectoryIdentity;
readonly directories: Map<string, DirectoryIdentity>;
readonly files: Map<string, FileIdentity>;
readonly preserveRoot?: boolean;
}): Promise<void> => {
const { outputRoot, rootIdentity, directories, files } = input;
const filesByDirectory = new Map<string, [string, FileIdentity][]>();
@@ -50,7 +51,7 @@ export const removeOwnedTree = async (input: {
);
await assertPathIdentity(outputRoot, rootIdentity);
await requireEmpty(outputRoot);
await rmdir(outputRoot);
if (!input.preserveRoot) await rmdir(outputRoot);
};
const removeFiles = async (
+42 -1
View File
@@ -1,5 +1,6 @@
import type { BigIntStats } from "node:fs";
import { lstat } from "node:fs/promises";
import { lstat, readdir } from "node:fs/promises";
import { basename, dirname } from "node:path";
import type { FileHandle } from "node:fs/promises";
import { ArtifactReaderFailure } from "./ArtifactReader.js";
@@ -7,6 +8,46 @@ import { ArtifactReaderFailure } from "./ArtifactReader.js";
export type DirectoryIdentity = Pick<BigIntStats, "dev" | "ino">;
export type FileIdentity = DirectoryIdentity;
/** Reject unowned, missing or replaced entries before an output tree is published. */
export const assertExactTreeEntries = async (
root: string,
directories: ReadonlyMap<string, DirectoryIdentity>,
files: ReadonlyMap<string, FileIdentity>,
): Promise<void> => {
const expected = new Map<string, Set<string>>();
expected.set(root, new Set());
for (const directory of directories.keys())
expected.set(directory, new Set());
for (const path of directories.keys())
expected.get(dirname(path))?.add(basename(path));
for (const path of files.keys())
expected.get(dirname(path))?.add(basename(path));
for (const [directory, names] of expected) {
const entries = await readdir(directory);
const remaining = new Map<string, number>();
for (const name of names) {
const key = name.normalize("NFC");
remaining.set(key, (remaining.get(key) ?? 0) + 1);
}
let unexpected = entries.length !== names.size;
for (const entry of entries) {
const key = entry.normalize("NFC");
const count = remaining.get(key) ?? 0;
if (count === 0) unexpected = true;
else remaining.set(key, count - 1);
}
// HFS+ can return an equivalent decomposed spelling. Counts avoid merging
// distinct Linux names; the original paths must still resolve to owned IDs.
if (unexpected)
throw new ArtifactReaderFailure(
"path",
`Unowned or missing output entries: ${directory}`,
);
}
for (const [path, identity] of files)
await assertFilePathIdentity(path, identity);
};
export const readDirectoryIdentity = async (
path: string,
): Promise<DirectoryIdentity> => {
+3 -3
View File
@@ -1,4 +1,4 @@
import { resolve } from "node:path";
import { absoluteCallerPath } from "../filesystem/callerPath.js";
import type { EnvironmentExportResult } from "../application/environment/environmentExport.js";
import { z } from "incur";
import type { Logger } from "pino";
@@ -200,12 +200,12 @@ const exportEnvironmentCommand = async (
const exported = await exportEnvironmentBundle({
...inputs.value,
source: source.value,
directory: resolve(input.directory),
directory: absoluteCallerPath(input.directory),
name: input.name,
packageTarball:
input.packageTarball === undefined
? undefined
: resolve(input.packageTarball),
: absoluteCallerPath(input.packageTarball),
});
return exported.ok
? exportDocument(exported.value)
+123 -1
View File
@@ -1,4 +1,11 @@
import { mkdir, readdir, readFile, writeFile } from "node:fs/promises";
import {
mkdir,
readdir,
readFile,
stat,
symlink,
writeFile,
} from "node:fs/promises";
import { join } from "node:path";
import { describe, expect } from "vitest";
@@ -14,6 +21,43 @@ import {
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { cliTest } from "../../support/cli/cliFixture.js";
cliTest(
"returns a tagged JSON error when a read input descriptor cannot be closed",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-export-input-close-cli-");
const files = await writeEnvironmentServeFiles(root);
const preload = join(root, "fail-close.mjs");
await writeFile(
preload,
`import fs from 'node:fs';\nimport { syncBuiltinESMExports } from 'node:module';\nconst original = fs.promises.open;\nfs.promises.open = async (...args) => {\n const handle = await original(...args);\n if (args[0] === ${JSON.stringify(files.manifestPath)}) handle.close = async () => { throw Object.assign(new Error('controlled persistent close failure'), { code: 'EIO' }); };\n return handle;\n};\nsyncBuiltinESMExports();\n`,
);
const directory = join(root, "bundle");
const result = await cli.run({
arguments: [
"environment-export",
files.manifestPath,
"--evidence",
files.capturePath,
directory,
"--json",
],
environment: {
NODE_OPTIONS: `--import=${preload} --max-old-space-size=768`,
NODE_DISABLE_COMPILE_CACHE: "1",
},
timeoutMs: 30_000,
});
expect(result.exitCode).toBe(1);
expect(result.json).toMatchObject({
details: { operation: "environment-export" },
});
expect(JSON.stringify(result.json)).toContain("cleanup");
await expect(readFile(join(directory, "README.md"))).rejects.toMatchObject({
code: "ENOENT",
});
},
);
const BUNDLE_FILES = [
"README.md",
"artifacts/check_grafana-example/verify.py",
@@ -160,3 +204,81 @@ describe("rea environment-export", () => {
},
);
});
cliTest.skipIf(process.platform === "win32")(
"produces readable private files independently of an owner-read-masking umask",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-export-umask-");
const files = await writeEnvironmentServeFiles(root);
const preload = join(root, "umask.mjs");
await writeFile(preload, "process.umask(0o400);\n");
const directory = join(root, "bundle");
const result = await cli.run({
arguments: [
"environment-export",
files.manifestPath,
"--evidence",
files.capturePath,
directory,
"--json",
],
environment: {
NODE_OPTIONS: `--import=${preload} --max-old-space-size=768`,
NODE_DISABLE_COMPILE_CACHE: "1",
},
timeoutMs: 30_000,
});
expect(result.exitCode, result.stderr).toBe(0);
for (const relative of BUNDLE_FILES) {
expect((await stat(join(directory, relative))).mode & 0o777).toBe(0o600);
expect(
(await readFile(join(directory, relative))).length,
).toBeGreaterThan(0);
}
expect(
await readFile(join(directory, "envs/grafana-example/evidence.json")),
).toEqual(await readFile(files.capturePath));
},
);
cliTest.skipIf(process.platform === "win32")(
"preserves the filesystem meaning of symlink/.. for the destination and tarball",
async ({ cli }) => {
const root = await createTestTempDirectory("rea-export-selected-path-");
const physical = join(root, "physical");
const nested = join(physical, "nested");
await mkdir(nested, { recursive: true });
await symlink(nested, join(root, "link"), "dir");
const files = await writeEnvironmentServeFiles(root);
const bytes = Buffer.from("selected packed package");
await writeFile(join(physical, "package.tgz"), bytes);
const directory = `${root}/link/../bundle`;
const packagePath = `${root}/link/../package.tgz`;
const result = await cli.run({
arguments: [
"environment-export",
files.manifestPath,
"--evidence",
files.capturePath,
directory,
"--package-tarball",
packagePath,
"--json",
],
timeoutMs: 30_000,
});
expect(result.exitCode, result.stderr).toBe(0);
expect(result.json).toMatchObject({
directory,
package_source: { kind: "tarball", path: packagePath },
});
expect(
await readFile(
join(physical, "bundle/envs/grafana-example/rea-agents.tgz"),
),
).toEqual(bytes);
await expect(
readFile(join(root, "bundle/README.md")),
).rejects.toMatchObject({ code: "ENOENT" });
},
);
@@ -810,6 +810,10 @@ it.each([
"src/application/OwnedJsonInput.ts",
{ code: true, agentenv: true, runtime: true, package: false },
],
[
"src/artifacts/SafeOutputTree.ts",
{ code: true, agentenv: true, runtime: true },
],
[
"src/cliJsonInput.ts",
{ code: true, agentenv: true, runtime: true, package: false },
@@ -24,6 +24,21 @@ import { SafeOutputTreeCreationFailure } from "../../../src/artifacts/SafeOutput
import { ArtifactReaderFailure } from "../../../src/artifacts/ArtifactReader.js";
describe("safe artifact output cleanup", () => {
it("adopts an empty caller directory without taking authority to remove it", async () => {
const parent = await createTestTempDirectory("rea-admitted-output-");
const output = join(parent, "output");
await mkdir(output);
const tree = await SafeOutputTree.create(output, process.platform, {
acceptExistingEmpty: true,
});
await tree.write("file", Readable.from([Buffer.from("owned")]), {
bytes: 5,
sha256: createHash("sha256").update("owned").digest("hex"),
});
await tree.rollback();
expect(await readdir(output)).toEqual([]);
await tree.closeHandles();
});
it.skipIf(process.platform !== "linux")(
"closes the parent descriptor when the output disappears before commit",
async () => {
@@ -0,0 +1,69 @@
import { createHash } from "node:crypto";
import { readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { Readable } from "node:stream";
import { expect, it, vi } from "vitest";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
import { SafeOutputTree } from "../../../src/artifacts/SafeOutputTree.js";
const boundary = vi.hoisted(() => ({ decomposed: false }));
vi.mock("node:fs/promises", async (original) => {
const actual = await original<typeof import("node:fs/promises")>();
return {
...actual,
readdir: async (...args: unknown[]) => {
const entries: unknown = await Reflect.apply(
actual.readdir,
undefined,
args,
);
return boundary.decomposed &&
Array.isArray(entries) &&
entries.every((name): name is string => typeof name === "string")
? entries.map((name) => name.normalize("NFD"))
: entries;
},
};
});
const writeAccent = async (tree: SafeOutputTree) => {
const bytes = Buffer.from("owned");
await tree.write("café.txt", Readable.from([bytes]), {
bytes: bytes.length,
sha256: createHash("sha256").update(bytes).digest("hex"),
});
};
it("accepts a decomposed directory listing while verifying the original owned file identity", async () => {
const root = await createTestTempDirectory("rea-output-unicode-");
const tree = await SafeOutputTree.create(join(root, "output"));
await writeAccent(tree);
boundary.decomposed = true;
try {
await tree.commit();
expect(tree.published).toBe(true);
} finally {
boundary.decomposed = false;
await tree.closeHandles();
}
});
it.skipIf(process.platform !== "linux")(
"does not collapse an additional canonically equivalent foreign filename",
async () => {
const root = await createTestTempDirectory("rea-output-unicode-foreign-");
const output = join(root, "output");
const tree = await SafeOutputTree.create(output);
await writeAccent(tree);
const foreign = join(output, "cafe\u0301.txt");
await writeFile(foreign, "foreign", { flag: "wx" });
try {
await expect(tree.commit()).rejects.toThrow("Unowned or missing");
expect(await readFile(foreign, "utf8")).toBe("foreign");
} finally {
await tree.closeHandles();
}
},
);
@@ -1,5 +1,5 @@
import { spawnSync } from "node:child_process";
import { readFile, writeFile } from "node:fs/promises";
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
@@ -7,6 +7,66 @@ import { expect, it } from "vitest";
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
it.skipIf(process.platform === "win32")(
"packs through the selected npm entrypoint even when PATH contains another npm",
async () => {
const root = await createTestTempDirectory("rea-agentenv-npm-selection-");
const bin = join(root, "bin");
await mkdir(bin);
const selected = join(root, "selected-npm.mjs");
const selectedMarker = join(root, "selected-called");
const pathMarker = join(root, "path-called");
const runner = join(root, "agent-env.mjs");
const docker = join(root, "docker.mjs");
await writeFile(
selected,
`import { writeFileSync } from 'node:fs';\nwriteFileSync(${JSON.stringify(selectedMarker)}, process.argv[2]);\nprocess.exit(23);\n`,
);
await writeFile(
join(bin, "npm"),
`#!/usr/bin/env node\nrequire('node:fs').writeFileSync(${JSON.stringify(pathMarker)}, 'called');\nprocess.exit(66);\n`,
{ mode: 0o755 },
);
await writeFile(
runner,
"#!/usr/bin/env node\nprocess.stdout.write('agent-env 0.9.1334\\n');\n",
{ mode: 0o755 },
);
await writeFile(docker, "#!/usr/bin/env node\nprocess.exit(0);\n", {
mode: 0o755,
});
const result = spawnSync(
process.execPath,
[
fileURLToPath(
new URL(
"../../../scripts/verify-agentenv-environment.mjs",
import.meta.url,
),
),
],
{
env: {
...process.env,
PATH: `${bin}:${process.env.PATH ?? ""}`,
npm_execpath: selected,
REA_AGENT_ENV_COMMAND: runner,
REA_DOCKER_COMMAND: docker,
},
encoding: "utf8",
timeout: 30_000,
maxBuffer: 1024 * 1024,
},
);
expect(result.error).toBeUndefined();
expect(result.status).toBe(1);
expect(await readFile(selectedMarker, "utf8")).toBe("pack");
await expect(readFile(pathMarker)).rejects.toMatchObject({
code: "ENOENT",
});
},
);
// The local framework and these executable shebang fixtures target POSIX hosts.
it.skipIf(process.platform === "win32").each([
["agent-env 0.9.9999", false],