mirror of
https://github.com/morluto/rea
synced 2026-10-25 22:08:11 -04:00
refactor(verification): separate readiness evidence scopes
Move the synthetic evaluator and its unchanged semantic tests out of the shipped runtime. Retain actual CLI and MCP Evidence separately from the synthetic nine-stage baseline in a versioned verifier artifact, execute its reported conformance checks, and assign the verifier to its CI lane. Validation: evaluator/state/classifier source suites, compiled CLI/MCP views, verify:readiness, static checks, build and docs passed. Readiness observed coverage is static analysis; fixture runtime behavior and reconstruction closure were not executed.
This commit is contained in:
@@ -789,7 +789,7 @@ jobs:
|
||||
id: npm-cache
|
||||
uses: ./.github/actions/npm-cache
|
||||
- run: npm ci
|
||||
- name: Run the public CLI/MCP reconstruction journey
|
||||
- name: Verify static CLI/MCP journeys and synthetic report conformance
|
||||
env:
|
||||
REA_READINESS_REPORT_PATH: readiness-report.json
|
||||
run: |
|
||||
|
||||
@@ -1,19 +1,25 @@
|
||||
# Reconstruction readiness conformance
|
||||
# Reconstruction readiness verification
|
||||
|
||||
Run the source-owned native, JavaScript CLI, Electron, Evidence, comparison,
|
||||
obligation, replay, and cleanup conformance verifier from a clean checkout:
|
||||
Run the public static-analysis journeys and synthetic report-evaluator checks
|
||||
from a clean checkout:
|
||||
|
||||
```sh
|
||||
npm ci
|
||||
npm run verify:readiness
|
||||
```
|
||||
|
||||
The command builds the redistributable native conformance fixtures, inventories
|
||||
the native executable through the public CLI, analyzes the generated JavaScript
|
||||
CLI and Electron fixtures through both CLI and MCP, evaluates an internal
|
||||
nine-stage conformance report, and proves stale/tampered replay rejection. The
|
||||
report evaluator is part of the repository verifier, not a public REA tool. It
|
||||
prints one compact JSON summary.
|
||||
The command builds the redistributable native conformance fixtures, inspects
|
||||
the native executable through the public CLI, and analyzes the JavaScript CLI
|
||||
and Electron fixture sources through both CLI and MCP. It checks matching
|
||||
CLI/MCP Evidence identities for those two application analyses. The fixture
|
||||
applications are not executed, and native MCP parity is not exercised.
|
||||
|
||||
Separately, the command evaluates a wholly synthetic nine-stage input and
|
||||
checks evaluator determinism, Evidence tampering rejection, and stale-source
|
||||
digest rejection. These checks concern report evaluation; they do not establish
|
||||
application runtime behavior, authority recovery, cancellation cleanup,
|
||||
behavioral equivalence, or reconstruction closure for the analyzed fixtures.
|
||||
The command prints one compact JSON summary.
|
||||
|
||||
Set `REA_READINESS_REPORT_PATH` to retain the complete machine-readable report:
|
||||
|
||||
@@ -24,15 +30,37 @@ REA_READINESS_REPORT_PATH=readiness-report.json npm run verify:readiness
|
||||
CI runs this command on Linux, publishes the compact summary, and retains the
|
||||
full report as the `reconstruction-readiness` artifact.
|
||||
|
||||
## Project conformance contract
|
||||
## Report scope and ownership
|
||||
|
||||
The verifier runs as `npm run verify:readiness`; it is not exposed as a public
|
||||
CLI command or MCP tool. The synthetic report input and evaluator exist to
|
||||
check project-owned conformance fixtures and fail-closed report behavior.
|
||||
CLI command or MCP tool. Its report separates two kinds of verification:
|
||||
|
||||
## Required stages
|
||||
- `observed_journeys` retains the actual static-analysis Evidence and the
|
||||
observed CLI/MCP identity checks, with their coverage limits.
|
||||
- `evaluator_conformance` retains the synthetic baseline report and the
|
||||
evaluator checks. Its nine-stage `pass` describes acceptance of supplied
|
||||
synthetic observations.
|
||||
|
||||
| Stage | Required proof |
|
||||
The verifier's overall success means its listed checks passed. It is not an
|
||||
end-to-end reconstruction-readiness verdict. Actual analysis Evidence is not
|
||||
grafted into the synthetic report or used to close its checks.
|
||||
|
||||
The retained artifact now uses `schema: "ReconstructionReadinessVerificationReport"`
|
||||
and `schema_version: 1`. The previous unversioned artifact was the nine-stage
|
||||
evaluator report itself. Consumers should read the two explicit sections; the
|
||||
synthetic nine-stage report now lives at `evaluator_conformance.baseline_report`.
|
||||
Observed CLI and MCP Evidence are retained in separate bundles under
|
||||
`observed_journeys.evidence_bundles`, preserving each surface's records.
|
||||
|
||||
The evaluator, schemas, synthetic input, and semantic tests live under
|
||||
`tests/conformance/readiness/report/`. They are development verification code,
|
||||
excluded from the compiled runtime package. The evaluator authenticates Evidence
|
||||
commitments and checks supplied observations for consistency; it does not
|
||||
independently observe the events described by those observations.
|
||||
|
||||
## Synthetic evaluator stages
|
||||
|
||||
| Stage | Required supplied observations |
|
||||
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
|
||||
| `discover-classify` | target-free startup, artifact inventory, compatible workflow selection, and exact visible limits |
|
||||
| `diagnose-environment` | broken runtime separated from analysis failure, version skew detected, and revision changed after recovery |
|
||||
@@ -44,13 +72,13 @@ check project-owned conformance fixtures and fail-closed report behavior.
|
||||
| `verify-reconstruction-closure` | incomplete reconstruction detected, incremental verifier closure, and authority-delegating candidates rejected |
|
||||
| `export-replay` | CLI/MCP parity, process/structured failure parity, deterministic replay, tamper rejection, and stale-input rejection |
|
||||
|
||||
Every stage and required check must occur exactly once and link to Evidence in
|
||||
the embedded Evidence bundle. `fail`, `unknown`, `unsupported`, `truncated`,
|
||||
and `skipped` never aggregate to `pass`.
|
||||
Within the synthetic input, every stage and required check must occur exactly
|
||||
once and link to Evidence in its embedded bundle. `fail`, `unknown`,
|
||||
`unsupported`, `truncated`, and `skipped` never aggregate to `pass`.
|
||||
|
||||
## Fail-closed rules
|
||||
|
||||
The evaluator rejects or withholds readiness when:
|
||||
The evaluator rejects or withholds a passing synthetic report when:
|
||||
|
||||
- a bounded capability omits its effective numeric limits;
|
||||
- a fixture lacks exactly one compatible selected workflow;
|
||||
@@ -63,8 +91,8 @@ The evaluator rejects or withholds readiness when:
|
||||
- contradiction Evidence or its affected comparison is missing;
|
||||
- the reconstruction obligation ledger is not `ready`, has no required
|
||||
obligation, or retains an unowned/unverified required obligation;
|
||||
- a process exceeds its declared target/lifecycle, cancellation loses diagnostics, or owned
|
||||
resources remain;
|
||||
- a denied or cancelled grant reports a process launch, a cancelled cleanup
|
||||
observation omits diagnostic Evidence, or owned resources remain;
|
||||
- replay is nondeterministic, tampering/staleness is not detected, or the
|
||||
expected source digest differs.
|
||||
|
||||
@@ -95,19 +123,27 @@ Differing transport status is checked on each observation separately.
|
||||
|
||||
## Fixtures
|
||||
|
||||
The verifier uses only source-owned, redistributable fixtures:
|
||||
The static-analysis journeys use source-owned, redistributable fixtures:
|
||||
|
||||
- `tests/conformance` native executables with stable symbols and deliberate
|
||||
version divergence;
|
||||
- `build/conformance/c`, built from `tests/conformance/c/fixture.c`, with stable
|
||||
symbols and strings;
|
||||
- `tests/conformance/readiness/javascript-cli`, which exposes prompts,
|
||||
alternate-screen behavior, configuration precedence, child process,
|
||||
persistence, loopback HTTP, and signal teardown to static reconstruction;
|
||||
- `tests/conformance/readiness/electron`, which exposes main, preload,
|
||||
renderer, utility, IPC, renderer restart, blocked shell effect, and deep-link
|
||||
surfaces;
|
||||
- the shared incomplete reconstruction and broken-runtime observations in
|
||||
`RECONSTRUCTION_READINESS_EXAMPLE`.
|
||||
surfaces to static analysis.
|
||||
|
||||
The synthetic example is a public typed input, not a global equivalence claim.
|
||||
It demonstrates that every deliberate fault remains represented in the final
|
||||
report and that only comparable Evidence can close the associated obligation.
|
||||
The separate `RECONSTRUCTION_READINESS_EXAMPLE` supplies synthetic incomplete
|
||||
reconstruction, broken-runtime, authority, comparison, cleanup, and closure
|
||||
observations. Its semantic tests preserve malformed-Evidence rejection,
|
||||
fail-closed stage results, contradiction handling, repeated CLI/MCP outcomes,
|
||||
and closure-history consistency. Run those source tests independently:
|
||||
|
||||
```sh
|
||||
npm run test:local -- tests/conformance/readiness/report/reconstructionReadiness.test.ts
|
||||
```
|
||||
|
||||
Passing evaluator tests does not establish real-provider or fixture runtime
|
||||
behavior. Use the applicable [real-toolchain lanes](testing.md#real-toolchain-verification-lanes)
|
||||
for those claims.
|
||||
|
||||
+10
-6
@@ -549,12 +549,16 @@ uses the same ownership rule to find retained fixtures under a private temporary
|
||||
root. Third-party caches remain outside that ownership rule.
|
||||
|
||||
Real Hopper, Ghidra, IDA, browser, package, and managed-code claims belong to their
|
||||
explicit `npm run verify:*` lanes. The reconstruction-readiness lane also
|
||||
checks deterministic rerun, tamper, and stale-input handling; those checks do
|
||||
not execute extracted JavaScript modules. When application runtime behavior is
|
||||
needed, exercise the actual target through browser, Electron, or process
|
||||
capture. Real model trials are manual; Vitest covers deterministic evaluator
|
||||
logic.
|
||||
explicit `npm run verify:*` lanes. The reconstruction-readiness lane separates
|
||||
observed native CLI inspection and JavaScript/Electron static CLI/MCP analysis
|
||||
from synthetic report-evaluator conformance. Its determinism, tamper, and
|
||||
stale-source checks concern the synthetic evaluator; they do not execute fixture
|
||||
applications or verify runtime authority, cleanup, comparison, or reconstruction
|
||||
closure. The evaluator and its semantic tests belong to
|
||||
`tests/conformance/readiness/report/`, outside the shipped runtime. When
|
||||
application runtime behavior is needed, exercise the actual target through
|
||||
browser, Electron, or process capture. Real model trials are manual; Vitest
|
||||
covers deterministic evaluator logic.
|
||||
|
||||
`verify:managed` runs the portable PE byte-fixture conformance entrypoint under
|
||||
`scripts/verify/managed/`, with its byte builder under
|
||||
|
||||
@@ -28,7 +28,6 @@ const FIXTURE_SOURCES = [
|
||||
"tests/conformance/swift/**",
|
||||
"tests/conformance/versions/**",
|
||||
];
|
||||
const READINESS_SOURCES = ["tests/conformance/readiness/**"];
|
||||
// These reporters affect CI output only; they do not select a host or provider
|
||||
// verification lane when changed on their own.
|
||||
const PORTABLE_CI_REPORTERS = new Set([
|
||||
@@ -281,7 +280,6 @@ if (tier === "reused") {
|
||||
selected.fixtures = true;
|
||||
selected.readiness = true;
|
||||
}
|
||||
if (matchesAny(file, READINESS_SOURCES)) selected.readiness = true;
|
||||
if (file.startsWith("tests/")) continue;
|
||||
if (
|
||||
/^(src\/process\/|src\/filesystem\/|src\/contracts\/tool|src\/server\/createServer|src\/cli\.ts|scripts\/ci\/)/.test(
|
||||
|
||||
@@ -2,6 +2,10 @@
|
||||
// Shared build inputs are classified once in plan.mjs rather than repeated in
|
||||
// every workflow.
|
||||
export const SCOPES = {
|
||||
readiness: [
|
||||
"tests/conformance/readiness/**",
|
||||
"scripts/verify-reconstruction-readiness.mjs",
|
||||
],
|
||||
reverse_tools: [
|
||||
".github/workflows/mutation.yml",
|
||||
"src/application/reverse/**",
|
||||
|
||||
@@ -6,14 +6,32 @@ import { promisify } from "node:util";
|
||||
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { tsImport } from "tsx/esm/api";
|
||||
|
||||
import { toolContract } from "../dist/contracts/toolContracts.js";
|
||||
import { RECONSTRUCTION_READINESS_EXAMPLE } from "../dist/contracts/reconstructionReadinessExample.js";
|
||||
import { parseEvidence } from "../dist/domain/evidence.js";
|
||||
import { createEvidenceBundle } from "../dist/domain/evidenceBundle.js";
|
||||
import { createReconstructionReadinessReport } from "../dist/domain/reconstructionReadiness.js";
|
||||
import { reconstructionReadinessReportSchema } from "../dist/domain/reconstructionReadinessSchemas.js";
|
||||
import { PACKAGE_METADATA } from "../dist/generatedPackageMetadata.js";
|
||||
import { closeVerifierResources } from "./lib/verifier-cleanup.mjs";
|
||||
|
||||
const [
|
||||
{ RECONSTRUCTION_READINESS_EXAMPLE },
|
||||
{ createReconstructionReadinessReport },
|
||||
{ reconstructionReadinessReportSchema },
|
||||
] = await Promise.all(
|
||||
[
|
||||
"reconstructionReadinessExample",
|
||||
"reconstructionReadiness",
|
||||
"reconstructionReadinessSchemas",
|
||||
].map((name) =>
|
||||
tsImport(
|
||||
new URL(
|
||||
`../tests/conformance/readiness/report/${name}.ts`,
|
||||
import.meta.url,
|
||||
).href,
|
||||
import.meta.url,
|
||||
),
|
||||
),
|
||||
);
|
||||
|
||||
const execute = promisify(execFile);
|
||||
const root = process.cwd();
|
||||
@@ -50,6 +68,8 @@ const client = new Client({
|
||||
capabilities: { elicitation: {} },
|
||||
});
|
||||
|
||||
let verificationReport;
|
||||
let primaryError;
|
||||
try {
|
||||
await client.connect(transport);
|
||||
const catalog = await client.listTools();
|
||||
@@ -61,6 +81,7 @@ try {
|
||||
{ name: "binary_session", arguments: {} },
|
||||
{ toolDefinition: sessionTool },
|
||||
);
|
||||
assert.notEqual(sessionReply.isError, true);
|
||||
const sessionStatus = toolContract("binary_session").outputSchema.parse(
|
||||
sessionReply.structuredContent,
|
||||
).result;
|
||||
@@ -77,76 +98,93 @@ try {
|
||||
"CLI/MCP Electron analysis Evidence identity drifted",
|
||||
);
|
||||
|
||||
const input = structuredClone(RECONSTRUCTION_READINESS_EXAMPLE);
|
||||
input.identity.cli_version = PACKAGE_METADATA.version;
|
||||
input.identity.server_version = PACKAGE_METADATA.version;
|
||||
input.identity.catalog_digest =
|
||||
sessionStatus.server_identity.catalog.digests.combined_sha256;
|
||||
const records = [
|
||||
...input.evidence_bundle.records,
|
||||
nativeEvidence,
|
||||
javascriptCliEvidence,
|
||||
electronCliEvidence,
|
||||
].map(parseEvidence);
|
||||
input.evidence_bundle = createEvidenceBundle(records);
|
||||
setFixtureDigest(input, "native", nativeEvidence);
|
||||
setFixtureDigest(input, "javascript-cli", javascriptCliEvidence);
|
||||
setFixtureDigest(input, "electron", electronCliEvidence);
|
||||
linkCheck(input, "discover-classify", "artifact-inventory", nativeEvidence);
|
||||
linkCheck(input, "static-analysis", "native-routed", nativeEvidence);
|
||||
linkCheck(
|
||||
input,
|
||||
"static-analysis",
|
||||
"javascript-routed",
|
||||
javascriptCliEvidence,
|
||||
);
|
||||
linkCheck(
|
||||
input,
|
||||
"reactive-scenarios",
|
||||
"electron-correlated",
|
||||
electronCliEvidence,
|
||||
);
|
||||
|
||||
const readinessReport = reconstructionReadinessReportSchema.parse(
|
||||
createReconstructionReadinessReport(input),
|
||||
);
|
||||
assert.equal(readinessReport.status, "pass");
|
||||
if (process.env.REA_READINESS_REPORT_PATH !== undefined)
|
||||
await writeFile(
|
||||
resolve(root, process.env.REA_READINESS_REPORT_PATH),
|
||||
`${JSON.stringify(readinessReport, null, 2)}\n`,
|
||||
);
|
||||
|
||||
const tampered = structuredClone(input);
|
||||
tampered.replay.expected_source_digest = readinessReport.source_digest;
|
||||
tampered.identity.server_version = "tampered";
|
||||
const tamperedReport = reconstructionReadinessReportSchema.parse(
|
||||
createReconstructionReadinessReport(tampered),
|
||||
);
|
||||
assert.equal(tamperedReport.status, "fail");
|
||||
assert.ok(
|
||||
tamperedReport.findings.some(
|
||||
({ code }) => code === "replay-source-digest-mismatch",
|
||||
),
|
||||
);
|
||||
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({
|
||||
status: readinessReport.status,
|
||||
report_id: readinessReport.report_id,
|
||||
report_digest: readinessReport.report_digest,
|
||||
stages: readinessReport.summary.stages,
|
||||
findings: readinessReport.summary.findings,
|
||||
native_evidence_id: nativeEvidence.evidence_id,
|
||||
javascript_evidence_id: javascriptCliEvidence.evidence_id,
|
||||
electron_evidence_id: electronCliEvidence.evidence_id,
|
||||
provider: "local-stdio",
|
||||
})}\n`,
|
||||
);
|
||||
verificationReport = {
|
||||
schema: "ReconstructionReadinessVerificationReport",
|
||||
schema_version: 1,
|
||||
status: "pass",
|
||||
observed_journeys: {
|
||||
evidence_kind: "observed",
|
||||
scope: "static-cli-mcp",
|
||||
status: "pass",
|
||||
server_identity: sessionStatus.server_identity,
|
||||
evidence_bundles: {
|
||||
cli: createEvidenceBundle([
|
||||
nativeEvidence,
|
||||
javascriptCliEvidence,
|
||||
electronCliEvidence,
|
||||
]),
|
||||
mcp: createEvidenceBundle([javascriptMcp, electronMcp]),
|
||||
},
|
||||
journeys: [
|
||||
{
|
||||
journey_id: "native-artifact-cli",
|
||||
operation: "inspect-artifact",
|
||||
surfaces: ["cli"],
|
||||
status: "pass",
|
||||
evidence_ids: [nativeEvidence.evidence_id],
|
||||
checks: ["cli-completed", "evidence-authenticated"],
|
||||
},
|
||||
...[
|
||||
["javascript-static-cli-mcp", javascriptCliEvidence],
|
||||
["electron-static-cli-mcp", electronCliEvidence],
|
||||
].map(([journeyId, evidence]) => ({
|
||||
journey_id: journeyId,
|
||||
operation: "analyze-javascript-application",
|
||||
surfaces: ["cli", "mcp"],
|
||||
status: "pass",
|
||||
evidence_ids: [evidence.evidence_id],
|
||||
checks: [
|
||||
"cli-completed",
|
||||
"mcp-completed",
|
||||
"evidence-authenticated",
|
||||
"evidence-identity-parity",
|
||||
],
|
||||
})),
|
||||
],
|
||||
limitations: [
|
||||
"Native MCP parity was not exercised.",
|
||||
"Fixture application runtime behavior was not executed.",
|
||||
"Authority recovery, cancellation cleanup, behavioral comparison, and reconstruction closure were not exercised.",
|
||||
],
|
||||
},
|
||||
evaluator_conformance: verifySyntheticEvaluator(),
|
||||
};
|
||||
} catch (cause) {
|
||||
primaryError = cause;
|
||||
throw cause;
|
||||
} finally {
|
||||
await client.close();
|
||||
await closeVerifierResources(
|
||||
[() => client.close(), () => transport.close()],
|
||||
primaryError,
|
||||
);
|
||||
}
|
||||
|
||||
if (process.env.REA_READINESS_REPORT_PATH !== undefined)
|
||||
await writeFile(
|
||||
resolve(root, process.env.REA_READINESS_REPORT_PATH),
|
||||
`${JSON.stringify(verificationReport, null, 2)}\n`,
|
||||
);
|
||||
process.stdout.write(
|
||||
`${JSON.stringify({
|
||||
schema: verificationReport.schema,
|
||||
schema_version: verificationReport.schema_version,
|
||||
status: verificationReport.status,
|
||||
observed_journeys: {
|
||||
status: verificationReport.observed_journeys.status,
|
||||
scope: verificationReport.observed_journeys.scope,
|
||||
journeys: verificationReport.observed_journeys.journeys,
|
||||
limitations: verificationReport.observed_journeys.limitations,
|
||||
},
|
||||
evaluator_conformance: {
|
||||
status: verificationReport.evaluator_conformance.status,
|
||||
evidence_kind: verificationReport.evaluator_conformance.evidence_kind,
|
||||
checks: verificationReport.evaluator_conformance.checks,
|
||||
report_digest:
|
||||
verificationReport.evaluator_conformance.baseline_report.report_digest,
|
||||
},
|
||||
})}\n`,
|
||||
);
|
||||
|
||||
async function runCli(arguments_) {
|
||||
const { stdout } = await execute(
|
||||
process.execPath,
|
||||
@@ -166,32 +204,50 @@ async function analyzeMcp(client_, inputPath) {
|
||||
arguments: { input_path: inputPath },
|
||||
});
|
||||
assert.notEqual(response.isError, true);
|
||||
const projected = requireObject(response.structuredContent);
|
||||
return { evidence_id: projected.evidence_id };
|
||||
}
|
||||
|
||||
function requireObject(value) {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value))
|
||||
throw new TypeError("Expected an object");
|
||||
return value;
|
||||
}
|
||||
|
||||
function setFixtureDigest(input, kind, evidence) {
|
||||
const fixture = input.fixtures.find(
|
||||
({ kind: fixtureKind }) => fixtureKind === kind,
|
||||
toolContract("analyze_javascript_application").outputSchema.parse(
|
||||
response.structuredContent,
|
||||
);
|
||||
if (fixture === undefined || evidence.subject === null)
|
||||
throw new TypeError(`Missing ${kind} fixture identity`);
|
||||
fixture.artifact_sha256 = evidence.subject.digest.sha256;
|
||||
return parseEvidence(response.structuredContent);
|
||||
}
|
||||
|
||||
function linkCheck(input, stageId, checkId, evidence) {
|
||||
const stage = input.stages.find(({ stage_id: id }) => id === stageId);
|
||||
const check = stage?.checks.find(({ check_id: id }) => id === checkId);
|
||||
if (stage === undefined || check === undefined)
|
||||
throw new TypeError(`Missing readiness check ${stageId}/${checkId}`);
|
||||
stage.evidence_ids = [
|
||||
...new Set([...stage.evidence_ids, evidence.evidence_id]),
|
||||
].sort();
|
||||
check.evidence_ids = [evidence.evidence_id];
|
||||
function verifySyntheticEvaluator() {
|
||||
const input = structuredClone(RECONSTRUCTION_READINESS_EXAMPLE);
|
||||
const evaluate = (value) =>
|
||||
reconstructionReadinessReportSchema.parse(
|
||||
createReconstructionReadinessReport(value),
|
||||
);
|
||||
const baseline = evaluate(input);
|
||||
assert.equal(baseline.status, "pass");
|
||||
assert.deepEqual(evaluate(structuredClone(input)), baseline);
|
||||
|
||||
const tampered = structuredClone(input);
|
||||
const record = tampered.evidence_bundle.records[0];
|
||||
assert.ok(record, "Synthetic conformance requires Evidence");
|
||||
record.operation += "_tampered";
|
||||
assert.throws(
|
||||
() => evaluate(tampered),
|
||||
/Evidence semantic identifier does not match its record/u,
|
||||
);
|
||||
|
||||
const stale = structuredClone(input);
|
||||
stale.replay.expected_source_digest = baseline.source_digest;
|
||||
stale.identity.server_version += "-changed";
|
||||
const staleReport = evaluate(stale);
|
||||
assert.equal(staleReport.status, "fail");
|
||||
assert.ok(
|
||||
staleReport.findings.some(
|
||||
({ code }) => code === "replay-source-digest-mismatch",
|
||||
),
|
||||
);
|
||||
return {
|
||||
evidence_kind: "synthetic",
|
||||
status: "pass",
|
||||
checks: [
|
||||
"baseline-accepted",
|
||||
"deterministic-evaluation",
|
||||
"tampered-evidence-rejected",
|
||||
"stale-source-rejected",
|
||||
].map((checkId) => ({ check_id: checkId, status: "pass" })),
|
||||
baseline_report: baseline,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1246,6 +1246,40 @@ it.each([
|
||||
"tests/conformance/readiness/javascript-cli/package.json",
|
||||
{ code: true, fixtures: false, readiness: true, package: false },
|
||||
],
|
||||
[
|
||||
"scripts/verify-reconstruction-readiness.mjs",
|
||||
{
|
||||
code: true,
|
||||
readiness: true,
|
||||
fixtures: false,
|
||||
runtime: false,
|
||||
package: false,
|
||||
},
|
||||
],
|
||||
[
|
||||
"tests/conformance/readiness/report/reconstructionReadiness.ts",
|
||||
{
|
||||
code: true,
|
||||
readiness: true,
|
||||
fixtures: false,
|
||||
runtime: false,
|
||||
package: false,
|
||||
},
|
||||
],
|
||||
[
|
||||
"tests/conformance/readiness/report/reconstructionReadinessExample.ts",
|
||||
{
|
||||
code: true,
|
||||
readiness: true,
|
||||
fixtures: false,
|
||||
runtime: false,
|
||||
package: false,
|
||||
},
|
||||
],
|
||||
[
|
||||
"tests/conformance/readiness/report/reconstructionReadiness.test.ts",
|
||||
{ code: true, readiness: false, runtime: false, package: false },
|
||||
],
|
||||
[
|
||||
"tests/conformance/ghidra/no-return.c",
|
||||
{ code: true, fixtures: false, readiness: false, package: false },
|
||||
|
||||
@@ -30,6 +30,9 @@ or network dependencies.
|
||||
|
||||
`readiness/` adds source-only generated JavaScript CLI and Electron fixtures for
|
||||
the repository's reconstruction-readiness verifier. `npm run verify:readiness`
|
||||
combines them with the compiled native fixtures, exercises CLI and MCP parity
|
||||
for real analysis operations, and checks the internal machine-readable report
|
||||
documented in `docs/reconstruction-readiness.md`.
|
||||
inspects the compiled native fixture through CLI and checks JavaScript/Electron
|
||||
static-analysis Evidence identity through CLI and MCP. It does not execute the
|
||||
fixture applications. The development-owned evaluator under `readiness/report/`
|
||||
separately checks synthetic report conformance; its nine stages do not describe
|
||||
observed runtime journeys. The report keeps these scopes separate as documented
|
||||
in `docs/reconstruction-readiness.md`.
|
||||
|
||||
+3
-3
@@ -1,14 +1,14 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { RECONSTRUCTION_READINESS_EXAMPLE } from "../contracts/reconstructionReadinessExample.js";
|
||||
import { RECONSTRUCTION_READINESS_EXAMPLE } from "./reconstructionReadinessExample.js";
|
||||
import { createReconstructionReadinessReport } from "./reconstructionReadiness.js";
|
||||
import {
|
||||
reconstructionObligationLedgerClosureDigest,
|
||||
reconstructionObligationLedgerStatus,
|
||||
reconstructionObligationOwnershipGraph,
|
||||
summarizeReconstructionObligations,
|
||||
} from "./reconstructionObligationLedgerIntegrity.js";
|
||||
import { reconstructionObligationLedgerSchema } from "./reconstructionObligationLedgerSchemas.js";
|
||||
} from "../../../../src/domain/reconstructionObligationLedgerIntegrity.js";
|
||||
import { reconstructionObligationLedgerSchema } from "../../../../src/domain/reconstructionObligationLedgerSchemas.js";
|
||||
import {
|
||||
reconstructionReadinessReportSchema,
|
||||
reconstructionReadinessInputSchema,
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
import { digestCanonicalValue } from "./canonicalDigest.js";
|
||||
import { authenticateParsedEvidenceBundle } from "./evidenceBundle.js";
|
||||
import { digestCanonicalValue } from "../../../../src/domain/canonicalDigest.js";
|
||||
import { authenticateParsedEvidenceBundle } from "../../../../src/domain/evidenceBundle.js";
|
||||
import { readinessFindingOrder } from "./reconstructionReadinessFindingHelpers.js";
|
||||
import { readinessJourneyFindings } from "./reconstructionReadinessJourneyFindings.js";
|
||||
import { readinessIntegrityFindings } from "./reconstructionReadinessIntegrityFindings.js";
|
||||
@@ -12,7 +12,7 @@ import {
|
||||
type ReconstructionReadinessReport,
|
||||
} from "./reconstructionReadinessSchemas.js";
|
||||
|
||||
/** Evaluate one public-contract journey into a deterministic fail-closed report. */
|
||||
/** Evaluate synthetic conformance observations into a deterministic fail-closed report. */
|
||||
export const createReconstructionReadinessReport = (
|
||||
inputValue: unknown,
|
||||
): ReconstructionReadinessReport => {
|
||||
+7
-7
@@ -1,15 +1,15 @@
|
||||
import { createEvidence } from "../domain/evidence.js";
|
||||
import { createEvidenceBundle } from "../domain/evidenceBundle.js";
|
||||
import { READINESS_REQUIRED_CHECKS } from "../domain/reconstructionReadinessJourneyFindings.js";
|
||||
import { createEvidence } from "../../../../src/domain/evidence.js";
|
||||
import { createEvidenceBundle } from "../../../../src/domain/evidenceBundle.js";
|
||||
import { READINESS_REQUIRED_CHECKS } from "./reconstructionReadinessJourneyFindings.js";
|
||||
import {
|
||||
reconstructionObligationLedgerClosureDigest,
|
||||
summarizeReconstructionObligations,
|
||||
} from "../domain/reconstructionObligationLedgerIntegrity.js";
|
||||
import type { ReconstructionObligationLedger } from "../domain/reconstructionObligationLedgerSchemas.js";
|
||||
} from "../../../../src/domain/reconstructionObligationLedgerIntegrity.js";
|
||||
import type { ReconstructionObligationLedger } from "../../../../src/domain/reconstructionObligationLedgerSchemas.js";
|
||||
import {
|
||||
readinessStageIdSchema,
|
||||
type ReconstructionReadinessInput,
|
||||
} from "../domain/reconstructionReadinessSchemas.js";
|
||||
} from "./reconstructionReadinessSchemas.js";
|
||||
|
||||
const HASH = "a".repeat(64);
|
||||
const sourceEvidence = (variant: string) =>
|
||||
@@ -159,7 +159,7 @@ const READINESS_OBLIGATION_LEDGER = (() => {
|
||||
};
|
||||
})();
|
||||
|
||||
/** Complete synthetic input shared by docs, tests, CLI, MCP, and verifier. */
|
||||
/** Complete synthetic input for the repository's evaluator conformance checks. */
|
||||
export const RECONSTRUCTION_READINESS_EXAMPLE: ReconstructionReadinessInput = {
|
||||
identity: {
|
||||
cli_version: "2.4.0",
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
import { compositeKey } from "./unicodeCodePointOrder.js";
|
||||
import { compositeKey } from "../../../../src/domain/unicodeCodePointOrder.js";
|
||||
import {
|
||||
addMissingEvidenceFindings,
|
||||
readinessFinding as finding,
|
||||
+7
-4
@@ -1,10 +1,13 @@
|
||||
import { z } from "zod";
|
||||
|
||||
import { STABLE_IDENTIFIER_PATTERN } from "./stringPatterns.js";
|
||||
import { STABLE_IDENTIFIER_PATTERN } from "../../../../src/domain/stringPatterns.js";
|
||||
|
||||
import { evidenceBundleSchema } from "./evidenceBundle.js";
|
||||
import { reconstructionObligationLedgerSchema } from "./reconstructionObligationLedgerSchemas.js";
|
||||
import { digestSchema, prefixedDigestSchema } from "./digests.js";
|
||||
import { evidenceBundleSchema } from "../../../../src/domain/evidenceBundle.js";
|
||||
import { reconstructionObligationLedgerSchema } from "../../../../src/domain/reconstructionObligationLedgerSchemas.js";
|
||||
import {
|
||||
digestSchema,
|
||||
prefixedDigestSchema,
|
||||
} from "../../../../src/domain/digests.js";
|
||||
|
||||
const evidenceIdSchema = prefixedDigestSchema("ev");
|
||||
const stableIdSchema = z.string().regex(STABLE_IDENTIFIER_PATTERN);
|
||||
Reference in New Issue
Block a user