mirror of
https://github.com/morluto/rea
synced 2026-10-25 22:08:11 -04:00
chore: reconcile current main boundary fixes
This commit is contained in:
morluto
@@ -154,9 +154,16 @@ server; older servers keep the complete request after schema discovery.
|
||||
- **Selected ABI.** AOT inspection applies the library budget to the selected
|
||||
ABI after selection; an oversized unselected library does not block it.
|
||||
Undefined dynamic-symbol imports are not treated as snapshot exports.
|
||||
A retained section directory without `SHT_DYNSYM` uses the same bounded
|
||||
`PT_DYNAMIC` symbol description as a section-stripped image. Malformed
|
||||
present symbol-table sections remain failures.
|
||||
Section-stripped images require a bounded symbol count from `DT_HASH` or
|
||||
`DT_GNU_HASH`; layouts without either are refused. Symbol names and their
|
||||
terminators must fit their linked string table.
|
||||
terminators must fit their linked string table. Only dynamic entries used for
|
||||
symbol discovery are interpreted as numeric bounds. ABI-ignored payloads,
|
||||
including [`DT_BIND_NOW`](https://gabi.xinuos.com/elf/08-dynamic.html#dynamic-section),
|
||||
do not change discovery; their stored bytes still contribute to the selected
|
||||
artifact's digest.
|
||||
- REA performs **no hash-to-SDK lookup**: mapping the snapshot hash to a
|
||||
Dart SDK release requires an external dataset and is deliberately out of
|
||||
scope for this family.
|
||||
|
||||
@@ -1012,6 +1012,12 @@ materialized; extraction fails as `unavailable` with that logical path.
|
||||
Extraction materializes regular files exposed by the active archive reader.
|
||||
When that reader exposes a nested archive as a file, extraction copies the
|
||||
archive file; inspect and extract that file separately to unpack its contents.
|
||||
Extraction retains inventory and materialization traversal limitations in its
|
||||
result and outer Evidence. If either traversal has unknown coverage, the
|
||||
observed regular files can still be copied, but the result does not establish
|
||||
a complete source copy. For example, a POSIX filename that is not valid UTF-8
|
||||
remains omitted with its raw-name explanation; it is not replacement-decoded
|
||||
into an output filename.
|
||||
|
||||
Packaging tools commonly sign or strip `.asar.unpacked` native binaries after
|
||||
writing the archive header, so their declared hashes no longer match. Electron
|
||||
|
||||
+80
-1
@@ -57,7 +57,8 @@ name, original type ID, dimensions, relative data offset, file offset and derive
|
||||
byte length. Alignment accepts every positive UINT32 multiple of eight, including
|
||||
non-powers of two. Partial Evidence retains explicit alignment once validated;
|
||||
the default alignment becomes known after metadata completes. Cancellation
|
||||
digests or file identity. Metadata keys preserve their spelling and use lowercase
|
||||
retains completed declarations with unknown digests and unverified final file
|
||||
identity. Metadata keys preserve their spelling and use lowercase
|
||||
ASCII alphanumeric components separated by dots, underscores or hyphens;
|
||||
standard numeric components such as `general.base_model.0.name` are preserved.
|
||||
Hyphenated components are retained for compatibility with the published
|
||||
@@ -80,6 +81,84 @@ Nonfinite floating metadata uses `NaN`, `+Infinity` or `-Infinity` with its
|
||||
declared float type; negative zero is `"-0"`. Unknown tensor types retain their numeric IDs and null byte
|
||||
lengths; affected totals and storage validation remain unknown.
|
||||
|
||||
Q8_1 (type ID 9) has two released layouts: 40 bytes per 32 elements in
|
||||
[llama.cpp b1500](https://github.com/ggml-org/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/ggml-quants.h#L49-L55)
|
||||
and 36 bytes in the pinned
|
||||
[modern ggml layout](https://github.com/ggml-org/ggml/blob/da8e2a54b919557b816b1c2beb432ccb766f7ce0/src/ggml-common.h#L242-L253). Both use GGUF v3 and
|
||||
quantization version 2, so those declarations do not select a storage layout.
|
||||
REA retains the known type, shape, parameter count and starting offsets, with
|
||||
`byte_length: null`, `storage_validation: "unknown"`, partial storage/range
|
||||
coverage and an explicit limitation for nonempty Q8_1 tensors. Per-type and
|
||||
overall byte totals remain unknown where affected. Padding, the next tensor's
|
||||
offset and file size do not establish a layout; retained queries preserve the
|
||||
same uncertainty. Empty Q8_1 tensors still have a derived zero-byte length,
|
||||
and the known 32-element row-block constraint remains enforced.
|
||||
|
||||
K-quantization IDs 10–15 also have incompatible released layouts. The
|
||||
[b1500 build option](https://github.com/ggml-org/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/CMakeLists.txt#L88)
|
||||
`LLAMA_QKK_64` selects `GGML_QKK_64`; the same release's default and
|
||||
[b3001](https://github.com/ggml-org/llama.cpp/blob/c429b33beb35f13934a4dfbe0c138d30b45e5d54/ggml-common.h)
|
||||
use 256-element blocks. The
|
||||
[b1500 structures](https://github.com/ggml-org/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/ggml-quants.h#L61-L169)
|
||||
give these source-derived sizes:
|
||||
|
||||
| ID | Stable name | Bytes per 64 elements, K64 | Bytes per 256 elements, K256 | Padded extent for 256 elements, either layout |
|
||||
| --- | ----------- | -------------------------- | ---------------------------- | --------------------------------------------- |
|
||||
| 10 | Q2_K | 24 | 84 | 96 |
|
||||
| 11 | Q3_K | 28 | 110 | 128 |
|
||||
| 12 | Q4_K | 38 | 144 | 160 |
|
||||
| 13 | Q5_K | 46 | 176 | 192 |
|
||||
| 14 | Q6_K | 54 | 210 | 224 |
|
||||
| 15 | Q8_K | 76 | 292 | 320 |
|
||||
|
||||
Both builds write GGUF v3 and quantization version 2. The
|
||||
[quantizer](https://github.com/ggml-org/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/llama.cpp#L7430-L7492)
|
||||
writes `general.file_type` directly from the selected `llama_ftype`, without
|
||||
encoding the K64 build option. `GGML_QNT_VERSION_FACTOR` is defined in the
|
||||
header but is not used by this GGUF serialization path. Global file type also
|
||||
does not identify individual tensor types: the quantizer
|
||||
[retains unselected tensors](https://github.com/ggml-org/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/llama.cpp#L7561-L7586).
|
||||
The [writer pads each payload](https://github.com/ggml-org/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/ggml.c#L19549-L19565)
|
||||
to the default 32-byte alignment; the final column shows why even the same file
|
||||
extents need not distinguish these layouts.
|
||||
REA preserves each K type's name and enforces the common 64-element row
|
||||
multiple, which does not establish a storage block size. Nonempty K tensor
|
||||
lengths and affected totals are now unknown, including common modern files.
|
||||
Empty K tensors still have derived zero-byte storage.
|
||||
|
||||
Integer IDs also shifted each time IQ types were added. These released
|
||||
`ggml.h` snapshots all declare GGUF v3 and quantization version 2:
|
||||
|
||||
| Released snapshot | I8 ID | I16 ID | I32 ID |
|
||||
| --------------------------------------------------------------------------------------------------- | ----- | ------ | ------ |
|
||||
| [b1500](https://github.com/ggml-org/llama.cpp/blob/57ad015dc3011b046ed5a23186c86ea55f987c54/ggml.h) | 16 | 17 | 18 |
|
||||
| [b1803](https://github.com/ggml-org/llama.cpp/blob/36e5a08b203542dca53cca4eaf172c5dc4bbc991/ggml.h) | 17 | 18 | 19 |
|
||||
| [b1871](https://github.com/ggml-org/llama.cpp/blob/bb0c1392479398f9aba86d9ec98db0b95ede6e6d/ggml.h) | 18 | 19 | 20 |
|
||||
| [b2100](https://github.com/ggml-org/llama.cpp/blob/4aa43fab569215a13495a7f1a0f8afc541b16d03/ggml.h) | 19 | 20 | 21 |
|
||||
| [b2172](https://github.com/ggml-org/llama.cpp/blob/bd2d4e393b2b7d2a1b2e201058e26017c9728ead/ggml.h) | 20 | 21 | 22 |
|
||||
| [b2221](https://github.com/ggml-org/llama.cpp/blob/a14679cc30c785e75d38028bae6ec39c6209ddef/ggml.h) | 21 | 22 | 23 |
|
||||
| [b2253](https://github.com/ggml-org/llama.cpp/blob/4c4cb30736582cacb1a164a9d4bc8e17b1014be7/ggml.h) | 22 | 23 | 24 |
|
||||
| [b2275](https://github.com/ggml-org/llama.cpp/blob/a33e6a0d2a66104ea9a906bdbf8a94d050189d91/ggml.h) | 23 | 24 | 25 |
|
||||
| [b2300](https://github.com/ggml-org/llama.cpp/blob/6ea0f010ff6967034528d9e0b8330b9b0f0b7c13/ggml.h) | 24 | 25 | 26 |
|
||||
|
||||
[Upstream PR 6050](https://github.com/ggml-org/llama.cpp/pull/6050) later pinned
|
||||
integer values explicitly. Modern IDs 16–23 mean IQ2_XXS, IQ2_XS, IQ3_XXS,
|
||||
IQ1_S, IQ4_NL, IQ3_S, IQ2_S and IQ4_XS, respectively. Thus every ID 16–25 has
|
||||
multiple released meanings. REA keeps its numeric `type_id`, uses
|
||||
`AMBIGUOUS_<id>` for `type`, and records the released candidate names in a
|
||||
limitation. Nonempty byte lengths are unknown. Empty declarations still derive
|
||||
zero-byte storage, while their textual identity remains ambiguous. Type filters
|
||||
match the reported `AMBIGUOUS_<id>` label. ID 26 remains I32 across these
|
||||
snapshots; later stable entries retain their existing interpretation. This audit
|
||||
does not establish compatibility with every producer fork or future revision.
|
||||
|
||||
The inventory does not select a historical layout from model names, global
|
||||
file types, padding or guessed producer provenance. A source revision together
|
||||
with its build configuration can establish an external layout fact; these
|
||||
headers do not carry that identification, and REA currently has no layout
|
||||
selection input. Authored regression fixtures use the source-derived sizes
|
||||
above; historical executables were not run.
|
||||
|
||||
Safetensors reports the header extent, exact string `__metadata__` values, dtype,
|
||||
shape and declared data ranges. A null `__metadata__` is treated as omitted;
|
||||
unknown fields on tensor entries are ignored within the reader's 32-level JSON
|
||||
|
||||
@@ -433,9 +433,10 @@ complete observation.
|
||||
its outcome. Paths outside the analyzed root, including shared-cache system
|
||||
libraries, stay undetermined. See
|
||||
[Apple application analysis](apple-application-analysis.md#dylib-load-resolution).
|
||||
Executable-root discovery preserves traversal uncertainty and skipped
|
||||
symlinks as partial coverage; it cannot establish an empty root set from
|
||||
unexamined names or symlink targets.
|
||||
Executable-root discovery preserves traversal uncertainty, including
|
||||
undecodable filenames and skipped symlinks, as partial coverage. Known
|
||||
executable roots remain available; unexamined names and symlink targets
|
||||
cannot establish an empty root set or the absence of other roots.
|
||||
- `inspect_asset_catalog` / `rea inspect-asset-catalog <app>` reads compiled
|
||||
`Assets.car` metadata through macOS `assetutil --info`. Catalog digests, raw
|
||||
rendition fields, pagination and exact UI resource-name matches are returned.
|
||||
@@ -793,6 +794,15 @@ unresolved runtime/value semantics remain visible in results.
|
||||
|
||||
## Interface Builder hierarchy coverage
|
||||
|
||||
Bundle archive selection preserves directory traversal uncertainty, including
|
||||
filenames that cannot be decoded as UTF-8. These names are not guessed or
|
||||
counted as known archive omissions. Recognized symlinked `.nib` entries are
|
||||
not followed; they retain their paths and reader limitations as counted archive
|
||||
omissions. Either gap makes `archive_decode` partial and the graph truncated
|
||||
while preserving decoded documents. Its `examined` count remains actual decode
|
||||
attempts, and asset-catalog UI joins retain their existing incomplete-graph
|
||||
qualification.
|
||||
|
||||
A document's `hierarchy_complete` flag follows its own hierarchy coverage.
|
||||
If graph limits omit a hierarchy source or destination, or prevent a hierarchy
|
||||
edge or traversal from being retained, that document reports partial hierarchy
|
||||
@@ -842,11 +852,29 @@ Integral reals stay numbers, and ambiguous or incompletely observed values retai
|
||||
explicit precision limitations. Before plist decoding, the existing aggregate
|
||||
decode budget reserves estimated room for number metadata and attribute
|
||||
projection; these estimates are not a hard process-memory bound. Keyed-archive
|
||||
object identities come from their original `$objects` table indices unless an
|
||||
explicit `objectID` or `object-id` is present. An authored `archiveUID` field
|
||||
remains ordinary object data and does not change identity. NIBArchive float32
|
||||
and float64 values retain non-finite values and negative zero through projection
|
||||
as the same typed real wrappers used for plist values; per-document limitations
|
||||
object identities always come from their original `$objects` table indices.
|
||||
Authored `objectID`, `object-id`, `archiveUID` and `id` fields remain ordinary
|
||||
object data and do not change inventory, reference, connector or hierarchy
|
||||
identity. This changes keyed-archive graph IDs that previously used authored
|
||||
`objectID` or `object-id` values: `interface_builder_object_id` and linked node
|
||||
IDs now use the decimal table UID, while the authored values remain in attributes.
|
||||
For example, an authored `objectID: "2"` on table entry 1 now identifies object
|
||||
`"1"`; its graph node ends in `object:["1"]`. Nonkeyed ibtool dictionary IDs
|
||||
and native NIBArchive IDs keep their existing source-specific meaning.
|
||||
Connector endpoints require encoded UID references to archived objects; strings
|
||||
and dictionaries that merely resemble resolved reference stubs do not establish
|
||||
identity. Inline hierarchy dictionaries have no table identity and count as
|
||||
omitted references, while independently referenced children remain available.
|
||||
The UID identity rule follows the
|
||||
[Foundation unarchiver's table lookup](https://github.com/swiftlang/swift-corelibs-foundation/blob/28249ced079d8e7ec03b821c039bf9868738c3fc/Sources/Foundation/NSKeyedUnarchiver.swift#L213-L223).
|
||||
UIKit keyed nibs select their explicitly named
|
||||
`UINibTopLevelObjectsKey` collection; generic `root` is used when that named key
|
||||
is absent. Unsupported or dangling named roots remain incomplete rather than
|
||||
falling back to another collection. The key spelling follows the independent
|
||||
[WinObjC UINib reader](https://github.com/microsoft/WinObjC/blob/94f6b5bfdc09d945e30607b56124b6f94a5b9d7b/Frameworks/UIKit/UINib.mm#L104-L109).
|
||||
|
||||
NIBArchive float32 and float64 values retain non-finite values and negative zero
|
||||
through projection as the same typed real wrappers used for plist values; per-document limitations
|
||||
count those values. NIBArchive nil remains JSON null.
|
||||
|
||||
Dates are reported as `{ "$plist_type": "date", "iso8601": "<timestamp>" }`.
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
import { compareUnicodeCodePoints } from "../domain/unicodeCodePointOrder.js";
|
||||
import { normalizeArtifactPath } from "./ArtifactPaths.js";
|
||||
import type {
|
||||
ArtifactReaderTraversal,
|
||||
ArtifactTraversalOmission,
|
||||
ArtifactTraversalOptions,
|
||||
} from "./ArtifactReader.js";
|
||||
|
||||
/** Retain producer coverage and omissions across one or more exhausted traversals. */
|
||||
export class ArtifactTraversalObservation implements ArtifactTraversalOptions {
|
||||
readonly #limitations = new Set<string>();
|
||||
#unknown = false;
|
||||
|
||||
readonly onCoverage = (coverage: ArtifactReaderTraversal): void => {
|
||||
this.#unknown ||= coverage.status === "unknown";
|
||||
for (const limitation of coverage.limitations)
|
||||
this.#limitations.add(limitation);
|
||||
};
|
||||
|
||||
readonly onOmission = (omission: ArtifactTraversalOmission): void => {
|
||||
const path = normalizeArtifactPath(omission.path);
|
||||
this.#unknown = true;
|
||||
this.#limitations.add(
|
||||
`Artifact traversal omitted ${omission.kind} at ${JSON.stringify(path)}.`,
|
||||
);
|
||||
};
|
||||
|
||||
snapshot(): ArtifactReaderTraversal {
|
||||
return {
|
||||
status: this.#unknown ? "unknown" : "complete",
|
||||
limitations: [...this.#limitations].sort(compareUnicodeCodePoints),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -193,6 +193,34 @@ describe("Apple asset catalog application workflow", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it.skipIf(process.platform === "win32")(
|
||||
"qualifies UI resource joins when a symlinked nib was not decoded",
|
||||
async () => {
|
||||
const bundle = await createTestTempDirectory("rea-asset-ui-symlink-");
|
||||
const resources = join(bundle, "Contents", "Resources");
|
||||
await mkdir(resources, { recursive: true });
|
||||
await writeFile(
|
||||
join(resources, "Panel.data"),
|
||||
encodeNibArchiveFixture({
|
||||
classes: ["NSView"],
|
||||
objects: [{ classIndex: 0, values: { title: "Panel" } }],
|
||||
}),
|
||||
);
|
||||
await symlink("Panel.data", join(resources, "Panel.nib"));
|
||||
const result = await analyzeAppleAssetCatalogs({
|
||||
bundlePath: bundle,
|
||||
environment: {},
|
||||
targetSha256: "a".repeat(64),
|
||||
runAssetUtil: () => {
|
||||
throw new Error("No catalog is present");
|
||||
},
|
||||
});
|
||||
expect(result.limitations).toContain(
|
||||
"Interface Builder resource-key joins may be incomplete because the decoded UI graph was truncated.",
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([0, 1, 5])(
|
||||
"keeps an empty assetutil inventory complete at offset %i",
|
||||
async (offset) => {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { lstat } from "node:fs/promises";
|
||||
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
|
||||
import { ArtifactTraversalObservation } from "../ArtifactTraversalObservation.js";
|
||||
import {
|
||||
ArtifactReaderFailure,
|
||||
sameArtifactSourceIdentity,
|
||||
@@ -137,18 +138,16 @@ const collectAssetCatalogs = async (input: {
|
||||
records: ReturnType<typeof parseAppleAssetCatalogRecords>;
|
||||
}[] = [];
|
||||
const uninspected: string[] = [];
|
||||
let status: ArtifactReaderTraversal["status"] = "complete";
|
||||
const limitations: string[] = [];
|
||||
const traversal = new ArtifactTraversalObservation();
|
||||
let symlinks = 0;
|
||||
let firstSymlink: string | undefined;
|
||||
let totalMetadataBytes = 0;
|
||||
try {
|
||||
for await (const entry of reader.entries(input.signal, {
|
||||
onCoverage: (coverage) => {
|
||||
if (coverage.status !== "complete") status = coverage.status;
|
||||
limitations.push(...coverage.limitations);
|
||||
},
|
||||
onOmission: ({ path, kind }) => {
|
||||
onCoverage: traversal.onCoverage,
|
||||
onOmission: (omission) => {
|
||||
traversal.onOmission(omission);
|
||||
const { path, kind } = omission;
|
||||
if (path.split("/").at(-1) === "Assets.car")
|
||||
uninspected.push(`${path} (${kind})`);
|
||||
},
|
||||
@@ -187,9 +186,14 @@ const collectAssetCatalogs = async (input: {
|
||||
await reader.close();
|
||||
}
|
||||
if (symlinks > 0)
|
||||
limitations.push(
|
||||
`Asset catalog discovery did not follow ${symlinks} symlink(s); catalogs beneath them may remain uninspected. First path: ${JSON.stringify(firstSymlink)}.`,
|
||||
);
|
||||
traversal.onCoverage({
|
||||
// Source entries are represented even when their link targets were not
|
||||
// inspected. Keep that inventory scope separate from catalog discovery.
|
||||
status: "complete",
|
||||
limitations: [
|
||||
`Asset catalog discovery did not follow ${symlinks} symlink(s); catalogs beneath them may remain uninspected. First path: ${JSON.stringify(firstSymlink)}.`,
|
||||
],
|
||||
});
|
||||
// Without any readable catalog, unmatched resource keys would overstate an
|
||||
// absence that a skipped catalog entry may contradict.
|
||||
if (catalogs.length === 0 && uninspected.length > 0)
|
||||
@@ -197,7 +201,7 @@ const collectAssetCatalogs = async (input: {
|
||||
"path",
|
||||
`Asset catalog entries are not regular files and were not inspected: ${uninspected.join(", ")}`,
|
||||
);
|
||||
return { catalogs, uninspected, traversal: { status, limitations } };
|
||||
return { catalogs, uninspected, traversal: traversal.snapshot() };
|
||||
};
|
||||
|
||||
const inspectAssetCatalogEntry = async (input: {
|
||||
|
||||
@@ -19,6 +19,7 @@ import {
|
||||
ArtifactReaderFailure,
|
||||
type ArtifactReaderTraversal,
|
||||
} from "../ArtifactReader.js";
|
||||
import { ArtifactTraversalObservation } from "../ArtifactTraversalObservation.js";
|
||||
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
|
||||
import { consumeWithOrderedReads } from "../OrderedReads.js";
|
||||
import {
|
||||
@@ -156,6 +157,7 @@ export const traceDylibResolution = async (options: {
|
||||
const trace = await traceDylibLoading(view, {
|
||||
roots,
|
||||
unclassified,
|
||||
...(traversal === undefined ? {} : { rootEnumeration: traversal }),
|
||||
...(parsed.data.architecture === undefined
|
||||
? {}
|
||||
: { architecture: parsed.data.architecture }),
|
||||
@@ -179,12 +181,6 @@ export const traceDylibResolution = async (options: {
|
||||
assertArtifactReadActive(options.signal);
|
||||
return dylibResolutionResultSchema.parse({
|
||||
...trace,
|
||||
coverage: {
|
||||
...trace.coverage,
|
||||
status:
|
||||
traversal?.status === "unknown" ? "partial" : trace.coverage.status,
|
||||
},
|
||||
limitations: [...trace.limitations, ...(traversal?.limitations ?? [])],
|
||||
root_path: options.rootPath,
|
||||
target_sha256: options.targetSha256,
|
||||
images: trace.images.map((image) => ({
|
||||
@@ -215,31 +211,18 @@ const executableRoots = async (
|
||||
}> => {
|
||||
const roots: string[] = [];
|
||||
const unclassified: string[] = [];
|
||||
let status: ArtifactReaderTraversal["status"] = "complete";
|
||||
const limitations: string[] = [];
|
||||
const traversal = new ArtifactTraversalObservation();
|
||||
let symlinks = 0;
|
||||
let firstSymlink: string | undefined;
|
||||
// Bundles hold tens of thousands of files and few Mach-O images, so the
|
||||
// magic reads overlap the walk while results are classified in walk order.
|
||||
await consumeWithOrderedReads(
|
||||
new DirectoryArtifactReader(root).entries(signal, {
|
||||
onCoverage: (coverage) => {
|
||||
if (coverage.status !== "complete") status = coverage.status;
|
||||
limitations.push(...coverage.limitations);
|
||||
},
|
||||
onOmission: ({ path, kind }) => {
|
||||
status = "unknown";
|
||||
limitations.push(
|
||||
`Executable-root traversal omitted ${kind} at ${JSON.stringify(path)}; its root role is unknown.`,
|
||||
);
|
||||
},
|
||||
}),
|
||||
new DirectoryArtifactReader(root).entries(signal, traversal),
|
||||
MAGIC_READ_CONCURRENCY,
|
||||
(entry) => {
|
||||
// The ordered-read owner skips consume() when read() returns undefined.
|
||||
// Discovery facts for skipped entries belong at this selection boundary.
|
||||
if (entry.kind === "symlink") {
|
||||
status = "unknown";
|
||||
symlinks += 1;
|
||||
firstSymlink ??= entry.path;
|
||||
}
|
||||
@@ -262,13 +245,16 @@ const executableRoots = async (
|
||||
},
|
||||
);
|
||||
if (symlinks > 0)
|
||||
limitations.push(
|
||||
`Executable-root discovery did not follow ${symlinks} symlink(s); their root roles and possible descendants are unknown. First path: ${JSON.stringify(firstSymlink)}.`,
|
||||
);
|
||||
traversal.onCoverage({
|
||||
status: "unknown",
|
||||
limitations: [
|
||||
`Executable-root discovery did not follow ${symlinks} symlink(s); their root roles and possible descendants are unknown. First path: ${JSON.stringify(firstSymlink)}.`,
|
||||
],
|
||||
});
|
||||
return {
|
||||
roots: roots.sort(compare),
|
||||
unclassified: unclassified.sort(compare),
|
||||
traversal: { status, limitations },
|
||||
traversal: traversal.snapshot(),
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
@@ -9,6 +9,7 @@ import {
|
||||
isUIKitRuntimeConnectionClass,
|
||||
} from "../../domain/apple/interfaceBuilderKeyedArchive.js";
|
||||
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
|
||||
import { ArtifactTraversalObservation } from "../ArtifactTraversalObservation.js";
|
||||
import {
|
||||
projectPlistValue,
|
||||
binaryDatePrecisionOverlapLimitation,
|
||||
@@ -62,6 +63,13 @@ import {
|
||||
const MAX_AGGREGATE_INPUT_BYTES = 32 * 1024 * 1024;
|
||||
const MAX_AGGREGATE_DECODE_BYTES = 256 * 1024 * 1024;
|
||||
const MAX_REPORTED_OMITTED_ARCHIVES = 64;
|
||||
|
||||
interface OmittedArchive {
|
||||
readonly path: string;
|
||||
readonly reason: string;
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
/** Decode compiled Interface Builder archives from a local app bundle. */
|
||||
export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
readonly bundlePath: string;
|
||||
@@ -74,7 +82,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
const reader = new DirectoryArtifactReader(input.bundlePath);
|
||||
const documents: InterfaceBuilderDocumentInput[] = [];
|
||||
const invalid: string[] = [];
|
||||
const omittedArchives: Array<{ path: string; reason: string }> = [];
|
||||
const omittedArchives: OmittedArchive[] = [];
|
||||
let unreportedOmittedArchives = 0;
|
||||
const incompleteHierarchies = new Map<string, number>();
|
||||
const unresolvedConnections = new Map<string, number>();
|
||||
@@ -87,16 +95,17 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
let omitted = 0;
|
||||
let attempted = 0;
|
||||
let aggregateInputBytes = 0;
|
||||
let traversal: ArtifactReaderTraversal = {
|
||||
status: "complete",
|
||||
limitations: [],
|
||||
};
|
||||
const traversal = new ArtifactTraversalObservation();
|
||||
const decodeBudget = new InterfaceBuilderDecodeBudget(
|
||||
MAX_AGGREGATE_DECODE_BYTES,
|
||||
);
|
||||
const recordOmittedArchive = (path: string, reason: string): void => {
|
||||
const recordOmittedArchive = (
|
||||
path: string,
|
||||
reason: string,
|
||||
limitations: readonly string[] = [],
|
||||
): void => {
|
||||
if (omittedArchives.length < MAX_REPORTED_OMITTED_ARCHIVES)
|
||||
omittedArchives.push({ path, reason });
|
||||
omittedArchives.push({ path, reason, limitations });
|
||||
else unreportedOmittedArchives += 1;
|
||||
};
|
||||
try {
|
||||
@@ -104,11 +113,17 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
reader,
|
||||
input.signal,
|
||||
limits.max_documents,
|
||||
traversal,
|
||||
);
|
||||
omitted = selection.omitted;
|
||||
traversal = selection.traversal;
|
||||
for (const { path, reason } of selection.omittedSamples)
|
||||
recordOmittedArchive(path, reason);
|
||||
for (const entry of selection.omittedSamples)
|
||||
recordOmittedArchive(entry.path, "max_documents");
|
||||
for (const entry of selection.symlinkSamples)
|
||||
recordOmittedArchive(
|
||||
entry.path,
|
||||
"symlink_not_followed",
|
||||
entry.limitations,
|
||||
);
|
||||
unreportedOmittedArchives += selection.unreportedOmitted;
|
||||
for (const entry of selection.selected) {
|
||||
// Plist representation reserved for this document before decoding.
|
||||
@@ -276,7 +291,7 @@ export const analyzeInterfaceBuilderBundle = async (input: {
|
||||
unreportedOmittedArchives,
|
||||
attempted,
|
||||
omitted,
|
||||
traversal,
|
||||
traversal: traversal.snapshot(),
|
||||
});
|
||||
};
|
||||
|
||||
@@ -285,47 +300,30 @@ const selectInterfaceBuilderCandidates = async (
|
||||
reader: DirectoryArtifactReader,
|
||||
signal: AbortSignal | undefined,
|
||||
maximumDocuments: number,
|
||||
traversal: ArtifactTraversalObservation,
|
||||
): Promise<{
|
||||
readonly selected: readonly ArtifactEntry[];
|
||||
readonly omittedSamples: readonly { path: string; reason: string }[];
|
||||
readonly omittedSamples: readonly ArtifactEntry[];
|
||||
readonly symlinkSamples: readonly ArtifactEntry[];
|
||||
readonly omitted: number;
|
||||
readonly unreportedOmitted: number;
|
||||
readonly traversal: ArtifactReaderTraversal;
|
||||
}> => {
|
||||
const capacity = maximumDocuments + MAX_REPORTED_OMITTED_ARCHIVES;
|
||||
const candidates: ArtifactEntry[] = [];
|
||||
const omittedSamples: Array<{ path: string; reason: string }> = [];
|
||||
const traversalLimitations: string[] = [];
|
||||
let traversalStatus: ArtifactReaderTraversal["status"] = "complete";
|
||||
let nonRegular = 0;
|
||||
const symlinkSamples: ArtifactEntry[] = [];
|
||||
let symlinks = 0;
|
||||
let unclassifiedSymlinks = 0;
|
||||
let firstUnclassifiedSymlink: string | undefined;
|
||||
let total = 0;
|
||||
const omit = (path: string, reason: string) => {
|
||||
nonRegular += 1;
|
||||
if (omittedSamples.length < MAX_REPORTED_OMITTED_ARCHIVES)
|
||||
omittedSamples.push({ path, reason });
|
||||
};
|
||||
for await (const entry of reader.entries(signal, {
|
||||
onCoverage: (coverage) => {
|
||||
if (coverage.status !== "complete") traversalStatus = coverage.status;
|
||||
traversalLimitations.push(...coverage.limitations);
|
||||
},
|
||||
onOmission: ({ path, kind }) => {
|
||||
if (isInterfaceBuilderArchive(path)) omit(path, `non_regular_${kind}`);
|
||||
},
|
||||
})) {
|
||||
// A directory-shaped nib is a container for objects.nib; its children
|
||||
// are visited normally. Symlinks cannot establish their own or subtree
|
||||
// archive coverage without following them.
|
||||
for await (const entry of reader.entries(signal, traversal)) {
|
||||
if (entry.kind === "symlink") {
|
||||
if (
|
||||
isInterfaceBuilderArchive(entry.path) ||
|
||||
entry.path.toLowerCase().endsWith(".storyboardc")
|
||||
)
|
||||
omit(entry.path, "non_regular_symlink");
|
||||
else {
|
||||
traversalStatus = "unknown";
|
||||
) {
|
||||
symlinks += 1;
|
||||
retainCandidate(symlinkSamples, entry, MAX_REPORTED_OMITTED_ARCHIVES);
|
||||
} else {
|
||||
unclassifiedSymlinks += 1;
|
||||
firstUnclassifiedSymlink ??= entry.path;
|
||||
}
|
||||
@@ -337,20 +335,22 @@ const selectInterfaceBuilderCandidates = async (
|
||||
retainCandidate(candidates, entry, capacity);
|
||||
}
|
||||
if (unclassifiedSymlinks > 0)
|
||||
traversalLimitations.push(
|
||||
`Interface Builder discovery did not follow ${unclassifiedSymlinks} symlink(s); whether they contain archives is unknown. First path: ${JSON.stringify(firstUnclassifiedSymlink)}.`,
|
||||
);
|
||||
const omitted = Math.max(0, total - maximumDocuments) + nonRegular;
|
||||
for (const entry of candidates.slice(maximumDocuments, capacity)) {
|
||||
if (omittedSamples.length === MAX_REPORTED_OMITTED_ARCHIVES) break;
|
||||
omittedSamples.push({ path: entry.path, reason: "max_documents" });
|
||||
}
|
||||
traversal.onCoverage({
|
||||
status: "unknown",
|
||||
limitations: [
|
||||
`Interface Builder discovery did not follow ${unclassifiedSymlinks} symlink(s); whether they contain archives is unknown. First path: ${JSON.stringify(firstUnclassifiedSymlink)}.`,
|
||||
],
|
||||
});
|
||||
const omittedSamples = candidates.slice(maximumDocuments, capacity);
|
||||
return {
|
||||
selected: candidates.slice(0, maximumDocuments),
|
||||
omittedSamples,
|
||||
omitted,
|
||||
unreportedOmitted: omitted - omittedSamples.length,
|
||||
traversal: { status: traversalStatus, limitations: traversalLimitations },
|
||||
symlinkSamples,
|
||||
omitted: Math.max(0, total - maximumDocuments) + symlinks,
|
||||
unreportedOmitted:
|
||||
Math.max(0, total - maximumDocuments - omittedSamples.length) +
|
||||
symlinks -
|
||||
symlinkSamples.length,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -388,7 +388,7 @@ const finalizeHierarchyCoverage = (
|
||||
nonFiniteReals: ReadonlyMap<string, number>;
|
||||
negativeZeroReals: ReadonlyMap<string, number>;
|
||||
invalid: readonly string[];
|
||||
omittedArchives: readonly { path: string; reason: string }[];
|
||||
omittedArchives: readonly OmittedArchive[];
|
||||
unreportedOmittedArchives: number;
|
||||
attempted: number;
|
||||
omitted: number;
|
||||
@@ -412,7 +412,7 @@ const finalizeHierarchyCoverage = (
|
||||
invalid.length > 0 ||
|
||||
prototypeKeyOmissions.size > 0 ||
|
||||
counts.omitted > 0 ||
|
||||
counts.traversal.status !== "complete";
|
||||
counts.traversal.status === "unknown";
|
||||
return {
|
||||
...result,
|
||||
documents: result.documents.map((document) => ({
|
||||
@@ -446,29 +446,25 @@ const finalizeHierarchyCoverage = (
|
||||
status: archiveDecodePartial
|
||||
? ("partial" as const)
|
||||
: ("complete" as const),
|
||||
reason:
|
||||
counts.traversal.status !== "complete"
|
||||
? "source_traversal_unknown"
|
||||
: omittedArchives.some(({ reason }) =>
|
||||
reason.startsWith("non_regular_"),
|
||||
reason: omittedArchives.some(
|
||||
({ reason }) => reason === "aggregate_decode_budget_exhausted",
|
||||
)
|
||||
? "aggregate_decode_budget_exhausted"
|
||||
: omittedArchives.some(({ reason }) => reason === "max_documents")
|
||||
? "document_limit_reached"
|
||||
: omittedArchives.some(
|
||||
({ reason }) => reason === "symlink_not_followed",
|
||||
)
|
||||
? "archive_not_regular_file"
|
||||
: omittedArchives.some(
|
||||
({ reason }) =>
|
||||
reason === "aggregate_decode_budget_exhausted",
|
||||
)
|
||||
? "aggregate_decode_budget_exhausted"
|
||||
: omittedArchives.some(
|
||||
({ reason }) => reason === "max_documents",
|
||||
)
|
||||
? "document_limit_reached"
|
||||
: omittedArchives.length > 0
|
||||
? "archive_decode_budget_exhausted"
|
||||
: invalid.length > 0
|
||||
? "one_or_more_archives_invalid"
|
||||
: prototypeKeyOmissions.size > 0
|
||||
? "dictionary_entries_omitted"
|
||||
: null,
|
||||
? "archive_symlink_not_followed"
|
||||
: omittedArchives.length > 0
|
||||
? "archive_decode_budget_exhausted"
|
||||
: invalid.length > 0
|
||||
? "one_or_more_archives_invalid"
|
||||
: prototypeKeyOmissions.size > 0
|
||||
? "dictionary_entries_omitted"
|
||||
: counts.traversal.status === "unknown"
|
||||
? "source_traversal_unknown"
|
||||
: null,
|
||||
examined: counts.attempted,
|
||||
omitted: counts.omitted,
|
||||
},
|
||||
@@ -516,10 +512,10 @@ const finalizeHierarchyCoverage = (
|
||||
...[...negativeZeroReals].map(
|
||||
([path, count]) => `${path}: ${negativeZeroRealLimitation(count)}`,
|
||||
),
|
||||
...omittedArchives.map(
|
||||
({ path, reason }) =>
|
||||
`${path}: omitted because ${reason.replaceAll("_", " ")}.`,
|
||||
),
|
||||
...omittedArchives.flatMap(({ path, reason, limitations }) => [
|
||||
`${path}: omitted because ${reason.replaceAll("_", " ")}.`,
|
||||
...limitations.map((limitation) => `${path}: ${limitation}`),
|
||||
]),
|
||||
...(unreportedOmittedArchives === 0
|
||||
? []
|
||||
: [
|
||||
|
||||
@@ -2,7 +2,7 @@ import { createHash } from "node:crypto";
|
||||
import { mkdir, realpath, rename, symlink, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import {
|
||||
CPU,
|
||||
@@ -31,12 +31,75 @@ import {
|
||||
readMachoImage,
|
||||
} from "./MachoLoadCommandReader.js";
|
||||
import { traceDylibResolution } from "./DylibResolutionReader.js";
|
||||
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
|
||||
import { createTestTempDirectory } from "../../../tests/fixtures/temporaryDirectory.js";
|
||||
import { javaClassFile } from "../../../tests/fixtures/javaClassFile.js";
|
||||
|
||||
const read = (bytes: Uint8Array) =>
|
||||
readMachoImage(readerOf(bytes), bytes.length);
|
||||
|
||||
it("retains unknown executable-root traversal after a complete callback and keeps known image facts", async () => {
|
||||
const root = await createTestTempDirectory("rea-dylib-traversal-callback-");
|
||||
const bytes = machoImage({ commands: [buildVersionCommand(1)] });
|
||||
const path = join(root, "main");
|
||||
await writeFile(path, bytes);
|
||||
const digest = createHash("sha256").update(bytes).digest("hex");
|
||||
const reason = "Controlled producer executable-root uncertainty";
|
||||
const entries = DirectoryArtifactReader.prototype.entries;
|
||||
const spy = vi
|
||||
.spyOn(DirectoryArtifactReader.prototype, "entries")
|
||||
.mockImplementation(async function* (
|
||||
this: DirectoryArtifactReader,
|
||||
signal,
|
||||
options,
|
||||
) {
|
||||
yield* entries.call(this, signal, options);
|
||||
if (typeof options === "function") return;
|
||||
options?.onCoverage?.({
|
||||
status: "unknown",
|
||||
examinedEntries: 1,
|
||||
limitations: [reason],
|
||||
});
|
||||
options?.onCoverage?.({
|
||||
status: "complete",
|
||||
examinedEntries: 1,
|
||||
limitations: [],
|
||||
});
|
||||
});
|
||||
try {
|
||||
const trace = await traceDylibResolution({
|
||||
rootPath: root,
|
||||
targetPath: path,
|
||||
targetSha256: digest,
|
||||
enumerateRoots: true,
|
||||
parameters: {},
|
||||
});
|
||||
expect(trace.coverage).toMatchObject({
|
||||
status: "partial",
|
||||
unparsed_images: [],
|
||||
roots_without_architecture: [],
|
||||
});
|
||||
expect(trace.roots).toEqual([{ image: "main", architecture: "arm64" }]);
|
||||
expect(trace.images).toContainEqual(
|
||||
expect.objectContaining({
|
||||
path: "main",
|
||||
sha256: digest,
|
||||
parse_status: "parsed",
|
||||
slices: [
|
||||
expect.objectContaining({
|
||||
architecture: "arm64",
|
||||
file_type: "execute",
|
||||
platform: 1,
|
||||
}),
|
||||
],
|
||||
}),
|
||||
);
|
||||
expect(trace.limitations).toContain(reason);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
// Legacy `LC_VERSION_MIN_*` commands carry no simulator flag: per dyld
|
||||
// rules Intel images are simulator, ARM images are device (tvOS simulator
|
||||
// is x86_64 only). A legacy simulator executable must not fail resolution
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
type ArtifactReader,
|
||||
type ArtifactReaderTraversal,
|
||||
} from "../ArtifactReader.js";
|
||||
import { ArtifactTraversalObservation } from "../ArtifactTraversalObservation.js";
|
||||
import { abortIfNeeded } from "../ArtifactHash.js";
|
||||
import { DirectoryArtifactReader } from "../DirectoryArtifactReader.js";
|
||||
import { SafeOutputTree } from "../SafeOutputTree.js";
|
||||
@@ -261,11 +262,17 @@ const materializeSelection = async ({
|
||||
if (cause instanceof SafeOutputTreeCreationFailure) output = cause.tree;
|
||||
throw cause;
|
||||
}
|
||||
const traversal = await writeSelectedEntries({
|
||||
const traversal = new ArtifactTraversalObservation();
|
||||
traversal.onCoverage({
|
||||
status: inventory.coverage.status,
|
||||
limitations: inventory.limitations,
|
||||
});
|
||||
await writeSelectedEntries({
|
||||
reader,
|
||||
output,
|
||||
byPath,
|
||||
extracted,
|
||||
traversal,
|
||||
signal,
|
||||
});
|
||||
localReaderOwner = undefined;
|
||||
@@ -282,7 +289,7 @@ const materializeSelection = async ({
|
||||
inventory,
|
||||
selected,
|
||||
extracted,
|
||||
traversal,
|
||||
traversal.snapshot(),
|
||||
reader.provenance(),
|
||||
);
|
||||
await output.commit(signal);
|
||||
@@ -314,29 +321,18 @@ const writeSelectedEntries = async ({
|
||||
output,
|
||||
byPath,
|
||||
extracted,
|
||||
traversal,
|
||||
signal,
|
||||
}: {
|
||||
readonly reader: ArtifactReader;
|
||||
readonly output: SafeOutputTree;
|
||||
readonly byPath: Map<string, SelectedOccurrence>;
|
||||
readonly extracted: ExtractedOccurrence[];
|
||||
readonly traversal: ArtifactTraversalObservation;
|
||||
readonly signal: AbortSignal | undefined;
|
||||
}): Promise<ArtifactReaderTraversal> => {
|
||||
}): Promise<void> => {
|
||||
const registry = new ArtifactPathRegistry();
|
||||
let status: ArtifactReaderTraversal["status"] = "complete";
|
||||
const limitations: string[] = [];
|
||||
for await (const entry of reader.entries(signal, {
|
||||
onCoverage: (coverage) => {
|
||||
if (coverage.status !== "complete") status = coverage.status;
|
||||
limitations.push(...coverage.limitations);
|
||||
},
|
||||
onOmission: ({ path, kind }) => {
|
||||
status = "unknown";
|
||||
limitations.push(
|
||||
`Materialization traversal omitted ${kind} at ${JSON.stringify(path)}.`,
|
||||
);
|
||||
},
|
||||
})) {
|
||||
for await (const entry of reader.entries(signal, traversal)) {
|
||||
const path = normalizeArtifactPath(entry.path);
|
||||
registry.add(path, entry.kind);
|
||||
// Logical declarations may share a query path with a real file. They do
|
||||
@@ -374,7 +370,6 @@ const writeSelectedEntries = async ({
|
||||
"integrity",
|
||||
`Inventoried regular artifact entries were not materialized: ${[...byPath.keys()].sort(compareUnicodeCodePoints).join(", ")}`,
|
||||
);
|
||||
return { status, limitations };
|
||||
};
|
||||
|
||||
const releaseExtractionOwners = async (
|
||||
@@ -438,24 +433,20 @@ const createExtractionResult = (
|
||||
provenance: [...inventory.provenance, ...materializationProvenance],
|
||||
coverage: {
|
||||
...inventory.coverage,
|
||||
status:
|
||||
inventory.coverage.status === "complete" &&
|
||||
traversal.status === "complete"
|
||||
? "complete"
|
||||
: "unknown",
|
||||
status: traversal.status,
|
||||
},
|
||||
integrity_contradictions: integrityContradictions,
|
||||
limitations: [
|
||||
...inventory.limitations,
|
||||
...traversal.limitations.filter(
|
||||
(limitation) => !inventory.limitations.includes(limitation),
|
||||
),
|
||||
...traversal.limitations,
|
||||
...selected.flatMap(({ occurrence }) =>
|
||||
occurrence.limitations.map(
|
||||
(limitation) => `${occurrence.logical_path}: ${limitation}`,
|
||||
),
|
||||
),
|
||||
"All selected inventoried regular files exposed by the active reader were materialized; nested archive contents remain represented by their containing file. Publication does not establish complete source traversal or content decoding.",
|
||||
traversal.status === "complete"
|
||||
? "All regular files in the active artifact were materialized; nested archive contents remain represented by their containing file."
|
||||
: "All inventoried regular files exposed by the active reader were materialized; source traversal coverage is unknown, so the output is not established as a complete source copy. Nested archive contents remain represented by their containing file.",
|
||||
"Publication verifies the selected inventoried regular files; it does not establish complete content decoding.",
|
||||
...(integrityContradictions.length === 0
|
||||
? []
|
||||
: [
|
||||
|
||||
@@ -16,11 +16,10 @@ import {
|
||||
type ArtifactEntry,
|
||||
type ArtifactReader,
|
||||
type ArtifactReaderTraversal,
|
||||
type ArtifactTraversalOmission,
|
||||
} from "../ArtifactReader.js";
|
||||
import { ArtifactTraversalObservation } from "../ArtifactTraversalObservation.js";
|
||||
import { AsarArtifactReader } from "../AsarArtifactReader.js";
|
||||
import type { ArtifactNode } from "../../domain/artifactGraph.js";
|
||||
import { compareUnicodeCodePoints } from "../../domain/unicodeCodePointOrder.js";
|
||||
import {
|
||||
classifyArtifactContent,
|
||||
classifyArtifactBytes,
|
||||
@@ -65,8 +64,7 @@ interface ScanContext {
|
||||
readonly registry: ArtifactPathRegistry;
|
||||
readonly expandedContainerIds: Set<string>;
|
||||
readonly ownedReaders: ArtifactReader[];
|
||||
readonly omissions: ArtifactTraversalOmission[];
|
||||
readonly coverage: ArtifactReaderTraversal[];
|
||||
readonly traversal: ArtifactTraversalObservation;
|
||||
}
|
||||
|
||||
export const scanReader = async (
|
||||
@@ -105,8 +103,7 @@ export const scanReader = async (
|
||||
registry: new ArtifactPathRegistry(),
|
||||
expandedContainerIds: new Set<string>(),
|
||||
ownedReaders,
|
||||
omissions: [],
|
||||
coverage: [],
|
||||
traversal: new ArtifactTraversalObservation(),
|
||||
};
|
||||
await visitArtifactEntries(context, reader, "");
|
||||
// Archive directories may appear after their children. Resolve containment
|
||||
@@ -118,30 +115,17 @@ export const scanReader = async (
|
||||
context.expandedContainerIds,
|
||||
);
|
||||
const structuralLimitations = [...parentResolution.limitations];
|
||||
context.traversal.onCoverage({
|
||||
status: structuralLimitations.length === 0 ? "complete" : "unknown",
|
||||
limitations: structuralLimitations,
|
||||
});
|
||||
return {
|
||||
nodes,
|
||||
occurrences,
|
||||
pendingContradictions,
|
||||
caseCollisionLimitation: annotateEnglishUnicodeCaseCollisions(occurrences),
|
||||
unknownParentOccurrences: parentResolution.unknownParentOccurrences,
|
||||
traversal: {
|
||||
status:
|
||||
context.omissions.length === 0 &&
|
||||
structuralLimitations.length === 0 &&
|
||||
context.coverage.every((value) => value.status === "complete")
|
||||
? "complete"
|
||||
: "unknown",
|
||||
limitations: [
|
||||
...new Set([
|
||||
...context.omissions.map(
|
||||
({ path, kind }) =>
|
||||
`Artifact traversal omitted ${kind} at ${JSON.stringify(path)}.`,
|
||||
),
|
||||
...structuralLimitations,
|
||||
...context.coverage.flatMap((value) => value.limitations),
|
||||
]),
|
||||
].sort(compareUnicodeCodePoints),
|
||||
},
|
||||
traversal: context.traversal.snapshot(),
|
||||
};
|
||||
};
|
||||
|
||||
@@ -152,13 +136,12 @@ const inventoryEntries = (
|
||||
): AsyncIterator<ArtifactEntry> =>
|
||||
reader
|
||||
.entries(context.signal, {
|
||||
onCoverage: (coverage) => context.coverage.push(coverage),
|
||||
onCoverage: context.traversal.onCoverage,
|
||||
onOmission: (omission) =>
|
||||
context.omissions.push({
|
||||
context.traversal.onOmission({
|
||||
...omission,
|
||||
path: normalizeArtifactPath(
|
||||
path:
|
||||
prefix.length === 0 ? omission.path : `${prefix}/${omission.path}`,
|
||||
),
|
||||
}),
|
||||
})
|
||||
[Symbol.asyncIterator]();
|
||||
|
||||
@@ -15,7 +15,7 @@ export const MODEL_TOOL_CONTRACTS = [
|
||||
...toolContractMetadata("inspect_model_artifact"),
|
||||
kind: "artifact-provider",
|
||||
description:
|
||||
"Inspect local GGUF v2/v3 or safetensors declarations by content, without loading models or executing code. Default hash header reads only the header: full-file SHA-256 and Evidence subject are unknown; artifact.header_sha256 is a scoped prefix commitment, not weight identity. hash full additionally streams all file bytes for SHA-256. Optional view returns a selected metadata/tensor/type page plus model context in this call. Summary retains complete Evidence for inspect_analysis_view kind model-artifact, collection metadata (keys/indices/fields) or tensors (names/types/indices/fields), with stable header-scoped indices and paging. Metadata arrays retain eight-element typed previews; scalar metadata is complete. Tensor counts, shapes, types and offsets are observed; parameter/storage totals are derived from declarations. Unknown GGUF types retain IDs with unknown byte lengths. Tensor payload, model behavior and inference quality are not inspected. One active inspection, 100 MB header ceiling and 32 MiB retained-facts budget; resource refusals are explicit. No network, external files, Python, model runtime, extraction or deserialization of pickle.",
|
||||
"Inspect local GGUF v2/v3 or safetensors declarations by content, without loading models or executing code. Default hash header reads only the header: full-file SHA-256 and Evidence subject are unknown; artifact.header_sha256 is a scoped prefix commitment, not weight identity. hash full additionally streams all file bytes for SHA-256. Optional view returns a selected metadata/tensor/type page plus model context in this call. Summary retains complete Evidence for inspect_analysis_view kind model-artifact, collection metadata (keys/indices/fields) or tensors (names/types/indices/fields), with stable header-scoped indices and paging. Metadata arrays retain eight-element typed previews; scalar metadata is complete. Tensor counts, shapes, numeric type IDs and offsets are observed; textual types and parameter/storage totals use established format facts. Unknown or historically ambiguous GGUF types retain IDs with unknown nonempty byte lengths; ambiguous identities use AMBIGUOUS_<id> labels. Unresolved Q8_1 and K-block layouts also keep nonempty byte lengths unknown. Tensor payload, model behavior and inference quality are not inspected. One active inspection, 100 MB header ceiling and 32 MiB retained-facts budget; resource refusals are explicit. No network, external files, Python, model runtime, extraction or deserialization of pickle.",
|
||||
inputSchema: inspectModelArtifactInputSchema,
|
||||
outputSchema: evidenceResultOf(
|
||||
z.union([
|
||||
|
||||
@@ -648,6 +648,12 @@ const traceProcess = async (
|
||||
return edges;
|
||||
};
|
||||
|
||||
/** Observed completeness of deriving executable roots from a directory walk. */
|
||||
export interface DylibRootEnumeration {
|
||||
readonly status: "complete" | "unknown";
|
||||
readonly limitations: readonly string[];
|
||||
}
|
||||
|
||||
const projectTraceResult = (input: {
|
||||
readonly images: ReadonlyMap<string, MachoImageFacts>;
|
||||
readonly roots: DylibTrace["roots"];
|
||||
@@ -657,6 +663,7 @@ const projectTraceResult = (input: {
|
||||
readonly environmentRoots: ReadonlySet<string>;
|
||||
readonly unknownPlatformCandidates: ReadonlySet<string>;
|
||||
readonly implicitMacOSPlatformCandidates: ReadonlySet<string>;
|
||||
readonly rootEnumeration: DylibRootEnumeration | undefined;
|
||||
}): DylibTrace => {
|
||||
const reported = [...input.images.entries()]
|
||||
.filter(([, facts]) => facts.status !== "not-mach-o")
|
||||
@@ -686,7 +693,8 @@ const projectTraceResult = (input: {
|
||||
unparsed.length === 0 &&
|
||||
input.withoutArchitecture.length === 0 &&
|
||||
input.environmentRoots.size === 0 &&
|
||||
input.unknownPlatformCandidates.size === 0
|
||||
input.unknownPlatformCandidates.size === 0 &&
|
||||
input.rootEnumeration?.status !== "unknown"
|
||||
? "complete"
|
||||
: "partial",
|
||||
unparsed_images: unparsed,
|
||||
@@ -694,6 +702,7 @@ const projectTraceResult = (input: {
|
||||
},
|
||||
limitations: [
|
||||
...DYLIB_RESOLUTION_LIMITATIONS,
|
||||
...(input.rootEnumeration?.limitations ?? []),
|
||||
...(input.environmentRoots.size === 0
|
||||
? []
|
||||
: [
|
||||
@@ -720,6 +729,7 @@ export const traceDylibLoading = async (
|
||||
readonly roots: readonly string[];
|
||||
/** Mach-O files whose root role could not be classified because they did not parse. */
|
||||
readonly unclassified?: readonly string[];
|
||||
readonly rootEnumeration?: DylibRootEnumeration;
|
||||
readonly architecture?: string;
|
||||
readonly signal?: AbortSignal;
|
||||
},
|
||||
@@ -796,6 +806,7 @@ export const traceDylibLoading = async (
|
||||
environmentRoots,
|
||||
unknownPlatformCandidates,
|
||||
implicitMacOSPlatformCandidates,
|
||||
rootEnumeration: request.rootEnumeration,
|
||||
});
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,332 @@
|
||||
import { expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
buildInterfaceBuilderAnalysis,
|
||||
interfaceBuilderLimitsSchema,
|
||||
} from "./interfaceBuilderGraph.js";
|
||||
import { parseInterfaceBuilderRecords } from "./interfaceBuilderKeyedArchive.js";
|
||||
|
||||
const uid = (value: number) => ({ UID: value });
|
||||
|
||||
const archive = (
|
||||
fields: Record<string, unknown> = {},
|
||||
top: Record<string, unknown> = { root: uid(1) },
|
||||
) => ({
|
||||
$archiver: "NSKeyedArchiver",
|
||||
$version: 100000,
|
||||
$objects: [
|
||||
"$null",
|
||||
{ $class: uid(4), NSSubviews: [uid(2)], ...fields },
|
||||
{ $class: uid(4) },
|
||||
{
|
||||
$class: uid(5),
|
||||
NSSource: uid(2),
|
||||
NSDestination: uid(1),
|
||||
NSLabel: "parent",
|
||||
},
|
||||
{ $classname: "NSView", $classes: ["NSView", "NSObject"] },
|
||||
{
|
||||
$classname: "NSNibOutletConnector",
|
||||
$classes: ["NSNibOutletConnector", "NSNibConnector", "NSObject"],
|
||||
},
|
||||
{ $class: uid(7), "NS.objects": [uid(1)] },
|
||||
{ $classname: "NSArray", $classes: ["NSArray", "NSObject"] },
|
||||
] as unknown[],
|
||||
$top: top,
|
||||
});
|
||||
|
||||
it.each([
|
||||
["id", "2"],
|
||||
["id", "record-17"],
|
||||
["recordID", "2"],
|
||||
["archiveUID", 2],
|
||||
["objectID", "2"],
|
||||
["object-id", "2"],
|
||||
["objectID", "record-17"],
|
||||
["object-id", "record-17"],
|
||||
])(
|
||||
"uses one inventory/reference/hierarchy identity for the authored %s field %s",
|
||||
(field, value) => {
|
||||
const parsed = parseInterfaceBuilderRecords(archive({ [field]: value }));
|
||||
expect(parsed.objects.map(({ id }) => id)).toEqual(["1", "2", "3", "6"]);
|
||||
expect(parsed.objects[0]?.attributes).toMatchObject({ [field]: value });
|
||||
expect(parsed.hierarchy).toEqual([
|
||||
{ objectID: "1", children: [{ objectID: "2", children: [] }] },
|
||||
]);
|
||||
expect(parsed.connections).toMatchObject([
|
||||
{ source_id: "2", destination_id: "1" },
|
||||
]);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([0, 8])(
|
||||
"resolves authored objectID UID %s as data without changing identity",
|
||||
(index) => {
|
||||
const source = archive({ objectID: uid(index) });
|
||||
source.$objects.push("archived-id");
|
||||
const parsed = parseInterfaceBuilderRecords(source);
|
||||
expect(parsed.objects[0]).toMatchObject({
|
||||
id: "1",
|
||||
attributes: { objectID: index === 0 ? null : "archived-id" },
|
||||
});
|
||||
expect(parsed.hierarchy).toEqual([
|
||||
{ objectID: "1", children: [{ objectID: "2", children: [] }] },
|
||||
]);
|
||||
expect(parsed.connections).toMatchObject([
|
||||
{ source_id: "2", destination_id: "1" },
|
||||
]);
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["objectID", "object-id"])(
|
||||
"keeps distinct graph nodes and links despite duplicate authored %s values",
|
||||
(field) => {
|
||||
const source = archive({ [field]: uid(8), title: "Parent" });
|
||||
source.$objects[2] = { $class: uid(4), [field]: uid(8), title: "Child" };
|
||||
// The authored string also collides with the child's numeric UID spelling.
|
||||
source.$objects.push("2");
|
||||
const result = buildInterfaceBuilderAnalysis({
|
||||
targetSha256: "a".repeat(64),
|
||||
toolVersion: "test",
|
||||
documents: [
|
||||
{
|
||||
relativePath: "Main.nib",
|
||||
archiveSha256: "b".repeat(64),
|
||||
documentKind: "nib",
|
||||
raw: source,
|
||||
},
|
||||
],
|
||||
limits: interfaceBuilderLimitsSchema.parse({}),
|
||||
});
|
||||
const byObjectId = new Map(
|
||||
result.graph.nodes.map((node) => [
|
||||
node.attributes.interface_builder_object_id,
|
||||
node,
|
||||
]),
|
||||
);
|
||||
expect(byObjectId.get("1")).toMatchObject({
|
||||
name: "Parent",
|
||||
attributes: { [field]: "2" },
|
||||
});
|
||||
expect(byObjectId.get("2")).toMatchObject({
|
||||
name: "Child",
|
||||
attributes: { [field]: "2" },
|
||||
});
|
||||
expect(result.graph.edges).toContainEqual(
|
||||
expect.objectContaining({
|
||||
from: byObjectId.get("1")?.id,
|
||||
to: byObjectId.get("2")?.id,
|
||||
relation: "contains",
|
||||
resolution: "observed",
|
||||
}),
|
||||
);
|
||||
const outlet = result.graph.nodes.find(({ kind }) => kind === "outlet");
|
||||
expect(result.graph.edges).toContainEqual(
|
||||
expect.objectContaining({
|
||||
from: byObjectId.get("2")?.id,
|
||||
to: outlet?.id,
|
||||
relation: "contains",
|
||||
}),
|
||||
);
|
||||
expect(result.graph.edges).toContainEqual(
|
||||
expect.objectContaining({
|
||||
from: outlet?.id,
|
||||
to: byObjectId.get("1")?.id,
|
||||
relation: "outlet_to",
|
||||
resolution: "observed",
|
||||
}),
|
||||
);
|
||||
expect(result.documents[0]).toMatchObject({
|
||||
object_count: 4,
|
||||
hierarchy_complete: true,
|
||||
});
|
||||
expect(result.graph.truncated).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
"2",
|
||||
{ objectID: "2" },
|
||||
{ "object-id": "2" },
|
||||
{ id: "2" },
|
||||
{ archiveUID: 2 },
|
||||
{ archiveUID: 2, className: "NSView" },
|
||||
{ UID: 2, archiveUID: 2 },
|
||||
{ UID: "2" },
|
||||
uid(-1),
|
||||
uid(99),
|
||||
uid(0),
|
||||
uid(4), // A class descriptor is not an archived connection endpoint.
|
||||
])("does not interpret a non-object-UID connector endpoint: %j", (endpoint) => {
|
||||
const source = archive();
|
||||
source.$objects[3] = {
|
||||
$class: uid(5),
|
||||
NSSource: endpoint,
|
||||
NSDestination: uid(1),
|
||||
};
|
||||
const parsed = parseInterfaceBuilderRecords(source);
|
||||
expect(parsed.connections).toEqual([]);
|
||||
expect(parsed.unresolvedConnections).toBe(1);
|
||||
|
||||
source.$objects[3] = {
|
||||
$class: uid(5),
|
||||
NSSource: uid(2),
|
||||
NSDestination: endpoint,
|
||||
};
|
||||
expect(parseInterfaceBuilderRecords(source).connections).toMatchObject([
|
||||
{ source_id: "2", destination_id: null },
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["source", "destination"],
|
||||
["from", "to"],
|
||||
["owner", "target"],
|
||||
])(
|
||||
"keeps AppKit %s/%s role aliases only for raw UIDs",
|
||||
(sourceField, destinationField) => {
|
||||
const source = archive();
|
||||
source.$objects[3] = {
|
||||
$class: uid(5),
|
||||
[sourceField]: uid(2),
|
||||
[destinationField]: uid(1),
|
||||
};
|
||||
expect(parseInterfaceBuilderRecords(source).connections).toMatchObject([
|
||||
{ source_id: "2", destination_id: "1" },
|
||||
]);
|
||||
source.$objects[3] = {
|
||||
$class: uid(5),
|
||||
[sourceField]: "2",
|
||||
[destinationField]: "1",
|
||||
};
|
||||
expect(parseInterfaceBuilderRecords(source).unresolvedConnections).toBe(1);
|
||||
},
|
||||
);
|
||||
|
||||
it("uses a real UIKit UID even when a role alias looks like a generated stub", () => {
|
||||
const source = archive();
|
||||
source.$objects[5] = {
|
||||
$classname: "UIRuntimeOutletConnection",
|
||||
$classes: ["UIRuntimeOutletConnection", "NSObject"],
|
||||
};
|
||||
source.$objects[3] = {
|
||||
$class: uid(5),
|
||||
UISource: uid(2),
|
||||
UIDestination: uid(1),
|
||||
source: { archiveUID: 1 },
|
||||
};
|
||||
expect(parseInterfaceBuilderRecords(source).connections).toMatchObject([
|
||||
{ source_id: "2", destination_id: "1" },
|
||||
]);
|
||||
source.$objects[3] = {
|
||||
$class: uid(5),
|
||||
UISource: { archiveUID: 2 },
|
||||
UIDestination: uid(1),
|
||||
source: uid(2),
|
||||
};
|
||||
expect(parseInterfaceBuilderRecords(source).unresolvedConnections).toBe(1);
|
||||
});
|
||||
|
||||
it("keeps a dangling class's owned object UID as a connector endpoint", () => {
|
||||
const source = archive();
|
||||
source.$objects[2] = { $class: uid(99) };
|
||||
expect(parseInterfaceBuilderRecords(source).connections).toMatchObject([
|
||||
{ source_id: "2", destination_id: "1" },
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ objectID: "1", children: [uid(2)] },
|
||||
{ "object-id": "1", children: [uid(2)] },
|
||||
{ archiveUID: 1, className: "NSView", children: [uid(2)] },
|
||||
])("marks unsupported inline hierarchy identity incomplete: %j", (inline) => {
|
||||
const source = archive({}, { root: inline });
|
||||
const parsed = parseInterfaceBuilderRecords(source);
|
||||
expect(parsed.hierarchy).toEqual([{ objectID: "2", children: [] }]);
|
||||
expect(parsed.omittedHierarchyReferences).toBe(1);
|
||||
const result = buildInterfaceBuilderAnalysis({
|
||||
targetSha256: "a".repeat(64),
|
||||
toolVersion: "test",
|
||||
documents: [
|
||||
{
|
||||
relativePath: "Main.nib",
|
||||
archiveSha256: "b".repeat(64),
|
||||
documentKind: "nib",
|
||||
raw: source,
|
||||
},
|
||||
],
|
||||
limits: interfaceBuilderLimitsSchema.parse({}),
|
||||
});
|
||||
expect(result.documents[0]?.hierarchy_complete).toBe(false);
|
||||
expect(result.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "hierarchy:Main.nib",
|
||||
status: "partial",
|
||||
reason: "archive_hierarchy_references_missing",
|
||||
examined: 1,
|
||||
omitted: 1,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("selects the named UIKit top-level collection before a generic root alias", () => {
|
||||
const parsed = parseInterfaceBuilderRecords(
|
||||
archive(
|
||||
{},
|
||||
{
|
||||
UINibTopLevelObjectsKey: uid(6),
|
||||
root: uid(2),
|
||||
},
|
||||
),
|
||||
);
|
||||
expect(parsed.hierarchy).toEqual([
|
||||
{ objectID: "1", children: [{ objectID: "2", children: [] }] },
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ UINibTopLevelObjectsKey: "opaque", root: uid(1) },
|
||||
{ UINibTopLevelObjectsKey: null, root: uid(1) },
|
||||
{ UITopLevelObjectsKey: uid(6) },
|
||||
])("keeps an unsupported named root incomplete: %j", (top) => {
|
||||
const result = buildInterfaceBuilderAnalysis({
|
||||
targetSha256: "a".repeat(64),
|
||||
toolVersion: "test",
|
||||
documents: [
|
||||
{
|
||||
relativePath: "Main.nib",
|
||||
archiveSha256: "b".repeat(64),
|
||||
documentKind: "nib",
|
||||
raw: archive({}, top),
|
||||
},
|
||||
],
|
||||
limits: interfaceBuilderLimitsSchema.parse({}),
|
||||
});
|
||||
expect(result.documents[0]?.hierarchy_complete).toBe(false);
|
||||
expect(result.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "hierarchy:Main.nib",
|
||||
status: "unsupported",
|
||||
reason: "archive_hierarchy_not_decoded",
|
||||
examined: 0,
|
||||
omitted: 0,
|
||||
}),
|
||||
);
|
||||
expect(
|
||||
result.graph.edges.filter(({ id }) => id.includes(":hierarchy:")),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it("counts a dangling named root without substituting a valid generic root", () => {
|
||||
const parsed = parseInterfaceBuilderRecords(
|
||||
archive(
|
||||
{},
|
||||
{
|
||||
UINibTopLevelObjectsKey: uid(99),
|
||||
root: uid(1),
|
||||
},
|
||||
),
|
||||
);
|
||||
expect(parsed.hierarchy).toEqual([]);
|
||||
expect(parsed.omittedHierarchyReferences).toBe(1);
|
||||
});
|
||||
@@ -245,14 +245,19 @@ const keyedArchiveTable = (objectTable: readonly unknown[]) => {
|
||||
? text
|
||||
: undefined;
|
||||
};
|
||||
const text = (value: unknown): string | null => {
|
||||
const uid = archiveUid(value);
|
||||
return firstString(
|
||||
uid === undefined ? value : archivedString(objectTable[uid]),
|
||||
);
|
||||
/** Only an encoded UID proves which archived object a connector names. */
|
||||
const objectReference = (...values: unknown[]): string | null => {
|
||||
for (const value of values) {
|
||||
const uid = archiveUid(value);
|
||||
if (
|
||||
uid !== undefined &&
|
||||
uid !== 0 &&
|
||||
Object.hasOwn(record(objectTable[uid]), "$class")
|
||||
)
|
||||
return String(uid);
|
||||
}
|
||||
return null;
|
||||
};
|
||||
const authoredId = (item: Record<string, unknown>): string | null =>
|
||||
firstString(text(item.objectID), text(item["object-id"]));
|
||||
const reference = (uid: number): unknown => {
|
||||
// UID 0 is the archived nil.
|
||||
const target = uid === 0 ? null : objectTable[uid];
|
||||
@@ -260,10 +265,8 @@ const keyedArchiveTable = (objectTable: readonly unknown[]) => {
|
||||
if (typeof target !== "object") return target;
|
||||
const string = archivedString(target);
|
||||
if (string !== undefined) return string;
|
||||
const objectId = authoredId(record(target));
|
||||
return {
|
||||
archiveUID: uid,
|
||||
...(objectId === null ? {} : { objectID: objectId }),
|
||||
className: classNameOf(target),
|
||||
};
|
||||
};
|
||||
@@ -291,21 +294,11 @@ const keyedArchiveTable = (objectTable: readonly unknown[]) => {
|
||||
];
|
||||
const appendNode = (
|
||||
item: Record<string, unknown>,
|
||||
uid: number | undefined,
|
||||
uid: number,
|
||||
output: unknown[],
|
||||
): void => {
|
||||
const objectId = firstString(
|
||||
authoredId(item),
|
||||
text(item.id),
|
||||
uid === undefined ? null : String(uid),
|
||||
);
|
||||
if (objectId === null) {
|
||||
for (const [key, child] of Object.entries(item).reverse())
|
||||
if (HIERARCHY_KEY.test(key)) pending.push({ value: child, output });
|
||||
return;
|
||||
}
|
||||
const children: unknown[] = [];
|
||||
output.push({ objectID: objectId, children });
|
||||
output.push({ objectID: String(uid), children });
|
||||
for (const [key, child] of Object.entries(item).reverse())
|
||||
if (HIERARCHY_KEY.test(key))
|
||||
pending.push({ value: child, output: children });
|
||||
@@ -352,7 +345,11 @@ const keyedArchiveTable = (objectTable: readonly unknown[]) => {
|
||||
if (child !== undefined) pending.push({ value: child, output });
|
||||
}
|
||||
} else if (typeof value === "object" && value !== null) {
|
||||
appendNode(record(value), undefined, output);
|
||||
// Inline dictionaries have no object-table identity. Preserve any
|
||||
// independently referenced children without inventing a parent ID.
|
||||
omittedReferences += 1;
|
||||
for (const [key, child] of Object.entries(value).reverse())
|
||||
if (HIERARCHY_KEY.test(key)) pending.push({ value: child, output });
|
||||
}
|
||||
}
|
||||
return { items: root, omittedReferences };
|
||||
@@ -381,6 +378,7 @@ const keyedArchiveTable = (objectTable: readonly unknown[]) => {
|
||||
hasClassSuperclass,
|
||||
classChainOf,
|
||||
classTable,
|
||||
objectReference,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -398,6 +396,7 @@ const parseKeyedArchive = (
|
||||
hasClassSuperclass,
|
||||
classChainOf,
|
||||
classTable,
|
||||
objectReference,
|
||||
} = keyedArchiveTable(objectTable);
|
||||
const objects: InterfaceBuilderObject[] = [];
|
||||
const connections: InterfaceBuilderConnection[] = [];
|
||||
@@ -416,11 +415,9 @@ const parseKeyedArchive = (
|
||||
const fields = { ...resolved };
|
||||
// Keep the unresolvable class reference itself, e.g. a dangling UID.
|
||||
if (className === null) fields.$class = record(raw).$class;
|
||||
const authoredId = firstString(fields.objectID, fields["object-id"]);
|
||||
// Object identity comes from the archive table. `archiveUID` is ordinary
|
||||
// application data when it appears on an archived object; only UID
|
||||
// reference stubs select object-table entries.
|
||||
const objectId = authoredId ?? String(index);
|
||||
// Authored objectID, object-id, id and archiveUID fields are data. Only
|
||||
// the original table position identifies an archived object.
|
||||
const objectId = String(index);
|
||||
const kind = classifyObject(className, classChainOf(raw));
|
||||
if (objects.length < 20_000)
|
||||
objects.push(
|
||||
@@ -450,13 +447,14 @@ const parseKeyedArchive = (
|
||||
// class-specific field namespaces; the source is the outlet owner.
|
||||
const connectionFields = interfaceBuilderConnectionFieldNames(className);
|
||||
if (isSourceLessAppKitAnnotationConnector(className, fields)) continue;
|
||||
const rawFields = record(raw);
|
||||
const source = uiKit
|
||||
? archivedObjectReference(fields[connectionFields.source])
|
||||
: firstObjectReference(
|
||||
controlAction ? fields.destination : fields.source,
|
||||
archivedObjectReference(fields[connectionFields.source]),
|
||||
controlAction ? fields.to : fields.from,
|
||||
controlAction ? fields.target : fields.owner,
|
||||
? objectReference(rawFields[connectionFields.source])
|
||||
: objectReference(
|
||||
controlAction ? rawFields.destination : rawFields.source,
|
||||
rawFields[connectionFields.source],
|
||||
controlAction ? rawFields.to : rawFields.from,
|
||||
controlAction ? rawFields.target : rawFields.owner,
|
||||
);
|
||||
if (source === null) {
|
||||
// A recognized connector that cannot name its owner is an unresolved
|
||||
@@ -471,12 +469,12 @@ const parseKeyedArchive = (
|
||||
kind,
|
||||
source_id: source,
|
||||
destination_id: uiKit
|
||||
? archivedObjectReference(fields[connectionFields.destination])
|
||||
: firstObjectReference(
|
||||
controlAction ? fields.source : fields.destination,
|
||||
archivedObjectReference(fields[connectionFields.destination]),
|
||||
controlAction ? fields.from : fields.to,
|
||||
controlAction ? fields.owner : fields.target,
|
||||
? objectReference(rawFields[connectionFields.destination])
|
||||
: objectReference(
|
||||
controlAction ? rawFields.source : rawFields.destination,
|
||||
rawFields[connectionFields.destination],
|
||||
controlAction ? rawFields.from : rawFields.to,
|
||||
controlAction ? rawFields.owner : rawFields.target,
|
||||
),
|
||||
label: uiKit
|
||||
? firstString(fields[connectionFields.label])
|
||||
@@ -494,7 +492,13 @@ const parseKeyedArchive = (
|
||||
}
|
||||
}
|
||||
const top = record(root.$top);
|
||||
const hierarchy = toHierarchy(top.root ?? top.UITopLevelObjectsKey);
|
||||
// UIKit names its top-level collection explicitly. A present but undecoded
|
||||
// named root must not be replaced by an unrelated generic root alias.
|
||||
const hierarchy = toHierarchy(
|
||||
Object.hasOwn(top, "UINibTopLevelObjectsKey")
|
||||
? top.UINibTopLevelObjectsKey
|
||||
: top.root,
|
||||
);
|
||||
const classes = classTable();
|
||||
return {
|
||||
objects,
|
||||
@@ -520,35 +524,6 @@ export const firstString = (...values: unknown[]): string | null => {
|
||||
return null;
|
||||
};
|
||||
|
||||
/**
|
||||
* The node identity of a resolved archive reference: its authored object ID,
|
||||
* otherwise its archive UID, which is how the referenced object's node is keyed.
|
||||
*/
|
||||
const archivedObjectReference = (value: unknown): string | null => {
|
||||
const item = record(value);
|
||||
if (typeof item.archiveUID !== "number") return null;
|
||||
return (
|
||||
firstString(item.objectID, item["object-id"]) ?? String(item.archiveUID)
|
||||
);
|
||||
};
|
||||
|
||||
const firstObjectReference = (...values: unknown[]): string | null => {
|
||||
for (const value of values) {
|
||||
const direct = firstString(value);
|
||||
if (direct !== null) return direct;
|
||||
const item = record(value);
|
||||
const objectId = firstString(
|
||||
item.objectID,
|
||||
item["object-id"],
|
||||
item.id,
|
||||
item.identifier,
|
||||
typeof item.archiveUID === "number" ? String(item.archiveUID) : null,
|
||||
);
|
||||
if (objectId !== null) return objectId;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
/**
|
||||
* Framework base classes whose presence in an archived `$classes` chain settles
|
||||
* an object's kind, most specific first: UIControl and NSControl are views.
|
||||
|
||||
@@ -537,12 +537,14 @@ describe("function collection collation ties", () => {
|
||||
});
|
||||
|
||||
describe("function comparison ordering controls", () => {
|
||||
const composed = procedure("0x2000", "caf\u00e9");
|
||||
const decomposed = procedure("0x3000", "cafe\u0301");
|
||||
|
||||
it("preserves duplicate Unicode collections and nullable endpoints", () => {
|
||||
const controlled = (base: "0x1000" | "0x2000", reverse: boolean) => {
|
||||
const endpoint = base === "0x1000" ? "0x1010" : "0x2010";
|
||||
const composed = procedure(endpoint, "caf\u00e9");
|
||||
const decomposed = procedure(
|
||||
base === "0x1000" ? "0x1020" : "0x2020",
|
||||
"cafe\u0301",
|
||||
);
|
||||
const edges = [
|
||||
{
|
||||
source_address: base,
|
||||
@@ -635,14 +637,10 @@ describe("function comparison ordering controls", () => {
|
||||
functionSnapshotFixture(makeEvidence("e", rightWithoutNativeApi)),
|
||||
);
|
||||
|
||||
for (const dimension of ["strings_names"] as const)
|
||||
for (const dimension of ["calls", "strings_names"] as const)
|
||||
expect(
|
||||
comparison.dimensions.find((item) => item.dimension === dimension),
|
||||
).toMatchObject({ status: "unchanged" });
|
||||
// Reference and call facets report the same name at different entries.
|
||||
expect(
|
||||
comparison.dimensions.find((item) => item.dimension === "calls"),
|
||||
).toMatchObject({ status: "unknown" });
|
||||
// Unnamed endpoints in different artifacts have no stable identity.
|
||||
expect(
|
||||
comparison.dimensions.find((item) => item.dimension === "references"),
|
||||
@@ -1417,6 +1415,128 @@ it.each([
|
||||
},
|
||||
);
|
||||
|
||||
describe("function call endpoint identity", () => {
|
||||
const calling = (
|
||||
direction: "callers" | "callees",
|
||||
entries: readonly ReturnType<typeof procedure>[],
|
||||
) =>
|
||||
functionDossierSchema.parse({
|
||||
...dossier("return helper();", "0x1000"),
|
||||
[direction]: entries,
|
||||
});
|
||||
const calls = (
|
||||
left: ReturnType<typeof dossier>,
|
||||
right: ReturnType<typeof dossier>,
|
||||
sameArtifact: boolean,
|
||||
) =>
|
||||
compareFunctions(
|
||||
functionSnapshotFixture(observe("a", left)),
|
||||
functionSnapshotFixture(observe(sameArtifact ? "a" : "b", right)),
|
||||
).dimensions.find(({ dimension }) => dimension === "calls");
|
||||
|
||||
it.each(["callers", "callees"] as const)(
|
||||
"uses known procedure entries rather than %s labels within an artifact",
|
||||
(direction) => {
|
||||
const left = calling(direction, [procedure("ram:0x1010", "helper")]);
|
||||
expect(
|
||||
calls(
|
||||
left,
|
||||
calling(direction, [procedure("ram:0x1020", "helper")]),
|
||||
true,
|
||||
),
|
||||
).toMatchObject({ status: "changed" });
|
||||
expect(
|
||||
calls(
|
||||
left,
|
||||
calling(direction, [procedure("ram:0X001010", "renamed")]),
|
||||
true,
|
||||
),
|
||||
).toMatchObject({ status: "unchanged" });
|
||||
expect(
|
||||
calls(
|
||||
left,
|
||||
calling(direction, [procedure("rom:0x1010", "helper")]),
|
||||
true,
|
||||
),
|
||||
).toMatchObject({ status: "changed" });
|
||||
expect(
|
||||
calls(
|
||||
left,
|
||||
calling(direction, [procedure("unparsed", "helper")]),
|
||||
true,
|
||||
),
|
||||
).toMatchObject({
|
||||
status: "unknown",
|
||||
limitations: [expect.stringContaining("could not be parsed")],
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["callers", "callees"] as const)(
|
||||
"keeps repeated cross-artifact %s labels ambiguous without reference edges",
|
||||
(direction) => {
|
||||
const left = calling(direction, [
|
||||
procedure("0x1010", "helper"),
|
||||
procedure("0x1020", "helper"),
|
||||
]);
|
||||
const right = calling(direction, [
|
||||
procedure("0x2010", "helper"),
|
||||
procedure("0x2020", "helper"),
|
||||
]);
|
||||
expect(calls(left, right, false)).toMatchObject({
|
||||
status: "unknown",
|
||||
left_count: 2,
|
||||
right_count: 2,
|
||||
left_digest: null,
|
||||
right_digest: null,
|
||||
limitations: [
|
||||
expect.stringContaining("unique comparable procedure entry"),
|
||||
],
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("retains duplicate counts when a name resolves to one entry per artifact", () => {
|
||||
const left = calling("callees", [
|
||||
procedure("0x1010", "helper"),
|
||||
procedure("0X001010", "helper"),
|
||||
]);
|
||||
const right = calling("callees", [
|
||||
procedure("0x2010", "helper"),
|
||||
procedure("0x2010", "helper"),
|
||||
]);
|
||||
expect(calls(left, right, false)).toMatchObject({
|
||||
status: "unchanged",
|
||||
left_count: 2,
|
||||
right_count: 2,
|
||||
});
|
||||
expect(
|
||||
calls(left, calling("callees", [procedure("0x2010", "helper")]), false),
|
||||
).toMatchObject({ status: "changed" });
|
||||
});
|
||||
|
||||
it("shares procedure-name ambiguity across call and reference observations", () => {
|
||||
const observed = (base: "0x1000" | "0x2000") =>
|
||||
functionDossierSchema.parse({
|
||||
...dossierWithReference(base, "call"),
|
||||
callees: [procedure(base === "0x1000" ? "0x1020" : "0x2020", "helper")],
|
||||
});
|
||||
const result = compareFunctions(
|
||||
functionSnapshotFixture(observe("a", observed("0x1000"))),
|
||||
functionSnapshotFixture(observe("b", observed("0x2000"))),
|
||||
);
|
||||
for (const dimension of ["calls", "references"])
|
||||
expect(
|
||||
result.dimensions.find((value) => value.dimension === dimension),
|
||||
).toMatchObject({
|
||||
status: "unknown",
|
||||
limitations: [
|
||||
expect.stringContaining("unique comparable procedure entry"),
|
||||
],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("function comparison unnamed reference endpoints", () => {
|
||||
it("distinguishes unnamed reference targets by address within one artifact", () => {
|
||||
const unnamedTarget = (target: string) =>
|
||||
|
||||
@@ -40,11 +40,16 @@ const imageWithSections = (): Buffer => {
|
||||
};
|
||||
|
||||
/** The same layout without section headers: symbols live in PT_DYNAMIC. */
|
||||
const strippedImageWithSections = (): Buffer => {
|
||||
const elf = buildStrippedElf64Image([
|
||||
{ name: "_kDartVmSnapshotData", value: 0x400, size: 52 },
|
||||
{ name: "_kDartIsolateSnapshotData", value: 0x800, size: 52 },
|
||||
]);
|
||||
const strippedImageWithSections = (
|
||||
extraDynamicEntries: ReadonlyArray<readonly [number, bigint]> = [],
|
||||
): Buffer => {
|
||||
const elf = buildStrippedElf64Image(
|
||||
[
|
||||
{ name: "_kDartVmSnapshotData", value: 0x400, size: 52 },
|
||||
{ name: "_kDartIsolateSnapshotData", value: 0x800, size: 52 },
|
||||
],
|
||||
extraDynamicEntries,
|
||||
);
|
||||
const section = snapshotSection(GALLERY_HASH);
|
||||
return Buffer.concat([
|
||||
elf,
|
||||
@@ -90,6 +95,38 @@ describe("readElfSymbol", () => {
|
||||
expect(header.kind).toBe(3);
|
||||
});
|
||||
|
||||
it.each([
|
||||
[16, "DT_SYMBOLIC"],
|
||||
[22, "DT_TEXTREL"],
|
||||
[24, "DT_BIND_NOW"],
|
||||
] as const)(
|
||||
"ignores the payload of %s (%s) without losing symbols",
|
||||
async (tag, _name) => {
|
||||
const names = ["_kDartVmSnapshotData", "_kDartIsolateSnapshotData"];
|
||||
const control = await readElfSymbols(
|
||||
strippedImageWithSections([[tag, 0n]]),
|
||||
names,
|
||||
);
|
||||
expect(control.failures).toEqual([]);
|
||||
expect(control.symbols).toHaveLength(2);
|
||||
const image = strippedImageWithSections([[tag, 0xffffffffffffffffn]]);
|
||||
const original = Buffer.from(image);
|
||||
expect(await readElfSymbols(image, names)).toEqual(control);
|
||||
expect(image).toEqual(original);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([4, 5, 6, 10, 11, 0x6ffffef5])(
|
||||
"keeps representability checks for consumed dynamic tag %s",
|
||||
async (tag) => {
|
||||
const image = strippedImageWithSections([[tag, 0xffffffffffffffffn]]);
|
||||
expect(await readElfSymbols(image, ["_kDartVmSnapshotData"])).toEqual({
|
||||
symbols: [],
|
||||
failures: [{ reason: "Dynamic entry is not representable" }],
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("refuses a stripped image whose PT_DYNAMIC carries no symbols", async () => {
|
||||
// Drop the PT_DYNAMIC program header by leaving only PT_LOAD.
|
||||
const image = strippedImageWithSections();
|
||||
@@ -144,6 +181,53 @@ describe("readElfSymbol", () => {
|
||||
);
|
||||
});
|
||||
|
||||
describe("dynamic symbols without a SHT_DYNSYM section", () => {
|
||||
it.each([false, true])(
|
||||
"uses PT_DYNAMIC with a retained directory (data sections: %s)",
|
||||
async (retainDataSections) => {
|
||||
const original = strippedImageWithSections();
|
||||
const image = appendElfRanges(
|
||||
original,
|
||||
"section",
|
||||
retainDataSections
|
||||
? [
|
||||
{ type: 1, offset: 0x400, size: 52 },
|
||||
{ type: 1, offset: 0x800, size: 52 },
|
||||
]
|
||||
: [],
|
||||
);
|
||||
const names = ["_kDartVmSnapshotData", "_kDartIsolateSnapshotData"];
|
||||
const control = await readElfSymbols(original, names);
|
||||
expect(control.failures).toEqual([]);
|
||||
expect(control.symbols).toHaveLength(2);
|
||||
const before = Buffer.from(image);
|
||||
expect(await readElfSymbols(image, names)).toEqual(control);
|
||||
expect(image).toEqual(before);
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps malformed present dynsym sections as failures despite usable PT_DYNAMIC", async () => {
|
||||
const image = appendElfRanges(strippedImageWithSections(), "section", [
|
||||
{ type: 11, offset: 0, size: 0 },
|
||||
]);
|
||||
expect(await readElfSymbols(image, ["_kDartVmSnapshotData"])).toEqual({
|
||||
symbols: [],
|
||||
failures: [{ reason: "Dynamic symbol table is malformed" }],
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps bounded dynamic metadata requirements with a retained directory", async () => {
|
||||
const image = appendElfRanges(strippedImageWithSections(), "section", []);
|
||||
// The first entry is DT_SYMTAB, whose address must remain representable.
|
||||
const dynamic = Number(image.readBigUInt64LE(64 + 56 + 8));
|
||||
image.writeBigUInt64LE(0xffffffffffffffffn, dynamic + 8);
|
||||
expect(await readElfSymbols(image, ["_kDartVmSnapshotData"])).toEqual({
|
||||
symbols: [],
|
||||
failures: [{ reason: "Dynamic entry is not representable" }],
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("ELF symbol-table ownership", () => {
|
||||
it.each([
|
||||
{ address: 0x400, offset: 0x700, size: 52 },
|
||||
|
||||
+35
-17
@@ -65,10 +65,12 @@ const PT_DYNAMIC = 2;
|
||||
/** Position-independent executable images carry PT_DYNAMIC. */
|
||||
const ET_DYN = 3;
|
||||
const DT_NULL = 0;
|
||||
const DT_HASH = 4;
|
||||
const DT_STRTAB = 5;
|
||||
const DT_SYMTAB = 6;
|
||||
const DT_STRSZ = 10;
|
||||
const DT_SYMENT = 11;
|
||||
const DT_GNU_HASH = 0x6ffffef5;
|
||||
|
||||
/** ELF32 needs 52 bytes, ELF64 needs 64; retain at most this prefix. */
|
||||
export const ELF_HEADER_MAX_BYTES = 64;
|
||||
@@ -115,17 +117,18 @@ export const readElfHeader = (
|
||||
count > 0 &&
|
||||
count <= Math.floor((totalBytes - offset) / width);
|
||||
const MIN_SHENT = is64 ? 64 : 40;
|
||||
// Section-stripped ET_DYN images keep their exported symbols reachable
|
||||
// through PT_DYNAMIC instead of section headers; every other shape still
|
||||
// needs a usable section header table.
|
||||
// Section-stripped ET_DYN images may omit the section directory. This
|
||||
// describes header availability, not which symbol-table source is usable.
|
||||
// A nonzero shoff with shnum=0 uses extended numbering, which this
|
||||
// parser does not support; it must not masquerade as a stripped image.
|
||||
const dynamicFallback = e_type === ET_DYN && e_shoff === 0 && e_shnum === 0;
|
||||
const sectionTableAbsent =
|
||||
e_type === ET_DYN && e_shoff === 0 && e_shnum === 0;
|
||||
if (
|
||||
e_shoff === null ||
|
||||
e_shentsize === null ||
|
||||
e_shnum === null ||
|
||||
(!dynamicFallback && !tableFits(e_shoff, e_shentsize, e_shnum, MIN_SHENT))
|
||||
(!sectionTableAbsent &&
|
||||
!tableFits(e_shoff, e_shentsize, e_shnum, MIN_SHENT))
|
||||
)
|
||||
return {
|
||||
failure: {
|
||||
@@ -149,7 +152,7 @@ export const readElfHeader = (
|
||||
return {
|
||||
header: {
|
||||
is64,
|
||||
dynamicFallback,
|
||||
sectionTableAbsent,
|
||||
e_shoff,
|
||||
e_shentsize,
|
||||
e_shnum,
|
||||
@@ -180,7 +183,7 @@ export const readElfSymbols = async (
|
||||
if (parsed.failure !== undefined) return failure(parsed.failure.reason);
|
||||
const {
|
||||
is64,
|
||||
dynamicFallback,
|
||||
sectionTableAbsent,
|
||||
e_shoff,
|
||||
e_shentsize,
|
||||
e_shnum,
|
||||
@@ -221,6 +224,7 @@ export const readElfSymbols = async (
|
||||
if (segments.length === 0)
|
||||
return failure("ELF image has no PT_LOAD segments");
|
||||
const tables: {
|
||||
source: "section" | "dynamic-segment";
|
||||
offset: number;
|
||||
count: number;
|
||||
stringOffset: number;
|
||||
@@ -248,7 +252,7 @@ export const readElfSymbols = async (
|
||||
? undefined
|
||||
: "section offset disagrees with its PT_LOAD mapping";
|
||||
};
|
||||
if (!dynamicFallback) {
|
||||
if (!sectionTableAbsent) {
|
||||
for (let index = 0; index < e_shnum; index += 1) {
|
||||
if (index % 4096 === 0) {
|
||||
await yieldToEventLoop();
|
||||
@@ -303,13 +307,18 @@ export const readElfSymbols = async (
|
||||
if (stringMapping !== undefined)
|
||||
return failure(`Dynamic symbol string table ${stringMapping}`);
|
||||
tables.push({
|
||||
source: "section",
|
||||
offset,
|
||||
count: size / entrySize,
|
||||
stringOffset,
|
||||
stringSize,
|
||||
});
|
||||
}
|
||||
} else if (dynamicSegment !== null) {
|
||||
}
|
||||
// A retained section directory may omit SHT_DYNSYM while PT_DYNAMIC still
|
||||
// describes the runtime symbols. A malformed present section already failed
|
||||
// above; it must not be bypassed by an alternative interpretation.
|
||||
if (tables.length === 0 && dynamicSegment !== null) {
|
||||
const tags = new Map<number, number>();
|
||||
const dynamicEntrySize = is64 ? 16 : 8;
|
||||
let terminated = false;
|
||||
@@ -330,14 +339,22 @@ export const readElfSymbols = async (
|
||||
terminated = true;
|
||||
break;
|
||||
}
|
||||
const value = is64 ? reader.u64(base + 8) : reader.u32(base + 4);
|
||||
if (value === null) return failure("Dynamic entry is not representable");
|
||||
// Interpret d_un only for the symbol-table description consumed here.
|
||||
// In particular, DT_SYMBOLIC, DT_TEXTREL and DT_BIND_NOW have ignored
|
||||
// payloads under the ELF ABI; their arbitrary bits are not numeric bounds.
|
||||
if (
|
||||
![DT_SYMTAB, DT_STRTAB, DT_STRSZ, DT_SYMENT, 4, 0x6ffffef5].includes(
|
||||
tag,
|
||||
)
|
||||
![
|
||||
DT_SYMTAB,
|
||||
DT_STRTAB,
|
||||
DT_STRSZ,
|
||||
DT_SYMENT,
|
||||
DT_HASH,
|
||||
DT_GNU_HASH,
|
||||
].includes(tag)
|
||||
)
|
||||
continue;
|
||||
const value = is64 ? reader.u64(base + 8) : reader.u32(base + 4);
|
||||
if (value === null) return failure("Dynamic entry is not representable");
|
||||
if (tags.has(tag) && tags.get(tag) !== value)
|
||||
return failure("Conflicting dynamic table tags");
|
||||
tags.set(tag, value);
|
||||
@@ -358,8 +375,8 @@ export const readElfSymbols = async (
|
||||
// DT_HASH's nchain is the dynsym count. GNU hash instead records the
|
||||
// first hashed index and bounded buckets/chains. No payload-to-EOF walk.
|
||||
let count: number;
|
||||
const sysv = tags.get(4);
|
||||
const gnu = tags.get(0x6ffffef5);
|
||||
const sysv = tags.get(DT_HASH);
|
||||
const gnu = tags.get(DT_GNU_HASH);
|
||||
if (sysv !== undefined) {
|
||||
const at = toOffset(sysv, 8);
|
||||
if (!at.ok) return failure(`DT_HASH header ${at.error.reason}`);
|
||||
@@ -420,6 +437,7 @@ export const readElfSymbols = async (
|
||||
if (!stringOffset.ok)
|
||||
return failure(`Dynamic string table ${stringOffset.error.reason}`);
|
||||
tables.push({
|
||||
source: "dynamic-segment",
|
||||
offset: offset.value,
|
||||
count,
|
||||
stringOffset: stringOffset.value,
|
||||
@@ -444,7 +462,7 @@ export const readElfSymbols = async (
|
||||
sectionIndex === 0xfff1 ||
|
||||
(sectionIndex > 0 &&
|
||||
sectionIndex < 0xff00 &&
|
||||
(dynamicFallback || sectionIndex < e_shnum));
|
||||
(table.source === "dynamic-segment" || sectionIndex < e_shnum));
|
||||
if (!defined || nameOffset >= table.stringSize) continue;
|
||||
const name = reader.cString(
|
||||
table.stringOffset + nameOffset,
|
||||
|
||||
@@ -179,13 +179,14 @@ export const buildElf32Image = (
|
||||
*/
|
||||
export const buildStrippedElf64Image = (
|
||||
symbols: ReadonlyArray<{ name: string; value: number; size: number }>,
|
||||
extraDynamicEntries: ReadonlyArray<readonly [number, bigint]> = [],
|
||||
): Buffer => {
|
||||
const ELF_HEADER = 64;
|
||||
const PHENT = 56;
|
||||
const PHNUM = 2;
|
||||
const dynamicEntry = 16;
|
||||
// DT_SYMTAB, DT_STRTAB, DT_STRSZ, DT_SYMENT, DT_HASH, DT_NULL.
|
||||
const DYNAMIC_ENTRIES = 6;
|
||||
const DYNAMIC_ENTRIES = 6 + extraDynamicEntries.length;
|
||||
const phoff = ELF_HEADER;
|
||||
const dynamicOff = phoff + PHENT * PHNUM;
|
||||
const dynsymOff = dynamicOff + DYNAMIC_ENTRIES * dynamicEntry;
|
||||
@@ -228,12 +229,13 @@ export const buildStrippedElf64Image = (
|
||||
image.writeBigUInt64LE(BigInt(dynamicOff), dyn + 8); // p_offset
|
||||
image.writeBigUInt64LE(BigInt(dynamicOff), dyn + 0x10); // p_vaddr
|
||||
image.writeBigUInt64LE(BigInt(DYNAMIC_ENTRIES * dynamicEntry), dyn + 0x20);
|
||||
const pairs: ReadonlyArray<readonly [number, number]> = [
|
||||
const pairs: ReadonlyArray<readonly [number, number | bigint]> = [
|
||||
[6, dynsymOff],
|
||||
[5, dynstrOff],
|
||||
[10, strtabBytes],
|
||||
[11, 24],
|
||||
[4, hashOff],
|
||||
...extraDynamicEntries,
|
||||
];
|
||||
pairs.forEach(([tag, value], index) => {
|
||||
const base = dynamicOff + index * dynamicEntry;
|
||||
@@ -259,6 +261,38 @@ export const buildStrippedElf64Image = (
|
||||
return image;
|
||||
};
|
||||
|
||||
/** Four snapshot exports and their pool, discovered without section headers. */
|
||||
export const strippedAotFixtureImage = (
|
||||
extraDynamicEntries: ReadonlyArray<readonly [number, bigint]> = [],
|
||||
): Buffer => {
|
||||
const symbols = buildStrippedElf64Image(
|
||||
[
|
||||
{ name: "_kDartVmSnapshotData", value: 0x1000, size: 52 },
|
||||
{ name: "_kDartVmSnapshotInstructions", value: 0x1800, size: 16 },
|
||||
{ name: "_kDartIsolateSnapshotData", value: 0x2000, size: 0x600 },
|
||||
{ name: "_kDartIsolateSnapshotInstructions", value: 0x2600, size: 16 },
|
||||
],
|
||||
extraDynamicEntries,
|
||||
);
|
||||
const vmSection = snapshotSection(REFERENCE_HASH);
|
||||
const isolateSection = Buffer.concat([
|
||||
snapshotSection(REFERENCE_HASH),
|
||||
Buffer.from([0]),
|
||||
Buffer.from("package:example/main.dart\0dart:core\0Example\0"),
|
||||
]);
|
||||
return Buffer.concat([
|
||||
symbols,
|
||||
Buffer.alloc(0x1000 - symbols.length),
|
||||
vmSection,
|
||||
Buffer.alloc(0x1800 - 0x1000 - vmSection.length),
|
||||
Buffer.alloc(16),
|
||||
Buffer.alloc(0x2000 - 0x1800 - 16),
|
||||
isolateSection,
|
||||
Buffer.alloc(0x600 - isolateSection.length),
|
||||
Buffer.alloc(16),
|
||||
]);
|
||||
};
|
||||
|
||||
/** A minimal Dart snapshot data section carrying a 32-hex hash. */
|
||||
export const snapshotSection = (hash: string): Buffer =>
|
||||
Buffer.concat([
|
||||
|
||||
+31
-9
@@ -46,6 +46,7 @@ export interface GgufProgress {
|
||||
onMetadataComplete(): void;
|
||||
onTensorsStarted(): void;
|
||||
onTensorDeclaration(row: ModelArtifactPartialTensor): void;
|
||||
onLimitation(message: string): void;
|
||||
}
|
||||
|
||||
/** Explicit-endian GGUF v2/v3 metadata and tensor-directory parser. */
|
||||
@@ -249,6 +250,7 @@ export class GgufHeader {
|
||||
tensor_data_offset: number;
|
||||
metadata: ModelMetadata[];
|
||||
tensors: ModelTensor[];
|
||||
limitations: string[];
|
||||
}> {
|
||||
const versionBytes = await this.reader.take(4);
|
||||
const littleVersion = versionBytes.readUInt32LE();
|
||||
@@ -331,6 +333,12 @@ export class GgufHeader {
|
||||
progress?.onTensorsStarted();
|
||||
const tensors: ModelArtifactPartialTensor[] = [];
|
||||
const names = new Set<string>();
|
||||
const limitations = new Set<string>();
|
||||
const note = (message: string): void => {
|
||||
if (limitations.has(message)) return;
|
||||
limitations.add(message);
|
||||
progress?.onLimitation(message);
|
||||
};
|
||||
for (let index = 0; index < tensorCount; index++) {
|
||||
this.reader.guarantee((tensorCount - index) * 20);
|
||||
const start = this.reader.position;
|
||||
@@ -355,17 +363,30 @@ export class GgufHeader {
|
||||
const type = ggmlType(id);
|
||||
let length: number | null = null;
|
||||
if (type !== undefined) {
|
||||
if (BigInt(shape[0] ?? "1") % BigInt(type[1]) !== 0n)
|
||||
const [, rowElementMultiple, blockBytes, limitation] = type;
|
||||
if (BigInt(shape[0] ?? "1") % BigInt(rowElementMultiple) !== 0n)
|
||||
rejectModel(
|
||||
`GGUF tensor ${name} first dimension is not divisible by the quantization block size`,
|
||||
`GGUF tensor ${name} first dimension must be a multiple of ${String(rowElementMultiple)} elements`,
|
||||
);
|
||||
const bytes = (parameters / BigInt(type[1])) * BigInt(type[2]);
|
||||
if (bytes > BigInt(this.reader.size))
|
||||
rejectModel(
|
||||
`GGUF tensor ${name} declared byte length exceeds the file`,
|
||||
);
|
||||
length = Number(bytes);
|
||||
}
|
||||
if (
|
||||
limitation !== undefined &&
|
||||
(parameters !== 0n || limitation.kind === "type-identity")
|
||||
)
|
||||
note(limitation.reason);
|
||||
if (parameters === 0n) length = 0;
|
||||
else if (blockBytes !== null) {
|
||||
const bytes =
|
||||
(parameters / BigInt(rowElementMultiple)) * BigInt(blockBytes);
|
||||
if (bytes > BigInt(this.reader.size))
|
||||
rejectModel(
|
||||
`GGUF tensor ${name} declared byte length exceeds the file`,
|
||||
);
|
||||
length = Number(bytes);
|
||||
}
|
||||
} else
|
||||
note(
|
||||
`Unknown tensor type UNKNOWN_${String(id)} prevents storage-size validation.`,
|
||||
);
|
||||
const tensor = {
|
||||
index,
|
||||
name,
|
||||
@@ -415,6 +436,7 @@ export class GgufHeader {
|
||||
tensor_data_offset,
|
||||
metadata,
|
||||
tensors: completeTensors,
|
||||
limitations: [...limitations],
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -46,7 +46,6 @@ import {
|
||||
modelCapacityRemediation,
|
||||
modelCapacityReportedLimits,
|
||||
} from "./ModelFormat.js";
|
||||
import { safetensorsBits } from "./ModelTensorTypes.js";
|
||||
import { jsonValueByteLength } from "../domain/jsonValueByteLength.js";
|
||||
import { ProviderCleanupError } from "../domain/providerCleanupError.js";
|
||||
import {
|
||||
@@ -305,6 +304,7 @@ export class ModelArtifactProvider implements ModelArtifactPort {
|
||||
partial.coverage.tensors = "partial";
|
||||
},
|
||||
onTensorDeclaration: (row) => partial.tensor_declarations.push(row),
|
||||
onLimitation: (message) => partial.limitations.push(message),
|
||||
})
|
||||
: await readSafetensorsHeader(reader, first);
|
||||
const intervals = parsed.tensors
|
||||
@@ -324,6 +324,9 @@ export class ModelArtifactProvider implements ModelArtifactPort {
|
||||
partial.declared_tensor_count = parsed.tensors.length;
|
||||
partial.tensor_declarations = [];
|
||||
partial.tensors = parsed.tensors;
|
||||
partial.limitations = [
|
||||
...new Set([...partial.limitations, ...parsed.limitations]),
|
||||
];
|
||||
partial.coverage = {
|
||||
header: "complete",
|
||||
metadata: "complete",
|
||||
@@ -384,18 +387,6 @@ export class ModelArtifactProvider implements ModelArtifactPort {
|
||||
: group.bytes + row.byte_length;
|
||||
groups.set(row.type, group);
|
||||
}
|
||||
const unknownTypes = [
|
||||
...new Set(
|
||||
parsed.tensors
|
||||
.filter(
|
||||
(row) =>
|
||||
row.byte_length === null ||
|
||||
(format === "safetensors" &&
|
||||
safetensorsBits(row.type) === undefined),
|
||||
)
|
||||
.map((row) => row.type),
|
||||
),
|
||||
];
|
||||
const report = {
|
||||
artifact: {
|
||||
path: input.path,
|
||||
@@ -444,11 +435,7 @@ export class ModelArtifactProvider implements ModelArtifactPort {
|
||||
"Metadata arrays retain at most eight elements per array; remaining values are omitted, with exact declared counts and source byte ranges.",
|
||||
]
|
||||
: []),
|
||||
...(unknownTypes.length
|
||||
? [
|
||||
`Unknown tensor types prevent storage-size validation: ${unknownTypes.join(", ")}`,
|
||||
]
|
||||
: []),
|
||||
...parsed.limitations,
|
||||
],
|
||||
};
|
||||
if (jsonValueByteLength(report, MODEL_REPORT_BYTES) > MODEL_REPORT_BYTES)
|
||||
|
||||
@@ -1,48 +1,85 @@
|
||||
/** ggml da8e2a54: enum ggml_type and block sizeof assertions in ggml-common.h.
|
||||
* Unknown future IDs retain their numeric identity and an unknown byte length.
|
||||
*/
|
||||
const GGML_TYPES: Readonly<Record<number, readonly [string, number, number]>> =
|
||||
{
|
||||
0: ["F32", 1, 4],
|
||||
1: ["F16", 1, 2],
|
||||
2: ["Q4_0", 32, 18],
|
||||
3: ["Q4_1", 32, 20],
|
||||
6: ["Q5_0", 32, 22],
|
||||
7: ["Q5_1", 32, 24],
|
||||
8: ["Q8_0", 32, 34],
|
||||
9: ["Q8_1", 32, 36],
|
||||
10: ["Q2_K", 256, 84],
|
||||
11: ["Q3_K", 256, 110],
|
||||
12: ["Q4_K", 256, 144],
|
||||
13: ["Q5_K", 256, 176],
|
||||
14: ["Q6_K", 256, 210],
|
||||
15: ["Q8_K", 256, 292],
|
||||
16: ["IQ2_XXS", 256, 66],
|
||||
17: ["IQ2_XS", 256, 74],
|
||||
18: ["IQ3_XXS", 256, 98],
|
||||
19: ["IQ1_S", 256, 50],
|
||||
20: ["IQ4_NL", 32, 18],
|
||||
21: ["IQ3_S", 256, 110],
|
||||
22: ["IQ2_S", 256, 82],
|
||||
23: ["IQ4_XS", 256, 136],
|
||||
24: ["I8", 1, 1],
|
||||
25: ["I16", 1, 2],
|
||||
26: ["I32", 1, 4],
|
||||
27: ["I64", 1, 8],
|
||||
28: ["F64", 1, 8],
|
||||
29: ["IQ1_M", 256, 56],
|
||||
30: ["BF16", 1, 2],
|
||||
34: ["TQ1_0", 256, 54],
|
||||
35: ["TQ2_0", 256, 66],
|
||||
39: ["MXFP4", 32, 17],
|
||||
40: ["NVFP4", 64, 36],
|
||||
41: ["Q1_0", 128, 18],
|
||||
42: ["Q2_0", 64, 18],
|
||||
};
|
||||
type GgmlType = readonly [
|
||||
name: string,
|
||||
rowElementMultiple: number,
|
||||
blockBytes: number | null,
|
||||
limitation?: {
|
||||
readonly kind: "storage" | "type-identity";
|
||||
readonly reason: string;
|
||||
},
|
||||
];
|
||||
|
||||
export const ggmlType = (
|
||||
id: number,
|
||||
): readonly [string, number, number] | undefined => GGML_TYPES[id];
|
||||
const kStorageLimitation = {
|
||||
kind: "storage",
|
||||
reason:
|
||||
"K-quantization types (IDs 10–15) have released 64-element and 256-element block layouts with the same GGUF and quantization versions. The common 64-element row constraint does not establish a storage block size. Nonempty tensor byte lengths and storage validation remain unknown; file extents and general.file_type do not identify the layout.",
|
||||
} as const;
|
||||
|
||||
const ambiguousGgmlType = (id: number, candidates: string): GgmlType => [
|
||||
`AMBIGUOUS_${String(id)}`,
|
||||
1,
|
||||
null,
|
||||
{
|
||||
kind: "type-identity",
|
||||
reason: `GGUF tensor type ID ${String(id)} has released ${candidates} meanings with the same GGUF and quantization versions. Textual identity and nonempty byte lengths remain unknown; file extents and general.file_type do not identify the per-tensor layout.`,
|
||||
},
|
||||
];
|
||||
|
||||
/** Modern names/sizes: ggml da8e2a54 enum and block sizeof assertions.
|
||||
* Historical collisions: llama.cpp b1500/b1803/b1871/b2100/b2172/b2221/
|
||||
* b2253/b2275/b2300. Their GGUF v3 / quantization v2 do not select a layout.
|
||||
* A common row multiple is necessary, but need not be a known storage block.
|
||||
*/
|
||||
const GGML_TYPES: Readonly<Record<number, GgmlType>> = {
|
||||
0: ["F32", 1, 4],
|
||||
1: ["F16", 1, 2],
|
||||
2: ["Q4_0", 32, 18],
|
||||
3: ["Q4_1", 32, 20],
|
||||
6: ["Q5_0", 32, 22],
|
||||
7: ["Q5_1", 32, 24],
|
||||
8: ["Q8_0", 32, 34],
|
||||
// llama.cpp b1500 (57ad015d) uses two float32 scales (40 bytes), while
|
||||
// ggml da8e2a54 uses two float16 scales (36 bytes). Both declare
|
||||
// GGUF version 3 and quantization version 2; neither selects the layout.
|
||||
9: [
|
||||
"Q8_1",
|
||||
32,
|
||||
null,
|
||||
{
|
||||
kind: "storage",
|
||||
reason:
|
||||
"Q8_1 has released 40-byte and 36-byte layouts per 32 elements with the same GGUF and quantization versions. Nonempty tensor byte lengths and storage validation remain unknown; offsets and file padding do not identify the layout.",
|
||||
},
|
||||
],
|
||||
10: ["Q2_K", 64, null, kStorageLimitation],
|
||||
11: ["Q3_K", 64, null, kStorageLimitation],
|
||||
12: ["Q4_K", 64, null, kStorageLimitation],
|
||||
13: ["Q5_K", 64, null, kStorageLimitation],
|
||||
14: ["Q6_K", 64, null, kStorageLimitation],
|
||||
15: ["Q8_K", 64, null, kStorageLimitation],
|
||||
16: ambiguousGgmlType(16, "I8 / IQ2_XXS"),
|
||||
17: ambiguousGgmlType(17, "I16 / I8 / IQ2_XS"),
|
||||
18: ambiguousGgmlType(18, "I32 / I16 / I8 / IQ3_XXS"),
|
||||
19: ambiguousGgmlType(19, "I32 / I16 / I8 / IQ1_S"),
|
||||
20: ambiguousGgmlType(20, "I32 / I16 / I8 / IQ4_NL"),
|
||||
21: ambiguousGgmlType(21, "I32 / I16 / I8 / IQ3_S"),
|
||||
22: ambiguousGgmlType(22, "I32 / I16 / I8 / IQ2_S"),
|
||||
23: ambiguousGgmlType(23, "I32 / I16 / I8 / IQ4_XS"),
|
||||
24: ambiguousGgmlType(24, "I32 / I16 / I8"),
|
||||
25: ambiguousGgmlType(25, "I32 / I16"),
|
||||
26: ["I32", 1, 4],
|
||||
27: ["I64", 1, 8],
|
||||
28: ["F64", 1, 8],
|
||||
29: ["IQ1_M", 256, 56],
|
||||
30: ["BF16", 1, 2],
|
||||
34: ["TQ1_0", 256, 54],
|
||||
35: ["TQ2_0", 256, 66],
|
||||
39: ["MXFP4", 32, 17],
|
||||
40: ["NVFP4", 64, 36],
|
||||
41: ["Q1_0", 128, 18],
|
||||
42: ["Q2_0", 64, 18],
|
||||
};
|
||||
|
||||
export const ggmlType = (id: number): GgmlType | undefined => GGML_TYPES[id];
|
||||
|
||||
/** safetensors e246a256 Dtype::bitsize, including packed sub-byte types. */
|
||||
const SAFETENSORS_BITS: Readonly<Record<string, number>> = {
|
||||
|
||||
@@ -209,6 +209,7 @@ export const readSafetensorsHeader = async (
|
||||
tensor_data_offset: number;
|
||||
metadata: ModelMetadata[];
|
||||
tensors: ModelTensor[];
|
||||
limitations: string[];
|
||||
}> => {
|
||||
const budget = new ModelReportBudget();
|
||||
// Lengths such as 640 (80 02 ...) also begin like a pickle PROTO opcode.
|
||||
@@ -275,6 +276,7 @@ export const readSafetensorsHeader = async (
|
||||
}
|
||||
}
|
||||
const tensors: ModelTensor[] = [];
|
||||
const unknownTypes = new Set<string>();
|
||||
const tensor_data_offset = 8 + size;
|
||||
const names = Object.keys(root)
|
||||
.filter((name) => name !== "__metadata__")
|
||||
@@ -312,6 +314,7 @@ export const readSafetensorsHeader = async (
|
||||
);
|
||||
}
|
||||
const bits = safetensorsBits(type);
|
||||
if (bits === undefined) unknownTypes.add(type);
|
||||
if (bits !== undefined) {
|
||||
const expected = parameters * BigInt(bits);
|
||||
if (expected % 8n !== 0n || expected / 8n !== BigInt(end - start))
|
||||
@@ -359,5 +362,11 @@ export const readSafetensorsHeader = async (
|
||||
tensor_data_offset,
|
||||
metadata,
|
||||
tensors,
|
||||
limitations:
|
||||
unknownTypes.size === 0
|
||||
? []
|
||||
: [
|
||||
`Unknown tensor types prevent storage-size validation: ${[...unknownTypes].join(", ")}`,
|
||||
],
|
||||
};
|
||||
};
|
||||
|
||||
@@ -0,0 +1,179 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { mkdir, readFile, rename, symlink, writeFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { expect } from "vitest";
|
||||
|
||||
import { encodeNibArchiveFixture } from "../../../src/artifacts/apple/NibArchive.fixture.js";
|
||||
import { machoImage } from "../../../src/artifacts/apple/MachoImage.fixture.js";
|
||||
import { interfaceBuilderAnalysisSchema } from "../../../src/domain/apple/interfaceBuilderGraph.js";
|
||||
import { dylibResolutionResultSchema } from "../../../src/domain/apple/dylibResolution.js";
|
||||
import { parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { cliTest, type TestCli } from "../../support/cli/cliFixture.js";
|
||||
|
||||
const bundle = async () => {
|
||||
const root = await createTestTempDirectory("rea-apple-traversal-");
|
||||
const app = join(root, "Fixture.app");
|
||||
const resources = join(app, "Contents", "Resources");
|
||||
const executables = join(app, "Contents", "MacOS");
|
||||
await mkdir(resources, { recursive: true });
|
||||
await mkdir(executables);
|
||||
await writeFile(
|
||||
join(app, "Contents", "Info.plist"),
|
||||
"<plist><dict><key>CFBundleExecutable</key><string>main</string></dict></plist>",
|
||||
);
|
||||
const executable = machoImage({});
|
||||
await writeFile(join(executables, "main"), executable);
|
||||
const nib = encodeNibArchiveFixture({
|
||||
classes: ["NSView"],
|
||||
objects: [{ classIndex: 0, values: { title: "Healthy view" } }],
|
||||
});
|
||||
await writeFile(join(resources, "Healthy.nib"), nib);
|
||||
return { app, resources, executables, executable, nib };
|
||||
};
|
||||
|
||||
const interfaceBuilderReports = async (cli: TestCli, app: string) => {
|
||||
const output = await cli.run({
|
||||
arguments: ["decode-interface-builder", app, "--json"],
|
||||
});
|
||||
expect(output.exitCode, output.stdout + output.stderr).toBe(0);
|
||||
const call = await cli.sharedMcpCall?.();
|
||||
if (call === undefined) throw new Error("Expected stdio MCP session");
|
||||
const opened = await call("open_bundle_resources", { path: app });
|
||||
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
|
||||
const reply = await call("decode_interface_builder", {});
|
||||
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
|
||||
return [
|
||||
parseEvidence(output.json),
|
||||
parseEvidence(reply.structuredContent),
|
||||
].map((evidence) => ({
|
||||
evidence,
|
||||
report: interfaceBuilderAnalysisSchema.parse(evidence.normalized_result),
|
||||
}));
|
||||
};
|
||||
|
||||
cliTest.skipIf(process.platform === "win32")(
|
||||
"reports symlinked nibs as unexamined while preserving healthy UI documents through CLI and MCP",
|
||||
async ({ cli }) => {
|
||||
const fixture = await bundle();
|
||||
const link = join(fixture.resources, "Panel.nib");
|
||||
await writeFile(join(fixture.resources, "Panel.data"), fixture.nib);
|
||||
await symlink("Panel.data", link);
|
||||
for (const { evidence, report } of await interfaceBuilderReports(
|
||||
cli,
|
||||
fixture.app,
|
||||
)) {
|
||||
expect(report.documents).toHaveLength(1);
|
||||
expect(report.documents[0]).toMatchObject({
|
||||
relative_path: "Contents/Resources/Healthy.nib",
|
||||
archive_sha256: createHash("sha256").update(fixture.nib).digest("hex"),
|
||||
});
|
||||
expect(report.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "archive_decode",
|
||||
status: "partial",
|
||||
reason: "archive_symlink_not_followed",
|
||||
examined: 1,
|
||||
omitted: 1,
|
||||
}),
|
||||
);
|
||||
expect(report.graph.truncated).toBe(true);
|
||||
for (const limitations of [report.limitations, evidence.limitations])
|
||||
expect(limitations).toContain(
|
||||
"Contents/Resources/Panel.nib: Symlink target was not followed or disclosed.",
|
||||
);
|
||||
}
|
||||
expect(await readFile(link)).toEqual(fixture.nib);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest.skipIf(process.platform !== "linux")(
|
||||
"retains unknown Apple resource and executable filename coverage and same-byte rename controls through CLI and MCP",
|
||||
async ({ cli }) => {
|
||||
const fixture = await bundle();
|
||||
const rawNib = Buffer.concat([
|
||||
Buffer.from(join(fixture.resources, "Panel")),
|
||||
Buffer.from([0xff]),
|
||||
Buffer.from(".nib"),
|
||||
]);
|
||||
const rawExecutable = Buffer.concat([
|
||||
Buffer.from(join(fixture.executables, "helper")),
|
||||
Buffer.from([0xff]),
|
||||
]);
|
||||
await writeFile(rawNib, fixture.nib);
|
||||
await writeFile(rawExecutable, fixture.executable);
|
||||
for (const { evidence, report } of await interfaceBuilderReports(
|
||||
cli,
|
||||
fixture.app,
|
||||
)) {
|
||||
expect(report.documents).toHaveLength(1);
|
||||
expect(report.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "archive_decode",
|
||||
status: "partial",
|
||||
reason: "source_traversal_unknown",
|
||||
examined: 1,
|
||||
omitted: 0,
|
||||
}),
|
||||
);
|
||||
expect(report.graph.truncated).toBe(true);
|
||||
for (const limitations of [report.limitations, evidence.limitations])
|
||||
expect(limitations).toContainEqual(
|
||||
expect.stringContaining("not valid UTF-8"),
|
||||
);
|
||||
}
|
||||
const call = await cli.sharedMcpCall?.();
|
||||
if (call === undefined) throw new Error("Expected stdio MCP session");
|
||||
const opened = await call("open_binary", { path: fixture.app });
|
||||
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
|
||||
const cliTrace = await cli.run({
|
||||
arguments: ["trace-dylib-resolution", fixture.app, "--json"],
|
||||
});
|
||||
expect(cliTrace.exitCode, cliTrace.stdout + cliTrace.stderr).toBe(0);
|
||||
const mcpTrace = await call("trace_dylib_resolution", {});
|
||||
expect(mcpTrace.isError, JSON.stringify(mcpTrace)).not.toBe(true);
|
||||
for (const evidence of [
|
||||
parseEvidence(cliTrace.json),
|
||||
parseEvidence(mcpTrace.structuredContent),
|
||||
]) {
|
||||
const trace = dylibResolutionResultSchema.parse(
|
||||
evidence.normalized_result,
|
||||
);
|
||||
expect(trace.roots).toEqual([
|
||||
{ image: "Contents/MacOS/main", architecture: "arm64" },
|
||||
]);
|
||||
expect(trace.coverage.status).toBe("partial");
|
||||
for (const limitations of [trace.limitations, evidence.limitations])
|
||||
expect(limitations).toContainEqual(
|
||||
expect.stringContaining("not valid UTF-8"),
|
||||
);
|
||||
}
|
||||
await rename(rawNib, join(fixture.resources, "Panel.nib"));
|
||||
await rename(rawExecutable, join(fixture.executables, "helper"));
|
||||
for (const { report } of await interfaceBuilderReports(cli, fixture.app)) {
|
||||
expect(report.documents).toHaveLength(2);
|
||||
expect(report.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "archive_decode",
|
||||
status: "complete",
|
||||
examined: 2,
|
||||
omitted: 0,
|
||||
}),
|
||||
);
|
||||
expect(report.limitations.join(" ")).not.toContain("not valid UTF-8");
|
||||
}
|
||||
const control = await cli.run({
|
||||
arguments: ["trace-dylib-resolution", fixture.app, "--json"],
|
||||
});
|
||||
expect(control.exitCode, control.stdout + control.stderr).toBe(0);
|
||||
const trace = dylibResolutionResultSchema.parse(
|
||||
parseEvidence(control.json).normalized_result,
|
||||
);
|
||||
expect(trace.roots.map(({ image }) => image)).toEqual([
|
||||
"Contents/MacOS/helper",
|
||||
"Contents/MacOS/main",
|
||||
]);
|
||||
expect(trace.coverage.status).toBe("complete");
|
||||
},
|
||||
);
|
||||
@@ -1,13 +1,15 @@
|
||||
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { build } from "plist";
|
||||
import { build, buildBinary } from "plist";
|
||||
import { expect } from "vitest";
|
||||
|
||||
import { interfaceBuilderAnalysisSchema } from "../../../src/domain/apple/interfaceBuilderGraph.js";
|
||||
import { parseEvidence } from "../../../src/domain/evidence.js";
|
||||
import { thinMach } from "../../../src/domain/binaryTarget.fixture.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { connectLocalToolsMcp } from "../../fixtures/localToolsMcp.js";
|
||||
import { cliTest } from "../../support/cli/cliFixture.js";
|
||||
|
||||
/** Write an app bundle whose only Interface Builder document is `Main.nib`. */
|
||||
@@ -94,6 +96,159 @@ cliTest(
|
||||
},
|
||||
);
|
||||
|
||||
cliTest.for(["xml", "binary"] as const)(
|
||||
"preserves UID identity despite colliding authored IDs in %s through CLI and MCP",
|
||||
async (format, { cli }) => {
|
||||
// Source-derived keyed-nib records, not output from an Apple compiler.
|
||||
const uid = (value: number) =>
|
||||
format === "binary" ? { UID: value } : { CF$UID: value };
|
||||
const source = {
|
||||
$archiver: "NSKeyedArchiver",
|
||||
$version: 100000,
|
||||
$objects: [
|
||||
"$null",
|
||||
{
|
||||
$class: uid(4),
|
||||
id: "2",
|
||||
archiveUID: 2,
|
||||
objectID: uid(8),
|
||||
"object-id": uid(9),
|
||||
title: "Root",
|
||||
NSSubviews: [uid(2)],
|
||||
},
|
||||
{
|
||||
$class: uid(4),
|
||||
objectID: uid(8),
|
||||
"object-id": uid(9),
|
||||
title: "Child",
|
||||
},
|
||||
{
|
||||
$class: uid(5),
|
||||
objectID: uid(8),
|
||||
UISource: uid(2),
|
||||
UIDestination: uid(1),
|
||||
UILabel: "parent",
|
||||
},
|
||||
{
|
||||
$classname: "ArchiveProbeView",
|
||||
$classes: ["ArchiveProbeView", "UIView", "NSObject"],
|
||||
},
|
||||
{
|
||||
$classname: "UIRuntimeOutletConnection",
|
||||
$classes: [
|
||||
"UIRuntimeOutletConnection",
|
||||
"UIRuntimeConnection",
|
||||
"NSObject",
|
||||
],
|
||||
},
|
||||
{ $class: uid(7), "NS.objects": [uid(1)] },
|
||||
{ $classname: "NSArray", $classes: ["NSArray", "NSObject"] },
|
||||
"2",
|
||||
"shared",
|
||||
],
|
||||
$top: { UINibTopLevelObjectsKey: uid(6) },
|
||||
};
|
||||
const bytes = Buffer.from(
|
||||
format === "binary" ? buildBinary(source) : build(source),
|
||||
);
|
||||
const app = await writeUIKitApp("rea-keyed-nib-hierarchy-", bytes);
|
||||
const path = join(app, "Contents/Resources/Main.nib");
|
||||
const output = await cli.run({
|
||||
arguments: ["decode-interface-builder", app, "--json"],
|
||||
environment: { REA_LOG_LEVEL: "silent" },
|
||||
});
|
||||
expect(output.exitCode, output.stderr).toBe(0);
|
||||
const evidence = parseEvidence(output.json);
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
const opened = await call("open_bundle_resources", { path: app });
|
||||
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
|
||||
const reply = await call("decode_interface_builder", {});
|
||||
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
|
||||
expect(parseEvidence(reply.structuredContent).normalized_result).toEqual(
|
||||
evidence.normalized_result,
|
||||
);
|
||||
const report = interfaceBuilderAnalysisSchema.parse(
|
||||
evidence.normalized_result,
|
||||
);
|
||||
const nodes = new Map(report.graph.nodes.map((node) => [node.id, node]));
|
||||
expect(
|
||||
report.graph.edges
|
||||
.filter(({ id }) => id.includes(":hierarchy:"))
|
||||
.map(({ from, to, resolution }) => ({
|
||||
from:
|
||||
nodes.get(from)?.attributes.interface_builder_object_id ??
|
||||
"document",
|
||||
to:
|
||||
to === null
|
||||
? null
|
||||
: nodes.get(to)?.attributes.interface_builder_object_id,
|
||||
resolution,
|
||||
})),
|
||||
).toEqual([
|
||||
{ from: "document", to: "1", resolution: "observed" },
|
||||
{ from: "1", to: "2", resolution: "observed" },
|
||||
]);
|
||||
expect(
|
||||
report.graph.nodes.find(
|
||||
({ attributes }) => attributes.interface_builder_object_id === "1",
|
||||
),
|
||||
).toMatchObject({
|
||||
attributes: {
|
||||
id: "2",
|
||||
archiveUID: 2,
|
||||
objectID: "2",
|
||||
"object-id": "shared",
|
||||
title: "Root",
|
||||
},
|
||||
});
|
||||
const root = report.graph.nodes.find(
|
||||
({ attributes }) => attributes.interface_builder_object_id === "1",
|
||||
);
|
||||
const child = report.graph.nodes.find(
|
||||
({ attributes }) => attributes.interface_builder_object_id === "2",
|
||||
);
|
||||
expect(child).toMatchObject({
|
||||
name: "Child",
|
||||
attributes: { objectID: "2", "object-id": "shared" },
|
||||
});
|
||||
const outlet = report.graph.nodes.find(({ kind }) => kind === "outlet");
|
||||
expect(outlet).toMatchObject({
|
||||
id: 'ib:"Contents/Resources/Main.nib":outlet:["archive:3"]',
|
||||
});
|
||||
expect(report.graph.edges).toContainEqual(
|
||||
expect.objectContaining({
|
||||
from: child?.id,
|
||||
to: outlet?.id,
|
||||
relation: "contains",
|
||||
resolution: "observed",
|
||||
}),
|
||||
);
|
||||
expect(report.graph.edges).toContainEqual(
|
||||
expect.objectContaining({
|
||||
from: outlet?.id,
|
||||
to: root?.id,
|
||||
relation: "outlet_to",
|
||||
resolution: "observed",
|
||||
}),
|
||||
);
|
||||
expect(report.documents[0]).toMatchObject({
|
||||
archive_sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
connection_count: 1,
|
||||
hierarchy_complete: true,
|
||||
});
|
||||
expect(report.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "hierarchy:Contents/Resources/Main.nib",
|
||||
status: "complete",
|
||||
examined: 2,
|
||||
omitted: 0,
|
||||
}),
|
||||
);
|
||||
expect(report.graph.truncated).toBe(false);
|
||||
expect(await readFile(path)).toEqual(bytes);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"classifies UIKit keyed-archive objects by their declared superclasses through CLI",
|
||||
async ({ cli }) => {
|
||||
|
||||
@@ -0,0 +1,189 @@
|
||||
import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { expect, onTestFinished } from "vitest";
|
||||
|
||||
import { artifactExtractionResultSchema } from "../../src/domain/artifactGraph.js";
|
||||
import { parseEvidence } from "../../src/domain/evidence.js";
|
||||
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
|
||||
import { cliTest } from "../support/cli/cliFixture.js";
|
||||
import {
|
||||
cArchiveExecutable,
|
||||
pyzArchive,
|
||||
PYTHON_39_MAGIC,
|
||||
} from "../../src/artifacts/pyinstaller/CArchive.fixture.js";
|
||||
|
||||
// APFS rejects the raw FF basename with EILSEQ; match the filename-byte lane.
|
||||
cliTest.skipIf(process.platform !== "linux").for([false, true])(
|
||||
"preserves extraction traversal uncertainty through CLI and stdio MCP (opaque name: %s)",
|
||||
async (withOpaqueName, { cli }) => {
|
||||
const root = await createTestTempDirectory("rea-extraction-coverage-");
|
||||
const source = join(root, "Fixture.app");
|
||||
await mkdir(join(source, "Contents", "MacOS"), { recursive: true });
|
||||
const macho = Buffer.alloc(32);
|
||||
macho.writeUInt32LE(0xfeedfacf, 0);
|
||||
macho.writeUInt32LE(0x0100000c, 4);
|
||||
macho.writeUInt32LE(2, 12);
|
||||
const files = new Map([
|
||||
[
|
||||
"Contents/Info.plist",
|
||||
Buffer.from(
|
||||
"<plist><dict><key>CFBundlePackageType</key><string>APPL</string><key>CFBundleExecutable</key><string>fixture</string></dict></plist>",
|
||||
),
|
||||
],
|
||||
["Contents/MacOS/fixture", macho],
|
||||
["Contents/good.txt", Buffer.from("retained bytes\n")],
|
||||
]);
|
||||
for (const [path, bytes] of files)
|
||||
await writeFile(join(source, path), bytes);
|
||||
const opaque = Buffer.concat([
|
||||
Buffer.from(join(source, "Contents", "opaque-")),
|
||||
Buffer.from([0xff]),
|
||||
Buffer.from(".txt"),
|
||||
]);
|
||||
if (withOpaqueName) await writeFile(opaque, "omitted bytes\n");
|
||||
|
||||
const cliResult = await cli.run({
|
||||
arguments: ["extract-artifact", source, "--json"],
|
||||
});
|
||||
expect(cliResult.exitCode, cliResult.stdout + cliResult.stderr).toBe(0);
|
||||
const cliEvidence = parseEvidence(cliResult.json);
|
||||
const cliExtraction = artifactExtractionResultSchema.parse(
|
||||
cliEvidence.normalized_result,
|
||||
);
|
||||
onTestFinished(() =>
|
||||
rm(cliExtraction.output_root, { recursive: true, force: true }),
|
||||
);
|
||||
const call = await cli.sharedMcpCall?.();
|
||||
if (call === undefined) throw new Error("Expected a stdio MCP session");
|
||||
const opened = await call("open_binary", { path: source });
|
||||
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
|
||||
const reply = await call("extract_artifact", {});
|
||||
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
|
||||
const mcpEvidence = parseEvidence(reply.structuredContent);
|
||||
const mcpExtraction = artifactExtractionResultSchema.parse(
|
||||
mcpEvidence.normalized_result,
|
||||
);
|
||||
onTestFinished(() =>
|
||||
rm(mcpExtraction.output_root, { recursive: true, force: true }),
|
||||
);
|
||||
for (const { evidence, result } of [
|
||||
{ evidence: cliEvidence, result: cliExtraction },
|
||||
{ evidence: mcpEvidence, result: mcpExtraction },
|
||||
]) {
|
||||
expect(
|
||||
result.artifacts.map(({ relative_path }) => relative_path),
|
||||
).toEqual([...files.keys()]);
|
||||
for (const [path, bytes] of files)
|
||||
expect(await readFile(join(result.output_root, path))).toEqual(bytes);
|
||||
for (const limitations of [result.limitations, evidence.limitations]) {
|
||||
expect(
|
||||
limitations.some((value) => value.includes("not valid UTF-8")),
|
||||
).toBe(withOpaqueName);
|
||||
expect(
|
||||
limitations.some((value) =>
|
||||
value.includes("source traversal coverage is unknown"),
|
||||
),
|
||||
).toBe(withOpaqueName);
|
||||
expect(
|
||||
limitations.some((value) => value.startsWith("All regular files")),
|
||||
).toBe(!withOpaqueName);
|
||||
}
|
||||
}
|
||||
if (withOpaqueName)
|
||||
expect(await readFile(opaque, "utf8")).toBe("omitted bytes\n");
|
||||
},
|
||||
);
|
||||
|
||||
cliTest.for([false, true])(
|
||||
"retains PYZ traversal coverage through CLI and stdio MCP extraction (opaque: %s)",
|
||||
async (opaque, { cli }) => {
|
||||
const root = await createTestTempDirectory("rea-extraction-pyz-coverage-");
|
||||
const source = join(root, "fixture.exe");
|
||||
const pyz = pyzArchive({
|
||||
magic: opaque ? Buffer.alloc(4) : PYTHON_39_MAGIC,
|
||||
modules: [],
|
||||
});
|
||||
const healthy = Buffer.from("retained sibling bytes\n");
|
||||
await writeFile(
|
||||
source,
|
||||
cArchiveExecutable({
|
||||
carrier: "pe",
|
||||
layout: "2.1",
|
||||
pythonVersion: 309,
|
||||
members: [
|
||||
{ name: "PYZ.pyz", typeCode: "z", payload: pyz, compress: true },
|
||||
{
|
||||
name: "good.txt",
|
||||
typeCode: "x",
|
||||
payload: healthy,
|
||||
compress: false,
|
||||
},
|
||||
],
|
||||
}),
|
||||
);
|
||||
const cliResult = await cli.run({
|
||||
arguments: ["extract-artifact", source, "--json"],
|
||||
});
|
||||
expect(cliResult.exitCode, cliResult.stdout + cliResult.stderr).toBe(0);
|
||||
const cliEvidence = parseEvidence(cliResult.json);
|
||||
const cliExtraction = artifactExtractionResultSchema.parse(
|
||||
cliEvidence.normalized_result,
|
||||
);
|
||||
onTestFinished(() =>
|
||||
rm(cliExtraction.output_root, { recursive: true, force: true }),
|
||||
);
|
||||
const call = await cli.sharedMcpCall?.();
|
||||
if (call === undefined) throw new Error("Expected a stdio MCP session");
|
||||
const opened = await call("open_binary", { path: source });
|
||||
expect(opened.isError, JSON.stringify(opened)).not.toBe(true);
|
||||
const reply = await call("extract_artifact", {});
|
||||
expect(reply.isError, JSON.stringify(reply)).not.toBe(true);
|
||||
const mcpEvidence = parseEvidence(reply.structuredContent);
|
||||
const mcpExtraction = artifactExtractionResultSchema.parse(
|
||||
mcpEvidence.normalized_result,
|
||||
);
|
||||
onTestFinished(() =>
|
||||
rm(mcpExtraction.output_root, { recursive: true, force: true }),
|
||||
);
|
||||
const files = opaque
|
||||
? ([
|
||||
["PYZ.pyz", pyz],
|
||||
["good.txt", healthy],
|
||||
] as const)
|
||||
: ([["good.txt", healthy]] as const);
|
||||
for (const { evidence, result } of [
|
||||
{ evidence: cliEvidence, result: cliExtraction },
|
||||
{ evidence: mcpEvidence, result: mcpExtraction },
|
||||
]) {
|
||||
expect(
|
||||
result.artifacts.map(({ relative_path }) => relative_path),
|
||||
).toEqual(files.map(([path]) => path));
|
||||
for (const [path, bytes] of files) {
|
||||
expect(await readFile(join(result.output_root, path))).toEqual(bytes);
|
||||
expect(
|
||||
result.artifacts.find((file) => file.relative_path === path),
|
||||
).toMatchObject({
|
||||
sha256: createHash("sha256").update(bytes).digest("hex"),
|
||||
bytes_written: bytes.length,
|
||||
});
|
||||
}
|
||||
for (const limitations of [result.limitations, evidence.limitations]) {
|
||||
expect(
|
||||
limitations.some((value) =>
|
||||
value.includes("PYZ archives remain opaque"),
|
||||
),
|
||||
).toBe(opaque);
|
||||
expect(
|
||||
limitations.some((value) =>
|
||||
value.includes("source traversal coverage is unknown"),
|
||||
),
|
||||
).toBe(opaque);
|
||||
expect(
|
||||
limitations.some((value) => value.startsWith("All regular files")),
|
||||
).toBe(!opaque);
|
||||
}
|
||||
}
|
||||
},
|
||||
);
|
||||
@@ -6,6 +6,7 @@ import { parseEvidence } from "../../src/domain/evidence.js";
|
||||
import { flutterResultSchemas } from "../../src/domain/flutter/flutterBuildAnalysis.js";
|
||||
import {
|
||||
aotFixtureImage,
|
||||
strippedAotFixtureImage,
|
||||
engineFixtureImage,
|
||||
REFERENCE_HASH,
|
||||
appendElfRanges,
|
||||
@@ -18,6 +19,61 @@ import { connectLocalToolsMcp } from "../fixtures/localToolsMcp.js";
|
||||
import { parseMcpToolError } from "../fixtures/mcpToolError.js";
|
||||
import { cliTest } from "../support/cli/cliFixture.js";
|
||||
|
||||
cliTest(
|
||||
"CLI and stdio MCP preserve AOT facts and exact artifact identity across ignored DT_BIND_NOW values",
|
||||
async ({ cli }) => {
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
const reports = [];
|
||||
const digests = [];
|
||||
for (const ignoredValue of [0n, 0xffffffffffffffffn]) {
|
||||
const image = strippedAotFixtureImage([[24, ignoredValue]]);
|
||||
const path = await createFlutterApk(
|
||||
`bind-now-${String(ignoredValue)}.apk`,
|
||||
[{ path: "lib/x86_64/libapp.so", bytes: image }],
|
||||
);
|
||||
const result = await cli.run({
|
||||
arguments: ["inspect-dart-aot", path, "--abi", "x86_64", "--json"],
|
||||
});
|
||||
expect(result.exitCode, result.stderr).toBe(0);
|
||||
const evidence = toolContract("inspect_dart_aot").outputSchema.parse(
|
||||
result.json,
|
||||
);
|
||||
const report = flutterResultSchemas.inspect_dart_aot.parse(
|
||||
evidence.normalized_result,
|
||||
);
|
||||
const response = await call("inspect_dart_aot", { path, abi: "x86_64" });
|
||||
expect(response.isError, JSON.stringify(response)).not.toBe(true);
|
||||
expect(parseEvidence(response.structuredContent)).toEqual(evidence);
|
||||
const digest = createHash("sha256").update(image).digest("hex");
|
||||
expect(report).toMatchObject({
|
||||
coverage: "complete",
|
||||
libapp: {
|
||||
sha256: digest,
|
||||
coverage: "complete",
|
||||
snapshot_hash: REFERENCE_HASH,
|
||||
snapshot_hash_sources: 2,
|
||||
},
|
||||
string_pool: {
|
||||
coverage: "complete",
|
||||
package_uris: ["package:example/main.dart"],
|
||||
},
|
||||
});
|
||||
expect(report.libapp.sections).toHaveLength(4);
|
||||
expect(evidence.subject).toMatchObject({
|
||||
local_path: path,
|
||||
digest: { sha256: report.target.sha256 },
|
||||
});
|
||||
reports.push({
|
||||
sections: report.libapp.sections,
|
||||
string_pool: report.string_pool,
|
||||
});
|
||||
digests.push(digest);
|
||||
}
|
||||
expect(reports[1]).toEqual(reports[0]);
|
||||
expect(digests[1]).not.toBe(digests[0]);
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"CLI and stdio MCP retain Flutter identity, selected ABI and duplicate-member refusals",
|
||||
async ({ cli }) => {
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { writeFile, unlink } from "node:fs/promises";
|
||||
import { readFile, writeFile, unlink } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect, it } from "vitest";
|
||||
import { toolContract } from "../../src/contracts/toolContracts.js";
|
||||
@@ -15,7 +15,12 @@ import { createEvidence, parseEvidence } from "../../src/domain/evidence.js";
|
||||
import { readEvidenceBundle } from "../../src/application/EvidenceBundleFiles.js";
|
||||
import { connectLocalToolsMcp } from "../fixtures/localToolsMcp.js";
|
||||
import { parseMcpToolError } from "../fixtures/mcpToolError.js";
|
||||
import { ggufFixture, safetensorsFixture } from "../fixtures/model/model.js";
|
||||
import {
|
||||
ggufFixture,
|
||||
ggufKLayouts,
|
||||
ggufShiftedTypeLayouts,
|
||||
safetensorsFixture,
|
||||
} from "../fixtures/model/model.js";
|
||||
import { createTestTempDirectory } from "../fixtures/temporaryDirectory.js";
|
||||
import { cliTest } from "../support/cli/cliFixture.js";
|
||||
|
||||
@@ -69,6 +74,363 @@ cliTest(
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"preserves unresolved Q8_1 layout through CLI, MCP summaries and retained tensor/type pages",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-model-q8-layout-");
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
for (const blockBytes of [36, 40]) {
|
||||
const path = join(root, `blocks-${String(blockBytes)}.gguf`);
|
||||
const base = ggufFixture();
|
||||
// Released layouts share GGUF v3/type 9/quantization version 2.
|
||||
// Trailing alignment space is not a storage-layout declaration.
|
||||
const fixture = ggufFixture({
|
||||
metadata: [
|
||||
{ key: "general.quantization_version", type: 4, value: base.u32(2) },
|
||||
],
|
||||
tensors: [
|
||||
{ name: "control", type: 0, shape: [4n], offset: 0n, bytes: 16 },
|
||||
{
|
||||
name: "weight",
|
||||
type: 9,
|
||||
shape: [320n],
|
||||
offset: 32n,
|
||||
bytes: 10 * blockBytes,
|
||||
},
|
||||
],
|
||||
});
|
||||
const bytes = Buffer.concat([fixture.bytes, Buffer.alloc(16)]);
|
||||
await writeFile(path, bytes);
|
||||
const cliResult = await cli.run({
|
||||
arguments: [
|
||||
"inspect-model-artifact",
|
||||
path,
|
||||
"--detail",
|
||||
"complete",
|
||||
"--hash",
|
||||
"full",
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(cliResult.exitCode, cliResult.stderr).toBe(0);
|
||||
const report = modelArtifactSchema.parse(
|
||||
parseEvidence(cliResult.json).normalized_result,
|
||||
);
|
||||
const sha256 = createHash("sha256").update(bytes).digest("hex");
|
||||
expect(report.artifact).toMatchObject({ path, sha256 });
|
||||
expect(report.tensors[0]).toMatchObject({
|
||||
byte_length: 16,
|
||||
storage_validation: "consistent",
|
||||
});
|
||||
const weight = {
|
||||
index: 1,
|
||||
name: "weight",
|
||||
type: "Q8_1",
|
||||
type_id: 9,
|
||||
shape: ["320"],
|
||||
file_offset: fixture.dataOffset + 32,
|
||||
byte_length: null,
|
||||
byte_length_state: "unknown",
|
||||
storage_validation: "unknown",
|
||||
};
|
||||
expect(report.tensors[1]).toMatchObject(weight);
|
||||
expect(report.totals).toMatchObject({
|
||||
parameter_count: "324",
|
||||
tensor_bytes: null,
|
||||
by_type: [
|
||||
{ type: "F32", bytes: 16 },
|
||||
{ type: "Q8_1", bytes: null },
|
||||
],
|
||||
});
|
||||
const coverage = { storage_sizes: "partial", tensor_ranges: "partial" };
|
||||
expect(report.coverage).toMatchObject(coverage);
|
||||
const limitation = expect.stringContaining(
|
||||
"Q8_1 has released 40-byte and 36-byte layouts",
|
||||
);
|
||||
expect(report.limitations).toContainEqual(limitation);
|
||||
|
||||
const response = await call("inspect_model_artifact", {
|
||||
path,
|
||||
hash: "full",
|
||||
});
|
||||
expect(response.isError, JSON.stringify(response)).not.toBe(true);
|
||||
const summary = modelArtifactSummaryViewResultSchema.parse(
|
||||
parseEvidence(response.structuredContent).normalized_result,
|
||||
);
|
||||
expect(summary.summary).toMatchObject({
|
||||
artifact: { path, sha256 },
|
||||
totals: { parameter_count: "324", tensor_bytes: null },
|
||||
coverage,
|
||||
});
|
||||
expect(summary.summary.limitations).toContainEqual(limitation);
|
||||
expect(await readFile(path)).toEqual(bytes);
|
||||
await unlink(path);
|
||||
|
||||
const source = {
|
||||
kind: "retained-evidence",
|
||||
evidence_id: summary.parent_evidence_id,
|
||||
};
|
||||
const tensorResponse = await call("inspect_analysis_view", {
|
||||
source,
|
||||
view: {
|
||||
kind: "model-artifact",
|
||||
collection: "tensors",
|
||||
types: ["Q8_1"],
|
||||
storage_validation: ["unknown"],
|
||||
fields: Object.keys(weight).filter((field) => field !== "index"),
|
||||
},
|
||||
});
|
||||
expect(tensorResponse.isError, JSON.stringify(tensorResponse)).not.toBe(
|
||||
true,
|
||||
);
|
||||
const tensorView = analysisViewResultSchema.parse(
|
||||
parseEvidence(tensorResponse.structuredContent).normalized_result,
|
||||
);
|
||||
if (tensorView.kind !== "model-artifact")
|
||||
throw new Error("Incorrect tensor view");
|
||||
expect(tensorView.items).toEqual([weight]);
|
||||
expect(tensorView.source).toMatchObject({ path, sha256 });
|
||||
expect(tensorView.source_coverage).toMatchObject(coverage);
|
||||
expect(tensorView.limitations).toContainEqual(limitation);
|
||||
|
||||
const typeResponse = await call("inspect_analysis_view", {
|
||||
source,
|
||||
view: {
|
||||
kind: "model-artifact",
|
||||
collection: "types",
|
||||
types: ["Q8_1"],
|
||||
fields: ["type", "parameter_count", "bytes"],
|
||||
},
|
||||
});
|
||||
expect(typeResponse.isError, JSON.stringify(typeResponse)).not.toBe(true);
|
||||
const typeView = analysisViewResultSchema.parse(
|
||||
parseEvidence(typeResponse.structuredContent).normalized_result,
|
||||
);
|
||||
if (typeView.kind !== "model-artifact")
|
||||
throw new Error("Incorrect type view");
|
||||
expect(typeView.items).toEqual([
|
||||
{ index: 1, type: "Q8_1", parameter_count: "320", bytes: null },
|
||||
]);
|
||||
expect(typeView.source_coverage).toMatchObject(coverage);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
const assertRetainedLayoutViews = async (
|
||||
call: Awaited<ReturnType<typeof connectLocalToolsMcp>>["call"],
|
||||
parentEvidenceId: string,
|
||||
report: ModelArtifact,
|
||||
sha256: string,
|
||||
): Promise<void> => {
|
||||
const { path } = report.artifact;
|
||||
const { header, tensor_ranges, storage_sizes } = report.coverage;
|
||||
const coverage = { header, tensor_ranges, storage_sizes };
|
||||
const source = {
|
||||
kind: "retained-evidence",
|
||||
evidence_id: parentEvidenceId,
|
||||
};
|
||||
const types = ["Q2_K", "AMBIGUOUS_16", "AMBIGUOUS_24", "AMBIGUOUS_25"];
|
||||
const tensorResponse = await call("inspect_analysis_view", {
|
||||
source,
|
||||
view: {
|
||||
kind: "model-artifact",
|
||||
collection: "tensors",
|
||||
types,
|
||||
storage_validation: ["unknown"],
|
||||
fields: [
|
||||
"type",
|
||||
"type_id",
|
||||
"shape",
|
||||
"file_offset",
|
||||
"byte_length",
|
||||
"storage_validation",
|
||||
],
|
||||
},
|
||||
});
|
||||
expect(tensorResponse.isError, JSON.stringify(tensorResponse)).not.toBe(true);
|
||||
const tensorView = analysisViewResultSchema.parse(
|
||||
parseEvidence(tensorResponse.structuredContent).normalized_result,
|
||||
);
|
||||
if (tensorView.kind !== "model-artifact")
|
||||
throw new Error("Incorrect tensor view");
|
||||
expect(tensorView.items).toEqual(
|
||||
report.tensors
|
||||
.filter((tensor) => types.includes(tensor.type))
|
||||
.map(
|
||||
({
|
||||
index,
|
||||
type,
|
||||
type_id,
|
||||
shape,
|
||||
file_offset,
|
||||
byte_length,
|
||||
storage_validation,
|
||||
}) => ({
|
||||
index,
|
||||
type,
|
||||
type_id,
|
||||
shape,
|
||||
file_offset,
|
||||
byte_length,
|
||||
storage_validation,
|
||||
}),
|
||||
),
|
||||
);
|
||||
expect(tensorView.source).toMatchObject({ path, sha256 });
|
||||
expect(tensorView.source_coverage).toMatchObject(coverage);
|
||||
const typeResponse = await call("inspect_analysis_view", {
|
||||
source,
|
||||
view: {
|
||||
kind: "model-artifact",
|
||||
collection: "types",
|
||||
types,
|
||||
fields: ["type", "bytes"],
|
||||
},
|
||||
});
|
||||
expect(typeResponse.isError, JSON.stringify(typeResponse)).not.toBe(true);
|
||||
const typeView = analysisViewResultSchema.parse(
|
||||
parseEvidence(typeResponse.structuredContent).normalized_result,
|
||||
);
|
||||
if (typeView.kind !== "model-artifact")
|
||||
throw new Error("Incorrect type view");
|
||||
expect(typeView.items).toEqual(
|
||||
report.totals.by_type.flatMap((row, index) =>
|
||||
types.includes(row.type) ? [{ index, type: row.type, bytes: null }] : [],
|
||||
),
|
||||
);
|
||||
expect(typeView.source_coverage).toMatchObject(coverage);
|
||||
};
|
||||
|
||||
cliTest(
|
||||
"preserves released K and shifted integer/IQ uncertainty through CLI and retained MCP views",
|
||||
async ({ cli }) => {
|
||||
const root = await createTestTempDirectory("rea-model-historical-layouts-");
|
||||
const { call } = await connectLocalToolsMcp();
|
||||
for (const layout of ["legacy", "modern"] as const) {
|
||||
const path = join(root, `${layout}.gguf`);
|
||||
const base = ggufFixture();
|
||||
// Authored bodies follow released source sizes; historical producers are
|
||||
// not executed. Both files carry the same version and global file type.
|
||||
const declarations = [
|
||||
{ name: "control", type: 0, shape: [4n], bytes: 16, label: "F32" },
|
||||
{
|
||||
name: "stable-integer",
|
||||
type: 26,
|
||||
shape: [3n],
|
||||
bytes: 12,
|
||||
label: "I32",
|
||||
},
|
||||
...ggufKLayouts.flatMap(({ id, name, bytes64, bytes256 }) =>
|
||||
(layout === "legacy" ? [64, 256] : [256]).map((elements) => ({
|
||||
name: `${name}-${String(elements)}`,
|
||||
type: id,
|
||||
shape: [BigInt(elements)],
|
||||
bytes: layout === "legacy" ? (elements / 64) * bytes64 : bytes256,
|
||||
label: name,
|
||||
})),
|
||||
),
|
||||
...ggufShiftedTypeLayouts.map(
|
||||
({ id, integerBytes, modernBytes256 }) => ({
|
||||
name: `shifted-${String(id)}`,
|
||||
type: id,
|
||||
shape: [256n],
|
||||
bytes: layout === "legacy" ? 256 * integerBytes : modernBytes256,
|
||||
label: `AMBIGUOUS_${String(id)}`,
|
||||
}),
|
||||
),
|
||||
];
|
||||
let nextOffset = 0n;
|
||||
const tensors = declarations.map((declaration) => {
|
||||
const tensor = { ...declaration, offset: nextOffset };
|
||||
nextOffset += BigInt(Math.ceil(declaration.bytes / 32) * 32);
|
||||
return tensor;
|
||||
});
|
||||
const fixture = ggufFixture({
|
||||
metadata: [
|
||||
{ key: "general.quantization_version", type: 4, value: base.u32(2) },
|
||||
{ key: "general.file_type", type: 4, value: base.u32(15) },
|
||||
],
|
||||
tensors,
|
||||
});
|
||||
await writeFile(path, fixture.bytes);
|
||||
const response = await cli.run({
|
||||
arguments: [
|
||||
"inspect-model-artifact",
|
||||
path,
|
||||
"--detail",
|
||||
"complete",
|
||||
"--json",
|
||||
],
|
||||
});
|
||||
expect(response.exitCode, response.stderr).toBe(0);
|
||||
const report = modelArtifactSchema.parse(
|
||||
parseEvidence(response.json).normalized_result,
|
||||
);
|
||||
expect(report.tensors.slice(0, 2)).toMatchObject([
|
||||
{ type: "F32", byte_length: 16, storage_validation: "consistent" },
|
||||
{
|
||||
type: "I32",
|
||||
type_id: 26,
|
||||
byte_length: 12,
|
||||
storage_validation: "consistent",
|
||||
},
|
||||
]);
|
||||
const unresolved = tensors.slice(2).map((tensor, index) => ({
|
||||
index: index + 2,
|
||||
type: tensor.label,
|
||||
type_id: tensor.type,
|
||||
shape: tensor.shape.map(String),
|
||||
file_offset: fixture.dataOffset + Number(tensor.offset),
|
||||
byte_length: null,
|
||||
storage_validation: "unknown",
|
||||
}));
|
||||
expect(report.tensors.slice(2)).toMatchObject(unresolved);
|
||||
const coverage = {
|
||||
header: "complete",
|
||||
tensor_ranges: "partial",
|
||||
storage_sizes: "partial",
|
||||
};
|
||||
expect(report.coverage).toMatchObject(coverage);
|
||||
expect(report.totals.tensor_bytes).toBeNull();
|
||||
for (const { id, modernName } of ggufShiftedTypeLayouts) {
|
||||
expect(report.limitations).toContainEqual(
|
||||
expect.stringMatching(
|
||||
new RegExp(`type ID ${String(id)} has released .*${modernName}`),
|
||||
),
|
||||
);
|
||||
}
|
||||
|
||||
const mcpResponse = await call("inspect_model_artifact", {
|
||||
path,
|
||||
hash: "full",
|
||||
});
|
||||
expect(mcpResponse.isError, JSON.stringify(mcpResponse)).not.toBe(true);
|
||||
const summary = modelArtifactSummaryViewResultSchema.parse(
|
||||
parseEvidence(mcpResponse.structuredContent).normalized_result,
|
||||
);
|
||||
const sha256 = createHash("sha256").update(fixture.bytes).digest("hex");
|
||||
expect(summary.summary).toMatchObject({
|
||||
artifact: { path, sha256 },
|
||||
totals: { tensor_bytes: null },
|
||||
coverage,
|
||||
});
|
||||
for (const limitation of report.limitations.filter(
|
||||
(reason) =>
|
||||
reason.includes("have released") || reason.includes("has released"),
|
||||
))
|
||||
expect(summary.summary.limitations).toContain(limitation);
|
||||
expect(await readFile(path)).toEqual(fixture.bytes);
|
||||
await unlink(path);
|
||||
await assertRetainedLayoutViews(
|
||||
call,
|
||||
summary.parent_evidence_id,
|
||||
report,
|
||||
sha256,
|
||||
);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
cliTest(
|
||||
"preserves exact large safetensors dimensions while bounding actual offsets",
|
||||
async ({ cli }) => {
|
||||
|
||||
@@ -7,9 +7,9 @@ import { buildBinary } from "plist";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
|
||||
import { analyzeInterfaceBuilderBundle } from "../../../src/artifacts/apple/InterfaceBuilderAnalysis.js";
|
||||
import { DirectoryArtifactReader } from "../../../src/artifacts/DirectoryArtifactReader.js";
|
||||
import { InterfaceBuilderDecodeBudget } from "../../../src/artifacts/apple/InterfaceBuilderDecodeBudget.js";
|
||||
import { ArtifactReaderFailure } from "../../../src/artifacts/ArtifactReader.js";
|
||||
import { DirectoryArtifactReader } from "../../../src/artifacts/DirectoryArtifactReader.js";
|
||||
import { encodeNibArchiveFixture } from "../../../src/artifacts/apple/NibArchive.fixture.js";
|
||||
import { thinMach } from "../../../src/domain/binaryTarget.fixture.js";
|
||||
import { xcodeIbtoolAvailable } from "../../fixtures/ibtool.js";
|
||||
@@ -190,6 +190,132 @@ const encodeNibArchiveWithSharedDataValue = (input: {
|
||||
return Buffer.concat([header, objects, keys, values, classes]);
|
||||
};
|
||||
|
||||
describe("compiled Interface Builder traversal coverage", () => {
|
||||
it.skipIf(process.platform === "win32")(
|
||||
"shares the 64-path omission budget between regular-document capacity and symlink source notices",
|
||||
async () => {
|
||||
const bundle = await createTestTempDirectory(
|
||||
"rea-ib-mixed-omission-bound-",
|
||||
);
|
||||
const bytes = encodeNibArchiveFixture({
|
||||
classes: ["NSView"],
|
||||
objects: [{ classIndex: 0, values: {} }],
|
||||
});
|
||||
await writeFile(join(bundle, "payload.data"), bytes);
|
||||
// One regular document is decoded; the other 61 regular documents and
|
||||
// all 65 symlinks are omitted. Exactly three symlink samples still fit.
|
||||
for (let index = 0; index < 62; index += 1)
|
||||
await writeFile(
|
||||
join(bundle, `${String(index).padStart(3, "0")}.nib`),
|
||||
bytes,
|
||||
);
|
||||
for (let index = 0; index < 65; index += 1)
|
||||
await symlink(
|
||||
"payload.data",
|
||||
join(bundle, `link-${String(index).padStart(3, "0")}.nib`),
|
||||
);
|
||||
const result = await analyzeInterfaceBuilderBundle({
|
||||
bundlePath: bundle,
|
||||
targetSha256: "a".repeat(64),
|
||||
limits: { max_documents: 1 },
|
||||
});
|
||||
expect(result.documents).toHaveLength(1);
|
||||
expect(result.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "archive_decode",
|
||||
status: "partial",
|
||||
reason: "document_limit_reached",
|
||||
examined: 1,
|
||||
omitted: 126,
|
||||
}),
|
||||
);
|
||||
const reasons = result.limitations.filter((value) =>
|
||||
value.includes(": omitted because "),
|
||||
);
|
||||
const sourceNotices = result.limitations.filter((value) =>
|
||||
value.endsWith(": Symlink target was not followed or disclosed."),
|
||||
);
|
||||
expect(reasons).toHaveLength(64);
|
||||
expect(sourceNotices).toEqual(
|
||||
[0, 1, 2].map(
|
||||
(index) =>
|
||||
`link-${String(index).padStart(3, "0")}.nib: Symlink target was not followed or disclosed.`,
|
||||
),
|
||||
);
|
||||
const reportedPaths = new Set(
|
||||
[...reasons, ...sourceNotices].map((value) =>
|
||||
value.slice(0, value.indexOf(":")),
|
||||
),
|
||||
);
|
||||
expect(reportedPaths.size).toBe(64);
|
||||
expect(
|
||||
result.limitations.some((value) => value.startsWith("link-003.nib:")),
|
||||
).toBe(false);
|
||||
expect(result.limitations).toContain(
|
||||
"62 additional omitted archive path(s) and their reasons were not retained; reporting is bounded to 64 paths.",
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it("preserves unknown traversal after a later complete callback without inventing archive omissions", async () => {
|
||||
const bundle = await createTestTempDirectory("rea-ib-traversal-callback-");
|
||||
const bytes = encodeNibArchiveFixture({
|
||||
classes: ["NSView"],
|
||||
objects: [{ classIndex: 0, values: { title: "Retained view" } }],
|
||||
});
|
||||
await writeFile(join(bundle, "Main.nib"), bytes);
|
||||
const reason = "Controlled producer traversal uncertainty";
|
||||
const entries = DirectoryArtifactReader.prototype.entries;
|
||||
const spy = vi
|
||||
.spyOn(DirectoryArtifactReader.prototype, "entries")
|
||||
.mockImplementation(async function* (
|
||||
this: DirectoryArtifactReader,
|
||||
signal,
|
||||
options,
|
||||
) {
|
||||
yield* entries.call(this, signal, options);
|
||||
if (typeof options === "function") return;
|
||||
options?.onCoverage?.({
|
||||
status: "unknown",
|
||||
examinedEntries: 1,
|
||||
limitations: [reason],
|
||||
});
|
||||
options?.onCoverage?.({
|
||||
status: "complete",
|
||||
examinedEntries: 1,
|
||||
limitations: [],
|
||||
});
|
||||
});
|
||||
try {
|
||||
const result = await analyzeInterfaceBuilderBundle({
|
||||
bundlePath: bundle,
|
||||
targetSha256: "a".repeat(64),
|
||||
});
|
||||
expect(result.documents).toHaveLength(1);
|
||||
expect(result.documents[0]).toMatchObject({ relative_path: "Main.nib" });
|
||||
expect(result.graph.nodes).toContainEqual(
|
||||
expect.objectContaining({
|
||||
kind: "view",
|
||||
attributes: expect.objectContaining({ class_name: "NSView" }),
|
||||
}),
|
||||
);
|
||||
expect(result.graph.coverage).toContainEqual(
|
||||
expect.objectContaining({
|
||||
facet: "archive_decode",
|
||||
status: "partial",
|
||||
reason: "source_traversal_unknown",
|
||||
examined: 1,
|
||||
omitted: 0,
|
||||
}),
|
||||
);
|
||||
expect(result.graph.truncated).toBe(true);
|
||||
expect(result.limitations).toContain(reason);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("compiled Interface Builder bundle reader", () => {
|
||||
it("reads nib plist archives and reports provenance", async () => {
|
||||
const root = await createTestTempDirectory("rea-ib-test-");
|
||||
@@ -438,7 +564,7 @@ describe("keyed archive hierarchy coverage", () => {
|
||||
]),
|
||||
);
|
||||
const rootId = analysis.graph.nodes.find(
|
||||
({ attributes }) => attributes.interface_builder_object_id === "root",
|
||||
({ attributes }) => attributes.interface_builder_object_id === "1",
|
||||
)?.id;
|
||||
const childNames = analysis.graph.edges
|
||||
.filter(
|
||||
@@ -448,7 +574,7 @@ describe("keyed archive hierarchy coverage", () => {
|
||||
id.includes(":hierarchy:"),
|
||||
)
|
||||
.map(({ to }) => (to === null ? null : archiveIdsByNodeId.get(to)));
|
||||
expect(childNames).toEqual(["first", "second"]);
|
||||
expect(childNames).toEqual(["3", "5"]);
|
||||
});
|
||||
|
||||
it.skipIf(process.platform !== "darwin")(
|
||||
@@ -592,7 +718,7 @@ describe("bounded Interface Builder archive decoding", () => {
|
||||
? null
|
||||
: kind === "invalid-byte-name" || kind === "subtree-symlink"
|
||||
? "source_traversal_unknown"
|
||||
: "archive_not_regular_file",
|
||||
: "archive_symlink_not_followed",
|
||||
examined: 0,
|
||||
omitted: kind === "nib-symlink" || kind === "storyboard-symlink" ? 1 : 0,
|
||||
});
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { unlink } from "node:fs/promises";
|
||||
import { createHash } from "node:crypto";
|
||||
import { Ajv2020 } from "ajv/dist/2020.js";
|
||||
import { expect, it, onTestFinished, vi } from "vitest";
|
||||
import { FlutterBuildAnalysisService } from "../../../src/application/flutter/FlutterBuildAnalysisService.js";
|
||||
@@ -14,40 +15,14 @@ import { flutterResultSchemas } from "../../../src/domain/flutter/flutterBuildAn
|
||||
import type { JsonValue } from "../../../src/domain/jsonValue.js";
|
||||
import {
|
||||
REFERENCE_HASH,
|
||||
appendElfRanges,
|
||||
aotFixtureImage,
|
||||
buildStrippedElf64Image,
|
||||
snapshotSection,
|
||||
strippedAotFixtureImage,
|
||||
} from "../../../src/flutter/FlutterBuild.fixture.js";
|
||||
import { createFlutterApk } from "../../fixtures/flutterApk.js";
|
||||
import { createApplicationMcpHarness } from "../../fixtures/applicationMcpHarness.js";
|
||||
import { parseMcpToolError } from "../../fixtures/mcpToolError.js";
|
||||
|
||||
const strippedAotImage = (): Buffer => {
|
||||
const symbols = buildStrippedElf64Image([
|
||||
{ name: "_kDartVmSnapshotData", value: 0x1000, size: 52 },
|
||||
{ name: "_kDartVmSnapshotInstructions", value: 0x1800, size: 16 },
|
||||
{ name: "_kDartIsolateSnapshotData", value: 0x2000, size: 0x600 },
|
||||
{ name: "_kDartIsolateSnapshotInstructions", value: 0x2600, size: 16 },
|
||||
]);
|
||||
const vmSection = snapshotSection(REFERENCE_HASH);
|
||||
const isolateSection = Buffer.concat([
|
||||
snapshotSection(REFERENCE_HASH),
|
||||
Buffer.from([0]),
|
||||
Buffer.from("package:example/main.dart\0dart:core\0Example\0"),
|
||||
]);
|
||||
return Buffer.concat([
|
||||
symbols,
|
||||
Buffer.alloc(0x1000 - symbols.length),
|
||||
vmSection,
|
||||
Buffer.alloc(0x1800 - 0x1000 - vmSection.length),
|
||||
Buffer.alloc(16),
|
||||
Buffer.alloc(0x2000 - 0x1800 - 16),
|
||||
isolateSection,
|
||||
Buffer.alloc(0x600 - isolateSection.length),
|
||||
Buffer.alloc(16),
|
||||
]);
|
||||
};
|
||||
|
||||
const dynamicSegmentOffset = (image: Buffer): number =>
|
||||
Number(image.readBigUInt64LE(64 + 56 + 8));
|
||||
|
||||
@@ -333,14 +308,24 @@ it("rejects unavailable retention, invalid delivery and cancellation before prod
|
||||
expect(execute).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([0n, 0xffffffffffffffffn])(
|
||||
"accepts a complete PT_DYNAMIC DT_NULL entry with ignored value %s through MCP",
|
||||
async (ignoredValue) => {
|
||||
const image = strippedAotImage();
|
||||
it.each([
|
||||
{ ignoredValue: 0n, retainedSections: false },
|
||||
{ ignoredValue: 0xffffffffffffffffn, retainedSections: false },
|
||||
{ ignoredValue: 0n, retainedSections: true },
|
||||
])(
|
||||
"accepts bounded PT_DYNAMIC through MCP (ignored value: $ignoredValue, retained sections: $retainedSections)",
|
||||
async ({ ignoredValue, retainedSections }) => {
|
||||
const image = strippedAotFixtureImage();
|
||||
const dynamic = dynamicSegmentOffset(image);
|
||||
image.writeBigUInt64LE(ignoredValue, dynamic + 5 * 16 + 8);
|
||||
const selected = retainedSections
|
||||
? appendElfRanges(image, "section", [
|
||||
{ type: 1, offset: 0x1000, size: 52 },
|
||||
{ type: 1, offset: 0x2000, size: 0x600 },
|
||||
])
|
||||
: image;
|
||||
const path = await createFlutterApk("dt-null.apk", [
|
||||
{ path: "lib/arm64-v8a/libapp.so", bytes: image },
|
||||
{ path: "lib/arm64-v8a/libapp.so", bytes: selected },
|
||||
]);
|
||||
const provider = createFlutterBuildAnalysisProvider();
|
||||
onTestFinished(() => provider.close());
|
||||
@@ -360,6 +345,12 @@ it.each([0n, 0xffffffffffffffffn])(
|
||||
coverage: "complete",
|
||||
snapshot_hash: REFERENCE_HASH,
|
||||
snapshot_hash_sources: 2,
|
||||
sha256: createHash("sha256").update(selected).digest("hex"),
|
||||
});
|
||||
expect(report.libapp.sections).toHaveLength(4);
|
||||
expect(report.string_pool).toMatchObject({
|
||||
coverage: "complete",
|
||||
package_uris: ["package:example/main.dart"],
|
||||
});
|
||||
},
|
||||
);
|
||||
@@ -378,7 +369,7 @@ it.each([
|
||||
])(
|
||||
"refuses unusable dynamic tables for $label",
|
||||
async ({ truncate, reason }) => {
|
||||
const image = strippedAotImage();
|
||||
const image = strippedAotFixtureImage();
|
||||
const dynamic = dynamicSegmentOffset(image);
|
||||
if (truncate) {
|
||||
image.writeBigUInt64LE(5n * 16n + 15n, 64 + 56 + 0x20);
|
||||
|
||||
@@ -191,11 +191,16 @@ const expectReferences = (
|
||||
)
|
||||
.toMatchObject({ status });
|
||||
|
||||
const verifyReferences = async (
|
||||
const verifyDimension = async (
|
||||
client: Client,
|
||||
comparisonEvidenceId: string,
|
||||
status: "pass" | "fail" | "unknown",
|
||||
dimension: "references" | "calls" | "identity" = "references",
|
||||
observedStatus: "unchanged" | "changed" | "unknown" = status === "pass"
|
||||
? "unchanged"
|
||||
: status === "fail"
|
||||
? "changed"
|
||||
: "unknown",
|
||||
) => {
|
||||
const claimId = `${dimension}-${comparisonEvidenceId}`;
|
||||
const response = await client.callTool({
|
||||
@@ -207,7 +212,7 @@ const verifyReferences = async (
|
||||
{
|
||||
kind: "structural-function",
|
||||
claim_id: claimId,
|
||||
title: "Reference facts agree",
|
||||
title: `${dimension} facts agree`,
|
||||
comparison_evidence_id: comparisonEvidenceId,
|
||||
dimension,
|
||||
},
|
||||
@@ -219,8 +224,6 @@ const verifyReferences = async (
|
||||
const verification = reconstructionVerificationResultSchema.parse(
|
||||
parseEvidence(response.structuredContent).normalized_result,
|
||||
);
|
||||
const observed =
|
||||
status === "pass" ? "unchanged" : status === "fail" ? "changed" : "unknown";
|
||||
expect.soft(verification).toMatchObject({
|
||||
status,
|
||||
summary: {
|
||||
@@ -234,7 +237,7 @@ const verifyReferences = async (
|
||||
expect.objectContaining({
|
||||
claim_id: claimId,
|
||||
status,
|
||||
observed_status: observed,
|
||||
observed_status: observedStatus,
|
||||
}),
|
||||
],
|
||||
},
|
||||
@@ -263,7 +266,7 @@ it("preserves reference identity uncertainty and same-artifact coordinates", asy
|
||||
);
|
||||
const namedToNull = await compare(client, named, unnamedElsewhere);
|
||||
expectReferences(namedToNull, "unknown");
|
||||
await verifyReferences(client, namedToNull.evidence.evidence_id, "unknown");
|
||||
await verifyDimension(client, namedToNull.evidence.evidence_id, "unknown");
|
||||
const nullToNamed = await compare(client, unnamedElsewhere, named);
|
||||
expectReferences(nullToNamed, "unknown");
|
||||
|
||||
@@ -379,6 +382,111 @@ it("keeps generated and ambiguous procedure names from proving cross-artifact id
|
||||
expect.soft(generatedFunctions.result.status).toBe("unknown");
|
||||
});
|
||||
|
||||
it("preserves call endpoint identity through public comparisons and selected reconstruction claims", async () => {
|
||||
const { client, session } = await createHarness();
|
||||
const observedCalls = (
|
||||
id: string,
|
||||
digest: string,
|
||||
entry: string,
|
||||
targets: readonly string[],
|
||||
includeReferences = true,
|
||||
) =>
|
||||
evidence(
|
||||
id,
|
||||
digest,
|
||||
includeReferences
|
||||
? targets.map((target) =>
|
||||
reference(entry, target, "call", { sourceName: "worker" }),
|
||||
)
|
||||
: [],
|
||||
{
|
||||
procedureAddress: entry,
|
||||
procedureName: "worker",
|
||||
callees: targets.map((target) => procedure(target, "helper")),
|
||||
},
|
||||
);
|
||||
const original = observedCalls("original-calls", "1", "0x1000", [
|
||||
"ram:0x2000",
|
||||
]);
|
||||
const cases = [
|
||||
{
|
||||
left: original,
|
||||
right: original,
|
||||
status: "unchanged",
|
||||
verification: "unknown",
|
||||
},
|
||||
{
|
||||
left: original,
|
||||
right: observedCalls("distinct-helpers", "1", "0x1100", ["ram:0x3000"]),
|
||||
status: "changed",
|
||||
verification: "fail",
|
||||
},
|
||||
{
|
||||
left: original,
|
||||
right: observedCalls("same-helper-coordinate", "1", "0x1000", [
|
||||
"ram:0X002000",
|
||||
]),
|
||||
status: "unchanged",
|
||||
verification: "pass",
|
||||
},
|
||||
{
|
||||
left: original,
|
||||
right: observedCalls("cross-build-helper", "2", "0x1100", ["ram:0x3000"]),
|
||||
status: "unchanged",
|
||||
verification: "pass",
|
||||
},
|
||||
...[true, false].map((includeReferences) => ({
|
||||
left: observedCalls(
|
||||
`duplicate-helpers-left-${includeReferences}`,
|
||||
"3",
|
||||
"0x1000",
|
||||
["0x2000", "0x3000"],
|
||||
includeReferences,
|
||||
),
|
||||
right: observedCalls(
|
||||
`duplicate-helpers-right-${includeReferences}`,
|
||||
"4",
|
||||
"0x1100",
|
||||
["0x4000", "0x5000"],
|
||||
includeReferences,
|
||||
),
|
||||
status: "unknown" as const,
|
||||
verification: "unknown" as const,
|
||||
})),
|
||||
] as const;
|
||||
for (const test of cases) {
|
||||
const compared = await compare(client, test.left, test.right);
|
||||
expect(
|
||||
compared.result.dimensions.find((value) => value.dimension === "calls"),
|
||||
).toMatchObject({
|
||||
status: test.status,
|
||||
evidence_links: [test.left.evidence_id, test.right.evidence_id],
|
||||
...(test.status === "unknown"
|
||||
? {
|
||||
left_count: 2,
|
||||
right_count: 2,
|
||||
left_digest: null,
|
||||
right_digest: null,
|
||||
limitations: [
|
||||
expect.stringContaining("unique comparable procedure entry"),
|
||||
],
|
||||
}
|
||||
: {}),
|
||||
});
|
||||
await verifyDimension(
|
||||
client,
|
||||
compared.evidence.evidence_id,
|
||||
test.verification,
|
||||
"calls",
|
||||
test.status,
|
||||
);
|
||||
for (const source of [test.left, test.right])
|
||||
expect(
|
||||
session.evidenceById(source.evidence_id)?.normalized_result,
|
||||
).toEqual(source.normalized_result);
|
||||
}
|
||||
});
|
||||
|
||||
it("normalizes numeric coordinates but preserves address spaces and source sites", async () => {
|
||||
const { client } = await createHarness();
|
||||
const at = (id: string, address: string) =>
|
||||
@@ -494,7 +602,7 @@ it("keeps known reference changes decisive beside unrelated unknown kinds", asyn
|
||||
);
|
||||
const compared = await compare(client, left, right);
|
||||
expectReferences(compared, "changed");
|
||||
await verifyReferences(client, compared.evidence.evidence_id, "fail");
|
||||
await verifyDimension(client, compared.evidence.evidence_id, "fail");
|
||||
});
|
||||
|
||||
it("keeps ambiguous duplicate edge kinds unknown independent of input order", async () => {
|
||||
@@ -663,7 +771,7 @@ it("uses shared endpoint identity for selected calls and references claims", asy
|
||||
scenario.name === "duplicate-name" && dimension === "references"
|
||||
? "fail"
|
||||
: scenario.expected;
|
||||
await verifyReferences(
|
||||
await verifyDimension(
|
||||
client,
|
||||
comparison.evidence.evidence_id,
|
||||
expected,
|
||||
@@ -686,7 +794,7 @@ it("keeps empty function labels ambiguous and selected identity unknown", async
|
||||
({ dimension }) => dimension === "identity",
|
||||
),
|
||||
).toMatchObject({ status: "unknown" });
|
||||
await verifyReferences(
|
||||
await verifyDimension(
|
||||
client,
|
||||
comparison.evidence.evidence_id,
|
||||
"unknown",
|
||||
@@ -703,7 +811,7 @@ it("retains caller entries and independent contradictions beside unknown endpoin
|
||||
callers: [procedure("0x1020", "caller")],
|
||||
});
|
||||
const retargeted = await compare(client, left, right);
|
||||
await verifyReferences(
|
||||
await verifyDimension(
|
||||
client,
|
||||
retargeted.evidence.evidence_id,
|
||||
"fail",
|
||||
@@ -723,7 +831,7 @@ it("retains caller entries and independent contradictions beside unknown endpoin
|
||||
knownAndUnknown("calls-mixed-left", "a", "known_left"),
|
||||
knownAndUnknown("calls-mixed-right", "b", "known_right"),
|
||||
);
|
||||
await verifyReferences(client, mixed.evidence.evidence_id, "fail", "calls");
|
||||
await verifyDimension(client, mixed.evidence.evidence_id, "fail", "calls");
|
||||
});
|
||||
|
||||
it("keeps equal coordinates in distinct same-architecture universal slices unknown", async () => {
|
||||
@@ -760,7 +868,7 @@ it("keeps equal coordinates in distinct same-architecture universal slices unkno
|
||||
atSlice(`slice-arm64-${id}`, selected),
|
||||
);
|
||||
for (const dimension of ["calls", "references"] as const)
|
||||
await verifyReferences(
|
||||
await verifyDimension(
|
||||
client,
|
||||
comparison.evidence.evidence_id,
|
||||
expected,
|
||||
|
||||
@@ -97,7 +97,7 @@ mcpTest(
|
||||
expect.objectContaining({
|
||||
facet: "archive_decode",
|
||||
status: "partial",
|
||||
reason: "archive_not_regular_file",
|
||||
reason: "archive_symlink_not_followed",
|
||||
examined: 2,
|
||||
omitted: 1,
|
||||
}),
|
||||
|
||||
@@ -17,7 +17,12 @@ import { GgufHeader } from "../../../src/model/GgufHeader.js";
|
||||
import { createModelArtifactService } from "../../../src/composition/model.js";
|
||||
import { modelArtifactSchema } from "../../../src/domain/model/modelArtifact.js";
|
||||
import { createTestTempDirectory } from "../../fixtures/temporaryDirectory.js";
|
||||
import { ggufFixture, safetensorsFixture } from "../../fixtures/model/model.js";
|
||||
import {
|
||||
ggufFixture,
|
||||
ggufKLayouts,
|
||||
ggufShiftedTypeLayouts,
|
||||
safetensorsFixture,
|
||||
} from "../../fixtures/model/model.js";
|
||||
|
||||
const hash = (bytes: Buffer) =>
|
||||
createHash("sha256").update(bytes).digest("hex");
|
||||
@@ -197,6 +202,7 @@ it.each([false, true])(
|
||||
it.each([
|
||||
{ name: "excess rank", type: 0, shape: [1n, 1n, 1n, 1n, 1n], bytes: 4 },
|
||||
{ name: "quantized scalar", type: 2, shape: [], bytes: 0 },
|
||||
{ name: "Q8_1 incomplete block", type: 9, shape: [31n], bytes: 40 },
|
||||
])(
|
||||
"rejects GGUF $name without relaxing representation constraints",
|
||||
async (tensor) => {
|
||||
@@ -443,6 +449,239 @@ it("returns unknown lengths for unknown GGUF types without inventing storage siz
|
||||
expect(result.value.totals.tensor_bytes).toBeNull();
|
||||
});
|
||||
|
||||
it.each([
|
||||
{ version: 2, bigEndian: false, blockBytes: 36 },
|
||||
{ version: 2, bigEndian: false, blockBytes: 40 },
|
||||
{ version: 3, bigEndian: false, blockBytes: 36 },
|
||||
{ version: 3, bigEndian: false, blockBytes: 40 },
|
||||
{ version: 3, bigEndian: true, blockBytes: 36 },
|
||||
{ version: 3, bigEndian: true, blockBytes: 40 },
|
||||
])(
|
||||
"preserves unresolved Q8_1 storage for GGUF v$version, endian=$bigEndian, block bytes=$blockBytes",
|
||||
async ({ version, bigEndian, blockBytes }) => {
|
||||
const base = ggufFixture({ bigEndian });
|
||||
const fixture = ggufFixture({
|
||||
version,
|
||||
bigEndian,
|
||||
metadata: [
|
||||
{ key: "general.quantization_version", type: 4, value: base.u32(2) },
|
||||
],
|
||||
tensors: [
|
||||
{ name: "control", type: 0, shape: [4n], offset: 0n, bytes: 16 },
|
||||
{
|
||||
name: "weight",
|
||||
type: 9,
|
||||
shape: [320n],
|
||||
offset: 32n,
|
||||
bytes: 10 * blockBytes,
|
||||
},
|
||||
],
|
||||
});
|
||||
const { provider, path } = await providerAt(
|
||||
Buffer.concat([fixture.bytes, Buffer.alloc(16)]),
|
||||
);
|
||||
const result = await provider.inspect({ path, hash: "header" });
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.tensors).toMatchObject([
|
||||
{ name: "control", byte_length: 16, storage_validation: "consistent" },
|
||||
{
|
||||
name: "weight",
|
||||
type: "Q8_1",
|
||||
type_id: 9,
|
||||
shape: ["320"],
|
||||
data_offset: 32,
|
||||
file_offset: fixture.dataOffset + 32,
|
||||
parameter_count: "320",
|
||||
byte_length: null,
|
||||
byte_length_state: "unknown",
|
||||
storage_validation: "unknown",
|
||||
},
|
||||
]);
|
||||
expect(result.value.totals).toMatchObject({
|
||||
parameter_count: "324",
|
||||
tensor_bytes: null,
|
||||
by_type: [
|
||||
{ type: "F32", bytes: 16, parameter_count: "4" },
|
||||
{ type: "Q8_1", bytes: null, parameter_count: "320" },
|
||||
],
|
||||
});
|
||||
expect(result.value.coverage).toMatchObject({
|
||||
header: "complete",
|
||||
tensor_ranges: "partial",
|
||||
storage_sizes: "partial",
|
||||
});
|
||||
expect(result.value.limitations).toContainEqual(
|
||||
expect.stringContaining("Q8_1 has released 40-byte and 36-byte layouts"),
|
||||
);
|
||||
expect(result.value.limitations.join(" ")).not.toContain(
|
||||
"Unknown tensor type Q8_1",
|
||||
);
|
||||
expect(result.value.artifact.bytes_read).toBe(fixture.header.length);
|
||||
},
|
||||
);
|
||||
|
||||
it("derives empty Q8_1 storage without choosing a historical block layout", async () => {
|
||||
const fixture = ggufFixture({
|
||||
metadata: [],
|
||||
tensors: [
|
||||
{ name: "empty", type: 9, shape: [32n, 0n], offset: 0n, bytes: 0 },
|
||||
],
|
||||
});
|
||||
const { provider, path } = await providerAt(fixture.bytes);
|
||||
const result = await provider.inspect({ path, hash: "header" });
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.tensors[0]).toMatchObject({
|
||||
type: "Q8_1",
|
||||
parameter_count: "0",
|
||||
byte_length: 0,
|
||||
byte_length_state: "derived",
|
||||
storage_validation: "consistent",
|
||||
});
|
||||
expect(result.value.totals.tensor_bytes).toBe(0);
|
||||
expect(result.value.coverage).toMatchObject({
|
||||
tensor_ranges: "complete",
|
||||
storage_sizes: "complete",
|
||||
});
|
||||
});
|
||||
|
||||
it.each(ggufKLayouts)(
|
||||
"preserves $name identity while leaving released K64/K256 storage unresolved",
|
||||
async ({ id, name, bytes64, bytes256 }) => {
|
||||
// The released GGUF writer pads each tensor to 32 bytes. For 256 elements,
|
||||
// both layouts have the same padded extent, despite different byte lengths.
|
||||
expect(Math.ceil((4 * bytes64) / 32)).toBe(Math.ceil(bytes256 / 32));
|
||||
for (const [elements, bytes] of [
|
||||
[64, bytes64],
|
||||
[256, 4 * bytes64],
|
||||
[256, bytes256],
|
||||
] as const) {
|
||||
const base = ggufFixture();
|
||||
const fixture = ggufFixture({
|
||||
metadata: [
|
||||
{ key: "general.quantization_version", type: 4, value: base.u32(2) },
|
||||
{ key: "general.file_type", type: 4, value: base.u32(15) },
|
||||
],
|
||||
tensors: [
|
||||
{
|
||||
name: "weight",
|
||||
type: id,
|
||||
shape: [BigInt(elements)],
|
||||
offset: 0n,
|
||||
bytes,
|
||||
},
|
||||
],
|
||||
});
|
||||
const { provider, path } = await providerAt(
|
||||
Buffer.concat([
|
||||
fixture.bytes,
|
||||
Buffer.alloc(Math.ceil(bytes / 32) * 32 - bytes),
|
||||
]),
|
||||
);
|
||||
const result = await provider.inspect({ path, hash: "header" });
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.tensors[0]).toMatchObject({
|
||||
type: name,
|
||||
type_id: id,
|
||||
shape: [String(elements)],
|
||||
parameter_count: String(elements),
|
||||
byte_length: null,
|
||||
storage_validation: "unknown",
|
||||
});
|
||||
expect(result.value.coverage).toMatchObject({
|
||||
tensor_ranges: "partial",
|
||||
storage_sizes: "partial",
|
||||
});
|
||||
expect(result.value.totals.tensor_bytes).toBeNull();
|
||||
expect(result.value.limitations).toContainEqual(
|
||||
expect.stringContaining("common 64-element row constraint"),
|
||||
);
|
||||
}
|
||||
const { provider, path } = await providerAt(
|
||||
ggufFixture({
|
||||
tensors: [
|
||||
{
|
||||
name: "invalid",
|
||||
type: id,
|
||||
shape: [32n],
|
||||
offset: 0n,
|
||||
bytes: bytes64,
|
||||
},
|
||||
],
|
||||
}).bytes,
|
||||
);
|
||||
const rejected = await provider.inspect({ path, hash: "header" });
|
||||
expect(rejected.ok).toBe(false);
|
||||
if (rejected.ok) throw new Error("Incomplete common K row admitted");
|
||||
expect(rejected.error).toMatchObject({
|
||||
constraint: "file_format_rejected",
|
||||
reason: expect.stringContaining("multiple of 64 elements"),
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it.each(ggufShiftedTypeLayouts)(
|
||||
"retains ambiguous numeric type $id for released integer and $modernName bodies",
|
||||
async ({ id, integerBytes, modernName, modernBytes256 }) => {
|
||||
for (const [elements, bytes] of [
|
||||
[3, 3 * integerBytes],
|
||||
[256, 256 * integerBytes],
|
||||
[256, modernBytes256],
|
||||
[0, 0],
|
||||
] as const) {
|
||||
const fixture = ggufFixture({
|
||||
tensors: [
|
||||
{
|
||||
name: "weight",
|
||||
type: id,
|
||||
shape: [BigInt(elements)],
|
||||
offset: 0n,
|
||||
bytes,
|
||||
},
|
||||
],
|
||||
});
|
||||
const { provider, path } = await providerAt(fixture.bytes);
|
||||
const result = await provider.inspect({ path, hash: "header" });
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.tensors[0]).toMatchObject({
|
||||
type: `AMBIGUOUS_${String(id)}`,
|
||||
type_id: id,
|
||||
shape: [String(elements)],
|
||||
parameter_count: String(elements),
|
||||
byte_length: elements === 0 ? 0 : null,
|
||||
storage_validation: elements === 0 ? "consistent" : "unknown",
|
||||
});
|
||||
expect(result.value.limitations).toContainEqual(
|
||||
expect.stringContaining(`type ID ${String(id)} has released`),
|
||||
);
|
||||
expect(result.value.limitations.join(" ")).toContain(modernName);
|
||||
expect(result.value.totals.tensor_bytes).toBe(elements === 0 ? 0 : null);
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("retains established storage for stable GGUF types and zero-byte K declarations", async () => {
|
||||
const fixture = ggufFixture({
|
||||
tensors: [
|
||||
{ name: "integer", type: 26, shape: [3n], offset: 0n, bytes: 12 },
|
||||
{ name: "empty-k", type: 10, shape: [64n, 0n], offset: 32n, bytes: 0 },
|
||||
{ name: "quant", type: 2, shape: [32n], offset: 32n, bytes: 18 },
|
||||
],
|
||||
});
|
||||
const { provider, path } = await providerAt(fixture.bytes);
|
||||
const result = await provider.inspect({ path, hash: "header" });
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.tensors).toMatchObject([
|
||||
{ type: "I32", byte_length: 12, storage_validation: "consistent" },
|
||||
{ type: "Q2_K", byte_length: 0, storage_validation: "consistent" },
|
||||
{ type: "Q4_0", byte_length: 18, storage_validation: "consistent" },
|
||||
]);
|
||||
expect(result.value.totals.tensor_bytes).toBe(30);
|
||||
expect(result.value.coverage).toMatchObject({
|
||||
tensor_ranges: "complete",
|
||||
storage_sizes: "complete",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps unknown safetensors storage validation separate from observed valid byte ranges", async () => {
|
||||
const fixture = safetensorsFixture(
|
||||
'{"future":{"dtype":"FUTURE4","shape":[2],"data_offsets":[0,4]}}',
|
||||
|
||||
@@ -131,7 +131,7 @@ it.each([undefined, 64])(
|
||||
},
|
||||
);
|
||||
|
||||
it("retains completed GGUF tensor declarations before later tensor bytes cancel", async () => {
|
||||
it("retains GGUF tensor declarations and layout limitations before later tensor bytes cancel", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "rea-model-tensor-partial-"));
|
||||
const path = join(root, "model.gguf");
|
||||
const tensorCount = 4000;
|
||||
@@ -139,10 +139,10 @@ it("retains completed GGUF tensor declarations before later tensor bytes cancel"
|
||||
metadata: [],
|
||||
tensors: Array.from({ length: tensorCount }, (_, index) => ({
|
||||
name: `tensor-${String(index).padStart(4, "0")}`,
|
||||
type: 0,
|
||||
shape: [1n],
|
||||
type: index === 0 ? 9 : 0,
|
||||
shape: index === 0 ? [320n] : [1n],
|
||||
offset: 0n,
|
||||
bytes: 4,
|
||||
bytes: index === 0 ? 400 : 4,
|
||||
})),
|
||||
});
|
||||
await writeFile(path, fixture.bytes);
|
||||
@@ -210,12 +210,18 @@ it("retains completed GGUF tensor declarations before later tensor bytes cancel"
|
||||
expect(partial.tensor_declarations.length).toBeGreaterThan(0);
|
||||
expect(partial.tensor_declarations[0]).toMatchObject({
|
||||
index: 0,
|
||||
type: "Q8_1",
|
||||
byte_length: null,
|
||||
byte_length_state: "unknown",
|
||||
file_offset: null,
|
||||
storage_validation: "not-established",
|
||||
});
|
||||
expect(partial.tensors).toEqual([]);
|
||||
expect(partial.coverage.tensors).toBe("partial");
|
||||
expect(evidence.normalized_result).toMatchObject({ alignment: 32 });
|
||||
expect(evidence.limitations).toContainEqual(
|
||||
expect.stringContaining("Q8_1 has released 40-byte and 36-byte layouts"),
|
||||
);
|
||||
expect(partial.artifact).toMatchObject({
|
||||
path,
|
||||
header_sha256: null,
|
||||
@@ -244,7 +250,7 @@ it.each([
|
||||
JSON.stringify({
|
||||
__metadata__: { author: "test" },
|
||||
weight: {
|
||||
dtype: "F32",
|
||||
dtype: "FUTURE_F32",
|
||||
shape: [524288],
|
||||
data_offsets: [0, 2 * 1024 * 1024],
|
||||
},
|
||||
@@ -330,6 +336,14 @@ it.each([
|
||||
file_identity: "unverified",
|
||||
},
|
||||
});
|
||||
if (format === "safetensors")
|
||||
expect(
|
||||
parseEvidence(result.error.partialObservation).limitations,
|
||||
).toContainEqual(
|
||||
expect.stringContaining(
|
||||
"Unknown tensor types prevent storage-size validation: FUTURE_F32",
|
||||
),
|
||||
);
|
||||
} finally {
|
||||
readSpy.mockRestore();
|
||||
await provider.close();
|
||||
|
||||
@@ -126,6 +126,150 @@ it("still refuses a regular file added after inventory and rolls back", async ()
|
||||
await expect(access(output)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it
|
||||
.skipIf(process.platform !== "linux")
|
||||
.each(["inventory", "materialization"] as const)(
|
||||
"retains unknown filename coverage established during %s",
|
||||
async (phase) => {
|
||||
const root = await createTestTempDirectory(
|
||||
"rea-extract-unknown-traversal-",
|
||||
);
|
||||
const source = join(root, "source");
|
||||
await mkdir(source);
|
||||
await writeFile(join(source, "known.txt"), "retained bytes\n");
|
||||
const opaque = Buffer.concat([
|
||||
Buffer.from(join(source, "opaque-")),
|
||||
Buffer.from([0xff]),
|
||||
]);
|
||||
if (phase === "inventory") await writeFile(opaque, "opaque bytes\n");
|
||||
const output = join(root, "output");
|
||||
const materialize = await inventoriedExtraction(source, output);
|
||||
if (phase === "inventory") await rm(opaque);
|
||||
else await writeFile(opaque, "opaque bytes\n");
|
||||
|
||||
const result = await materialize();
|
||||
expect(result.artifacts.map(({ relative_path }) => relative_path)).toEqual([
|
||||
"known.txt",
|
||||
]);
|
||||
expect(await readFile(join(output, "known.txt"), "utf8")).toBe(
|
||||
"retained bytes\n",
|
||||
);
|
||||
expect(result.limitations).toContainEqual(
|
||||
expect.stringContaining("not valid UTF-8"),
|
||||
);
|
||||
expect(result.limitations).toContainEqual(
|
||||
expect.stringContaining("source traversal coverage is unknown"),
|
||||
);
|
||||
expect(
|
||||
result.limitations.some((value) => value.startsWith("All regular files")),
|
||||
).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
it.each(["inventory", "materialization"] as const)(
|
||||
"retains typed unknown coverage from only the %s producer stage",
|
||||
async (phase) => {
|
||||
const root = await createTestTempDirectory("rea-extract-stage-coverage-");
|
||||
const source = join(root, "source");
|
||||
await mkdir(source);
|
||||
await writeFile(join(source, "known.txt"), "retained bytes\n");
|
||||
const output = join(root, "output");
|
||||
let stage: "inventory" | "materialization" = "inventory";
|
||||
const limitation = `Producer traversal coverage is unknown during ${phase}.`;
|
||||
const entries = DirectoryArtifactReader.prototype.entries;
|
||||
const spy = vi
|
||||
.spyOn(DirectoryArtifactReader.prototype, "entries")
|
||||
.mockImplementation(async function* (
|
||||
this: DirectoryArtifactReader,
|
||||
signal,
|
||||
selection,
|
||||
) {
|
||||
yield* entries.call(this, signal, selection);
|
||||
if (typeof selection === "function") return;
|
||||
if (stage === phase)
|
||||
selection?.onCoverage?.({
|
||||
status: "unknown",
|
||||
limitations: [limitation],
|
||||
});
|
||||
// A later complete observation must not erase established uncertainty.
|
||||
selection?.onCoverage?.({ status: "complete", limitations: [] });
|
||||
});
|
||||
try {
|
||||
const materialize = await inventoriedExtraction(source, output);
|
||||
stage = "materialization";
|
||||
const result = await materialize();
|
||||
expect(result.limitations).toContain(limitation);
|
||||
expect(result.limitations).toContainEqual(
|
||||
expect.stringContaining("source traversal coverage is unknown"),
|
||||
);
|
||||
expect(
|
||||
result.limitations.some((value) =>
|
||||
value.startsWith("All regular files"),
|
||||
),
|
||||
).toBe(false);
|
||||
expect(
|
||||
result.artifacts.map(({ relative_path }) => relative_path),
|
||||
).toEqual(["known.txt"]);
|
||||
expect(await readFile(join(output, "known.txt"), "utf8")).toBe(
|
||||
"retained bytes\n",
|
||||
);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
it("normalizes an NFD omission once across inventory and materialization", async () => {
|
||||
const root = await createTestTempDirectory("rea-extract-omission-nfc-");
|
||||
const source = join(root, "source");
|
||||
await mkdir(source);
|
||||
await writeFile(join(source, "known.txt"), "retained bytes\n");
|
||||
const output = join(root, "output");
|
||||
const entries = DirectoryArtifactReader.prototype.entries;
|
||||
const spy = vi
|
||||
.spyOn(DirectoryArtifactReader.prototype, "entries")
|
||||
.mockImplementation(async function* (
|
||||
this: DirectoryArtifactReader,
|
||||
signal,
|
||||
selection,
|
||||
) {
|
||||
yield* entries.call(this, signal, selection);
|
||||
if (typeof selection === "function") return;
|
||||
selection?.onOmission?.({ path: "cafe\u0301.pipe", kind: "fifo" });
|
||||
selection?.onCoverage?.({ status: "complete", limitations: [] });
|
||||
});
|
||||
try {
|
||||
const snapshot = await scanArtifactInventory(source);
|
||||
expect(snapshot.coverage.status).toBe("unknown");
|
||||
const result = await materializeArtifactInventory(
|
||||
{
|
||||
inputPath: source,
|
||||
inputFormat: "asar",
|
||||
outputRoot: output,
|
||||
environment: process.env,
|
||||
integrityPolicy: "fail",
|
||||
},
|
||||
source,
|
||||
snapshot,
|
||||
);
|
||||
const omission = 'Artifact traversal omitted fifo at "caf\u00e9.pipe".';
|
||||
for (const limitations of [snapshot.limitations, result.limitations])
|
||||
expect(
|
||||
limitations.filter((value) =>
|
||||
value.startsWith("Artifact traversal omitted"),
|
||||
),
|
||||
).toEqual([omission]);
|
||||
expect(result.limitations).toContainEqual(
|
||||
expect.stringContaining("source traversal coverage is unknown"),
|
||||
);
|
||||
expect(await readFile(join(output, "known.txt"), "utf8")).toBe(
|
||||
"retained bytes\n",
|
||||
);
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("rolls back when cancellation arrives while closing the materialization reader", async () => {
|
||||
const root = await createTestTempDirectory("rea-extract-cancel-close-");
|
||||
const source = join(root, "source");
|
||||
|
||||
Vendored
+28
@@ -1,3 +1,31 @@
|
||||
// Released llama.cpp b1500 GGML_QKK_64 blocks and default/b3001 blocks.
|
||||
// These are source-derived payload sizes, not output from historical binaries.
|
||||
export const ggufKLayouts = [
|
||||
{ id: 10, name: "Q2_K", bytes64: 24, bytes256: 84 },
|
||||
{ id: 11, name: "Q3_K", bytes64: 28, bytes256: 110 },
|
||||
{ id: 12, name: "Q4_K", bytes64: 38, bytes256: 144 },
|
||||
{ id: 13, name: "Q5_K", bytes64: 46, bytes256: 176 },
|
||||
{ id: 14, name: "Q6_K", bytes64: 54, bytes256: 210 },
|
||||
{ id: 15, name: "Q8_K", bytes64: 76, bytes256: 292 },
|
||||
] as const;
|
||||
|
||||
// Integer IDs shifted as IQ types were added, before upstream PR 6050 pinned
|
||||
// integers at 24–26. Oldest colliding integer width and modern bytes/256 values
|
||||
// are independent goldens from b1500/b1803/b1871/b2100/b2172/b2221/b2253/b2275
|
||||
// and b3001. An authored body does not establish producer provenance.
|
||||
export const ggufShiftedTypeLayouts = [
|
||||
{ id: 16, integerBytes: 1, modernName: "IQ2_XXS", modernBytes256: 66 },
|
||||
{ id: 17, integerBytes: 2, modernName: "IQ2_XS", modernBytes256: 74 },
|
||||
{ id: 18, integerBytes: 4, modernName: "IQ3_XXS", modernBytes256: 98 },
|
||||
{ id: 19, integerBytes: 4, modernName: "IQ1_S", modernBytes256: 50 },
|
||||
{ id: 20, integerBytes: 4, modernName: "IQ4_NL", modernBytes256: 144 },
|
||||
{ id: 21, integerBytes: 4, modernName: "IQ3_S", modernBytes256: 110 },
|
||||
{ id: 22, integerBytes: 4, modernName: "IQ2_S", modernBytes256: 82 },
|
||||
{ id: 23, integerBytes: 4, modernName: "IQ4_XS", modernBytes256: 136 },
|
||||
{ id: 24, integerBytes: 4, modernName: "I8", modernBytes256: 256 },
|
||||
{ id: 25, integerBytes: 4, modernName: "I16", modernBytes256: 512 },
|
||||
] as const;
|
||||
|
||||
/** Source-built declarations with independent expected byte layouts; no model loader. */
|
||||
export const ggufFixture = (
|
||||
options: {
|
||||
|
||||
Reference in New Issue
Block a user