diff --git a/docs/kuna-provider.md b/docs/kuna-provider.md index 9d653287d8..a2b7d684cb 100644 --- a/docs/kuna-provider.md +++ b/docs/kuna-provider.md @@ -64,7 +64,7 @@ Single-function subprocess output and each cached source facet share a 16 MiB bu ## Strings -`list_strings` and `search_strings` share one lazy `kuna strings --json --encoding all --min-length 5 --termination any --no-xrefs` scan of the digest-checked private input. Skipping references avoids program lifting, a call graph and a C export. REA validates the input, scan policy, every row and the total before caching any facts. Encoding, character count, byte length and terminator facts must agree; supplementary UTF-8 characters count as Unicode scalars. Malformed or oversized output is an error, not an empty search. JSON admits at most 16 MiB before parsing. String scans use the whole-program `REA_KUNA_PROJECT_TIMEOUT_MS` deadline; `REA_KUNA_TIMEOUT_MS` applies to single-function decompilation. The parsed inventory belongs to one selected target/executable profile and expires on session close; deterministic output-buffer failures are retained until close, while other failures are not cached. +`list_strings` and `search_strings` share one lazy `kuna strings --json --encoding all --min-length 5 --termination any --no-xrefs` scan of the digest-checked private input. Skipping references avoids program lifting, a call graph and a C export. REA validates the input, scan policy, every row and the total before caching any facts. Encoding, character count, byte length and terminator facts must agree; supplementary UTF-8 characters count as Unicode scalars. Malformed output is an error, not an empty search. REA parses the JSON as Kuna writes it instead of buffering it, so the scan has no fixed size: Kuna writes about 290 bytes per row, 31 MB for a 20 MB Ghostty binary and 179 MB (611k rows) for an arm64 Firefox `XUL`. Rows are validated one at a time, and the first contradiction stops Kuna. Text REA does not keep is dropped unassembled; kept text and retained rows are admitted against the REA process's live heap headroom as they arrive; a refusal is a `memory` resource constraint with `retryable: false`, and closing other targets or starting REA with a larger Node.js heap makes room. String scans use the whole-program `REA_KUNA_PROJECT_TIMEOUT_MS` deadline; `REA_KUNA_TIMEOUT_MS` applies to single-function decompilation. The parsed inventory belongs to one selected target/executable profile and expires on session close; Kuna diagnostics beyond their 8 MiB buffer are retained as a failure until close, while other failures are not cached. Results keep exact u64 virtual addresses, decoded text, native encoding, terminator status and byte length including any terminator. Kuna scans mapped sections or segments for runs of at least five characters; unterminated fragments can include instruction bytes. `all` includes ASCII, UTF-8 and the engine's Latin-range UTF-16LE reading. Other text, including short strings, non-Latin wide text and UTF-16BE, can remain undiscovered. An empty result does not prove absence. @@ -78,7 +78,7 @@ The first call query runs `kuna decompile-graph --functions ''`, admitting no C `xrefs` runs native `xrefs --to ADDRESS --json` on demand. Kuna unifies forwarding aliases and suppresses edges internal to their class; REA filters to the requested address and records the excluded-row count. Duplicate source sites appear once. An empty result cannot establish the absence of references. Addresses are explicit u64 values or exact, unambiguous procedure names; Kuna has no cursor. -Graph JSON admits at most 32 MiB before parsing; xref JSON at most 16 MiB. A large image can exceed the graph budget: an arm64 Firefox `XUL` (404k functions) does, while a 14 MB Sublime Text binary produced 31.8 MB. Output beyond a buffer is reported as a `memory` resource constraint with `retryable: false`, because the same target and executable produce the same output. The session remembers a graph that overflowed, so later call queries fail at once instead of repeating the whole-program walk. The graph is cached for the selected target and executable profile within one session. Xref reuse is an LRU charged by source bytes plus metadata, limited to 16 MiB and 128 entries; at most eight distinct native xref queries may be pending. Reference commands run one at a time per session and share the provider's foreground budget with single-function decompiles. An unproven process stop retains its admission slots until cleanup succeeds. Concurrent callers share work, individual cancellation releases only that waiter, and the last departing waiter cancels its producer and receives any unproven cleanup. Session close aborts and drains producers, releases parsed caches and retries any retained process/file cleanup. +REA parses the graph as Kuna writes it, like the string scan, so the graph has no fixed size: a 53 MB Raycast binary produces 157 MB (180k functions, 773k edges), which REA retains in about 93 MB after a whole-program walk of about a minute. Unrequested bodies are refused before any of their text is assembled, and edges are indexed as they arrive, so retained rows and their indexes are admitted against live heap headroom. Xref JSON admits at most 16 MiB before parsing. Output or Kuna diagnostics beyond a buffer are reported as a `memory` resource constraint with `retryable: false`, because the same target and executable produce the same output. The session remembers such a graph failure, so later call queries fail at once instead of repeating the whole-program walk. The graph is cached for the selected target and executable profile within one session. Xref reuse is an LRU charged by source bytes plus metadata, limited to 16 MiB and 128 entries; at most eight distinct native xref queries may be pending. Reference commands run one at a time per session and share the provider's foreground budget with single-function decompiles. An unproven process stop retains its admission slots until cleanup succeeds. Concurrent callers share work, individual cancellation releases only that waiter, and the last departing waiter cancels its producer and receives any unproven cleanup. Session close aborts and drains producers, releases parsed caches and retries any retained process/file cleanup. Evidence includes only selected addresses or dossier neighbors, the producing JSON's SHA-256/byte count, coverage and relevant function facts; unrequested program-wide JSON and instruction text are omitted. Reference caches are in memory and expire on close: a fresh CLI process or MCP server rebuilds them. A first call query therefore costs a whole-program reference walk even when a C export is warm. These operations do not establish full Ghidra/IDA feature parity. diff --git a/src/kuna/KunaCommand.ts b/src/kuna/KunaCommand.ts index 46f7af5d5c..a1629feca3 100644 --- a/src/kuna/KunaCommand.ts +++ b/src/kuna/KunaCommand.ts @@ -1,7 +1,7 @@ import { createHash, randomUUID } from "node:crypto"; import { createReadStream, createWriteStream } from "node:fs"; import { stat } from "node:fs/promises"; -import { Transform } from "node:stream"; +import { Transform, type Writable } from "node:stream"; import { pipeline } from "node:stream/promises"; import { AnalysisError } from "../domain/analysisErrorBase.js"; @@ -54,7 +54,10 @@ export interface KunaCommandRequest { readonly timeoutMs: number; /** The deadline callers configured, reported when the run times out. */ readonly reportedTimeoutMs: number; + /** Buffered stderr bytes, plus stdout bytes unless `stdout` consumes them. */ readonly outputBytes: number; + /** Parses stdout as it arrives instead of buffering it; see `runOwnedCommand`. */ + readonly stdout?: Writable | undefined; readonly signal: AbortSignal; readonly retainFailedFileClose: ( owner: OwnedFileHandle, @@ -220,6 +223,7 @@ export const runKunaCommand = async ( { timeoutMs: Math.max(1, request.timeoutMs), diagnosticBytes: request.outputBytes, + ...(request.stdout === undefined ? {} : { stdout: request.stdout }), }, { signal: request.signal, diff --git a/src/kuna/KunaJson.ts b/src/kuna/KunaJson.ts index 31bb234888..c5f9c17f1f 100644 --- a/src/kuna/KunaJson.ts +++ b/src/kuna/KunaJson.ts @@ -1,3 +1,20 @@ +/** Read a numeric u64 address from its exact JSON token, before rounding. */ +export const kunaAddressFromJsonNumber = ( + source: string | undefined, + key: string, +): string => { + if ( + source === undefined || + source.length > 20 || + !/^(?:0|[1-9][0-9]*)$/u.test(source) + ) + throw new Error(`invalid numeric address in ${key}`); + const exact = BigInt(source); + if (exact > 0xffffffffffffffffn) + throw new Error(`address in ${key} exceeds u64`); + return `0x${exact.toString(16)}`; +}; + /** Parse numeric u64 addresses from their exact JSON tokens, before rounding. */ export const parseKunaJson = ( text: string, @@ -14,18 +31,9 @@ export const parseKunaJson = ( context?: { readonly source?: string }, ): unknown => { if (!addressKeys.has(key) || value === null) return value; - const source = context?.source; - if ( - typeof value !== "number" || - source === undefined || - source.length > 20 || - !/^(?:0|[1-9][0-9]*)$/u.test(source) - ) + if (typeof value !== "number") throw new Error(`invalid numeric address in ${key}`); - const exact = BigInt(source); - if (exact > 0xffffffffffffffffn) - throw new Error(`address in ${key} exceeds u64`); - return `0x${exact.toString(16)}`; + return kunaAddressFromJsonNumber(context?.source, key); }, ); }; diff --git a/src/kuna/KunaJsonStream.ts b/src/kuna/KunaJsonStream.ts new file mode 100644 index 0000000000..e88d8f0a65 --- /dev/null +++ b/src/kuna/KunaJsonStream.ts @@ -0,0 +1,554 @@ +import { createHash } from "node:crypto"; +import { Writable } from "node:stream"; +import { getHeapStatistics } from "node:v8"; + +import { getManyValues, isMany, none, type Many } from "stream-chain/defs.js"; +import * as streamJsonParser from "stream-json/core/parser.js"; +import type { ParserOptions, Token } from "stream-json/core/parser.js"; +import type { z } from "zod"; + +import type { AnalysisError } from "../domain/analysisErrorBase.js"; +import { + AnalysisOutputError, + AnalysisResourceConstraintError, +} from "../domain/analysisErrorCore.js"; +import { err, ok, type Result } from "../domain/result.js"; +import { kunaAddressFromJsonNumber } from "./KunaJson.js"; + +/** + * stream-json 3's synchronous tokenizer behind `parser`, which its + * declarations omit. Each call returns the tokens `text` completes; `none` + * flushes the end of input. + */ +const { jsonParser } = streamJsonParser as unknown as { + readonly jsonParser: ( + options: ParserOptions, + ) => (text: string | typeof none) => Many | typeof none; +}; + +/** One array of flat row objects in a Kuna JSON document. */ +export interface KunaJsonRows { + /** Scalar fields kept from each row; other fields are skipped unassembled. */ + readonly fields: ReadonlySet; + /** Kept numeric u64 fields, read from their exact tokens as hex, or null. */ + readonly addresses: ReadonlySet; + /** Array fields that must be empty, kept as `[]`. */ + readonly empty: ReadonlySet; + /** Fields that must be null, rejected before any other value is assembled. */ + readonly nulls: ReadonlySet; +} + +/** The members of one Kuna JSON document that a reader interprets. */ +export interface KunaJsonDocument { + /** Root scalar members kept in the header. */ + readonly header: ReadonlySet; + /** Root object members whose listed scalar fields are kept in the header. */ + readonly sections: ReadonlyMap>; + /** Root array members of rows, each passed to `acceptRow`. */ + readonly rows: ReadonlyMap; +} + +/** What the document established once Kuna stopped writing. */ +export interface KunaJsonCompletion { + readonly header: Readonly>; + readonly sha256: string; + readonly bytes: number; +} + +type Frame = + | { readonly kind: "root"; readonly values: Record } + | { + readonly kind: "section"; + readonly name: string; + readonly fields: ReadonlySet; + readonly values: Record; + } + | { + readonly kind: "rows"; + readonly name: string; + readonly spec: KunaJsonRows; + } + | { + readonly kind: "row"; + readonly member: string; + readonly spec: KunaJsonRows; + readonly values: Record; + } + | { readonly kind: "empty"; readonly field: string }; + +/** Charge for one retained row beyond its UTF-16 text. */ +const ROW_OVERHEAD_BYTES = 256; +/** Projected retention admitted between heap samples. */ +const ADMISSION_INTERVAL_BYTES = 64 * 1024; +/** Per-row reserve for the owner's arrays, sorting and projections. */ +const ROW_REFERENCE_BYTES = 16; + +const describe = (cause: unknown): string => + cause instanceof Error ? cause.message : String(cause); + +/** + * Longest key or kept number assembled. Kuna's field names and u64 decimals + * are far shorter, so a longer key names no kept field and a longer kept + * number is not one Kuna writes. + */ +const MAX_TOKEN_CODE_UNITS = 64; + +/** Parse one row with its schema, naming the first field that is not Kuna's. */ +export const parseKunaRow = ( + schema: Schema, + member: string, + fields: unknown, +): z.infer => { + const parsed = schema.safeParse(fields); + if (parsed.success) return parsed.data; + throw new Error( + `${member} row field ${parsed.error.issues[0]?.path.join(".") ?? ""} is not Kuna's`, + ); +}; + +/** V8 copies shorter substrings instead of slicing them. */ +const SLICED_STRING_MIN_LENGTH = 13; + +/** + * Copy kept text out of the decoded chunk the tokenizer sliced it from, so a + * retained row does not keep that whole chunk alive. UTF-16 code units copy + * losslessly, including lone surrogates. + */ +const detached = (text: string): string => + text.length < SLICED_STRING_MIN_LENGTH + ? text + : Buffer.from(text, "utf16le").toString("utf16le"); + +/** + * Validate a Kuna JSON document as Kuna writes it, without buffering it. + * + * Every scalar arrives in chunks and is assembled only as far as it can + * matter: keys up to the longest field name, kept numbers up to the longest + * u64, and kept strings, which can carry target-derived text of any length, + * admitted against live heap headroom as they grow, as are retained rows. + * Ignored values are dropped unassembled. The first contradiction aborts + * `rejected`, letting the owner stop Kuna early. The writable side never + * fails, so the command's own outcome stays authoritative; `complete` reports + * what the document established. + */ +export abstract class KunaJsonDocumentReader extends Writable { + readonly #hash = createHash("sha256"); + readonly #decoder = new TextDecoder("utf-8", { fatal: true }); + readonly #tokens = jsonParser({ + packKeys: false, + packNumbers: false, + packStrings: false, + streamValues: true, + }); + readonly #rejection = new AbortController(); + readonly #frames: Frame[] = []; + readonly #rowsSeen = new Set(); + #header: Record = {}; + #bytes = 0; + #failure: AnalysisError | undefined; + #closed = false; + #key: string | undefined; + /** Containers still open inside an ignored value. */ + #skipping = 0; + /** Key the scalar being read is kept under; undefined discards it. */ + #target: string | undefined; + /** Whether `#target` holds a numeric u64 address. */ + #targetIsAddress = false; + /** Whether the scalar being assembled is a key. */ + #inKey = false; + /** The key or kept scalar assembled so far. */ + #text = ""; + #rows = 0; + #projectedBytes = 0; + #nextAdmission = 0; + + constructor( + private readonly producer: string, + private readonly label: string, + private readonly document: KunaJsonDocument, + ) { + super(); + } + + /** Validate and retain one row; throw to reject the document. */ + protected abstract acceptRow( + member: string, + row: Readonly>, + ): void; + + /** Release retained rows once the document is rejected. */ + protected abstract discardRows(): void; + + /** The validated document, once Kuna has exited successfully on `input`. */ + abstract finish(input: string): Result; + + /** Aborted at the first contradiction or refused retention. */ + get rejected(): AbortSignal { + return this.#rejection.signal; + } + + /** Why `rejected` aborted, if it has. */ + get rejection(): AnalysisError | undefined { + return this.#rejection.signal.aborted ? this.#failure : undefined; + } + + /** An output error naming this document. */ + protected unusable(reason: string): AnalysisOutputError { + return new AnalysisOutputError(this.producer, reason, { + userMessage: `${this.label} cannot be used: ${reason}`, + }); + } + + override _write( + chunk: Buffer, + _encoding: BufferEncoding, + callback: (error?: Error | null) => void, + ): void { + if (this.#failure === undefined) + try { + this.#bytes += chunk.byteLength; + this.#hash.update(chunk); + this.#feed(this.#decoder.decode(chunk, { stream: true })); + } catch (cause: unknown) { + this.#reject(cause); + this.#rejection.abort(); + } + callback(); + } + + override _final(callback: (error?: Error | null) => void): void { + // Kuna has stopped writing, so an incomplete document is recorded for + // `complete` without signalling: its exit status may explain it. + if (this.#failure === undefined) + try { + this.#feed(this.#decoder.decode()); + this.#feed(none); + } catch (cause: unknown) { + this.#reject(cause); + } + callback(); + } + + /** The kept header and source identity, once Kuna has exited successfully. */ + protected complete(): Result { + if (this.#failure !== undefined) return err(this.#failure); + if ( + !this.writableFinished || + !this.#closed || + [...this.document.rows.keys()].some((name) => !this.#rowsSeen.has(name)) + ) + return err(this.unusable(`${this.label} ended before it was complete`)); + return ok({ + header: this.#header, + sha256: this.#hash.digest("hex"), + bytes: this.#bytes, + }); + } + + #reject(cause: unknown): void { + this.#failure ??= + cause instanceof AnalysisResourceConstraintError + ? cause + : this.unusable(describe(cause)); + this.#text = ""; + this.#inKey = false; + this.#header = {}; + this.discardRows(); + } + + #feed(text: string | typeof none): void { + const tokens = this.#tokens(text); + if (isMany(tokens)) + for (const token of getManyValues(tokens)) this.#consume(token); + } + + /** Consume a token inside an ignored value; false once none is open. */ + #skip(token: Token): boolean { + if (this.#skipping === 0) return false; + if (token.name === "startObject" || token.name === "startArray") + this.#skipping += 1; + else if (token.name === "endObject" || token.name === "endArray") + this.#skipping -= 1; + return true; + } + + #consume(token: Token): void { + if (this.#skip(token) || this.#assemble(token)) return; + switch (token.name) { + case "startKey": + this.#inKey = true; + return; + case "endKey": + this.#key = this.#text; + this.#text = ""; + this.#inKey = false; + return; + case "startString": + this.#aim("string"); + return; + case "endString": + if (this.#target !== undefined) + this.#store(detached(this.#text), "string"); + this.#text = ""; + return; + case "startNumber": + this.#aim("number"); + return; + case "endNumber": + this.#store(this.#text, "number"); + this.#text = ""; + return; + case "nullValue": + this.#aim("null"); + this.#store(null, "literal"); + return; + case "trueValue": + case "falseValue": + this.#aim("boolean"); + this.#store(token.value, "literal"); + return; + case "startObject": + case "startArray": + this.#open(token.name === "startObject" ? "object" : "array"); + return; + case "endObject": + case "endArray": + this.#close(); + return; + default: + throw new Error(`unexpected JSON ${token.name}`); + } + } + + /** Append a scalar fragment within its bound; false for other tokens. */ + #assemble(token: Token): boolean { + if (token.name !== "stringChunk" && token.name !== "numberChunk") + return false; + const chunk = token.value; + if (this.#inKey) { + // A key longer than every kept field name is truncated unmatched. + if (this.#text.length <= MAX_TOKEN_CODE_UNITS) + this.#text += chunk.slice( + 0, + MAX_TOKEN_CODE_UNITS + 1 - this.#text.length, + ); + } else if (this.#target !== undefined) { + this.#text += chunk; + if (token.name === "stringChunk") + this.#charge(chunk.length * 2, this.#text.length); + else if (this.#text.length > MAX_TOKEN_CODE_UNITS) + throw new Error(`${this.#target} is not a number Kuna writes`); + } + return true; + } + + /** Decide where a scalar value goes, before any of it is assembled. */ + #aim(kind: "string" | "number" | "null" | "boolean"): void { + const frame = this.#frames.at(-1); + const key = this.#key!; + this.#key = undefined; + this.#targetIsAddress = false; + if (frame === undefined) + throw new Error(`${this.label} is not one JSON object`); + switch (frame.kind) { + case "root": + if (this.document.rows.has(key) || this.document.sections.has(key)) + throw new Error(`${this.label} member ${key} is not a container`); + this.#target = this.#kept(frame.values, key, this.document.header); + return; + case "section": + this.#target = this.#kept(frame.values, key, frame.fields); + return; + case "rows": + throw new Error(`${frame.name} row is not an object`); + case "row": + if (frame.spec.empty.has(key)) + throw new Error(`${frame.member} row ${key} is not an array`); + if (frame.spec.nulls.has(key)) { + if (kind !== "null") + throw new Error(`${frame.member} row ${key} must be null`); + this.#target = this.#kept(frame.values, key, frame.spec.nulls); + return; + } + if (frame.spec.addresses.has(key)) { + if (kind !== "number" && kind !== "null") + throw new Error(`invalid numeric address in ${key}`); + this.#target = this.#kept(frame.values, key, frame.spec.addresses); + this.#targetIsAddress = kind === "number"; + return; + } + this.#target = this.#kept(frame.values, key, frame.spec.fields); + return; + case "empty": + throw new Error(`${frame.field} must be empty`); + } + } + + /** The key a kept field is stored under, or undefined to discard it. */ + #kept( + values: Record, + key: string, + fields: ReadonlySet, + ): string | undefined { + if (!fields.has(key)) return undefined; + this.#claim(values, key); + return key; + } + + #claim(values: Record, key: string): void { + if (Object.hasOwn(values, key)) + throw new Error(`${this.label} repeats ${key}`); + } + + #store( + value: string | boolean | null, + kind: "string" | "number" | "literal", + ): void { + const key = this.#target; + if (key === undefined) return; + this.#target = undefined; + this.#values()[key] = this.#targetIsAddress + ? kunaAddressFromJsonNumber(value as string, key) + : kind === "number" + ? Number(value) + : value; + } + + #values(): Record { + const frame = this.#frames.at(-1); + if (frame === undefined || frame.kind === "rows" || frame.kind === "empty") + throw new Error(`${this.label} value has no member`); + return frame.values; + } + + /** Open a root member: rows, a section, or a value this reader skips. */ + #openMember( + kind: "object" | "array", + key: string, + root: Record, + ): void { + const rows = this.document.rows.get(key); + const section = this.document.sections.get(key); + if (rows !== undefined) { + if (kind !== "array" || this.#rowsSeen.has(key)) + throw new Error(`${this.label} rows ${key} are not one array`); + this.#frames.push({ kind: "rows", name: key, spec: rows }); + } else if (section !== undefined) { + if (kind !== "object") throw new Error(`${key} is not an object`); + this.#claim(root, key); + const values: Record = {}; + root[key] = values; + this.#frames.push({ + kind: "section", + name: key, + fields: section, + values, + }); + } else if (this.document.header.has(key)) + throw new Error(`${this.label} ${key} is not a scalar`); + else this.#skipping = 1; + } + + #open(kind: "object" | "array"): void { + const frame = this.#frames.at(-1); + const key = this.#key; + this.#key = undefined; + if (frame === undefined) { + if (kind !== "object" || this.#closed) + throw new Error(`${this.label} is not one JSON object`); + this.#frames.push({ kind: "root", values: this.#header }); + return; + } + switch (frame.kind) { + case "root": + this.#openMember(kind, key!, frame.values); + return; + case "rows": + if (kind !== "object") + throw new Error(`${frame.name} row is not an object`); + this.#frames.push({ + kind: "row", + member: frame.name, + spec: frame.spec, + values: {}, + }); + return; + case "row": + if (frame.spec.empty.has(key!)) { + if (kind !== "array") + throw new Error(`${frame.member} row ${key!} is not an array`); + this.#claim(frame.values, key!); + this.#frames.push({ kind: "empty", field: key! }); + } else if ( + frame.spec.fields.has(key!) || + frame.spec.addresses.has(key!) || + frame.spec.nulls.has(key!) + ) + throw new Error(`${frame.member} row ${key!} is not a scalar`); + else this.#skipping = 1; + return; + case "section": + if (frame.fields.has(key!)) + throw new Error(`${frame.name} ${key!} is not a scalar`); + this.#skipping = 1; + return; + case "empty": + throw new Error(`${frame.field} must be empty`); + } + } + + #close(): void { + const frame = this.#frames.pop(); + switch (frame?.kind) { + case "root": + this.#closed = true; + return; + case "rows": + this.#rowsSeen.add(frame.name); + return; + case "row": + this.#rows += 1; + this.#charge(ROW_OVERHEAD_BYTES, 0); + this.acceptRow(frame.member, frame.values); + return; + case "empty": { + const row = this.#frames.at(-1); + if (row?.kind === "row") row.values[frame.field] = []; + return; + } + default: + return; + } + } + + /** Admit growth against live headroom rather than a fixed document size. */ + #charge(bytes: number, assemblingCodeUnits: number): void { + this.#projectedBytes += bytes; + if (this.#projectedBytes < this.#nextAdmission) return; + const heap = getHeapStatistics(); + // Reserve the next interval, the owner's per-row references, and a flat + // copy of the string still being assembled. + const requiredBytes = + ADMISSION_INTERVAL_BYTES + + this.#rows * ROW_REFERENCE_BYTES + + assemblingCodeUnits * 2; + if (requiredBytes > heap.total_available_size) + throw new AnalysisResourceConstraintError( + this.producer, + "memory", + `Insufficient heap headroom to retain more of ${this.label} after ${String(this.#rows)} rows`, + { + retained_rows: this.#rows, + projected_retained_bytes: this.#projectedBytes, + required_headroom_bytes: requiredBytes, + available_heap_bytes: heap.total_available_size, + heap_size_limit_bytes: heap.heap_size_limit, + }, + { + remediationAction: + "Close other targets in this REA process, or start REA with a larger Node.js heap (--max-old-space-size), then reopen the target.", + }, + ); + this.#nextAdmission = this.#projectedBytes + ADMISSION_INTERVAL_BYTES; + } +} diff --git a/src/kuna/KunaLimits.ts b/src/kuna/KunaLimits.ts index 6af5e4fe2e..2a9202da87 100644 --- a/src/kuna/KunaLimits.ts +++ b/src/kuna/KunaLimits.ts @@ -1,6 +1,7 @@ /** Shared buffered-text capacity for a subprocess answer or one cached source facet. */ export const KUNA_FUNCTION_TEXT_BYTES = 16 * 1024 * 1024; -/** One buffered, retained string scan; independent of decompilation. */ -export const KUNA_STRING_OUTPUT_BYTES = 16 * 1024 * 1024; +/** Kuna's notes and summary lines for one run; those of a large image stay far below this. */ +export const KUNA_DIAGNOSTIC_BYTES = 8 * 1024 * 1024; + export const KUNA_STRING_MIN_LENGTH = 5; diff --git a/src/kuna/KunaProjectExport.ts b/src/kuna/KunaProjectExport.ts index b6077ae4c2..5ca1cb1352 100644 --- a/src/kuna/KunaProjectExport.ts +++ b/src/kuna/KunaProjectExport.ts @@ -22,6 +22,7 @@ import { type KunaProcessLauncher, } from "./KunaCommand.js"; import type { KunaInstallation } from "./KunaInstallation.js"; +import { KUNA_DIAGNOSTIC_BYTES } from "./KunaLimits.js"; import { parseKunaExportSummary, parseKunaInventory, @@ -36,9 +37,6 @@ import { RegularFileCleanupFailure } from "../filesystem/OwnedRegularFile.js"; /** Operation name every export and cache failure reports. */ export const KUNA_EXPORT_OPERATION = "kuna:decompile-project"; -/** Kuna prints one summary line per export; notes on a large image stay far below this. */ -const DIAGNOSTIC_BYTES = 8 * 1024 * 1024; - const INVENTORY_FILE = "functions.json"; /** Identity of one file an export consists of. */ @@ -258,7 +256,8 @@ export const exportKunaProject = async ( "--jobs", String(configuration.jobs), ], - DIAGNOSTIC_BYTES, + // Kuna prints one summary line per export on stdout. + KUNA_DIAGNOSTIC_BYTES, ); if (!exported.ok) return exported.outcome; diff --git a/src/kuna/KunaQueryWork.ts b/src/kuna/KunaQueryWork.ts index 4150f197ed..2d5afedc0f 100644 --- a/src/kuna/KunaQueryWork.ts +++ b/src/kuna/KunaQueryWork.ts @@ -1,8 +1,13 @@ import type { AnalysisError } from "../domain/analysisErrorBase.js"; -import { AnalysisCancelledError } from "../domain/analysisErrorCore.js"; +import { + AnalysisCancelledError, + AnalysisResourceConstraintError, +} from "../domain/analysisErrorCore.js"; import { err, type Result } from "../domain/result.js"; import { ABORTED, waitForAbortable } from "../process/AbortablePromise.js"; import { matchRegexValues } from "../process/RegexMatcher.js"; +import type { KunaJsonDocumentReader } from "./KunaJsonStream.js"; +import { KUNA_DIAGNOSTIC_BYTES } from "./KunaLimits.js"; import { KunaRunSlots } from "./KunaRunSlots.js"; import type { KunaTargetCommands, @@ -62,6 +67,43 @@ export class KunaQueryWork { } } + /** + * Run a whole-program command whose stdout `reader` validates as it + * arrives. A rejected document stops Kuna and reports why; diagnostics + * beyond their buffer recur for this target and executable, so they are + * passed to `retain` for the session to keep. + */ + async scan( + request: Omit< + KunaTargetCommandRequest, + "releaseRun" | "outputBytes" | "stdout" | "signal" + >, + reader: KunaJsonDocumentReader, + signal: AbortSignal, + retain: (failure: AnalysisError) => void, + ): Promise> { + const run = await this.run({ + ...request, + outputBytes: KUNA_DIAGNOSTIC_BYTES, + stdout: reader, + signal: AbortSignal.any([signal, reader.rejected]), + }); + if (run.ok) return reader.finish(run.value.input); + const rejection = reader.rejection; + if ( + rejection !== undefined && + run.error instanceof AnalysisCancelledError && + !signal.aborted + ) + return err(rejection); + if ( + run.error instanceof AnalysisResourceConstraintError && + run.error.resource === "memory" + ) + retain(run.error); + return run; + } + /** Bound inventory copies and regex workers; close cancels and drains both. */ match(input: RegexQuery): ReturnType { const signal = AbortSignal.any([ diff --git a/src/kuna/KunaReferenceFiles.ts b/src/kuna/KunaReferenceFiles.ts index b12d48d324..6170b5c252 100644 --- a/src/kuna/KunaReferenceFiles.ts +++ b/src/kuna/KunaReferenceFiles.ts @@ -2,22 +2,16 @@ import { createHash } from "node:crypto"; import { z } from "zod"; +import type { AnalysisError } from "../domain/analysisErrorBase.js"; import { err, ok, type Result } from "../domain/result.js"; import { compareKunaAddresses } from "./KunaAddress.js"; import { parseKunaJson } from "./KunaJson.js"; +import { KunaJsonDocumentReader, parseKunaRow } from "./KunaJsonStream.js"; -/** Raw JSON budgets also bound the materialized indexes; no decompiled bodies are admitted. */ -export const KUNA_GRAPH_OUTPUT_BYTES = 32 * 1024 * 1024; +/** The raw xref budget also bounds its materialized rows. */ export const KUNA_XREF_OUTPUT_BYTES = 16 * 1024 * 1024; -const addressKeys = new Set([ - "address", - "callerAddress", - "calleeAddress", - "forwardsTo", - "from_address", - "to_address", -]); +const addressKeys = new Set(["address", "from_address", "to_address"]); const address = z.string().regex(/^0x(?:0|[1-9a-f][0-9a-f]{0,15})$/u); const count = z.number().int().min(0).max(Number.MAX_SAFE_INTEGER); const kind = z.enum(["call", "jump", "data", "read", "write"]); @@ -29,8 +23,6 @@ const graphHeader = z.object({ functionCount: count, edgeCount: count, }), - functions: z.array(z.unknown()), - edges: z.array(z.unknown()), }); const graphNode = z.object({ address, @@ -60,75 +52,160 @@ export interface KunaCallGraph { readonly callees: ReadonlyMap; } -/** Validate once at the producer boundary and retain only facts used by queries. */ -export const parseKunaCallGraph = ( - text: string, - input: string, -): Result => { - try { - const header = graphHeader.parse( - parseKunaJson(text, KUNA_GRAPH_OUTPUT_BYTES, addressKeys), - ); - if (header.binary.sourcePath !== input) - throw new Error("graph names a different input"); - if ( - header.binary.functionCount !== header.functions.length || - header.binary.edgeCount !== header.edges.length - ) - throw new Error("graph counts disagree with examined rows"); - const nodes = new Map(); - let hasIndirectCalls = false; - for (const raw of header.functions) { - const node = graphNode.parse(raw); - if (nodes.has(node.address)) - throw new Error("graph repeats a function address"); - nodes.set(node.address, node); - hasIndirectCalls ||= node.hasIndirectCalls; - } - const callers = new Map(); - const callees = new Map(); - const pairs = new Set(); - const orders = new Map>(); - for (const raw of header.edges) { - const edge = graphEdge.parse(raw); - if (!nodes.has(edge.callerAddress) || !nodes.has(edge.calleeAddress)) - throw new Error("graph edge has an unknown endpoint"); - const pair = `${edge.callerAddress}:${edge.calleeAddress}`; - if (pairs.has(pair)) throw new Error("graph repeats an edge"); - pairs.add(pair); - const order = orders.get(edge.callerAddress) ?? new Set(); - if (order.has(edge.calleeOrder)) - throw new Error("graph repeats a callee order"); - order.add(edge.calleeOrder); - orders.set(edge.callerAddress, order); - if (edge.kind !== "call") continue; - const incoming = callers.get(edge.calleeAddress) ?? []; - incoming.push(edge.callerAddress); - callers.set(edge.calleeAddress, incoming); - const outgoing = callees.get(edge.callerAddress) ?? []; - outgoing.push(edge.calleeAddress); - callees.set(edge.callerAddress, outgoing); - } - for (const order of orders.values()) - for (let index = 0; index < order.size; index += 1) - if (!order.has(index)) - throw new Error("graph callee orders are incomplete"); - for (const list of [...callers.values(), ...callees.values()]) - list.sort(compareKunaAddresses); - return ok({ - nodes, - callers, - callees, - edgeCount: header.edges.length, - hasIndirectCalls, - bytes: Buffer.byteLength(text), - sha256: createHash("sha256").update(text).digest("hex"), - }); - } catch (cause: unknown) { - return err(cause instanceof Error ? cause.message : String(cause)); - } +const fieldsOf = (schema: z.ZodObject, excluded: readonly string[]) => + new Set(Object.keys(schema.shape).filter((key) => !excluded.includes(key))); + +const GRAPH_DOCUMENT = { + header: new Set(["schemaVersion"]), + sections: new Map([ + ["binary", new Set(Object.keys(graphHeader.shape.binary.shape))], + ]), + rows: new Map([ + [ + "functions", + { + fields: fieldsOf(graphNode, [ + "address", + "forwardsTo", + "codeC", + "assembly", + "error", + ]), + addresses: new Set(["address", "forwardsTo"]), + empty: new Set(), + // `--functions ''` requests no bodies; one is refused unassembled. + nulls: new Set(["codeC", "assembly", "error"]), + }, + ], + [ + "edges", + { + fields: fieldsOf(graphEdge, ["callerAddress", "calleeAddress"]), + addresses: new Set(["callerAddress", "calleeAddress"]), + empty: new Set(), + nulls: new Set(), + }, + ], + ]), }; +/** + * Validates `kuna decompile-graph --functions ''` as Kuna writes it and + * retains only the facts call queries use. + */ +export class KunaCallGraphReader extends KunaJsonDocumentReader { + readonly #nodes = new Map(); + /** Callees of every edge by caller, which also detects repeated edges. */ + readonly #targets = new Map>(); + /** Callee orders by caller; each caller's must be 0..n-1. */ + readonly #orders = new Map>(); + readonly #callers = new Map(); + readonly #callees = new Map(); + #edgeCount = 0; + #hasIndirectCalls = false; + + constructor() { + super("kuna:call-graph", "Kuna's call graph", GRAPH_DOCUMENT); + } + + override finish(input: string): Result { + const completed = this.complete(); + if (!completed.ok) return completed; + const header = graphHeader.safeParse(completed.value.header); + if (!header.success) + return err(this.unusable("the graph header is not Kuna's schema 4")); + if (header.data.binary.sourcePath !== input) + return err(this.unusable("graph names a different input")); + if ( + header.data.binary.functionCount !== this.#nodes.size || + header.data.binary.edgeCount !== this.#edgeCount + ) + return err(this.unusable("graph counts disagree with examined rows")); + const invalid = this.#invalidEdges(); + // These indexes only validate edges; queries never read them. + this.#targets.clear(); + this.#orders.clear(); + if (invalid !== undefined) return err(this.unusable(invalid)); + for (const index of [this.#callers, this.#callees]) + for (const list of index.values()) list.sort(compareKunaAddresses); + return ok({ + nodes: this.#nodes, + callers: this.#callers, + callees: this.#callees, + edgeCount: this.#edgeCount, + hasIndirectCalls: this.#hasIndirectCalls, + bytes: completed.value.bytes, + sha256: completed.value.sha256, + }); + } + + protected override acceptRow( + member: string, + fields: Readonly>, + ): void { + if (member === "edges") { + this.#index(parseKunaRow(graphEdge, "graph edge", fields)); + return; + } + const node = parseKunaRow(graphNode, "graph function", fields); + if (this.#nodes.has(node.address)) + throw new Error("graph repeats a function address"); + this.#nodes.set(node.address, node); + this.#hasIndirectCalls ||= node.hasIndirectCalls; + } + + protected override discardRows(): void { + for (const index of [ + this.#nodes, + this.#targets, + this.#orders, + this.#callers, + this.#callees, + ]) + index.clear(); + } + + /** + * Index one edge as it arrives, so the working set grows under the row + * admission instead of after it; endpoints are checked once every node is + * known. + */ + #index(edge: z.infer): void { + const { callerAddress: caller, calleeAddress: callee } = edge; + this.#edgeCount += 1; + const targets = this.#targets.get(caller) ?? new Set(); + if (targets.has(callee)) throw new Error("graph repeats an edge"); + targets.add(callee); + this.#targets.set(caller, targets); + const order = this.#orders.get(caller) ?? new Set(); + if (order.has(edge.calleeOrder)) + throw new Error("graph repeats a callee order"); + order.add(edge.calleeOrder); + this.#orders.set(caller, order); + if (edge.kind !== "call") return; + const incoming = this.#callers.get(callee) ?? []; + incoming.push(caller); + this.#callers.set(callee, incoming); + const outgoing = this.#callees.get(caller) ?? []; + outgoing.push(callee); + this.#callees.set(caller, outgoing); + } + + /** Why the edges contradict the nodes, if they do. */ + #invalidEdges(): string | undefined { + for (const [caller, targets] of this.#targets) { + if (!this.#nodes.has(caller)) return "graph edge has an unknown endpoint"; + for (const callee of targets) + if (!this.#nodes.has(callee)) + return "graph edge has an unknown endpoint"; + } + for (const order of this.#orders.values()) + for (let index = 0; index < order.size; index += 1) + if (!order.has(index)) return "graph callee orders are incomplete"; + return undefined; + } +} + const xrefFunction = z.object({ name: z.string().min(1), address, diff --git a/src/kuna/KunaReferenceQueries.ts b/src/kuna/KunaReferenceQueries.ts index 44256f705e..ae422b5d4f 100644 --- a/src/kuna/KunaReferenceQueries.ts +++ b/src/kuna/KunaReferenceQueries.ts @@ -10,9 +10,8 @@ import { ProviderAdapterError } from "../domain/providerAdapterError.js"; import { err, ok, type Result } from "../domain/result.js"; import type { KunaInstallation } from "./KunaInstallation.js"; import { - KUNA_GRAPH_OUTPUT_BYTES, KUNA_XREF_OUTPUT_BYTES, - parseKunaCallGraph, + KunaCallGraphReader, parseKunaIncomingReferences, type KunaCallGraph, type KunaGraphNode, @@ -108,33 +107,25 @@ export class KunaReferenceQueries { } if (this.#pendingGraph === undefined) { const pending = this.work.start( - async (producerSignal) => { - const run = await this.work.run({ - installation, - operation: "kuna:call-graph", - arguments: (input) => [ - "decompile-graph", - input, - "--functions", - "", - ], - timeoutMs: installation.configuration.projectTimeoutMs, - outputBytes: KUNA_GRAPH_OUTPUT_BYTES, - signal: producerSignal, - }); - if (!run.ok) { - if ( - run.error instanceof AnalysisResourceConstraintError && - run.error.resource === "memory" - ) - this.#graphFailure = run.error; - return run; - } - return this.#parsed( - parseKunaCallGraph(run.value.stdout.text, run.value.input), - "call graph", - ); - }, + (producerSignal) => + this.work.scan( + { + installation, + operation: "kuna:call-graph", + arguments: (input) => [ + "decompile-graph", + input, + "--functions", + "", + ], + timeoutMs: installation.configuration.projectTimeoutMs, + }, + new KunaCallGraphReader(), + producerSignal, + (failure) => { + this.#graphFailure = failure; + }, + ), (value) => { this.#graph = value; }, diff --git a/src/kuna/KunaStringInventory.ts b/src/kuna/KunaStringInventory.ts index 295f7379d4..978f5b270f 100644 --- a/src/kuna/KunaStringInventory.ts +++ b/src/kuna/KunaStringInventory.ts @@ -1,20 +1,14 @@ -import { createHash } from "node:crypto"; - import { z } from "zod"; +import type { AnalysisError } from "../domain/analysisErrorBase.js"; import type { analysisStringSchema } from "../domain/hopperValues.js"; import { err, ok, type Result } from "../domain/result.js"; import { compareKunaAddresses } from "./KunaAddress.js"; -import { parseKunaJson } from "./KunaJson.js"; - -import { - KUNA_STRING_OUTPUT_BYTES, - KUNA_STRING_MIN_LENGTH, -} from "./KunaLimits.js"; +import { KunaJsonDocumentReader, parseKunaRow } from "./KunaJsonStream.js"; +import { KUNA_STRING_MIN_LENGTH } from "./KunaLimits.js"; const count = z.number().int().min(0).max(Number.MAX_SAFE_INTEGER); const address = z.string().regex(/^0x(?:0|[1-9a-f][0-9a-f]{0,15})$/u); -const addressKeys = new Set(["address"]); const header = z.object({ binary: z.string(), encoding: z.literal("all"), @@ -25,7 +19,6 @@ const header = z.object({ scanned: z.enum(["sections", "segments"]), xrefs: z.literal(false), count, - strings: z.array(z.unknown()), }); const row = z.object({ address, @@ -40,6 +33,28 @@ const row = z.object({ functions: z.tuple([]), }); +const ROWS = "strings"; +const DOCUMENT = { + header: new Set(Object.keys(header.shape)), + sections: new Map(), + rows: new Map([ + [ + ROWS, + { + fields: new Set( + Object.keys(row.shape).filter( + (key) => key !== "address" && key !== "functions", + ), + ), + addresses: new Set(["address"]), + // Rows are scanned with --no-xrefs, so `functions` must be empty. + empty: new Set(["functions"]), + nulls: new Set(), + }, + ], + ]), +} as const; + /** JSON text omits terminators; native lengths describe the encoded source. */ const validateStringFacts = (entry: z.infer): void => { let characters = 0; @@ -84,43 +99,30 @@ export interface KunaStringInventory { readonly scanned: "sections" | "segments"; } -/** Validate the producer's complete scan before publishing any selected row. */ -export const parseKunaStringInventory = ( - text: string, - input: string, -): Result => { - try { - const document = header.parse( - parseKunaJson(text, KUNA_STRING_OUTPUT_BYTES, addressKeys), - ); - if (document.binary !== input) - throw new Error("string inventory names a different input"); - if (document.count !== document.strings.length) - throw new Error("string inventory count disagrees with examined rows"); - const seen = new Set(); - const items = document.strings.map((value): KunaStringItem => { - const entry = row.parse(value); - validateStringFacts(entry); - if (entry.address !== entry.address_hex) - throw new Error("string address disagrees with its hex field"); - const identity = `${entry.address}:${entry.encoding}`; - if (seen.has(identity)) - throw new Error("string inventory repeats an identity"); - seen.add(identity); - return { - address: entry.address, - value: entry.text, - string: { - encoding: entry.encoding, - encoding_status: "observed", - termination: entry.nul_terminated - ? "present_or_not_required" - : "missing", - byte_length: entry.byte_length, - }, - }; - }); - items.sort( +/** Validates `kuna strings --json` row by row as Kuna writes it. */ +export class KunaStringInventoryReader extends KunaJsonDocumentReader { + readonly #items: KunaStringItem[] = []; + readonly #identities = new Set(); + + constructor() { + super("kuna:strings", "Kuna's string inventory", DOCUMENT); + } + + override finish(input: string): Result { + const completed = this.complete(); + if (!completed.ok) return completed; + const fields = header.safeParse(completed.value.header); + if (!fields.success) + return err( + this.unusable("the string inventory header is not Kuna's scan"), + ); + if (fields.data.binary !== input) + return err(this.unusable("string inventory names a different input")); + if (fields.data.count !== this.#items.length) + return err( + this.unusable("string inventory count disagrees with examined rows"), + ); + this.#items.sort( (left, right) => compareKunaAddresses(left.address, right.address) || (left.string?.encoding ?? "").localeCompare( @@ -129,12 +131,41 @@ export const parseKunaStringInventory = ( ), ); return ok({ - items, - sha256: createHash("sha256").update(text).digest("hex"), - bytes: Buffer.byteLength(text), - scanned: document.scanned, + items: this.#items, + sha256: completed.value.sha256, + bytes: completed.value.bytes, + scanned: fields.data.scanned, }); - } catch (cause: unknown) { - return err(cause instanceof Error ? cause.message : String(cause)); } -}; + + protected override acceptRow( + _member: string, + fields: Readonly>, + ): void { + const entry = parseKunaRow(row, "string", fields); + validateStringFacts(entry); + if (entry.address !== entry.address_hex) + throw new Error("string address disagrees with its hex field"); + const identity = `${entry.address}:${entry.encoding}`; + if (this.#identities.has(identity)) + throw new Error("string inventory repeats an identity"); + this.#identities.add(identity); + this.#items.push({ + address: entry.address, + value: entry.text, + string: { + encoding: entry.encoding, + encoding_status: "observed", + termination: entry.nul_terminated + ? "present_or_not_required" + : "missing", + byte_length: entry.byte_length, + }, + }); + } + + protected override discardRows(): void { + this.#items.length = 0; + this.#identities.clear(); + } +} diff --git a/src/kuna/KunaStringQueries.ts b/src/kuna/KunaStringQueries.ts index 71e30b797e..4cf49c5a7a 100644 --- a/src/kuna/KunaStringQueries.ts +++ b/src/kuna/KunaStringQueries.ts @@ -2,8 +2,6 @@ import type { AnalysisError } from "../domain/analysisErrorBase.js"; import { AnalysisCancelledError, AnalysisInputError, - AnalysisOutputError, - AnalysisResourceConstraintError, } from "../domain/analysisErrorCore.js"; import { analysisSearchPredicate, @@ -16,12 +14,9 @@ import { normalizeKunaAddress } from "./KunaAddress.js"; import type { KunaInstallation } from "./KunaInstallation.js"; import type { KunaPendingQuery, KunaQueryWork } from "./KunaQueryWork.js"; import { kunaQueryFailureForCaller } from "./KunaQueryErrors.js"; +import { KUNA_STRING_MIN_LENGTH } from "./KunaLimits.js"; import { - KUNA_STRING_MIN_LENGTH, - KUNA_STRING_OUTPUT_BYTES, -} from "./KunaLimits.js"; -import { - parseKunaStringInventory, + KunaStringInventoryReader, type KunaStringInventory, type KunaStringItem, } from "./KunaStringInventory.js"; @@ -129,46 +124,31 @@ export class KunaStringQueries { } if (this.#pending === undefined) { const pending = this.work.start( - async (producerSignal) => { - const run = await this.work.run({ - installation, - operation: "kuna:strings", - arguments: (input) => [ - "strings", - input, - "--json", - "--encoding", - "all", - "--min-length", - String(KUNA_STRING_MIN_LENGTH), - "--termination", - "any", - "--no-xrefs", - ], - timeoutMs: installation.configuration.projectTimeoutMs, - outputBytes: KUNA_STRING_OUTPUT_BYTES, - signal: producerSignal, - }); - if (!run.ok) { - if ( - run.error instanceof AnalysisResourceConstraintError && - run.error.resource === "memory" - ) - this.#failure = run.error; - return run; - } - const parsed = parseKunaStringInventory( - run.value.stdout.text, - run.value.input, - ); - return parsed.ok - ? parsed - : err( - new AnalysisOutputError("kuna:strings", parsed.error, { - userMessage: `Kuna's string inventory cannot be used: ${parsed.error}`, - }), - ); - }, + (producerSignal) => + this.work.scan( + { + installation, + operation: "kuna:strings", + arguments: (input) => [ + "strings", + input, + "--json", + "--encoding", + "all", + "--min-length", + String(KUNA_STRING_MIN_LENGTH), + "--termination", + "any", + "--no-xrefs", + ], + timeoutMs: installation.configuration.projectTimeoutMs, + }, + new KunaStringInventoryReader(), + producerSignal, + (failure) => { + this.#failure = failure; + }, + ), (value) => { this.#inventory = value; }, diff --git a/src/kuna/KunaTargetCommands.ts b/src/kuna/KunaTargetCommands.ts index b99d21b5ac..a3e11c296d 100644 --- a/src/kuna/KunaTargetCommands.ts +++ b/src/kuna/KunaTargetCommands.ts @@ -1,5 +1,6 @@ import { basename, join } from "node:path"; import { realpath } from "node:fs/promises"; +import type { Writable } from "node:stream"; import type { AnalysisError } from "../domain/analysisErrorBase.js"; import { @@ -44,6 +45,8 @@ export interface KunaTargetCommandRequest { readonly operation: string; readonly arguments: (input: string) => readonly string[]; readonly outputBytes: number; + /** Parses stdout as it arrives instead of buffering it in the snapshot. */ + readonly stdout?: Writable; readonly timeoutMs: number; readonly signal: AbortSignal; /** Admission remains owned until the native process is proven stopped. */ diff --git a/src/process/OwnedCommand.ts b/src/process/OwnedCommand.ts index 3a77c73425..13acb77288 100644 --- a/src/process/OwnedCommand.ts +++ b/src/process/OwnedCommand.ts @@ -1,3 +1,6 @@ +import type { Readable, Writable } from "node:stream"; +import { pipeline } from "node:stream/promises"; + import type { OwnedProviderProcessSpawnOptions, ProviderProcessSnapshot, @@ -13,6 +16,7 @@ import { type OwnedProcessGroup, type ProcessCleanupFailure, } from "./ProcessOwnership.js"; +import { ABORTED, waitForAbortable } from "./AbortablePromise.js"; import { ProviderStartupDeadline, waitForAbortableDelay, @@ -52,15 +56,25 @@ export const ownedCommandResources = ( `process-group:${ownership.processGroupId}`, ]; +/** Output handling for one owned command. */ +interface OwnedCommandLimits { + /** Retained stderr bytes, plus stdout bytes unless a sink consumes stdout. */ + readonly diagnosticBytes: number; + /** Accept an ordinary nonzero exit after output closes without other process errors. */ + readonly acceptNonZeroExit?: boolean; + /** + * Receives stdout instead of the retained snapshot, for output parsed as it + * arrives. The command completes only after the sink finishes. The sink + * records its own rejections and keeps accepting, so the command's outcome + * stays authoritative; a failing stream is a collection failure. + */ + readonly stdout?: Writable; +} + /** Execute a single owned process with independent cleanup and bounded diagnostic retention. */ export const runOwnedCommand = async ( spawn: OwnedProviderProcessSpawnOptions, - limits: { - readonly timeoutMs: number; - readonly diagnosticBytes: number; - /** Accept an ordinary nonzero exit after output closes without other process errors. */ - readonly acceptNonZeroExit?: boolean; - }, + limits: OwnedCommandLimits & { readonly timeoutMs: number }, options: { readonly signal?: AbortSignal | undefined; /** Synchronously accept unresolved cleanup ownership; execution still returns its original outcome. Must not throw. */ @@ -96,11 +110,7 @@ export const runOwnedCommand = async ( const collectOwnedCommand = async ( spawn: OwnedProviderProcessSpawnOptions, - limits: { - readonly diagnosticBytes: number; - /** Accept an ordinary nonzero exit after output closes without other process errors. */ - readonly acceptNonZeroExit?: boolean; - }, + limits: OwnedCommandLimits, launcher: ( input: OwnedProviderProcessSpawnOptions, ) => Promise, @@ -143,6 +153,7 @@ const collectOwnedCommand = async ( (() => cleanupOwnedProcessGroup(launched.ownership)), }, { + captureStdout: limits.stdout === undefined, maxDiagnosticBytes: limits.diagnosticBytes, onDiagnostic: (event) => { if (event.type === "output" && event.truncated === true) @@ -151,6 +162,16 @@ const collectOwnedCommand = async ( }, }, ); + // A command that fails or is interrupted destroys its sink with its stdout. + const abandoned = new AbortController(); + const consumed = consumeStdout( + launched.process.stdout, + limits.stdout, + AbortSignal.any([signal, abandoned.signal]), + ); + void consumed.then((failed) => { + processFailure ??= failed; + }); let failure: OwnedCommandFailure | undefined; try { while (!(await supervisor.waitForOutputClose(10))) { @@ -173,6 +194,9 @@ const collectOwnedCommand = async ( }); await waitForAbortableDelay(25, signal); } + // Output has closed, but the sink may still be parsing what it buffered. + const sinkFailure = await waitForAbortable(consumed, signal); + if (sinkFailure !== ABORTED) processFailure ??= sinkFailure; const snapshot = supervisor.snapshot(); const interruption = deadline.interruption; if (interruption !== undefined) @@ -220,6 +244,8 @@ const collectOwnedCommand = async ( { cause }, ); } + // A finished sink ignores this; an unfinished one is destroyed. + abandoned.abort(); const stopped = await supervisor.stop(); const snapshot = supervisor.snapshot(); if (stopped.status === "incomplete" && retainCleanup !== undefined) @@ -261,3 +287,24 @@ const collectOwnedCommand = async ( throw new OwnedCommandFailure("cancelled", "Command cancelled.", snapshot); return snapshot; }; + +/** + * Pipe stdout into its sink until `signal` aborts, which destroys both; settles + * with a failure message, never rejects. + */ +const consumeStdout = ( + stdout: Readable | null, + sink: Writable | undefined, + signal: AbortSignal, +): Promise => { + if (sink === undefined) return Promise.resolve(undefined); + if (stdout === null) { + sink.destroy(); + return Promise.resolve("stdout stream is unavailable"); + } + return pipeline(stdout, sink, { signal }).then( + () => undefined, + (cause: unknown) => + `stdout stream failed: ${cause instanceof Error ? cause.message : String(cause)}`, + ); +}; diff --git a/tests/boundary/process/ownedCommand.test.ts b/tests/boundary/process/ownedCommand.test.ts index 33c1d09552..8eda618907 100644 --- a/tests/boundary/process/ownedCommand.test.ts +++ b/tests/boundary/process/ownedCommand.test.ts @@ -1,6 +1,6 @@ import { randomUUID } from "node:crypto"; import { EventEmitter } from "node:events"; -import { PassThrough } from "node:stream"; +import { PassThrough, Writable } from "node:stream"; import { setTimeout as delay } from "node:timers/promises"; import { beforeAll, expect, it, vi } from "vitest"; import { @@ -101,6 +101,75 @@ it("collects a real owned command and verifies its exit before returning diagnos }); }); +it("streams stdout beyond the diagnostic budget into a sink and returns after it finishes", async () => { + let received = 0; + let finished = false; + const sink = new Writable({ + write(chunk: Buffer, _encoding, callback) { + received += chunk.byteLength; + callback(); + }, + final(callback) { + // A sink still parsing after output closes delays the result. + setTimeout(() => { + finished = true; + callback(); + }, 50); + }, + }); + const result = await runOwnedCommand( + command( + 'process.stdout.write("a".repeat(256 * 1024)); process.stderr.write("diagnostic")', + ), + { timeoutMs: 2_000, diagnosticBytes: 1024, stdout: sink }, + ); + expect(finished).toBe(true); + expect(received).toBe(256 * 1024); + expect(result).toMatchObject({ + exitCode: 0, + stdout: { text: "", bytes: 0 }, + stderr: { text: "diagnostic" }, + }); +}); + +it("destroys a sink still finishing when the command deadline elapses", async () => { + const sink = new Writable({ + write(_chunk, _encoding, callback) { + callback(); + }, + final(callback) { + setTimeout(callback, 5_000); + }, + }); + await expect( + runOwnedCommand(command('process.stdout.write("observed-output")'), { + timeoutMs: 300, + diagnosticBytes: 1024, + stdout: sink, + }), + ).rejects.toMatchObject({ reason: "timeout" }); + expect(sink.destroyed).toBe(true); +}); + +it("reports a failed stdout sink as a collection failure", async () => { + const sink = new Writable({ + write(_chunk, _encoding, callback) { + callback(new Error("fixture sink failure")); + }, + }); + await expect( + runOwnedCommand(command('process.stdout.write("observed-output")'), { + timeoutMs: 2_000, + diagnosticBytes: 1024, + stdout: sink, + }), + ).rejects.toMatchObject({ + reason: "process", + collectionFailure: true, + message: "stdout stream failed: fixture sink failure", + }); +}); + it("fails an accepted nonzero command when an output stream errors and cleans it up", async () => { const stdout = new PassThrough(); const stderr = new PassThrough(); diff --git a/tests/boundary/providers/kuna/fakeKuna.ts b/tests/boundary/providers/kuna/fakeKuna.ts index 082abfe2e8..0965bc2f58 100644 --- a/tests/boundary/providers/kuna/fakeKuna.ts +++ b/tests/boundary/providers/kuna/fakeKuna.ts @@ -47,22 +47,34 @@ export interface FakeKunaBehavior { readonly inventoryDelayMs?: number; /** Milliseconds the export takes before writing its files. */ readonly delayMs?: number; - /** `oversized` writes more graph output than REA buffers (32 MiB). */ + /** `noisy` writes more diagnostics than REA buffers (8 MiB) before the graph. */ readonly graph?: | "ok" | "hang" | "not-json" | "miscounted" | "with-bodies" - | "oversized"; + | "noisy"; readonly graphIndirectCalls?: boolean; + /** Bytes of an extra root member in the graph JSON. */ + readonly graphPadding?: number; readonly xrefs?: "ok" | "hang" | "not-json"; readonly referenceDelayMs?: number; readonly referencePadding?: number; - readonly strings?: "ok" | "hang" | "not-json" | "miscounted" | "exit-failure"; + /** `invalid-then-hang` writes a contradictory first row, then never finishes. */ + readonly strings?: + | "ok" + | "hang" + | "not-json" + | "miscounted" + | "exit-failure" + | "invalid-then-hang"; readonly stringRows?: readonly FakeKunaString[]; readonly stringDelayMs?: number; + /** Bytes of an extra root member in the string JSON. */ readonly stringPadding?: number; + /** Bytes written to stderr before the string JSON. */ + readonly stringStderrBytes?: number; } /** @@ -96,7 +108,9 @@ if (args[0] === "--version") { if (behavior.strings === "exit-failure") { process.stderr.write("fixture string scan failed\n"); process.exit(7); } else if (behavior.strings === "hang") hang(); else if (behavior.strings === "not-json") process.stdout.write("not string JSON"); + else if (behavior.strings === "invalid-then-hang") { process.stdout.write('{"binary":' + JSON.stringify(args[1]) + ',"strings":[{"address":12288,"address_hex":"0x3001"}'); hang(); } else { + if (behavior.stringStderrBytes) process.stderr.write(Buffer.alloc(behavior.stringStderrBytes, 0x78)); if (!args.includes("--no-xrefs") || args[args.indexOf("--encoding") + 1] !== "all") throw new Error("strings must skip reference analysis and include all supported encodings"); const rows = (behavior.stringRows ?? [{ address: 12288, text: "REA fixture prompt" }, { address: 12320, text: "Δelta", encoding: "utf8", terminated: false }]).map((item) => ({ address: item.address, address_hex: "0x" + item.address.toString(16), text: item.text, length: [...item.text].length, byte_length: Buffer.byteLength(item.text, item.encoding === "utf16" ? "utf16le" : "utf8") + (item.terminated === false ? 0 : item.encoding === "utf16" ? 2 : 1), nul_terminated: item.terminated !== false, encoding: item.encoding ?? "ascii", section: ".rodata", xrefs_count: 0, functions: [] })); const document = { binary: args[1], encoding: "all", min_length: 5, filter: null, section: null, termination: "any", scanned: "sections", xrefs: false, count: rows.length + (behavior.strings === "miscounted" ? 1 : 0), strings: rows, padding: "x".repeat(behavior.stringPadding ?? 0) }; @@ -105,15 +119,15 @@ if (args[0] === "--version") { } else if (args[0] === "decompile-graph") { if (behavior.graph === "hang") hang(); else if (behavior.graph === "not-json") process.stdout.write("not graph JSON"); - else if (behavior.graph === "oversized") process.stdout.write(Buffer.alloc(33 * 1024 * 1024, 0x20)); else { + if (behavior.graph === "noisy") process.stderr.write(Buffer.alloc(9 * 1024 * 1024, 0x78)); if (args[args.indexOf("--functions") + 1] !== "") throw new Error("graph must request no bodies"); const nodes = [...functions, { name: "puts", addr: "0x2000" }].map((fn) => ({ address: parseInt(fn.addr, 16), name: fn.name, kind: "normal", hasIndirectCalls: behavior.graphIndirectCalls !== false && fn === functions[0], forwardsTo: null, codeC: behavior.graph === "with-bodies" ? "unrequested code" : null, assembly: null, error: null })); const edges = []; if (functions[1]) edges.push({ callerAddress: nodes[0].address, calleeAddress: nodes[1].address, kind: "call", calleeOrder: 0 }); if (functions[2]) edges.push({ callerAddress: nodes[0].address, calleeAddress: nodes[2].address, kind: functions[2].error ? "jump" : "call", calleeOrder: 1 }); if (functions[1]?.name === "parse") edges.push({ callerAddress: nodes[1].address, calleeAddress: 8192, kind: "call", calleeOrder: 0 }); - setTimeout(() => process.stdout.write(JSON.stringify({ schemaVersion: 4, binary: { sourcePath: args[1], functionCount: nodes.length + (behavior.graph === "miscounted" ? 1 : 0), edgeCount: edges.length }, functions: nodes, edges })), behavior.referenceDelayMs ?? 0); + setTimeout(() => process.stdout.write(JSON.stringify({ schemaVersion: 4, binary: { sourcePath: args[1], functionCount: nodes.length + (behavior.graph === "miscounted" ? 1 : 0), edgeCount: edges.length }, functions: nodes, edges, padding: "x".repeat(behavior.graphPadding ?? 0) })), behavior.referenceDelayMs ?? 0); } } else if (args[0] === "xrefs") { if (behavior.xrefs === "hang") hang(); diff --git a/tests/boundary/providers/kuna/kunaCommandFailures.test.ts b/tests/boundary/providers/kuna/kunaCommandFailures.test.ts index dce2a60196..80ec77dd89 100644 --- a/tests/boundary/providers/kuna/kunaCommandFailures.test.ts +++ b/tests/boundary/providers/kuna/kunaCommandFailures.test.ts @@ -53,9 +53,38 @@ it("reports output beyond REA's fixed buffer as a deterministic memory bound", a } }); -it("reports a graph beyond REA's buffer once per session instead of rebuilding it", async () => { +it("parses a graph larger than any fixed output buffer as it streams", async () => { + const padding = 40 * 1024 * 1024; + const { open } = await session("rea-kuna-graph-large-", { + graphPadding: padding, + }); + const client = open(); + try { + const result = await client.execute("procedure_callees", { + procedure: "main", + }); + expect(result).toMatchObject({ + ok: true, + value: { + result: ["0x1040"], + rawResult: { + references: { kind: "kuna-call-graph", functions: 4, edges: 3 }, + }, + }, + }); + if (!result.ok) return; + expect( + (result.value.rawResult as { references: { bytes: number } }).references + .bytes, + ).toBeGreaterThan(padding); + } finally { + expect(await client.close()).toEqual({ ok: true, value: null }); + } +}); + +it("reports graph diagnostics beyond REA's buffer once per session instead of rebuilding it", async () => { const { kuna, open } = await session("rea-kuna-graph-overflow-", { - graph: "oversized", + graph: "noisy", }); const client = open(); try { @@ -63,7 +92,7 @@ it("reports a graph beyond REA's buffer once per session instead of rebuilding i const result = await client.execute("procedure_callers", { procedure: "0x1000", }); - if (result.ok) throw new Error("the oversized graph was admitted"); + if (result.ok) throw new Error("the overflowing graph was admitted"); expect(result.error).toMatchObject({ _tag: "AnalysisResourceConstraintError", resource: "memory", diff --git a/tests/boundary/providers/kuna/kunaReferenceFiles.test.ts b/tests/boundary/providers/kuna/kunaReferenceFiles.test.ts index cb5fac03d5..12eee495b7 100644 --- a/tests/boundary/providers/kuna/kunaReferenceFiles.test.ts +++ b/tests/boundary/providers/kuna/kunaReferenceFiles.test.ts @@ -1,11 +1,17 @@ -import { expect, it } from "vitest"; +import { Readable } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import * as v8 from "node:v8"; + +import { expect, it, vi } from "vitest"; import { normalizeKunaAddress } from "../../../../src/kuna/KunaAddress.js"; import { - parseKunaCallGraph, + KunaCallGraphReader, parseKunaIncomingReferences, } from "../../../../src/kuna/KunaReferenceFiles.js"; +vi.mock("node:v8", { spy: true }); + const input = "/snapshot/program"; // Fixture-only numeric placeholders preserve decimal u64 tokens in JSON text. const wire = (value: unknown) => @@ -44,8 +50,19 @@ const graph = () => ({ ], }); -it("preserves u64 numeric tokens and excludes non-call edges", () => { - const result = parseKunaCallGraph(wire(graph()), input); +/** Stream a graph through the reader in small pieces, as Kuna's stdout arrives. */ +const parseKunaCallGraph = async (text: string, selected: string) => { + const bytes = Buffer.from(text); + const chunks: Buffer[] = []; + for (let offset = 0; offset < bytes.byteLength; offset += 7) + chunks.push(bytes.subarray(offset, offset + 7)); + const reader = new KunaCallGraphReader(); + await pipeline(Readable.from(chunks), reader); + return reader.finish(selected); +}; + +it("preserves u64 numeric tokens and excludes non-call edges", async () => { + const result = await parseKunaCallGraph(wire(graph()), input); expect(result.ok).toBe(true); if (!result.ok) return; expect([...result.value.nodes.keys()]).toEqual([ @@ -62,9 +79,9 @@ it("preserves u64 numeric tokens and excludes non-call edges", () => { expect(result.value.callers.has("0x1")).toBe(false); }); -it("keeps a forwarding instruction address separate from function identity", () => { +it("keeps a forwarding instruction address separate from function identity", async () => { const value = graph(); - const result = parseKunaCallGraph( + const result = await parseKunaCallGraph( wire({ ...value, functions: value.functions.map((fn, index) => @@ -135,16 +152,21 @@ it.each([ ], ] as const)( "rejects graph %s rather than returning an empty relationship set", - (_name, mutate) => { + async (_name, mutate) => { const value = graph(); mutate(value); - expect(parseKunaCallGraph(wire(value), input).ok).toBe(false); + expect(await parseKunaCallGraph(wire(value), input)).toMatchObject({ + ok: false, + error: { _tag: "AnalysisOutputError", operation: "kuna:call-graph" }, + }); }, ); -it("rejects unrequested bodies and a document for a different input", () => { +it("rejects unrequested bodies and a document for a different input", async () => { const value = graph(); - expect(parseKunaCallGraph(wire(value), "/other/program").ok).toBe(false); + expect((await parseKunaCallGraph(wire(value), "/other/program")).ok).toBe( + false, + ); const body = { ...value, functions: value.functions.map((fn) => ({ @@ -152,7 +174,79 @@ it("rejects unrequested bodies and a document for a different input", () => { codeC: "must not be materialized as reference data", })), }; - expect(parseKunaCallGraph(wire(body), input).ok).toBe(false); + expect(await parseKunaCallGraph(wire(body), input)).toMatchObject({ + ok: false, + error: { reason: "functions row codeC must be null" }, + }); +}); + +it("admits the edge index while edges stream and refuses it under low headroom", async () => { + const nodes = Array.from({ length: 300 }, (_, index) => + node(String(index + 1), `f${String(index)}`), + ); + const edges = Array.from({ length: 3000 }, (_, index) => ({ + callerAddress: `@${String((index % 10) + 1)}`, + calleeAddress: `@${String(Math.floor(index / 10) + 1)}`, + kind: "call", + calleeOrder: Math.floor(index / 10), + })); + const text = wire({ + schemaVersion: 4, + binary: { sourcePath: input, functionCount: 300, edgeCount: 3000 }, + functions: nodes, + edges, + }); + expect(await parseKunaCallGraph(text, input)).toMatchObject({ + ok: true, + value: { edgeCount: 3000 }, + }); + const heap = v8.getHeapStatistics(); + let samples = 0; + // Headroom runs out after the node rows have been admitted. + vi.mocked(v8.getHeapStatistics).mockImplementation(() => ({ + ...heap, + total_available_size: + (samples += 1) <= 2 ? heap.total_available_size : 1024, + })); + try { + const refused = await parseKunaCallGraph(text, input); + expect(refused).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisResourceConstraintError", + resource: "memory", + }, + }); + if (refused.ok) return; + const retained = ( + refused.error as unknown as { + reportedLimits: { retained_rows: number }; + } + ).reportedLimits.retained_rows; + expect(retained).toBeGreaterThan(nodes.length); + expect(retained).toBeLessThan(nodes.length + edges.length); + } finally { + vi.mocked(v8.getHeapStatistics).mockReset(); + } +}); + +it("skips fields it does not interpret, including nested parameters", async () => { + const value = graph(); + const result = await parseKunaCallGraph( + wire({ + ...value, + binary: { ...value.binary, name: "program", analysisImageBase: 0 }, + functions: value.functions.map((fn) => ({ + ...fn, + size: 16, + parameters: [{ name: "a0", type: "int" }], + signature: "int f(int a0)", + isEntryPoint: false, + })), + }), + input, + ); + expect(result).toMatchObject({ ok: true, value: { edgeCount: 2 } }); }); const identity = (name: string, value: number) => ({ diff --git a/tests/boundary/providers/kuna/kunaReferences.test.ts b/tests/boundary/providers/kuna/kunaReferences.test.ts index b15ee8eea5..e1dbe3d84d 100644 --- a/tests/boundary/providers/kuna/kunaReferences.test.ts +++ b/tests/boundary/providers/kuna/kunaReferences.test.ts @@ -271,7 +271,10 @@ it.each(["not-json", "miscounted", "with-bodies"] as const)( try { expect( await client.execute("procedure_callees", { procedure: "main" }), - ).toMatchObject({ ok: false, error: { _tag: "ProviderAdapterError" } }); + ).toMatchObject({ + ok: false, + error: { _tag: "AnalysisOutputError", operation: "kuna:call-graph" }, + }); await setFakeKunaBehavior(kuna, { functions: FUNCTIONS }); expect( (await run(client, "procedure_callees", { procedure: "main" })).result, diff --git a/tests/boundary/providers/kuna/kunaStringInventory.test.ts b/tests/boundary/providers/kuna/kunaStringInventory.test.ts index 850c1fc1fb..4bc1cb40bf 100644 --- a/tests/boundary/providers/kuna/kunaStringInventory.test.ts +++ b/tests/boundary/providers/kuna/kunaStringInventory.test.ts @@ -1,6 +1,13 @@ -import { expect, it } from "vitest"; +import { createHash } from "node:crypto"; +import { Readable } from "node:stream"; +import { pipeline } from "node:stream/promises"; +import * as v8 from "node:v8"; -import { parseKunaStringInventory } from "../../../../src/kuna/KunaStringInventory.js"; +import { expect, it, vi } from "vitest"; + +import { KunaStringInventoryReader } from "../../../../src/kuna/KunaStringInventory.js"; + +vi.mock("node:v8", { spy: true }); const input = "/private/program"; const row = { @@ -30,12 +37,25 @@ const document = () => ({ const wire = (value: unknown) => JSON.stringify(value).replace(/"@(\d+)"/gu, "$1"); -it("preserves exact u64 string identity, multiline text and typed source facts", () => { - const result = parseKunaStringInventory(wire(document()), input); +/** Stream `text` through a reader in `chunkBytes` pieces, as Kuna's stdout arrives. */ +const read = async (text: string, chunkBytes = 64 * 1024) => { + const bytes = Buffer.from(text); + const chunks: Buffer[] = []; + for (let offset = 0; offset < bytes.byteLength; offset += chunkBytes) + chunks.push(bytes.subarray(offset, offset + chunkBytes)); + const reader = new KunaStringInventoryReader(); + await pipeline(Readable.from(chunks), reader); + return { reader, result: reader.finish(input) }; +}; + +it("preserves exact u64 string identity, multiline text and typed source facts", async () => { + const text = wire(document()); + const { result } = await read(text, 1); expect(result).toMatchObject({ ok: true, value: { - sha256: expect.stringMatching(/^[0-9a-f]{64}$/u), + sha256: createHash("sha256").update(text).digest("hex"), + bytes: Buffer.byteLength(text), scanned: "sections", items: [ { @@ -53,6 +73,18 @@ it("preserves exact u64 string identity, multiline text and typed source facts", }); }); +it("skips members it does not interpret without retaining them", async () => { + const { result } = await read( + wire({ + ...document(), + padding: "x".repeat(1024), + notes: [{ a: [1] }], + strings: [{ ...row, score: 1, origin: { tool: [1] } }], + }).replace('"functions":[]', '"functions":[],"__proto__":{"text":1}'), + ); + expect(result).toMatchObject({ ok: true, value: { items: [{}] } }); +}); + it.each([ ["input", { binary: "/other/program" }], ["count", { count: 2 }], @@ -60,29 +92,136 @@ it.each([ ["xrefs", { xrefs: true }], ["encoding", { encoding: "ascii" }], ["scan scope", { scanned: "unknown" }], + ["header shape", { section: [] }], + ["missing rows", { strings: undefined }], [ "address identity", { strings: [{ ...row, address_hex: "0x20000000000000" }] }, ], ["u64 overflow", { strings: [{ ...row, address: "@18446744073709551616" }] }], + ["textual address", { strings: [{ ...row, address: "0x20000000000001" }] }], ["partial row", { strings: [{ ...row, byte_length: null }] }], ["unexpected references", { strings: [{ ...row, xrefs_count: 1 }] }], + [ + "referencing functions", + { strings: [{ ...row, functions: [{ name: "main" }] }] }, + ], ["duplicate identity", { count: 2, strings: [row, row] }], ])( "rejects inconsistent %s before publishing any strings", - (_label, changes) => { - expect( - parseKunaStringInventory(wire({ ...document(), ...changes }), input).ok, - ).toBe(false); + async (_label, changes) => { + const { result } = await read(wire({ ...document(), ...changes })); + expect(result).toMatchObject({ + ok: false, + error: { _tag: "AnalysisOutputError", operation: "kuna:strings" }, + }); }, ); +it.each([ + ["truncated", wire(document()).slice(0, -2)], + ["trailing", `${wire(document())} {}`], + ["array root", "[]"], + ["empty", ""], +])("rejects a %s document", async (_label, text) => { + expect((await read(text)).result.ok).toBe(false); +}); + +it("rejects invalid UTF-8 instead of substituting characters", async () => { + const reader = new KunaStringInventoryReader(); + const text = Buffer.from(wire(document())); + const at = text.indexOf("Δ"); + text[at] = 0xff; + await pipeline(Readable.from([text]), reader); + expect(reader.finish(input).ok).toBe(false); +}); + +it("signals the first contradiction while Kuna is still writing", async () => { + const reader = new KunaStringInventoryReader(); + const rows = [{ ...row, byte_length: 1 }, row]; + const text = wire({ ...document(), count: 2, strings: rows }); + reader.write(Buffer.from(text.slice(0, text.lastIndexOf("{")))); + await new Promise((resolve) => setImmediate(resolve)); + expect(reader.rejected.aborted).toBe(true); + expect(reader.rejection).toMatchObject({ _tag: "AnalysisOutputError" }); + reader.end(Buffer.from(text.slice(text.lastIndexOf("{")))); + await new Promise((resolve) => reader.once("finish", resolve)); + expect(reader.finish(input)).toMatchObject({ + ok: false, + error: reader.rejection, + }); +}); + +it("refuses retention beyond live heap headroom and stops the scan", async () => { + const heap = v8.getHeapStatistics(); + // The first retained row samples headroom. + vi.mocked(v8.getHeapStatistics).mockReturnValueOnce({ + ...heap, + total_available_size: 1024, + }); + const { reader, result } = await read(wire(document())); + expect(reader.rejected.aborted).toBe(true); + expect(result).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisResourceConstraintError", + operation: "kuna:strings", + resource: "memory", + reportedLimits: { retained_rows: 0, available_heap_bytes: 1024 }, + }, + }); +}); + +it("refuses a kept string while it is assembled, before Kuna finishes it", async () => { + const heap = v8.getHeapStatistics(); + const text = wire(document()); + const opening = text.slice(0, text.indexOf("a\\nΔelta")); + const reader = new KunaStringInventoryReader(); + reader.write(Buffer.from(opening)); + vi.mocked(v8.getHeapStatistics).mockReturnValueOnce({ + ...heap, + total_available_size: 1024, + }); + // One sampling interval of text, still inside one unterminated string. + reader.write(Buffer.from("x".repeat(64 * 1024))); + await new Promise((resolve) => setImmediate(resolve)); + expect(reader.rejection).toMatchObject({ + _tag: "AnalysisResourceConstraintError", + resource: "memory", + }); + reader.destroy(); +}); + +it("bounds keys and numbers: overlong keys match no field, overlong kept numbers reject", async () => { + const longKey = `count${"x".repeat(1024 * 1024)}`; + const ignored = wire({ ...document(), [longKey]: "@1" + "0".repeat(4096) }); + expect((await read(ignored, 4096)).result).toMatchObject({ + ok: true, + value: { items: [{}] }, + }); + const kept = wire({ ...document(), count: "@1" + "0".repeat(64) }); + expect((await read(kept)).result).toMatchObject({ + ok: false, + error: { reason: "count is not a number Kuna writes" }, + }); +}); + +it("drops ignored strings unassembled, without sampling headroom", async () => { + const sampled = vi.mocked(v8.getHeapStatistics).mock.calls.length; + const { result } = await read( + wire({ ...document(), padding: "x".repeat(4 * 1024 * 1024) }), + ); + expect(result.ok).toBe(true); + // Only the one retained row's admission is sampled. + expect(vi.mocked(v8.getHeapStatistics).mock.calls.length - sampled).toBe(1); +}); + it.each([ { encoding: "utf8", text: "😀😀😀😀😀", length: 5, byte_length: 21 }, { encoding: "utf16", text: "café!", length: 5, byte_length: 12 }, -])("counts $encoding source units without losing text", (facts) => { +])("counts $encoding source units without losing text", async (facts) => { const value = { ...document(), strings: [{ ...row, ...facts }] }; - expect(parseKunaStringInventory(wire(value), input)).toMatchObject({ + expect((await read(wire(value), 3)).result).toMatchObject({ ok: true, value: { items: [ @@ -102,7 +241,7 @@ it.each([ { encoding: "utf8", text: "hello\u0000", length: 6, byte_length: 7 }, { encoding: "utf8", text: "hello\ud800", length: 6, byte_length: 9 }, { encoding: "utf16", text: "Δelta", length: 5, byte_length: 12 }, -])("rejects conflicting encoded string facts: %j", (facts) => { +])("rejects conflicting encoded string facts: %j", async (facts) => { const value = { ...document(), strings: [{ ...row, ...facts }] }; - expect(parseKunaStringInventory(wire(value), input).ok).toBe(false); + expect((await read(wire(value))).result.ok).toBe(false); }); diff --git a/tests/boundary/providers/kuna/kunaStrings.test.ts b/tests/boundary/providers/kuna/kunaStrings.test.ts index 7bd6b837b9..ebb18a8b87 100644 --- a/tests/boundary/providers/kuna/kunaStrings.test.ts +++ b/tests/boundary/providers/kuna/kunaStrings.test.ts @@ -1,5 +1,6 @@ import { expect, it as test, vi } from "vitest"; import { readKunaConfiguration } from "../../../../src/kuna/KunaConfiguration.js"; +import { KUNA_DIAGNOSTIC_BYTES } from "../../../../src/kuna/KunaLimits.js"; import { projectAnalysisError } from "../../../../src/domain/analysisErrorProjection.js"; import { ProviderCleanupError } from "../../../../src/domain/providerCleanupError.js"; @@ -115,6 +116,57 @@ it.each(["not-json", "miscounted"] as const)( }, ); +it("parses a scan larger than any fixed output buffer as it streams", async () => { + const padding = 24 * 1024 * 1024; + const { open } = await session("rea-kuna-strings-large-", { + stringPadding: padding, + }); + const client = open(); + try { + const listed = await run(client, "list_strings"); + expect(listed.result).toHaveLength(2); + expect(listed.rawResult).toMatchObject({ + strings: { + bytes: expect.any(Number), + inventory_count: 2, + }, + }); + expect( + (listed.rawResult as { strings: { bytes: number } }).strings.bytes, + ).toBeGreaterThan(padding); + } finally { + expect(await client.close()).toEqual({ ok: true, value: null }); + } +}); + +it("stops Kuna at the first contradictory row instead of awaiting the rest", async () => { + const { open, kuna } = await session( + "rea-kuna-strings-reject-", + { strings: "invalid-then-hang" }, + { REA_KUNA_PROJECT_TIMEOUT_MS: "60000" }, + ); + const client = open(); + try { + const started = Date.now(); + expect( + await client.execute("search_strings", { pattern: "fixture" }), + ).toMatchObject({ + ok: false, + error: { + _tag: "AnalysisOutputError", + operation: "search_strings", + diagnostics: { producer_operation: "kuna:strings" }, + }, + }); + expect(Date.now() - started).toBeLessThan(30_000); + await setFakeKunaBehavior(kuna, {}); + expect((await run(client, "list_strings")).result).toHaveLength(2); + expect(await scans(kuna)).toHaveLength(2); + } finally { + expect(await client.close()).toEqual({ ok: true, value: null }); + } +}); + it("cancels one waiter without stopping the shared scan, and rebuilds in a fresh session", async () => { const { open, kuna } = await session("rea-kuna-strings-waiters-", { stringDelayMs: 200, @@ -283,7 +335,7 @@ it.each([ const { open, kuna } = await session( "rea-kuna-shared-string-failure-", mode === "oversize" - ? { stringPadding: 16 * 1024 * 1024 } + ? { stringStderrBytes: KUNA_DIAGNOSTIC_BYTES + 1 } : { strings: mode }, { REA_KUNA_PROJECT_TIMEOUT_MS: mode === "hang" ? "500" : "5000" }, );