Files
rea/src/process/OwnedCommand.ts
T
Kaoru0822 8ab5024397 fix(kuna): stream string scans instead of capping them at 16 MiB (#4318)
* fix(kuna): stream string scans instead of capping them at 16 MiB

`list_strings` and `search_strings` buffered Kuna's whole string JSON and
refused anything above 16 MiB. Kuna writes about 290 bytes per row, so
every app above roughly 10 MB failed with a non-retryable memory error:
a 20 MB Ghostty binary produces 31 MB, Raycast 30 MB and an arm64
Firefox XUL 179 MB.

Parse the scan as Kuna writes it. `runOwnedCommand` accepts a stdout
sink, and `KunaStringInventoryReader` owns the interpretation: it
tokenizes synchronously with stream-json, keeps exact u64 address
tokens, validates each row before retaining it, skips members it does
not interpret, and admits retained rows against live heap headroom
rather than a fixed scan size. The first contradiction stops Kuna
instead of waiting for the rest; an incomplete document is judged only
after Kuna's own exit status. Diagnostics keep the 8 MiB Kuna budget,
now shared with the project export.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): admit string scan text as it streams and drop sinks with their command

Review found that packed string tokens were assembled whole before the
reader could discard them or check heap headroom, and that a stdout sink
could outlive a cancelled or timed-out command.

Move the token mechanics into `KunaJsonDocumentReader`: keys and numbers
(Kuna's own fields) stay packed, while strings, which carry target-derived
text, arrive in chunks. Ignored strings are dropped unassembled and kept
ones are admitted against live heap headroom as they grow. Kept text is
copied out of the decoded chunk it was sliced from, so retained rows no
longer pin whole chunks (XUL: 278 MB retained before, 156 MB after).
`KunaStringInventoryReader` keeps only the string-scan semantics.

`runOwnedCommand` now ties the stdout pipeline to the command's deadline
signal and aborts it when collection fails, destroying the sink.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): type the stream-json tokenizer where it is imported

The Reverse tools quality lane type-checks a partial project that reaches
KunaJsonStream without the separate streamJsonParser.d.ts augmentation,
so jsonParser had no declaration there. Declare the tokenizer's shape at
its only import site and remove the augmentation file, so every
compilation of the reader is self-contained. Split two helpers out of
the token dispatcher and simplify the stdout sink wait in
runOwnedCommand, which keeps the added complexity minimal.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): bound key and number assembly in streamed Kuna JSON

Review found that packed keys (and packed numbers) were assembled whole
by the tokenizer before the reader could discard or constrain them, so a
malformed scan with one enormous key could still exhaust the heap.

Stream every scalar. Keys are assembled only up to 64 code units, past
every field name the reader keeps, so a longer key is skipped unmatched;
kept numbers longer than that are rejected, and ignored numbers are
dropped unassembled. XUL's 611k-row scan parses in the same time.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): stream the call graph instead of capping it at 32 MiB (#4329)

* fix(kuna): stream the call graph instead of capping it at 32 MiB

Call relationships buffered `kuna decompile-graph` and refused output
above 32 MiB. A 53 MB Raycast binary produces a 157 MB graph (180k
functions, 773k edges), so every call query failed with a non-retryable
memory error after Kuna's minute-long whole-program walk.

`KunaCallGraphReader` validates the graph through the shared streaming
document reader. It retains only the facts queries use (126 MB for
Raycast) and checks edges against the complete node set once Kuna
exits. Unrequested bodies are refused before any of their text is
assembled, through new must-be-null row fields; address fields may be
null, as `forwardsTo` is. Malformed graphs are now `AnalysisOutputError`
like the string scan.

`KunaQueryWork.scan` owns the run-and-validate sequence both producers
shared: diagnostics budget, early rejection and retention of
deterministic overflow.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(kuna): index call-graph edges as they stream

Review found that the reader kept every parsed edge and built the edge
indexes only after streaming, outside any heap admission, so a large
graph could exhaust the heap instead of returning a memory constraint.

Index each edge when it arrives: duplicate pairs and callee orders are
checked per caller, call edges go straight into the caller/callee
indexes, and no edge array is kept, so the whole working set grows under
the per-row admission. Endpoints and order completeness are checked once
all nodes are known, and the validation-only indexes are released then.
Raycast's graph now peaks at 346 MB instead of 503 MB and retains 93 MB
instead of 126 MB, in the same time.

Also drop imports the shared scan made unused.

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(kuna): state the graph's retained size after incremental indexing

Refs #1957

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 21:53:59 -07:00

311 lines
10 KiB
TypeScript

import type { Readable, Writable } from "node:stream";
import { pipeline } from "node:stream/promises";
import type {
OwnedProviderProcessSpawnOptions,
ProviderProcessSnapshot,
ProviderProcessStopResult,
SpawnedOwnedProviderProcess,
} from "./ProviderProcess.js";
import {
ProviderProcessSupervisor,
spawnOwnedProviderProcess,
} from "./ProviderProcess.js";
import {
cleanupOwnedProcessGroup,
type OwnedProcessGroup,
type ProcessCleanupFailure,
} from "./ProcessOwnership.js";
import { ABORTED, waitForAbortable } from "./AbortablePromise.js";
import {
ProviderStartupDeadline,
waitForAbortableDelay,
} from "./ProviderDeadline.js";
/** A short-lived command failed its lifecycle or complete-output contract. */
export class OwnedCommandFailure extends Error {
readonly cleanupResult: ProcessCleanupFailure | undefined;
readonly collectionFailure: boolean;
constructor(
readonly reason: "cancelled" | "timeout" | "output-limit" | "process",
message: string,
readonly snapshot: ProviderProcessSnapshot | null = null,
readonly cleanupFailure: string | null = null,
options?: ErrorOptions & {
readonly cleanupResult?: ProcessCleanupFailure;
readonly collectionFailure?: boolean;
},
readonly resources: readonly string[] = [],
readonly cleanupOwner?: ProviderProcessSupervisor,
) {
super(message, options);
this.cleanupResult = options?.cleanupResult;
this.collectionFailure = options?.collectionFailure ?? false;
}
}
/** Report recorded launch identities; these labels do not authorize signaling. */
export const ownedCommandResources = (
ownership: OwnedProcessGroup,
directory?: string,
): readonly string[] => [
ownership.runId,
...(directory === undefined ? [] : [directory]),
`pid:${ownership.leaderPid}`,
`process-group:${ownership.processGroupId}`,
];
/** Output handling for one owned command. */
interface OwnedCommandLimits {
/** Retained stderr bytes, plus stdout bytes unless a sink consumes stdout. */
readonly diagnosticBytes: number;
/** Accept an ordinary nonzero exit after output closes without other process errors. */
readonly acceptNonZeroExit?: boolean;
/**
* Receives stdout instead of the retained snapshot, for output parsed as it
* arrives. The command completes only after the sink finishes. The sink
* records its own rejections and keeps accepting, so the command's outcome
* stays authoritative; a failing stream is a collection failure.
*/
readonly stdout?: Writable;
}
/** Execute a single owned process with independent cleanup and bounded diagnostic retention. */
export const runOwnedCommand = async (
spawn: OwnedProviderProcessSpawnOptions,
limits: OwnedCommandLimits & { readonly timeoutMs: number },
options: {
readonly signal?: AbortSignal | undefined;
/** Synchronously accept unresolved cleanup ownership; execution still returns its original outcome. Must not throw. */
readonly retainCleanup?: (
owner: ProviderProcessSupervisor,
result: Extract<ProviderProcessStopResult, { status: "incomplete" }>,
) => void;
readonly launcher?: (
input: OwnedProviderProcessSpawnOptions,
) => Promise<SpawnedOwnedProviderProcess>;
} = {},
): Promise<ProviderProcessSnapshot> => {
const callerSignal = options.signal ?? spawn.signal;
if (callerSignal?.aborted)
throw new OwnedCommandFailure(
"cancelled",
"Command cancelled before launch.",
);
const deadline = new ProviderStartupDeadline(limits.timeoutMs, callerSignal);
try {
return await collectOwnedCommand(
spawn,
limits,
options.launcher ?? spawnOwnedProviderProcess,
deadline,
callerSignal,
options.retainCleanup,
);
} finally {
deadline.dispose();
}
};
const collectOwnedCommand = async (
spawn: OwnedProviderProcessSpawnOptions,
limits: OwnedCommandLimits,
launcher: (
input: OwnedProviderProcessSpawnOptions,
) => Promise<SpawnedOwnedProviderProcess>,
deadline: ProviderStartupDeadline,
callerSignal?: AbortSignal,
retainCleanup?: (
owner: ProviderProcessSupervisor,
result: Extract<ProviderProcessStopResult, { status: "incomplete" }>,
) => void,
): Promise<ProviderProcessSnapshot> => {
const signal = deadline.signal;
let launched: SpawnedOwnedProviderProcess;
try {
launched = await launcher({
...spawn,
signal,
});
} catch (cause: unknown) {
const interruption = deadline.interruption;
if (interruption !== undefined)
throw new OwnedCommandFailure(
interruption,
interruption === "cancelled"
? "Command cancelled during ownership preparation."
: "Command deadline elapsed during ownership preparation.",
null,
null,
{ cause },
);
throw cause;
}
let exceeded = false;
let processFailure: string | undefined;
const supervisor = new ProviderProcessSupervisor(
{
...launched,
ownsProcessLifetime: true,
cleanup:
launched.cleanup ??
(() => cleanupOwnedProcessGroup(launched.ownership)),
},
{
captureStdout: limits.stdout === undefined,
maxDiagnosticBytes: limits.diagnosticBytes,
onDiagnostic: (event) => {
if (event.type === "output" && event.truncated === true)
exceeded = true;
if (event.type === "error") processFailure = event.message;
},
},
);
// A command that fails or is interrupted destroys its sink with its stdout.
const abandoned = new AbortController();
const consumed = consumeStdout(
launched.process.stdout,
limits.stdout,
AbortSignal.any([signal, abandoned.signal]),
);
void consumed.then((failed) => {
processFailure ??= failed;
});
let failure: OwnedCommandFailure | undefined;
try {
while (!(await supervisor.waitForOutputClose(10))) {
const interruption = deadline.interruption;
if (interruption !== undefined)
throw new OwnedCommandFailure(
interruption,
interruption === "cancelled"
? "Command cancelled."
: "Command deadline elapsed.",
);
if (exceeded)
throw new OwnedCommandFailure(
"output-limit",
"Command diagnostic output exceeded its complete-output budget.",
);
if (processFailure !== undefined)
throw new OwnedCommandFailure("process", processFailure, null, null, {
collectionFailure: true,
});
await waitForAbortableDelay(25, signal);
}
// Output has closed, but the sink may still be parsing what it buffered.
const sinkFailure = await waitForAbortable(consumed, signal);
if (sinkFailure !== ABORTED) processFailure ??= sinkFailure;
const snapshot = supervisor.snapshot();
const interruption = deadline.interruption;
if (interruption !== undefined)
throw new OwnedCommandFailure(
interruption,
interruption === "cancelled"
? "Command cancelled."
: "Command deadline elapsed.",
snapshot,
);
if (exceeded)
throw new OwnedCommandFailure(
"output-limit",
"Command diagnostic output exceeded its complete-output budget.",
snapshot,
);
const acceptedNonzeroExit =
limits.acceptNonZeroExit === true &&
snapshot.exitCode !== null &&
snapshot.exitCode !== undefined &&
snapshot.exitCode !== 0 &&
snapshot.signal === null;
if (
processFailure !== undefined ||
((snapshot.exitCode !== 0 || snapshot.signal !== null) &&
!acceptedNonzeroExit)
)
throw new OwnedCommandFailure(
"process",
processFailure ??
"Command failed; inspect its exit status and diagnostics.",
snapshot,
null,
{ collectionFailure: processFailure !== undefined },
);
} catch (cause: unknown) {
failure =
cause instanceof OwnedCommandFailure
? cause
: new OwnedCommandFailure(
"process",
"Command collection failed.",
null,
null,
{ cause },
);
}
// A finished sink ignores this; an unfinished one is destroyed.
abandoned.abort();
const stopped = await supervisor.stop();
const snapshot = supervisor.snapshot();
if (stopped.status === "incomplete" && retainCleanup !== undefined)
retainCleanup(supervisor, stopped);
else if (stopped.status === "incomplete")
throw new OwnedCommandFailure(
failure?.reason ?? "process",
failure?.message ?? "Command cleanup failed.",
snapshot,
stopped.reason,
{
...(failure === undefined ? {} : { cause: failure }),
...(stopped.cleanupResult === undefined
? {}
: { cleanupResult: stopped.cleanupResult }),
...(failure?.collectionFailure === true
? { collectionFailure: true }
: {}),
},
ownedCommandResources(launched.ownership, spawn.cwd),
supervisor,
);
if (failure !== undefined)
throw new OwnedCommandFailure(
failure.reason,
failure.message,
snapshot,
failure.cleanupFailure,
{
cause: failure,
...(failure.cleanupResult === undefined
? {}
: { cleanupResult: failure.cleanupResult }),
...(failure.collectionFailure ? { collectionFailure: true } : {}),
},
failure.resources,
);
if (callerSignal?.aborted)
throw new OwnedCommandFailure("cancelled", "Command cancelled.", snapshot);
return snapshot;
};
/**
* Pipe stdout into its sink until `signal` aborts, which destroys both; settles
* with a failure message, never rejects.
*/
const consumeStdout = (
stdout: Readable | null,
sink: Writable | undefined,
signal: AbortSignal,
): Promise<string | undefined> => {
if (sink === undefined) return Promise.resolve(undefined);
if (stdout === null) {
sink.destroy();
return Promise.resolve("stdout stream is unavailable");
}
return pipeline(stdout, sink, { signal }).then(
() => undefined,
(cause: unknown) =>
`stdout stream failed: ${cause instanceof Error ? cause.message : String(cause)}`,
);
};