fix(ida): wait out a transient busy worker health in headless sessions (#1778)

* test(ida): expect a transient busy worker health to be waited out

* fix(ida): wait out a transient busy worker health before reading identity

* docs(ida): record the verified macOS arm64 headless workflow and the busy-health wait

* test(ida): pin the busy wait deadline, cause and reported tool, and assert clean release

* fix(ida): repeat only a tool-name-shaped busy tool and keep the wait in its own module

* test(ida): drive the busy wait with a virtual clock and pin its bound and cancellation

* fix(ida): stop the busy wait at its deadline without another read and honour a late abort

* docs(ida): state when the busy wait times out, cancels or reports an unknown tool
This commit is contained in:
Mykola Dementiev
2026-10-11 11:25:27 +02:00
committed by GitHub
parent 6e38529333
commit 0bdc7a40ac
7 changed files with 524 additions and 12 deletions
+2 -2
View File
@@ -92,7 +92,7 @@ Use a package version that actually includes this adapter; repository main can b
{"name":"close_binary","arguments":{}}
```
Opening an MCP target starts its selected provider session; headless opening waits for auto-analysis. Set the agent's MCP request deadline long enough for the target. The registration's optional `timeoutMs` defaults to 300000 for upstream requests. Cancellation drains the current upstream request before cleanup; it does not force-stop GUI analysis. An open request that times out without a completed response retains the private workspace because worker inventory alone cannot prove the request has stopped. `binary_session` reports actual tool availability. Repeated doctor/setup calls are unnecessary when the registration already works.
Opening an MCP target starts its selected provider session; headless opening waits for auto-analysis. Set the agent's MCP request deadline long enough for the target. The registration's optional `timeoutMs` defaults to 300000 for upstream requests. A freshly opened headless worker can briefly report `status: "busy"` from `server_health`; REA re-reads the health every 100 ms until the worker answers, the caller cancels, or `timeoutMs` has elapsed since the first read. Only expiry while the worker is still busy produces a retryable `provider_timeout`; no further read is started once the deadline has passed. That error names the busy tool of the last observation, or `unknown` when the worker reported none or a value that is not tool-name-shaped. Cancellation during the wait produces `cancelled`. Any other unexpected health payload, including a busy payload with an invalid shape, remains `unreadable_output`. Cancellation drains the current upstream request before cleanup; it does not force-stop GUI analysis. An open request that times out without a completed response retains the private workspace because worker inventory alone cannot prove the request has stopped. `binary_session` reports actual tool availability. Repeated doctor/setup calls are unnecessary when the registration already works.
For guided registration, set `REA_IDA_MCP_CONFIG` before `rea setup --client <client>`. Setup includes that file reference in its reviewed plan and preserves it after approval; it does not copy upstream credentials into the agent configuration. `rea doctor --provider ida --json` validates the registration without starting IDA and explicitly leaves runtime compatibility and target binding for analysis.
@@ -110,4 +110,4 @@ Complete body ranges, typed reference edges, CFG, referenced data, source locals
Run `npm run verify:ida -- --target /absolute/path/to/program --procedure main` with a configured upstream server. It exercises production CLI and MCP tools, validates their output contracts and Evidence, checks original-input preservation and session cleanup, and keeps local target data out of its console summary. Supply `--package-root` to verify an extracted or installed package. See [testing](testing.md) for the distinction between adapter tests and real-provider evidence.
The initial real workflows use stdio registrations for the legacy Windows GUI profile and a Windows x64 headless supervisor with IDA 9.3. Streamable HTTP is tested against a loopback MCP fixture with the pinned client SDK; a real IDA HTTP-supervisor workflow remains unverified. Linux/macOS headless engine operation, other IDA releases, architectures, and modern attached GUI tools require their corresponding real workflows before support is claimed.
The initial real workflows use stdio registrations for the legacy Windows GUI profile and a Windows x64 headless supervisor with IDA 9.3. The headless workflow has also passed on macOS arm64 with IDA Professional 9.1 (build 9.1.250226) against the same upstream commit `c133c3853faa111a9b00ee615c013b720d0c4acd`, using a source-built arm64 Mach-O fixture; Mach-O C symbols carry a leading underscore, so the procedure name is `_rea_fixture_add`. Streamable HTTP is tested against a loopback MCP fixture with the pinned client SDK; a real IDA HTTP-supervisor workflow remains unverified. Linux headless engine operation, other IDA releases, architectures, and modern attached GUI tools require their corresponding real workflows before support is claimed.
+9 -3
View File
@@ -739,9 +739,15 @@ entries, external callees, target switches, cancellation draining, snapshot
replay exclusion, ownership failures, and incomplete cleanup. They do not
establish real IDA operation. The initial real workflows cover legacy upstream
1.4.0 on a Windows GUI and the modern supervisor at upstream commit
`c133c3853faa111a9b00ee615c013b720d0c4acd` with Windows x64 IDA 9.3.
Linux/macOS headless, modern attached GUI tools, other engine versions and
architectures remain unverified; see the [provider guide](ida-provider.md).
`c133c3853faa111a9b00ee615c013b720d0c4acd` with Windows x64 IDA 9.3, and the
headless profile on macOS arm64 with IDA Professional 9.1 (build 9.1.250226)
at the same upstream commit, using `tests/conformance/ida/inventory.c` built
with `clang -O0 -arch arm64` and the procedure `_rea_fixture_add` (Mach-O C
symbols carry a leading underscore). Before the headless session waited out
the transient `busy` health a worker can report right after opening, that lane
failed intermittently on that host. Linux headless, modern attached GUI tools,
other engine versions and architectures remain unverified; see the
[provider guide](ida-provider.md).
## DOS Ghidra analysis
+133
View File
@@ -0,0 +1,133 @@
import { expect, it } from "vitest";
import type { JsonValue } from "../domain/jsonValue.js";
import type { IdaMcpConnection } from "./IdaMcpConnection.js";
import { readSettledHealth } from "./IdaBusyHealth.js";
const busy = (tool: string): JsonValue => ({
status: "busy",
busy_tool: tool,
busy_sec: 0,
queued_calls: 0,
});
const ok: JsonValue = { status: "ok", module: "target" };
/** Connection that answers each health read from a script and counts reads. */
const scripted = (answers: readonly JsonValue[]) => {
const reads: number[] = [];
const connection: IdaMcpConnection = {
connect: async () => [],
serverInfo: () => null,
close: async () => undefined,
call: async () => {
reads.push(reads.length);
const answer = answers[Math.min(reads.length - 1, answers.length - 1)];
if (answer === undefined) throw new Error("empty script");
return answer;
},
};
return { connection, reads };
};
/** Virtual clock: sleeping advances time, nothing really waits. */
const virtualClock = (onSleep?: (ms: number) => void) => {
let time = 0;
const sleeps: number[] = [];
return {
sleeps,
clock: {
now: () => time,
delay: async (ms: number) => {
sleeps.push(ms);
time += ms;
onSleep?.(ms);
},
},
};
};
it("sleeps the fixed retry delay between busy reads and returns the settled answer", async () => {
const { connection, reads } = scripted([busy("a"), busy("a"), ok]);
const { clock, sleeps } = virtualClock();
const raw = await readSettledHealth(connection, {
database: "db",
timeoutMs: 10_000,
clock,
});
expect(raw, "the settled payload is returned unchanged").toEqual(ok);
expect(reads.length, "one read per poll").toBe(3);
expect(sleeps, "two fixed 100 ms pauses").toEqual([100, 100]);
});
it("clamps the last pause to the remaining budget and stops without another read at expiry", async () => {
const { connection, reads } = scripted([busy("config_json_get")]);
const { clock, sleeps } = virtualClock();
const failure = await readSettledHealth(connection, {
database: "db",
timeoutMs: 350,
clock,
}).catch((cause: unknown) => cause);
expect(sleeps, "three full pauses, then the 50 ms remainder").toEqual([
100, 100, 100, 50,
]);
expect(reads.length, "no read is started once the budget is spent").toBe(4);
expect(failure, "expiry while busy is a timeout").toMatchObject({
_tag: "AnalysisTimeoutError",
timeoutMs: 350,
});
});
it("does not read again when the only pause consumes the whole budget", async () => {
const { connection, reads } = scripted([busy("config_json_get")]);
const { clock } = virtualClock();
const failure = await readSettledHealth(connection, {
database: "db",
timeoutMs: 100,
clock,
}).catch((cause: unknown) => cause);
expect(reads.length, "the single read before the pause is the only one").toBe(
1,
);
expect(failure, "the last busy payload ends the wait").toMatchObject({
_tag: "AnalysisTimeoutError",
});
});
it("reports the busy tool of the last observation", async () => {
const { connection } = scripted([busy("first_tool"), busy("last_tool")]);
const { clock } = virtualClock();
const failure = await readSettledHealth(connection, {
database: "db",
timeoutMs: 150,
clock,
}).catch((cause: unknown) => cause);
expect(failure, "the operation names the tool seen last").toMatchObject({
operation: "server_health: headless worker still busy in last_tool",
cause: busy("last_tool"),
});
});
it("rejects without another read when the signal aborts after the pause has fulfilled", async () => {
const controller = new AbortController();
const { connection, reads } = scripted([busy("config_json_get")]);
const { clock } = virtualClock(() => controller.abort());
const failure = await readSettledHealth(connection, {
database: "db",
timeoutMs: 10_000,
signal: controller.signal,
clock,
}).catch((cause: unknown) => cause);
expect(reads.length, "the abort wins over the next read").toBe(1);
expect(failure, "the abort is surfaced, not a timeout").toMatchObject({
name: "AbortError",
});
});
+74
View File
@@ -0,0 +1,74 @@
import { setTimeout as sleep } from "node:timers/promises";
import { AnalysisTimeoutError } from "../domain/analysisErrorCore.js";
import type { JsonValue } from "../domain/jsonValue.js";
import type { IdaMcpConnection } from "./IdaMcpConnection.js";
import { busyHealthSchema } from "./IdaProtocolValues.js";
/** Pause between health reads while a freshly opened worker reports itself busy. */
const BUSY_HEALTH_RETRY_MS = 100;
/** Only a tool-name-shaped value is repeated; other producer text is not echoed into errors. */
const BUSY_TOOL_NAME = /^[A-Za-z0-9_.:-]{1,64}$/u;
/** Time source and pause used by the wait; tests substitute a virtual clock. */
export interface BusyHealthClock {
now(): number;
delay(ms: number, signal: AbortSignal | undefined): Promise<void>;
}
const monotonicClock: BusyHealthClock = {
now: () => performance.now(),
delay: (ms, signal) => sleep(ms, undefined, { signal }),
};
const busyToolName = (tool: string | undefined): string =>
tool !== undefined && BUSY_TOOL_NAME.test(tool) ? tool : "unknown";
const isBusyStatus = (raw: JsonValue): boolean =>
typeof raw === "object" &&
raw !== null &&
!Array.isArray(raw) &&
raw["status"] === "busy";
/**
* Reads `server_health` until the worker stops reporting a transient busy
* state. The wait ends when the worker answers, the caller aborts, or the
* registration's request deadline has elapsed; once it has elapsed no further
* read is started and the last busy observation is reported. Any other payload
* is returned for the caller's strict parse, except an invalid busy payload,
* which keeps its own diagnostic.
*/
export const readSettledHealth = async (
connection: IdaMcpConnection,
{
database,
timeoutMs,
signal,
clock = monotonicClock,
}: {
readonly database: string;
readonly timeoutMs: number;
readonly signal?: AbortSignal | undefined;
readonly clock?: BusyHealthClock;
},
): Promise<JsonValue> => {
const started = clock.now();
const remaining = (): number => timeoutMs - (clock.now() - started);
for (;;) {
const raw = await connection.call("server_health", { database });
const busy = busyHealthSchema.safeParse(raw);
if (!busy.success) {
if (isBusyStatus(raw)) throw busy.error;
return raw;
}
const expired = (): AnalysisTimeoutError =>
new AnalysisTimeoutError(
`server_health: headless worker still busy in ${busyToolName(busy.data.busy_tool)}`,
timeoutMs,
{ cause: raw },
);
if (remaining() <= 0) throw expired();
await clock.delay(Math.min(BUSY_HEALTH_RETRY_MS, remaining()), signal);
signal?.throwIfAborted();
if (remaining() <= 0) throw expired();
}
};
+7
View File
@@ -37,6 +37,13 @@ export const legacyMetadataSchema = z
base: idaAddressSchema,
})
.passthrough();
/** A worker that is still busy right after opening reports this transient shape instead of its identity. */
export const busyHealthSchema = z
.object({
status: z.literal("busy"),
busy_tool: z.string().optional(),
})
.passthrough();
export const modernHealthSchema = z
.object({
status: z.literal("ok"),
+292 -3
View File
@@ -8,14 +8,18 @@ import {
} from "../../tests/fixtures/idaMcp.js";
import { IdaSessionClient } from "./IdaSessionClient.js";
import { functionDossierSchema } from "../domain/hopperValues.js";
import { jsonValueSchema } from "../domain/jsonValue.js";
import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { projectAnalysisError } from "../domain/analysisErrorProjection.js";
const roots: string[] = [];
afterEach(async () => {
for (const root of roots.splice(0))
await rm(root, { recursive: true, force: true });
});
const fixture = async (mode: "attached" | "headless" = "attached") => {
const fixture = async (
mode: "attached" | "headless" = "attached",
timeoutMs = 1000,
) => {
const { root, target } = await createIdaTarget();
roots.push(root);
const producer = new RecordingIdaMcp(target, mode);
@@ -25,7 +29,7 @@ const fixture = async (mode: "attached" | "headless" = "attached") => {
args: [],
env: {},
mode,
timeoutMs: 1000,
timeoutMs,
workspaceRoot: root,
},
target,
@@ -328,6 +332,291 @@ describe("attached IDA function observations", () => {
});
});
const busyHealth = (busyTool?: JsonValue): JsonValue => ({
status: "busy",
uptime_sec: 1.041,
...(busyTool === undefined ? {} : { busy_tool: busyTool }),
busy_sec: 0,
queued_calls: 0,
});
const healthCalls = (producer: { calls: { name: string }[] }) =>
producer.calls.filter(({ name }) => name === "server_health").length;
const expectCleanClose = async (
producer: { closes: number },
client: { close: () => Promise<{ ok: boolean }> },
): Promise<void> => {
expect((await client.close()).ok, "the owned worker is released").toBe(true);
expect(producer.closes, "the upstream connection is closed once").toBe(1);
};
describe("headless IDA transient busy health", () => {
it("re-reads server_health until the worker reports ok", async () => {
const { producer, client } = await fixture("headless");
let busyReads = 2;
producer.overrideCall = (name) =>
name === "server_health" && busyReads-- > 0
? busyHealth("config_json_get")
: undefined;
const result = await client.execute("health", {});
expect(result.ok, "a transient busy worker is waited out").toBe(true);
expect(
healthCalls(producer),
"two busy reads and the ok read that settles the identity",
).toBe(3);
await expectCleanClose(producer, client);
});
it("returns a retryable timeout naming the busy tool at the registration deadline", async () => {
const { producer, client } = await fixture("headless", 300);
const busy = busyHealth("config_json_get");
producer.overrideCall = (name) =>
name === "server_health" ? busy : undefined;
const started = Date.now();
const result = await client.execute("health", {});
const elapsed = Date.now() - started;
if (result.ok) throw new Error("expected a busy timeout");
expect(result.error._tag, "the neutral timeout class is used").toBe(
"AnalysisTimeoutError",
);
const projected = projectAnalysisError(result.error);
expect(projected.code, "the outcome is a provider timeout").toBe(
"provider_timeout",
);
expect(projected.retryable, "the caller may retry").toBe(true);
expect(
projected.details,
"the observed busy tool is named exactly in the operation",
).toMatchObject({
operation: "server_health: headless worker still busy in config_json_get",
timeout_ms: 300,
});
expect(
result.error.message,
"the error message names the busy tool",
).toContain("config_json_get");
expect(
`${result.error.message} ${projected.message}`,
"no Hopper wording leaks in",
).not.toContain("Hopper");
expect(result.error.cause, "the raw busy payload is the cause").toEqual(
busy,
);
expect(
elapsed,
"the wait runs to the deadline, not less",
).toBeGreaterThanOrEqual(290);
expect(
healthCalls(producer),
"reads are spaced by the retry delay, not a tight loop",
).toBeLessThanOrEqual(6);
await expectCleanClose(producer, client);
});
it("never invents a busy tool the worker did not report", async () => {
const { producer, client } = await fixture("headless", 300);
producer.overrideCall = (name) =>
name === "server_health" ? busyHealth() : undefined;
const result = await client.execute("health", {});
if (result.ok) throw new Error("expected a busy timeout");
expect(
projectAnalysisError(result.error).details,
"an omitted busy_tool is reported as exactly unknown",
).toMatchObject({
operation: "server_health: headless worker still busy in unknown",
});
await expectCleanClose(producer, client);
});
it.each([
["free-form text", "token=abc123 /private/home/user secret"],
["an over-long name", "x".repeat(200)],
])(
"reports %s from the worker as unknown instead of echoing it",
async (_label, tool) => {
const { producer, client } = await fixture("headless", 300);
producer.overrideCall = (name) =>
name === "server_health" ? busyHealth(tool) : undefined;
const result = await client.execute("health", {});
if (result.ok) throw new Error("expected a busy timeout");
expect(
projectAnalysisError(result.error).details,
"only a tool-name-shaped value is repeated",
).toMatchObject({
operation: "server_health: headless worker still busy in unknown",
});
expect(
result.error.message,
"the producer text never reaches the error message",
).not.toContain(tool);
await expectCleanClose(producer, client);
},
);
it("ends the wait at once when the caller aborts", async () => {
const { producer, client } = await fixture("headless", 30_000);
producer.overrideCall = (name) =>
name === "server_health" ? busyHealth("config_json_get") : undefined;
const controller = new AbortController();
const started = Date.now();
const call = client.execute("health", {}, { signal: controller.signal });
while (healthCalls(producer) < 2) {
if (Date.now() - started > 5_000) throw new Error("never polled");
await new Promise((resolve) => setTimeout(resolve, 10));
}
controller.abort();
const result = await call;
if (result.ok) throw new Error("expected cancellation");
expect(result.error._tag, "cancellation keeps today's outcome").toBe(
"AnalysisCancelledError",
);
expect(
Date.now() - started,
"the wait does not run to the registration deadline",
).toBeLessThan(5_000);
await expectCleanClose(producer, client);
});
});
describe("headless IDA busy health after an operation", () => {
const withInventory = (
producer: RecordingIdaMcp,
afterObservation: (read: number) => JsonValue | undefined,
) => {
let health = 0;
producer.overrideCall = (name) => {
if (name === "list_funcs")
return [
{
data: [{ addr: "0x1000", name: "main", size: "0x20" }],
next_offset: null,
},
];
if (name !== "server_health") return undefined;
health += 1;
return health < 2 ? undefined : afterObservation(health);
};
};
it("waits out a busy observation taken after the operation", async () => {
const { producer, client } = await fixture("headless");
withInventory(producer, (read) =>
read === 2 ? busyHealth("config_json_get") : undefined,
);
const result = await client.execute("list_procedures", {});
expect(result.ok, "the operation succeeds once the worker settles").toBe(
true,
);
expect(healthCalls(producer), "before, busy after, ok after").toBe(3);
await expectCleanClose(producer, client);
});
it("maps a cancellation during that wait to cancelled", async () => {
const { producer, client } = await fixture("headless", 30_000);
withInventory(producer, () => busyHealth("config_json_get"));
const controller = new AbortController();
const call = client.execute(
"list_procedures",
{},
{ signal: controller.signal },
);
const started = Date.now();
while (healthCalls(producer) < 3) {
if (Date.now() - started > 5_000) throw new Error("never polled");
await new Promise((resolve) => setTimeout(resolve, 10));
}
controller.abort();
const result = await call;
if (result.ok) throw new Error("expected cancellation");
expect(result.error._tag, "the after-observation wait is cancellable").toBe(
"AnalysisCancelledError",
);
await expectCleanClose(producer, client);
});
});
describe("headless IDA non-busy health payloads", () => {
it.each([
["an unknown status", { status: "starting" }],
["a non-string busy_tool", busyHealth(5)],
["a payload without a status", { uptime_sec: 1 }],
])(
"keeps %s as unreadable output without waiting",
async (_label, payload) => {
const { producer, client } = await fixture("headless");
producer.overrideCall = (name) =>
name === "server_health" ? jsonValueSchema.parse(payload) : undefined;
const result = await client.execute("health", {});
if (result.ok) throw new Error("expected unreadable output");
expect(
projectAnalysisError(result.error).code,
"only the narrow busy shape enters the wait",
).toBe("unreadable_output");
expect(healthCalls(producer), "no re-read happened").toBe(1);
await expectCleanClose(producer, client);
},
);
it("keeps the busy-shape reason when a busy payload is invalid", async () => {
const { producer, client } = await fixture("headless");
producer.overrideCall = (name) =>
name === "server_health" ? busyHealth(5) : undefined;
const result = await client.execute("health", {});
if (result.ok) throw new Error("expected unreadable output");
expect(
projectAnalysisError(result.error).details,
"the diagnostic names the invalid busy_tool, not the ok-schema fields",
).toMatchObject({ reason: expect.stringContaining("busy_tool") });
expect(
JSON.stringify(projectAnalysisError(result.error).details),
"the generic ok-schema field errors are not reported for a busy payload",
).not.toContain("input_path");
await expectCleanClose(producer, client);
});
it("stops waiting as soon as a later read is malformed", async () => {
const { producer, client } = await fixture("headless");
let reads = 0;
producer.overrideCall = (name) =>
name === "server_health"
? reads++ === 0
? busyHealth("config_json_get")
: busyHealth(5)
: undefined;
const result = await client.execute("health", {});
if (result.ok) throw new Error("expected unreadable output");
expect(
projectAnalysisError(result.error).code,
"a malformed re-read is not treated as busy",
).toBe("unreadable_output");
expect(
healthCalls(producer),
"exactly the busy read and the bad read",
).toBe(2);
await expectCleanClose(producer, client);
});
});
describe("headless IDA lifecycle boundaries", () => {
it("never closes an unowned worker even when open echoes the requested private identity", async () => {
const { producer, client } = await fixture("headless");
+7 -4
View File
@@ -20,6 +20,7 @@ import { jsonValueSchema, type JsonValue } from "../domain/jsonValue.js";
import { err, ok, type Result } from "../domain/result.js";
import type { IdaConfiguration } from "./IdaConfiguration.js";
import type { IdaMcpConnection } from "./IdaMcpConnection.js";
import { readSettledHealth } from "./IdaBusyHealth.js";
import { IdaOperationRunner } from "./IdaOperationRunner.js";
import {
IDA_LIMITATIONS,
@@ -183,7 +184,7 @@ export class IdaSessionClient implements AnalysisClient {
);
try {
await this.#start();
const before = await this.#observe();
const before = await this.#observe(options);
if (wasCancelled(options))
return err(new AnalysisCancelledError(operation));
if (operation === "health") return this.#health(before);
@@ -200,7 +201,7 @@ export class IdaSessionClient implements AnalysisClient {
const result = await runner.run(operation, parameters);
if (wasCancelled(options))
return err(new AnalysisCancelledError(operation));
const after = await this.#observe();
const after = await this.#observe(options);
if (wasCancelled(options))
return err(new AnalysisCancelledError(operation));
return ok(
@@ -339,7 +340,7 @@ export class IdaSessionClient implements AnalysisClient {
}
}
async #observe(): Promise<JsonValue> {
async #observe(options?: ExecutionOptions): Promise<JsonValue> {
let identity: string;
if (this.config.mode === "attached") {
const raw = await this.connection.call("get_metadata", {});
@@ -359,8 +360,10 @@ export class IdaSessionClient implements AnalysisClient {
throw new AnalysisProtocolError(
"IDA headless session has no database binding.",
);
const raw = await this.connection.call("server_health", {
const raw = await readSettledHealth(this.connection, {
database: this.#database,
timeoutMs: this.config.timeoutMs,
signal: options?.signal,
});
const metadata = modernHealthSchema.parse(raw);
if (