mirror of
https://github.com/morluto/rea
synced 2026-10-25 14:01:13 -04:00
perf(model): avoid decoding omitted GGUF strings (#5781)
Validate omitted GGUF strings without materializing discarded text while preserving full UTF-8 checks and retained Evidence. Record real-model costs and workflow coverage; preserve failed experiment receipts through cleanup. Fixes #5774.
This commit is contained in:
gensokyo-dev
@@ -1,5 +1,13 @@
|
||||
# Analysis efficiency measurements
|
||||
|
||||
The [2026-10-21 coverage reconciliation](coverage-review-20261021.md) maps all
|
||||
46 source directories to concrete workflow evidence, including original feature
|
||||
PRs, historical no-benefit results and remaining verification boundaries.
|
||||
|
||||
The [GGUF validation checkpoint](model-utf8-validation.md) removes decoding of
|
||||
omitted strings while retaining full UTF-8 validation. It records real-model
|
||||
phase gains, unchanged task token costs and the higher main-task memory peak.
|
||||
|
||||
The [WAT framing checkpoint](wasm-string-framing.md) measures structural scanning
|
||||
on real SQLite and VS Code components, with unchanged Evidence and token costs,
|
||||
small whole-task timing gains and no memory improvement.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# Workflow coverage reconciliation, 2026-10-21
|
||||
|
||||
The checked snapshot is main `264862bf2`, with **46 first-level `src/`
|
||||
directories and 212 production stdio MCP tools**. Directory counts and tool
|
||||
counts are different inventories: one application workflow can cross several
|
||||
directories, and one provider can expose many independently costly operations.
|
||||
This is a source/history review, not a new runtime qualification of all tools.
|
||||
|
||||
The search includes this experiment directory, owning contracts and source
|
||||
history, original feature PRs and related open/closed issues. Missing experiment
|
||||
files do not establish missing audits. In particular, the model and IL2CPP
|
||||
feature PRs contain substantial real-task evidence absent from this directory.
|
||||
Historical measurements retain their original revisions, platforms and limits.
|
||||
|
||||
| Directories / workflow | Evidence already present | Remaining coverage boundary |
|
||||
| ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `kuna`, `ghidra` | Earlier native entry work and [Ghidra projection](ghidra-evidence-projection.md) | Selected questions only; database reuse is a separate [research issue #3136](https://github.com/morluto/rea/issues/3136). |
|
||||
| `ida` | [Windows native correctness and completed-task costs](ida-native-windows.md), including real `whoami.exe` | Warm selected-task gains do not establish cold analysis, all string formats, all operations or other IDA platforms. |
|
||||
| `hopper`, `jeb` | [Hopper string records](hopper-string-records.md), [Linux runtime](hopper-linux-python-runtime.md), [real JEB CE](jeb-linux-native.md) | Preserve negative whole-task results; other editions, platforms and workflows are separate scopes. |
|
||||
| `rizin`, `cutter` | Existing command/session adapters and lifecycle fixes | No comparable dedicated real completed-task performance record located in the searched records. Actual engine/GUI availability must be established before selecting a task. |
|
||||
| `gdb`, `objdump` | [Real Bash command observations](command-observation-views.md) | Selected read-only disassembly is measured; other commands, large outputs and running-target observation are separate tasks. |
|
||||
| `adb`, `frida`, `inspector` | [Frida native status/history experiment](frida-status-views.md) | Frida status does not qualify arbitrary scripts, ADB/device transfers, browser Inspector events or live target freshness. |
|
||||
| `go`, `rust` | [Go build/dependency selection on real binaries](go-build-views.md); Rust cargo-auditable support [#2456](https://github.com/morluto/rea/pull/2456) | Go is already measured. No comparable dedicated Rust dependency-investigation cost record located in this review. |
|
||||
| `javascript`, `browser`, `electron` | [Agent fact workflow](javascript-facts-agent-history.json), lifecycle experiments and [#4415](https://github.com/morluto/rea/issues/4415) | Real Joplin preparation remains exploratory. Individual JS features and browser/runtime workflows need their own evidence. |
|
||||
| `android`, `apktool` | [Package summary](android-package-summary.md), [three-APK shared resource decoding](apktool-string-tables.md) | These do not qualify all JADX class/method/reference operations or physical-device workflows. #4777 is already fixed. |
|
||||
| `dotnet`, `il2cpp` | [Managed summaries/pages](managed-artifact-summary.md), IL2CPP metadata [#3807](https://github.com/morluto/rea/pull/3807) and native links/seeds [#5327](https://github.com/morluto/rea/pull/5327) | IL2CPP already has real Windows/Android row comparisons, task costs and Ghidra import. Unsupported versions/generics/platforms remain separate correctness and coverage work. |
|
||||
| `model` | GGUF/safetensors [#5603](https://github.com/morluto/rea/pull/5603): real models, independent readers and complete-task costs | Existing delivery is already audited. [#5774](https://github.com/morluto/rea/issues/5774) isolates omitted-string validation cost within that producer. |
|
||||
| `native`, `artifacts` | [Native inventory](native-inventory-views.md), [entry views](native-entry-views.md); real NIB output reductions [#1460](https://github.com/morluto/rea/pull/1460), [#1477](https://github.com/morluto/rea/pull/1477) | Inventory does not qualify Apple UI/resource, signatures, dyld or runtime traces. NIB byte reductions alone do not establish token, time, CPU or memory savings for a completed question. |
|
||||
| `firmware` | Region pilot plus real full OpenWrt extraction/publication [#4602](https://github.com/morluto/rea/pull/4602) | Extraction is already measured; noisy timings establish no general speedup. Other image layouts, extraction trees and publication filesystems remain unqualified. |
|
||||
| `flutter`, `sqlite`, `wasm`, `evm`, `pcap`, `signals` | [Dart AOT](flutter-aot-summary.md), [SQLite controls](sqlite-existing-controls.md), [WAT framing](wasm-string-framing.md), [EVM](evm-interface-summary.md), [PCAP](pcap-small-captures.md), [signals](signal-selection-workflows.md) | Extend a specific question. Preserve cases where existing controls suffice and small tasks regress; these records do not qualify each format or tool exhaustively. |
|
||||
| `application`, `domain`, `server`, `cli`, `contracts` | [Mixed retained investigations](retained-investigation-workflows.md), [immutable parent validation](immutable-parent-validation.md), [capture validation](retained-capture-validation.md) | Shared selected-query paths are already optimized. Reconstruction/environment composition and new fact families require their own end-to-end questions. |
|
||||
| `browser` / recorded captures | [SAZ selection](saz-selection-workflows.md) and [real qlog producer/retained reads](retained-capture-validation.md) | [#3133](https://github.com/morluto/rea/issues/3133) still includes actual model/task questions; producer measurements do not close those requirements. |
|
||||
| `filesystem`, `process`, `reference`, `providers`, `captures`, `windows`, `main`, `composition`, `config`, `evaluation` | Cross-workflow resource/cleanup checks, [process traces](process-trace-validation.md), [MCP profile](mcp-profile-identity.md) and Windows lifecycle checkpoints | Infrastructure coverage follows the exercised workflow. No blanket startup/resource claim; reference imports and `compare_source_to_bundle` need a dedicated completed source-tree question. |
|
||||
|
||||
Every one of the 46 directories appears above; repeated `browser` highlights a
|
||||
distinct workflow, not another directory. The table records located evidence,
|
||||
not proof that every historical PR was inspected. Open issue state by itself
|
||||
does not prove missing implementation or absent measurements.
|
||||
|
||||
## Three bounded follow-up questions
|
||||
|
||||
1. **Rust dependency identity:** inspect a public cargo-auditable binary, find
|
||||
exact dependency/version edges and retain complete source Evidence. Compare
|
||||
existing delivery first; independently decode its auditable section. Include
|
||||
a small binary and account for all discovery, queries and serialization.
|
||||
2. **Apple UI graph lookup:** on macOS, locate a specific control/outlet in a
|
||||
real Transmission/XLD NIB graph using existing summary/document pages.
|
||||
Compare equivalent complete questions and independently check archive
|
||||
records. Historical byte reductions alone are not the completion criterion.
|
||||
3. **Reference/source comparison:** import a pinned public source tree and
|
||||
answer one source-to-bundle identity question through existing workflows.
|
||||
Independently check file manifests, symlink/unknown semantics and selected
|
||||
matches; measure repeated traversal and retained representation before
|
||||
proposing an index or cache.
|
||||
|
||||
Each lane stops with a measured benefit, an evidenced no-change decision, or an
|
||||
explicit unavailable prerequisite. Rizin/Cutter and device/live workflows stay
|
||||
visible as additional gaps; no new execution, injection or mutation authority is
|
||||
implied by this review. No all-provider run or new API is justified by counts.
|
||||
@@ -0,0 +1,435 @@
|
||||
{
|
||||
"baseline_source": "264862bf2e3612c9cb6fcfb861843cbba4e1893f",
|
||||
"candidate_code": "c2550343173c5abd8f34e925eac1bcc12b7ee6a2",
|
||||
"toolchain": {
|
||||
"node": "24.18.0",
|
||||
"npm": "11.16.0"
|
||||
},
|
||||
"runtime_filetables_sha256": {
|
||||
"baseline": "81c293ff8997017cfe9db2a65a8b70b188e3e6d20cef07902599c69ade5d9fac",
|
||||
"candidate": "b4b654a7151f6affbb3ab8541bff74b643d9f78073d338b8feb2b095dd12b6b5"
|
||||
},
|
||||
"lock_sha256": "699a14749e3825c62430faeb18224190fee1c9347760494e38e334e180bea6ec",
|
||||
"gguf_source_sha256": {
|
||||
"baseline": "9c387d5fba10db7830e74ec74a73f4b3d2e1347cca665e948b7891881b186007",
|
||||
"candidate": "c43e5f8eebe04c0f82e0a970606f20ddf409c7a5fd0e298c88b09bcd0e7864a9"
|
||||
},
|
||||
"driver_sha256": "9732f45b8b550eef47c1fc993b435129108934f3ec3501a34df1784e6ee37b44",
|
||||
"resource_guard_sha256": {
|
||||
"resource-run.py": "7a9642ab86ba34290564f636eb080cacf38b4185a254028680b238bff172993b",
|
||||
"resource_payload.py": "6f089e645459eba80c5d73e39c21606c78173a44daf49fabc3b88c4631ce51c6"
|
||||
},
|
||||
"oracle": {
|
||||
"gguf": "0.19.0",
|
||||
"safetensors": "0.8.0",
|
||||
"receipts": [
|
||||
{
|
||||
"file": "stories15M-q4_0.gguf",
|
||||
"sha256": "66967fbece6dbe97886593fdbb73589584927e29119ec31f08090732d1861739",
|
||||
"metadata": 20,
|
||||
"tensors": 57,
|
||||
"status": "matched"
|
||||
},
|
||||
{
|
||||
"file": "stories260K.gguf",
|
||||
"sha256": "270cba1bd5109f42d03350f60406024560464db173c0e387d91f0426d3bd256d",
|
||||
"metadata": 19,
|
||||
"tensors": 48,
|
||||
"status": "matched"
|
||||
},
|
||||
{
|
||||
"file": "tiny-bert.safetensors",
|
||||
"sha256": "965f02b6a7e5520fc12f710e4e3b6132f697f1c8f648819553c5ade86752d2de",
|
||||
"metadata": 2,
|
||||
"tensors": 100,
|
||||
"status": "matched"
|
||||
}
|
||||
]
|
||||
},
|
||||
"tokenizer": "tiktoken 0.12.0 cl100k_base; independent compact JSON chunks, catalog once + each named request/reply; not actual model usage or initialization/framing",
|
||||
"cases": [
|
||||
{
|
||||
"artifact": {
|
||||
"repo": "ggml-org/models",
|
||||
"revision": "499bc8821c6b12b4e53c5bffcb21ec206f212d81",
|
||||
"member": "tinyllamas/stories15M-q4_0.gguf",
|
||||
"file": "stories15M-q4_0.gguf",
|
||||
"url": "https://huggingface.co/ggml-org/models/resolve/499bc8821c6b12b4e53c5bffcb21ec206f212d81/tinyllamas/stories15M-q4_0.gguf",
|
||||
"bytes": 19077344
|
||||
},
|
||||
"normalized_sha256": "a894614da7ce9486b07eb2ede05ce11307d4726089722278bc4c95043feb6b66",
|
||||
"answers_sha256": "64277f01f43440bc99a94ab0787a4ddde22866ed20140d87d3942b1723cbb067",
|
||||
"catalog_sha256": "859b1433ef7c45ea23ba4073715d25ed7a76fd9df6d74285d20a5ea39e8e2898",
|
||||
"mcp": [
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.667,
|
||||
"cpu_seconds": 6.480601,
|
||||
"peak_family_mib": 769.34,
|
||||
"catalog_plus_calls_tokens": 895583,
|
||||
"calls_ms": [367.272, 1936.653, 162.868, 12.072, 19.236]
|
||||
},
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.867,
|
||||
"cpu_seconds": 6.690645,
|
||||
"peak_family_mib": 816.34,
|
||||
"catalog_plus_calls_tokens": 895583,
|
||||
"calls_ms": [324.806, 1822.885, 153.167, 11.988, 18.093]
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.77,
|
||||
"cpu_seconds": 6.555891,
|
||||
"peak_family_mib": 798.98,
|
||||
"catalog_plus_calls_tokens": 895583,
|
||||
"calls_ms": [456.467, 1823.786, 160.038, 11.296, 18.607]
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.565,
|
||||
"cpu_seconds": 6.3467,
|
||||
"peak_family_mib": 813.566,
|
||||
"catalog_plus_calls_tokens": 895583,
|
||||
"calls_ms": [329.862, 1767.086, 152.299, 10.948, 17.523]
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.666,
|
||||
"cpu_seconds": 6.49779,
|
||||
"peak_family_mib": 771.02,
|
||||
"catalog_plus_calls_tokens": 895583,
|
||||
"calls_ms": [345.895, 1854.963, 156.895, 11.476, 15.852]
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.567,
|
||||
"cpu_seconds": 6.386103,
|
||||
"peak_family_mib": 818.891,
|
||||
"catalog_plus_calls_tokens": 895583,
|
||||
"calls_ms": [342.734, 1827.835, 157.363, 10.122, 17.6]
|
||||
}
|
||||
],
|
||||
"phase": [
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "baseline",
|
||||
"first_ms": 154.027,
|
||||
"repeat_median_ms": 56.14,
|
||||
"wall_seconds": 1.028,
|
||||
"cpu_seconds": 0.87953,
|
||||
"peak_family_mib": 75.867
|
||||
},
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "candidate",
|
||||
"first_ms": 121.41,
|
||||
"repeat_median_ms": 35.429,
|
||||
"wall_seconds": 0.925,
|
||||
"cpu_seconds": 0.717987,
|
||||
"peak_family_mib": 77.945
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "baseline",
|
||||
"first_ms": 159.766,
|
||||
"repeat_median_ms": 53.575,
|
||||
"wall_seconds": 1.027,
|
||||
"cpu_seconds": 0.854739,
|
||||
"peak_family_mib": 75.586
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "candidate",
|
||||
"first_ms": 121.383,
|
||||
"repeat_median_ms": 38.773,
|
||||
"wall_seconds": 0.925,
|
||||
"cpu_seconds": 0.733397,
|
||||
"peak_family_mib": 75.125
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "baseline",
|
||||
"first_ms": 148.447,
|
||||
"repeat_median_ms": 53.58,
|
||||
"wall_seconds": 1.026,
|
||||
"cpu_seconds": 0.848173,
|
||||
"peak_family_mib": 75.363
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "candidate",
|
||||
"first_ms": 121.712,
|
||||
"repeat_median_ms": 37.191,
|
||||
"wall_seconds": 0.926,
|
||||
"cpu_seconds": 0.728712,
|
||||
"peak_family_mib": 77.887
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"artifact": {
|
||||
"repo": "ggml-org/models",
|
||||
"revision": "499bc8821c6b12b4e53c5bffcb21ec206f212d81",
|
||||
"member": "tinyllamas/stories260K.gguf",
|
||||
"file": "stories260K.gguf",
|
||||
"url": "https://huggingface.co/ggml-org/models/resolve/499bc8821c6b12b4e53c5bffcb21ec206f212d81/tinyllamas/stories260K.gguf",
|
||||
"bytes": 1185376
|
||||
},
|
||||
"normalized_sha256": "15f4dbdc30dd9bd936747057031c6391c38b4c1717d56ae33ed2f0cba4184405",
|
||||
"answers_sha256": "99257428a01aa513b435c95ca7217c69f4641d9021372a6333e7cb9e2cb35d63",
|
||||
"catalog_sha256": "859b1433ef7c45ea23ba4073715d25ed7a76fd9df6d74285d20a5ea39e8e2898",
|
||||
"mcp": [
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.568,
|
||||
"cpu_seconds": 6.375324,
|
||||
"peak_family_mib": 743.887,
|
||||
"catalog_plus_calls_tokens": 895421,
|
||||
"calls_ms": [223.11, 1938.73, 102.018, 10.759, 16.11]
|
||||
},
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.566,
|
||||
"cpu_seconds": 6.338327,
|
||||
"peak_family_mib": 744.852,
|
||||
"catalog_plus_calls_tokens": 895421,
|
||||
"calls_ms": [248.979, 1886.809, 105.801, 10.738, 15.653]
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.365,
|
||||
"cpu_seconds": 6.206216,
|
||||
"peak_family_mib": 783.84,
|
||||
"catalog_plus_calls_tokens": 895421,
|
||||
"calls_ms": [248.196, 1845.737, 112.874, 10.724, 17.594]
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 7.383,
|
||||
"cpu_seconds": 7.211823,
|
||||
"peak_family_mib": 742.281,
|
||||
"catalog_plus_calls_tokens": 895421,
|
||||
"calls_ms": [233.155, 1909.721, 104.803, 11.603, 19.357]
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.565,
|
||||
"cpu_seconds": 6.323702,
|
||||
"peak_family_mib": 740.84,
|
||||
"catalog_plus_calls_tokens": 895421,
|
||||
"calls_ms": [233.664, 1918.301, 104.389, 14.561, 15.227]
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.566,
|
||||
"cpu_seconds": 6.315785,
|
||||
"peak_family_mib": 778.133,
|
||||
"catalog_plus_calls_tokens": 895421,
|
||||
"calls_ms": [232.37, 1851.929, 78.318, 21.857, 16.438]
|
||||
}
|
||||
],
|
||||
"phase": [
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "baseline",
|
||||
"first_ms": 35.366,
|
||||
"repeat_median_ms": 7.784,
|
||||
"wall_seconds": 0.625,
|
||||
"cpu_seconds": 0.429065,
|
||||
"peak_family_mib": 62.922
|
||||
},
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "candidate",
|
||||
"first_ms": 30.009,
|
||||
"repeat_median_ms": 7.934,
|
||||
"wall_seconds": 0.624,
|
||||
"cpu_seconds": 0.392848,
|
||||
"peak_family_mib": 59.664
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "baseline",
|
||||
"first_ms": 32.0,
|
||||
"repeat_median_ms": 8.772,
|
||||
"wall_seconds": 0.624,
|
||||
"cpu_seconds": 0.412059,
|
||||
"peak_family_mib": 60.422
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "candidate",
|
||||
"first_ms": 32.032,
|
||||
"repeat_median_ms": 9.36,
|
||||
"wall_seconds": 0.624,
|
||||
"cpu_seconds": 0.412412,
|
||||
"peak_family_mib": 59.441
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "baseline",
|
||||
"first_ms": 33.628,
|
||||
"repeat_median_ms": 9.496,
|
||||
"wall_seconds": 0.624,
|
||||
"cpu_seconds": 0.396232,
|
||||
"peak_family_mib": 60.41
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "candidate",
|
||||
"first_ms": 30.506,
|
||||
"repeat_median_ms": 8.172,
|
||||
"wall_seconds": 0.622,
|
||||
"cpu_seconds": 0.404398,
|
||||
"peak_family_mib": 61.672
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"artifact": {
|
||||
"repo": "hf-internal-testing/tiny-random-bert",
|
||||
"revision": "f171d7baecaf37b5da5a3616d8833b9969753535",
|
||||
"member": "model.safetensors",
|
||||
"file": "tiny-bert.safetensors",
|
||||
"url": "https://huggingface.co/hf-internal-testing/tiny-random-bert/resolve/f171d7baecaf37b5da5a3616d8833b9969753535/model.safetensors",
|
||||
"bytes": 520212
|
||||
},
|
||||
"normalized_sha256": "8658d90261cdbaaf1f5f8d2e9408ad301b40ad3e6b7aab7ac9c5d4be71ca36d5",
|
||||
"answers_sha256": "82072404a0aa24e3b42979397421f455da2589c2ea649f28ba7279d43e6c3bc8",
|
||||
"catalog_sha256": "859b1433ef7c45ea23ba4073715d25ed7a76fd9df6d74285d20a5ea39e8e2898",
|
||||
"mcp": [
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.667,
|
||||
"cpu_seconds": 6.46146,
|
||||
"peak_family_mib": 741.602,
|
||||
"catalog_plus_calls_tokens": 893755,
|
||||
"calls_ms": [251.201, 1965.785, 114.055, 10.837, 28.862]
|
||||
},
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.767,
|
||||
"cpu_seconds": 6.519606,
|
||||
"peak_family_mib": 736.367,
|
||||
"catalog_plus_calls_tokens": 893755,
|
||||
"calls_ms": [255.557, 1979.227, 86.726, 10.093, 24.531]
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.565,
|
||||
"cpu_seconds": 6.379249,
|
||||
"peak_family_mib": 730.273,
|
||||
"catalog_plus_calls_tokens": 893755,
|
||||
"calls_ms": [245.517, 1900.802, 94.351, 10.0, 25.635]
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.569,
|
||||
"cpu_seconds": 6.388131,
|
||||
"peak_family_mib": 785.746,
|
||||
"catalog_plus_calls_tokens": 893755,
|
||||
"calls_ms": [250.715, 1846.568, 103.314, 21.618, 27.261]
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "baseline",
|
||||
"wall_seconds": 6.771,
|
||||
"cpu_seconds": 6.530512,
|
||||
"peak_family_mib": 745.098,
|
||||
"catalog_plus_calls_tokens": 893755,
|
||||
"calls_ms": [255.441, 1949.189, 110.897, 10.679, 24.345]
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "candidate",
|
||||
"wall_seconds": 6.466,
|
||||
"cpu_seconds": 6.296937,
|
||||
"peak_family_mib": 741.348,
|
||||
"catalog_plus_calls_tokens": 893755,
|
||||
"calls_ms": [243.919, 1895.978, 113.995, 10.66, 27.269]
|
||||
}
|
||||
],
|
||||
"phase": [
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "baseline",
|
||||
"first_ms": 27.648,
|
||||
"repeat_median_ms": 8.11,
|
||||
"wall_seconds": 0.623,
|
||||
"cpu_seconds": 0.394405,
|
||||
"peak_family_mib": 57.281
|
||||
},
|
||||
{
|
||||
"pair": 1,
|
||||
"side": "candidate",
|
||||
"first_ms": 27.003,
|
||||
"repeat_median_ms": 8.082,
|
||||
"wall_seconds": 0.523,
|
||||
"cpu_seconds": 0.383678,
|
||||
"peak_family_mib": 57.266
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "baseline",
|
||||
"first_ms": 26.941,
|
||||
"repeat_median_ms": 8.04,
|
||||
"wall_seconds": 0.626,
|
||||
"cpu_seconds": 0.401321,
|
||||
"peak_family_mib": 59.504
|
||||
},
|
||||
{
|
||||
"pair": 2,
|
||||
"side": "candidate",
|
||||
"first_ms": 28.012,
|
||||
"repeat_median_ms": 8.006,
|
||||
"wall_seconds": 0.625,
|
||||
"cpu_seconds": 0.392541,
|
||||
"peak_family_mib": 59.012
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "baseline",
|
||||
"first_ms": 28.734,
|
||||
"repeat_median_ms": 8.499,
|
||||
"wall_seconds": 0.624,
|
||||
"cpu_seconds": 0.411118,
|
||||
"peak_family_mib": 59.23
|
||||
},
|
||||
{
|
||||
"pair": 3,
|
||||
"side": "candidate",
|
||||
"first_ms": 29.961,
|
||||
"repeat_median_ms": 8.632,
|
||||
"wall_seconds": 0.625,
|
||||
"cpu_seconds": 0.41875,
|
||||
"peak_family_mib": 57.5
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"published_equivalent_source": "2f091ccc734d7bd945bd61432d41cf5ba68be12b",
|
||||
"candidate_reproduction": "Use baseline_source plus the byte-identical src/model/GgufHeader.ts from published_equivalent_source; driver SHA-256 is pinned separately. Measured local candidate predates the integration rebase.",
|
||||
"post_review_receipt_fix": {
|
||||
"measured_driver_version": "2f091ccc734d7bd945bd61432d41cf5ba68be12b",
|
||||
"driver_sha256": "449860fb973418f2e3e51183a81be1e3111ffa6453de8208a4b175d3d410322f",
|
||||
"publication_owner_sha256": "901092ae6dc7fda1a8869d62362a0a9f560e03d5ef5667a5c6ed660db000b9a2",
|
||||
"validation": "3 new fault-injection regressions reject original driver and pass current; 11 shared receipt regressions pass. Real stories15M MCP and seven-inspection phase reruns qualify; original paired samples retain their measured driver hash."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
# Omitted GGUF string validation
|
||||
|
||||
[Issue #5774](https://github.com/morluto/rea/issues/5774) follows the existing
|
||||
real-model and completed-task audit in [#5603](https://github.com/morluto/rea/pull/5603).
|
||||
The [coverage reconciliation](coverage-review-20261021.md) explains why neither
|
||||
model inspection nor IL2CPP, firmware extraction or IDA should be described as
|
||||
wholly unaudited. This experiment changes only one model producer hot path.
|
||||
|
||||
## Owner and preserved semantics
|
||||
|
||||
GGUF arrays retain eight preview values. Previously, each omitted string still
|
||||
constructed a fatal UTF-8 decoder and materialized discarded text. `GgufHeader`
|
||||
now validates an omitted value of at most one existing 65,536-byte chunk with
|
||||
Node's `isUtf8`. Longer values retain the streaming fatal decoder, including
|
||||
multi-byte characters split across chunks. Retained values use their original
|
||||
decoder, preserving BOMs and exact text.
|
||||
|
||||
Every admitted byte is still read, hashed and validated. Header bounds,
|
||||
representation admission, cancellation checks, array-loop fairness, source
|
||||
offsets, partial-progress semantics, preview omission and complete retained
|
||||
Evidence are unchanged. There is no cache, new contract, selector or default.
|
||||
CLI and MCP share this producer through the existing application workflow.
|
||||
|
||||
## Corpus and correctness
|
||||
|
||||
The [curated receipts](model-utf8-validation.json) pin source commits, runtime
|
||||
file tables, driver/resource sources, immutable public URLs, artifact digests,
|
||||
complete report/answer fingerprints and all final samples. Baseline is main
|
||||
`264862bf2e3612c9cb6fcfb861843cbba4e1893f`; measured candidate code is
|
||||
`c2550343173c5abd8f34e925eac1bcc12b7ee6a2`. Later documentation/integration
|
||||
commits do not relabel that cohort. That measured local commit predates the
|
||||
rebase: its production file is byte-identical to published
|
||||
`2f091ccc734d7bd945bd61432d41cf5ba68be12b`, with both source hashes recorded.
|
||||
|
||||
| Artifact | Bytes | Metadata / tensors | Independent reader |
|
||||
| ------------------------------------------------ | ---------: | -----------------: | ------------------ |
|
||||
| ggml-org stories15M Q4_0 GGUF | 19,077,344 | 20 / 57 | gguf 0.19.0 |
|
||||
| ggml-org stories260K GGUF | 1,185,376 | 19 / 48 | gguf 0.19.0 |
|
||||
| hf-internal-testing tiny-random-bert safetensors | 520,212 | 2 / 100 | safetensors 0.8.0 |
|
||||
|
||||
Official independent readers match all exposed metadata values/array previews
|
||||
and counts, tensor names, types, shapes and parameter counts. GGUF reader
|
||||
offsets/lengths match every tensor; safetensors reader geometry and independent
|
||||
Python JSON offsets match its file ranges. Python byte hashing checks every
|
||||
header digest. No weights are executed or loaded into a model framework.
|
||||
The main GGUF has 32,000 vocabulary entries; the smaller GGUF and safetensors
|
||||
are controls, not evidence for larger vocabularies or other GGUF versions.
|
||||
|
||||
## Completed public tasks
|
||||
|
||||
Linux x64, Node 24.18.0/npm 11.16.0. Three alternating baseline/candidate,
|
||||
candidate/baseline, baseline/candidate pairs per artifact run serially. Each
|
||||
fresh production stdio MCP task discovers all 212 tools, directly selects two
|
||||
metadata records, retrieves two exact tensors from retained Evidence, repeats
|
||||
both calls, exports complete Evidence, validates answers and closes the
|
||||
transport. OS caches are unflushed; the controller reads inputs for hashing.
|
||||
Fresh process does not mean cold disk or cold model-engine analysis.
|
||||
|
||||
Medians of three whole tasks per side, including SDK/catalog/schema validation,
|
||||
all five calls, source checks, export and teardown:
|
||||
|
||||
| Artifact | Wall seconds, before → after | Family CPU seconds, before → after | Peak family MiB, before → after |
|
||||
| ----------- | ---------------------------: | ---------------------------------: | ------------------------------: |
|
||||
| stories15M | 6.667 → 6.567 | 6.498 → 6.386 | 771.0 → 816.3 |
|
||||
| stories260K | 6.565 → 6.566 | 6.324 → 6.338 | 743.9 → 744.9 |
|
||||
| tiny BERT | 6.667 → 6.569 | 6.461 → 6.388 | 741.6 → 741.3 |
|
||||
|
||||
The primary task's observed wall reduction is about 1.5%, but the unchanged
|
||||
safetensors control has similar variation. No stable whole-task speedup is
|
||||
established. Main-model peak memory rises: ranges 769.3–799.0 → 813.6–818.9
|
||||
MiB; its cause is unmeasured. No peak-memory improvement is claimed.
|
||||
|
||||
Separate fresh-process provider cohorts perform one first inspection and six
|
||||
repeated inspections, comparing the complete report after every operation:
|
||||
|
||||
| Artifact | First / repeat median ms, before | First / repeat median ms, after |
|
||||
| ----------- | -------------------------------: | ------------------------------: |
|
||||
| stories15M | 154.03 / 53.58 | 121.41 / 37.19 |
|
||||
| stories260K | 33.63 / 8.77 | 30.51 / 8.17 |
|
||||
| tiny BERT | 27.65 / 8.11 | 28.01 / 8.08 |
|
||||
|
||||
For stories15M this is about 21% faster first inspection and 31% faster repeated
|
||||
inspection. Median CPU for the seven-inspection process is 0.855 → 0.729 s;
|
||||
peak family memory is 75.6 → 77.9 MiB. Phase gains do not include discovery
|
||||
or substitute for the complete-task results above.
|
||||
|
||||
One catalog plus all named request/reply JSON chunks estimates median
|
||||
895,583 / 895,421 / 893,755 `cl100k_base` tokens for the three artifacts in
|
||||
both implementations, using tiktoken 0.12.0. This includes serialized output
|
||||
schemas, not merely named input schemas. It is an SDK serialization proxy;
|
||||
actual model usage, repeated host injection, negotiation and framing are
|
||||
unmeasured. No token saving is claimed.
|
||||
|
||||
All 18 public tasks and 18 provider processes completed under one CPU, 4 GiB
|
||||
hard / 3 GiB early-stop family memory, no swap and 192 tasks. Payload exit,
|
||||
counters and owned cleanup are verified; no limit/OOM events occurred. Kernel
|
||||
cgroup peak includes file cache. Terminal whole-scope counters after scope
|
||||
removal remain unknown. Raw inputs, exports, profiles and transcripts stay
|
||||
outside Git.
|
||||
|
||||
## Reproduction and validation boundaries
|
||||
|
||||
Build the baseline and the same baseline with the published GGUF source file
|
||||
using the pinned toolchain; this reconstructs the measured local candidate.
|
||||
Capture a complete
|
||||
baseline normalized report from existing CLI/MCP Evidence using the same
|
||||
absolute input path. Run `model-utf8-validation.mjs` with absolute `--runtime`,
|
||||
`--artifact`, `--expected` and a new `--output` directory under `resource-run.py
|
||||
--observe-payload-exit`. Add `--phase` for the seven-inspection provider probe.
|
||||
The driver validates advertised named schemas through the actual SDK and
|
||||
compares full exported parents, preview facts and tensor selections. A failing
|
||||
receipt or cleanup invalidates a task; it is not silently retried as success.
|
||||
|
||||
Five new producer regressions cover empty omitted strings, exact-chunk strings,
|
||||
split multi-byte characters, both byte orders, retained BOMs and rejection of
|
||||
short/long malformed omitted values. Existing malformed-header, admission,
|
||||
cancellation, partial-progress and public CLI/MCP tests remain required.
|
||||
An initial new-test assertion incorrectly hashed alignment padding; the
|
||||
corrected oracle uses the actual declared header, matching existing semantics.
|
||||
The first test launcher refused built projects; the exact built-project gate
|
||||
was then used. Neither attempt is a passing check. The initial resource-wrapper
|
||||
copy failed source-hash lookup before launching a build; corrected guarded
|
||||
builds qualified before measurement.
|
||||
|
||||
Integrated onto main `71b696582`: all 68 tests pass again, and three real public
|
||||
CLI complete reports plus three production SDK tasks match the original
|
||||
normalized reports. Typecheck, lint/module boundaries, formatting, metadata,
|
||||
runtime build, docs checks (five tests) and separate architecture guards pass.
|
||||
`npm run check` does **not** pass: Knip reports an unlisted `g++` invocation in
|
||||
the Windows test added by main's #5772. An independent main source checkout,
|
||||
with matching compiled worker artifacts and an uncached Knip run, reproduces
|
||||
only that finding. It is outside this change and is left unchanged. Initial
|
||||
control attempts lacked resolvable worker artifacts and are not clean controls.
|
||||
Development builds/checks used separate 6/8 GiB hard guards; runtime experiments
|
||||
retain their 4 GiB limit. No guard stop or OOM occurred.
|
||||
|
||||
Greptile identified a receipt defect in the phase driver, not the model parser:
|
||||
failed inspections/assertions were not recorded before cleanup, and a close
|
||||
failure could suppress publication. Each attempted phase now records its cost
|
||||
and primary failure before cleanup. Phase and MCP paths share one atomic receipt
|
||||
publisher; cleanup failures are independent facts and cannot replace the primary
|
||||
failure. Three new fault-injection tests fail on the original driver and pass
|
||||
after the correction; all 11 existing shared receipt tests pass. These stand-ins
|
||||
qualify failure accounting only. Real stories15M MCP and seven-inspection phase
|
||||
reruns also pass with verified counters/cleanup. The original 36 paired samples
|
||||
keep their measured driver hash; the receipt records the post-review driver and
|
||||
publication-owner hashes separately. No timing cohort is relabeled.
|
||||
@@ -0,0 +1,243 @@
|
||||
/** Equivalent model tasks; inputs, expected reports and raw receipts stay outside Git. */
|
||||
import assert from "node:assert/strict";
|
||||
import { createHash } from "node:crypto";
|
||||
import { createWriteStream } from "node:fs";
|
||||
import { mkdir, mkdtemp, readFile, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join, resolve } from "node:path";
|
||||
import { finished } from "node:stream/promises";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { parseArgs } from "node:util";
|
||||
import { Client } from "@modelcontextprotocol/client";
|
||||
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
|
||||
import { createRetainedQueryCaller } from "./retained-query-calls.mjs";
|
||||
import {
|
||||
finalizeRetainedQueryReceipt,
|
||||
publishRetainedQueryReceipt,
|
||||
} from "./retained-query-finalization.mjs";
|
||||
|
||||
const { values } = parseArgs({
|
||||
options: {
|
||||
...Object.fromEntries(
|
||||
["runtime", "artifact", "expected", "output"].map((name) => [
|
||||
name,
|
||||
{ type: "string" },
|
||||
]),
|
||||
),
|
||||
phase: { type: "boolean", default: false },
|
||||
},
|
||||
});
|
||||
for (const name of ["runtime", "artifact", "expected", "output"])
|
||||
assert.ok(values[name] && values[name] === resolve(values[name]), name);
|
||||
const hash = (value) => createHash("sha256").update(value).digest("hex");
|
||||
const expected = JSON.parse(await readFile(values.expected, "utf8"));
|
||||
assert.equal(expected.artifact.path, values.artifact);
|
||||
await mkdir(values.output, { mode: 0o700 });
|
||||
const report = {
|
||||
artifact_sha256: hash(await readFile(values.artifact)),
|
||||
normalized_sha256: hash(JSON.stringify(expected)),
|
||||
calls: [],
|
||||
answers: [],
|
||||
};
|
||||
|
||||
if (values.phase) {
|
||||
const { ModelArtifactProvider } = await import(
|
||||
pathToFileURL(join(values.runtime, "dist/model/ModelArtifactProvider.js"))
|
||||
);
|
||||
const provider = new ModelArtifactProvider();
|
||||
try {
|
||||
for (let index = 0; index < 7; index++) {
|
||||
const sample = {
|
||||
phase: index === 0 ? "fresh" : "repeat",
|
||||
is_error: true,
|
||||
};
|
||||
const start = performance.now();
|
||||
try {
|
||||
const result = await provider.inspect({
|
||||
path: values.artifact,
|
||||
hash: "header",
|
||||
});
|
||||
sample.elapsed_ms = performance.now() - start;
|
||||
assert.ok(result.ok, JSON.stringify(result));
|
||||
assert.deepEqual(result.value, expected);
|
||||
sample.is_error = false;
|
||||
} catch (cause) {
|
||||
sample.elapsed_ms ??= performance.now() - start;
|
||||
sample.failure = String(cause?.stack ?? cause);
|
||||
throw cause;
|
||||
} finally {
|
||||
report.calls.push(sample);
|
||||
}
|
||||
}
|
||||
report.status = "passed";
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.failure = String(cause?.stack ?? cause);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
report.cleanup_failures = [];
|
||||
try {
|
||||
await provider.close();
|
||||
} catch (cause) {
|
||||
report.cleanup_failures.push(`provider: ${String(cause)}`);
|
||||
report.status = "failed";
|
||||
}
|
||||
if (!(await publishRetainedQueryReceipt({ report, output: values.output })))
|
||||
process.exitCode = 1;
|
||||
}
|
||||
} else {
|
||||
await runMcpTask();
|
||||
}
|
||||
|
||||
async function runMcpTask() {
|
||||
const account = await mkdtemp(join(tmpdir(), "rea-model-utf8-"));
|
||||
const client = new Client({ name: "model-utf8-validation", version: "1" });
|
||||
const transport = new StdioClientTransport({
|
||||
command: process.execPath,
|
||||
args: [join(values.runtime, "scripts/rea.mjs"), "mcp"],
|
||||
env: {
|
||||
HOME: account,
|
||||
PATH: process.env.PATH,
|
||||
NODE_OPTIONS: "--max-old-space-size=2048",
|
||||
UV_THREADPOOL_SIZE: "1",
|
||||
REA_LOG_LEVEL: "silent",
|
||||
TMPDIR: account,
|
||||
},
|
||||
stderr: "pipe",
|
||||
});
|
||||
const serverLog = createWriteStream(join(values.output, "server.log"));
|
||||
const serverLogFinished = finished(serverLog);
|
||||
serverLogFinished.catch(() => {});
|
||||
transport.stderr?.pipe(serverLog);
|
||||
const transcript = [];
|
||||
const call = createRetainedQueryCaller(client, (receipt, event) => {
|
||||
report.calls.push(receipt);
|
||||
transcript.push(event);
|
||||
});
|
||||
const observe = async (name, input, phase) => {
|
||||
const reply = await call(name, input, phase);
|
||||
assert.notEqual(reply.isError, true, JSON.stringify(reply));
|
||||
assert.ok(reply.structuredContent);
|
||||
return reply.structuredContent;
|
||||
};
|
||||
try {
|
||||
await client.connect(transport);
|
||||
const catalog = await client.listTools();
|
||||
assert.equal(catalog.nextCursor, undefined);
|
||||
await writeFile(
|
||||
join(values.output, "catalog.json"),
|
||||
JSON.stringify(catalog),
|
||||
);
|
||||
report.catalog_sha256 = hash(JSON.stringify(catalog));
|
||||
report.catalog_tools = catalog.tools.length;
|
||||
const definitions = new Map(catalog.tools.map((tool) => [tool.name, tool]));
|
||||
const originalCall = client.callTool.bind(client);
|
||||
client.callTool = (input, options) =>
|
||||
originalCall(input, {
|
||||
...options,
|
||||
toolDefinition: definitions.get(input.name),
|
||||
});
|
||||
const keys = [
|
||||
...new Set(expected.metadata.slice(0, 2).map((row) => row.key)),
|
||||
];
|
||||
const names = expected.tensors.slice(0, 2).map((row) => row.name);
|
||||
const fields = [
|
||||
"key",
|
||||
"type",
|
||||
"value",
|
||||
"element_count",
|
||||
"preview_complete",
|
||||
];
|
||||
const tensorFields = [
|
||||
"name",
|
||||
"type",
|
||||
"shape",
|
||||
"file_offset",
|
||||
"byte_length",
|
||||
];
|
||||
const select = (rows, selectedFields) =>
|
||||
rows.map((row) =>
|
||||
Object.fromEntries(
|
||||
["index", ...selectedFields].map((key) => [key, row[key]]),
|
||||
),
|
||||
);
|
||||
const observations = [];
|
||||
for (const phase of ["fresh", "repeat"]) {
|
||||
const metadata = await observe(
|
||||
"inspect_model_artifact",
|
||||
{
|
||||
path: values.artifact,
|
||||
view: { collection: "metadata", keys, fields },
|
||||
},
|
||||
phase,
|
||||
);
|
||||
const parentId = metadata.normalized_result.parent_evidence_id;
|
||||
assert.equal(typeof parentId, "string");
|
||||
assert.deepEqual(
|
||||
metadata.normalized_result.items,
|
||||
select(
|
||||
expected.metadata.filter((row) => keys.includes(row.key)),
|
||||
fields,
|
||||
),
|
||||
);
|
||||
const tensors = await observe(
|
||||
"inspect_analysis_view",
|
||||
{
|
||||
source: { kind: "retained-evidence", evidence_id: parentId },
|
||||
view: {
|
||||
kind: "model-artifact",
|
||||
collection: "tensors",
|
||||
names,
|
||||
fields: tensorFields,
|
||||
},
|
||||
},
|
||||
`${phase}:tensors`,
|
||||
);
|
||||
assert.equal(tensors.normalized_result.parent_evidence_id, parentId);
|
||||
assert.deepEqual(
|
||||
tensors.normalized_result.items,
|
||||
select(expected.tensors.slice(0, 2), tensorFields),
|
||||
);
|
||||
observations.push(parentId);
|
||||
report.answers.push({
|
||||
parent_evidence_id: parentId,
|
||||
metadata: metadata.normalized_result.items,
|
||||
tensors: tensors.normalized_result.items,
|
||||
});
|
||||
}
|
||||
await observe(
|
||||
"export_evidence_bundle",
|
||||
{ path: join(values.output, "evidence.json") },
|
||||
"export",
|
||||
);
|
||||
const bundle = JSON.parse(
|
||||
await readFile(join(values.output, "evidence.json"), "utf8"),
|
||||
);
|
||||
for (const id of observations) {
|
||||
const parent = bundle.records.find((record) => record.evidence_id === id);
|
||||
assert.ok(parent);
|
||||
assert.deepEqual(parent.normalized_result, expected);
|
||||
}
|
||||
assert.deepEqual(report.answers[0], report.answers[1]);
|
||||
assert.equal(hash(await readFile(values.artifact)), report.artifact_sha256);
|
||||
report.status = "passed";
|
||||
} catch (cause) {
|
||||
report.status = "failed";
|
||||
report.failure = String(cause?.stack ?? cause);
|
||||
process.exitCode = 1;
|
||||
} finally {
|
||||
if (
|
||||
!(await finalizeRetainedQueryReceipt({
|
||||
report,
|
||||
client,
|
||||
transport,
|
||||
account,
|
||||
serverLog,
|
||||
serverLogFinished,
|
||||
output: values.output,
|
||||
transcript,
|
||||
}))
|
||||
)
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
@@ -18,7 +18,7 @@ Current history and the existing evidence support these narrower conclusions:
|
||||
| Apktool | [#4777](https://github.com/morluto/rea/issues/4777) is closed. Shared decoding and optional primary-string omission have [three-APK evidence](apktool-string-tables.md). Do not implement another decode cache from the old issue list. |
|
||||
| Hopper | [#5017](https://github.com/morluto/rea/pull/5017) already reuses decoded string records. Its [checkpoint](hopper-string-records.md) preserves faster warm queries and the negative whole-task result. |
|
||||
| JEB | The [Linux CE audit](jeb-linux-native.md) verifies selected real CLI/MCP reads. It does not justify another wrapper cache; discovery and other platforms remain separate coverage gaps. |
|
||||
| IDA | Existing [native Windows verification](ida-native-windows.md) qualifies selected correctness/authority paths. It does not establish a comparable full performance audit. |
|
||||
| IDA | Existing [native Windows verification](ida-native-windows.md) includes selected real completed-task performance and correctness/authority checks. It does not qualify every operation or platform. |
|
||||
| JavaScript/browser/Electron | Prior experiments and retained pages already exist. [#4415](https://github.com/morluto/rea/issues/4415) remains exploratory; no safe general Joplin preparation speedup is established. |
|
||||
| Other static formats | Managed, Flutter, SQLite, WASM, EVM, PCAP, signals and SAZ already have checkpoints in this directory. Extend a specific measured question rather than create duplicate selectors or caches. |
|
||||
| Other providers, languages and infrastructure | Rizin/Cutter/GDB/objdump, Go/Rust, ADB/Frida/inspector and startup/resource owners still need bounded real-workflow questions. Historical local improvements do not prove complete coverage or establish another bottleneck. |
|
||||
|
||||
@@ -58,6 +58,16 @@ export const finalizeRetainedQueryReceipt = async ({
|
||||
);
|
||||
if (report.cleanup_failures.length > 0) report.retained_workspace = account;
|
||||
|
||||
return publishRetainedQueryReceipt({ report, output, transcript, stdout });
|
||||
};
|
||||
|
||||
/** Share atomic publication and failure accounting with non-MCP phase probes. */
|
||||
export const publishRetainedQueryReceipt = async ({
|
||||
report,
|
||||
output,
|
||||
transcript,
|
||||
stdout = process.stdout,
|
||||
}) => {
|
||||
report.receipt_failures = [];
|
||||
if (transcript !== undefined)
|
||||
await attempt(report.receipt_failures, "transcript-file", () =>
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Phase receipt fault injection only; these stand-ins do not qualify a model provider."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
|
||||
class ModelPhaseReceipts(unittest.TestCase):
|
||||
def run_phase(self, inspect_body, close_body):
|
||||
driver = Path(__file__).with_name('model-utf8-validation.mjs').resolve()
|
||||
with tempfile.TemporaryDirectory(prefix='rea-model-phase-test-') as directory:
|
||||
root = Path(directory)
|
||||
artifact = root / 'input'
|
||||
artifact.write_bytes(b'receipt fixture, not a model')
|
||||
expected = {'artifact': {'path': str(artifact)}}
|
||||
(root / 'expected.json').write_text(json.dumps(expected))
|
||||
(root / 'package.json').write_text('{"type":"module"}')
|
||||
module = root / 'dist' / 'model'
|
||||
module.mkdir(parents=True)
|
||||
(module / 'ModelArtifactProvider.js').write_text(
|
||||
'export class ModelArtifactProvider {\n'
|
||||
'async inspect() {' + inspect_body.replace('EXPECTED', json.dumps(expected)) + '}\n'
|
||||
'async close() {' + close_body + '}\n}\n')
|
||||
result = subprocess.run([
|
||||
'node', str(driver), '--phase', '--runtime', str(root),
|
||||
'--artifact', str(artifact), '--expected', str(root / 'expected.json'),
|
||||
'--output', str(root / 'output'),
|
||||
], capture_output=True, text=True, timeout=15)
|
||||
self.assertEqual(result.returncode, 1, result.stderr)
|
||||
report = json.loads((root / 'output' / 'report.json').read_text())
|
||||
self.assertEqual(json.loads(result.stdout), report)
|
||||
self.assertEqual(report['status'], 'failed')
|
||||
self.assertEqual(report['receipt_failures'], [])
|
||||
return report
|
||||
|
||||
def test_rejected_inspection_keeps_cost_and_primary_failure_after_close_failure(self):
|
||||
report = self.run_phase(
|
||||
'throw new Error("inspection failed");',
|
||||
'throw new Error("cleanup failed");')
|
||||
self.assertEqual(len(report['calls']), 1)
|
||||
sample = report['calls'][0]
|
||||
self.assertEqual(sample['phase'], 'fresh')
|
||||
self.assertTrue(sample['is_error'])
|
||||
self.assertGreaterEqual(sample['elapsed_ms'], 0)
|
||||
self.assertIn('inspection failed', sample['failure'])
|
||||
self.assertIn('inspection failed', report['failure'])
|
||||
self.assertEqual(report['cleanup_failures'], ['provider: Error: cleanup failed'])
|
||||
|
||||
def test_mismatch_keeps_inspection_time_and_failure(self):
|
||||
report = self.run_phase('return {ok:true,value:{wrong:true}};', '')
|
||||
self.assertEqual(len(report['calls']), 1)
|
||||
self.assertTrue(report['calls'][0]['is_error'])
|
||||
self.assertGreaterEqual(report['calls'][0]['elapsed_ms'], 0)
|
||||
self.assertIn('AssertionError', report['failure'])
|
||||
self.assertEqual(report['cleanup_failures'], [])
|
||||
|
||||
def test_close_failure_keeps_all_successful_samples_and_publishes_failed_receipt(self):
|
||||
report = self.run_phase('return {ok:true,value:EXPECTED};',
|
||||
'throw new Error("cleanup failed");')
|
||||
self.assertEqual(len(report['calls']), 7)
|
||||
self.assertTrue(all(not sample['is_error'] for sample in report['calls']))
|
||||
self.assertNotIn('failure', report)
|
||||
self.assertEqual(report['cleanup_failures'], ['provider: Error: cleanup failed'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -1,3 +1,4 @@
|
||||
import { isUtf8 } from "node:buffer";
|
||||
import { TextDecoder } from "node:util";
|
||||
import { setImmediate } from "node:timers/promises";
|
||||
import type { JsonValue } from "../domain/jsonValue.js";
|
||||
@@ -73,6 +74,13 @@ export class GgufHeader {
|
||||
rejectModel("GGUF name exceeds its format byte limit");
|
||||
if (retain) this.#budget.charge(6 * length + 2);
|
||||
this.reader.guarantee(length);
|
||||
// Omitted preview entries still need validation, but not decoded strings.
|
||||
// Keep the streaming decoder for values spanning multiple read chunks.
|
||||
if (!retain && length <= 65536) {
|
||||
if (!isUtf8(await this.reader.take(length)))
|
||||
rejectModel("GGUF string is not valid UTF-8");
|
||||
return "";
|
||||
}
|
||||
const decoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
|
||||
let result = "";
|
||||
try {
|
||||
|
||||
@@ -368,6 +368,91 @@ it("validates booleans beyond an array preview without allocating the full value
|
||||
});
|
||||
});
|
||||
|
||||
it.each([false, true])(
|
||||
"validates omitted UTF-8 strings across chunk boundaries, big endian=%s",
|
||||
async (bigEndian) => {
|
||||
const base = ggufFixture({ bigEndian });
|
||||
const preview = Array.from({ length: 8 }, (_, n) => `\uFEFFtoken-${n} 🦙`);
|
||||
const strings = [
|
||||
...preview,
|
||||
"",
|
||||
"a".repeat(65533) + "€",
|
||||
"a".repeat(65535) + "🦙",
|
||||
"\0tail",
|
||||
];
|
||||
const fixture = ggufFixture({
|
||||
bigEndian,
|
||||
metadata: [
|
||||
{
|
||||
key: "custom.strings",
|
||||
type: 9,
|
||||
value: Buffer.concat([
|
||||
base.u32(8),
|
||||
base.u64(BigInt(strings.length)),
|
||||
...strings.map(base.text),
|
||||
]),
|
||||
},
|
||||
],
|
||||
});
|
||||
const { path, provider } = await providerAt(fixture.bytes);
|
||||
const result = await provider.inspect({ path, hash: "header" });
|
||||
if (!result.ok) throw result.error;
|
||||
expect(result.value.metadata[0]).toMatchObject({
|
||||
value: preview,
|
||||
element_count: strings.length,
|
||||
preview_complete: false,
|
||||
});
|
||||
expect(result.value.artifact).toMatchObject({
|
||||
bytes_read: fixture.header.length,
|
||||
header_sha256: createHash("sha256").update(fixture.header).digest("hex"),
|
||||
});
|
||||
expect(result.value.tensors).toHaveLength(2);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["short", Buffer.from([0xc0, 0x80])],
|
||||
[
|
||||
"chunk-sized",
|
||||
Buffer.concat([Buffer.alloc(65535, 0x61), Buffer.from([0xe2])]),
|
||||
],
|
||||
[
|
||||
"cross-chunk",
|
||||
Buffer.concat([
|
||||
Buffer.alloc(65535, 0x61),
|
||||
Buffer.from([0xf0, 0x9f, 0x61, 0x80]),
|
||||
]),
|
||||
],
|
||||
] as const)(
|
||||
"rejects malformed omitted %s UTF-8 strings",
|
||||
async (_, invalid) => {
|
||||
const base = ggufFixture();
|
||||
const fixture = ggufFixture({
|
||||
metadata: [
|
||||
{
|
||||
key: "custom.strings",
|
||||
type: 9,
|
||||
value: Buffer.concat([
|
||||
base.u32(8),
|
||||
base.u64(9n),
|
||||
...Array.from({ length: 8 }, () => base.text("valid")),
|
||||
base.u64(BigInt(invalid.length)),
|
||||
invalid,
|
||||
]),
|
||||
},
|
||||
],
|
||||
});
|
||||
const { path, provider } = await providerAt(fixture.bytes);
|
||||
const result = await provider.inspect({ path, hash: "header" });
|
||||
if (result.ok) throw new Error("Invalid omitted UTF-8 string was accepted");
|
||||
expect(result.error).toMatchObject({
|
||||
_tag: "BinaryTargetError",
|
||||
constraint: "file_format_rejected",
|
||||
message: expect.stringContaining("GGUF string is not valid UTF-8"),
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it("refuses an oversized declared header before reading or expanding it", async () => {
|
||||
const bytes = Buffer.alloc(9);
|
||||
bytes.writeBigUInt64LE(100_000_001n);
|
||||
|
||||
Reference in New Issue
Block a user