mirror/rea
1
0
Fork 0
mirror of https://github.com/morluto/rea synced 2026-10-26 06:16:17 -04:00
Code Issues Packages Projects Releases Wiki Activity

fix(cli): count emitted documents and normalize token windows

Browse Source
This commit is contained in:
morluto
2026-10-24 02:28:56 +08:00
parent cd128795eb
commit fc404f008d
5 changed files with 172 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cli.md
+10
View File
@@ -909,3 +909,13 @@ REA's failure status:
set -o pipefail
rea inspect-artifact ./app.asar --json | jq . > inspection.json
```
`--token-count` reports a tokenx estimate for the selected emitted document,
including a requested full-output envelope and metadata. JSON uses the same
terminal indentation or compact piped representation as normal output; JSONL
uses its emitted line representation. The final newline is included (tokenx
assigns whitespace no tokens). This is an estimate, not a model tokenizer count.
Counting retains the existing bounded formatted-string allocation after capacity
admission; omit the flag to stream large JSON documents without materializing
the complete formatted text. Exhausted token windows display an empty range at
the total token count and have no next offset.