mirror of
https://github.com/morluto/rea
synced 2026-10-26 06:16:17 -04:00
docs: codify boundary regression checks
Turn recurring parser and projection failures into a selectable regression guide covering format assumptions, raw identity, missing values, partial work, independent limits, representation expansion, snapshots, and public delivery. Link it from the agent instructions and PR template without adding checkboxes. Clarify completed-observation ownership, digest representations, and exclusive success/failure budget accounting. Preserve capacity fixtures and use byte comparisons for large buffers to avoid diagnostic allocation blowups.
This commit is contained in:
@@ -68,6 +68,10 @@ source, link, or rendered-document checks. -->
|
||||
- User-visible CLI/MCP output (if applicable):
|
||||
- Remaining proof gaps:
|
||||
|
||||
For boundary fixes, identify the applicable [boundary regressions](https://github.com/morluto/rea/blob/main/docs/testing.md#boundary-regressions)
|
||||
and [lifecycle regressions](https://github.com/morluto/rea/blob/main/docs/testing.md#lifecycle-regressions)
|
||||
covered by the evidence above, and explain relevant gaps.
|
||||
|
||||
For evidence-bearing changes:
|
||||
|
||||
- [ ] Observed, derived, and inferred claims remain distinguishable.
|
||||
|
||||
@@ -80,7 +80,7 @@ Coordinate expensive checks across agents. If a test unexpectedly consumes heavy
|
||||
|
||||
For tool design, follow [docs/tool-design.md](docs/tool-design.md): start from the analyst question, reuse existing contracts, preserve caller choices, and return useful evidence inline. Limits must follow real format, protocol, authority, or resource constraints. Account for representation expansion before allocation; derive completeness from examined coverage.
|
||||
|
||||
For test changes or provider claims, use [docs/testing.md](docs/testing.md). Prefer real public CLI/MCP workflows, then production-boundary integration, then producer goldens. Keep module tests for distinct semantics stronger workflows cannot reliably reproduce. Preserve malformed-input, cancellation, cleanup, permission, capacity, and target-identity coverage when pruning. Report unverified coverage explicitly; simulated providers do not establish real-provider behavior.
|
||||
For test changes or provider claims, use [docs/testing.md](docs/testing.md). Select applicable [boundary regressions](docs/testing.md#boundary-regressions) for parser, inventory, snapshot, and projection fixes. Prefer real public CLI/MCP workflows, then production-boundary integration, then producer goldens. Keep module tests for distinct semantics stronger workflows cannot reliably reproduce. Preserve malformed-input, cancellation, cleanup, permission, capacity, and target-identity coverage when pruning. Report unverified coverage explicitly; simulated providers do not establish real-provider behavior.
|
||||
|
||||
Use Conventional Commit subjects and PR titles. Mark breaking changes with `!` or a `BREAKING CHANGE:` footer. Describe behavior changes, relevant validation, and real-provider verification scope.
|
||||
|
||||
|
||||
@@ -200,6 +200,10 @@ work from fixture generation, schema compilation, subprocess startup and cleanup
|
||||
Choose elapsed, CPU and process-family memory limits before the run; a worker
|
||||
count or test timeout alone does not enforce those limits.
|
||||
|
||||
Keep capacity fixtures intact when diagnosing a test failure. For byte equality
|
||||
of large buffers, use `Buffer.equals` or `Buffer.compare`; a generic structural
|
||||
diff can allocate far more than the production operation.
|
||||
|
||||
Remove demonstrated repeated work while preserving distinct assertions and
|
||||
resource ownership. Share immutable fixture preparation where safe; keep fresh
|
||||
sessions when lifecycle or concurrency is the behavior under test. Compare the
|
||||
@@ -335,6 +339,30 @@ that a real loader accepts the artifact. If multiple producer revisions share a
|
||||
format identifier but differ in behavior, record that ambiguity rather than
|
||||
using one installed version as an oracle for all of them.
|
||||
|
||||
### Boundary regressions
|
||||
|
||||
For parser, inventory, snapshot, or projection fixes, select the applicable
|
||||
cases below. Assert the producing boundary's facts and their meaning in affected
|
||||
consumers. Add [lifecycle regressions](#lifecycle-regressions) when resource
|
||||
ownership or finalization changes; a single malformed fixture cannot establish
|
||||
the whole contract.
|
||||
|
||||
| Failure class | Regression that catches it |
|
||||
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Format assumptions | Pair valid padding, offsets, or names containing delimiters with malformed input. Identify the producer revision when behavior varies by version; an unrelated runtime API's range is not an on-disk format limit. |
|
||||
| Raw identity and normalization | Exercise allowed empty or unusual source names/metadata keys and values that normalize to one lookup or display value. Preserve admitted raw spelling under the existing redactions for transport credentials and explicitly marked sensitive values. Distinguish local paths, file URLs and HTTP paths; assert source identity through consumers. |
|
||||
| Missing, empty, and unknown values | Distinguish omitted, null, empty, zero, and false where the format admits them. Pair a malformed optional field with usable sibling facts; assert uncertainty without inventing absence or discarding the row. |
|
||||
| Completed work followed by failure | Complete one input, then fail or cancel a later input. Assert earlier digests, metadata, and parsed facts survive with the later failure reason and accurate processed/unprocessed coverage. For cancellation, assert the exact caller abort reason; where a primary failure was already produced, assert it survives. |
|
||||
| Independent limits and failures | Trigger a sample/output limit alongside a read or parse failure. Assert both dimensions survive; reaching a limit does not erase sampling failures or prove complete coverage. |
|
||||
| Representation expansion and capacity | Keep a valid set near the limit and a large malformed input followed by a valid input. Cover JSON escaping, expanded paths or decoded rows as applicable. Assert serialized bounds, retained identities, later valid results, and unknown omissions without charging discarded interpretations as retained output. |
|
||||
| Snapshots, caches, and comparison | Vary metadata that affects the result while keeping payload bytes equal. Exercise historical records with newly introduced fields omitted, current caller inputs on replay, and reuse across differing selections. Assert correct invalidation or unknown coverage, current request provenance, and unchanged saved Evidence and IDs. |
|
||||
| Public success and error delivery | Exercise affected public CLI/MCP paths, including partial Evidence on failure. Assert retained facts and named contracts; retrieve partial Evidence when MCP returns a reference. If SDK cancellation suppresses the reply, check server settlement and any retained Evidence. An error-code assertion alone misses lost observations. |
|
||||
|
||||
When producer order is unspecified, synchronize fixtures on the observed request
|
||||
or completion event. A filename, generated key, or fixed delay does not prove
|
||||
which item completed before cancellation. Preserve real producer ordering and
|
||||
use a controlled seam for the phase that needs deterministic failure.
|
||||
|
||||
Packed bridge validation uses Python's standard-library AST and tar readers
|
||||
with `-I -S`: caller-local modules, `PYTHONPATH`, and site startup hooks must
|
||||
not replace those readers or execute during validation. Controlled `ast.py`
|
||||
|
||||
+23
-4
@@ -85,6 +85,17 @@ satisfy. A plan without execution does not establish runtime behavior.
|
||||
|
||||
### Interpret boundary facts once
|
||||
|
||||
The format or protocol reader owns interpretation. Pass its parsed facts and
|
||||
explicit uncertainty to consumers; do not make adapters reparse display text,
|
||||
reconstruct omitted values, or infer success from missing diagnostics.
|
||||
|
||||
A completed read can establish byte identity even when decoding or format
|
||||
validation fails. State which representation a digest identifies, such as file
|
||||
bytes, Fetch response-body bytes, or UTF-8 text. Preserve admitted source values
|
||||
separately from normalized lookup and display values. When the retention
|
||||
contract keeps only a digest and size, state that the bytes or text are
|
||||
unavailable; the digest does not establish format validity.
|
||||
|
||||
Comparison needs separate proofs for lookup, correspondence, content, and
|
||||
coverage. A hash cannot supply the proof that two retained values belong to
|
||||
corresponding objects.
|
||||
@@ -103,10 +114,6 @@ Complete traversal or inventory does not make unavailable values equal. A
|
||||
partial aggregate fingerprint difference cannot establish modification when
|
||||
omissions alone explain it; independently proven differences still survive.
|
||||
|
||||
The format or protocol reader owns interpretation. Pass its parsed facts and
|
||||
explicit uncertainty to consumers; do not make adapters reparse display text,
|
||||
reconstruct omitted values, or infer success from missing diagnostics.
|
||||
|
||||
Keep format recognition separate from provider admission. A recognizable header
|
||||
can supply useful observations while later structure remains unresolved or a
|
||||
provider cannot accept the artifact. Report the reason at the affected boundary
|
||||
@@ -133,6 +140,12 @@ establish each transition. Include temporary files and one-shot commands, as
|
||||
well as persistent engine sessions. Use the existing provider contracts;
|
||||
providers need not expose identical state machines.
|
||||
|
||||
In multi-stage work, retain each completed, admitted observation before awaiting
|
||||
the next stage. Later reads, validation, cancellation, or finalization can fail
|
||||
without erasing those facts. Carry them through typed failure outcomes with
|
||||
explicit coverage of unperformed work; do not construct a successful result for
|
||||
stages that never ran.
|
||||
|
||||
- Keep target admission, provider readiness, active work, observed process exit,
|
||||
and resource release separate. A valid process handle, successful launcher,
|
||||
or selected target establishes only its own fact. Publish available identity
|
||||
@@ -259,6 +272,12 @@ completeness from examined records and exhaustiveness, rather than diagnostic
|
||||
prose or an empty failure list. Preserve observations independently of cleanup
|
||||
success.
|
||||
|
||||
Price mutually exclusive success and failure representations separately.
|
||||
Replace a reserved failure envelope when pricing success, and release discarded
|
||||
decoding state from retained-output accounting. Retaining additional failed-body
|
||||
text changes the capacity available to later results; treat that as a contract
|
||||
change and cover its effect on later valid items.
|
||||
|
||||
For example, if ten applicable records are discovered, eight are examined, and
|
||||
five are returned, the five returned records do not establish complete coverage
|
||||
of the ten. Keep discovered, applicable, examined, and retained scope distinct.
|
||||
|
||||
Reference in New Issue
Block a user