Commit Graph

2 Commits

Author SHA1 Message Date
Kaoru0822 77be0f6666 fix(capture): end special-scheme URL authorities at a backslash (#2767)
* fix(capture): end special-scheme URL authorities at a backslash

The native mitmproxy adapter removed transport userinfo with
`^(...//)[^/?#]*@`, so the authority ran to the last "@" before "/", "?"
or "#". For `https://example.test\part@tail` it treated
`example.test\part` as userinfo, reported `https://tail` as the
Location, and withdrew the original bytes as a transport credential.
WHATWG URL parsing, used by the shared TypeScript owner
(recordedTransportUrl.ts) and by HAR captures, ends a special-scheme or
scheme-relative authority at a backslash, so that "@" is ordinary path
text.

Locate the authority with the same boundary rule before removing
userinfo. Bytes are handled as Latin-1 text so unrelated bytes stay
identical. Userinfo before the backslash is still removed and redacted.
Add source unit tests for the adapter projection to test:python.

Fixes #2671

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(capture): skip special-scheme leading slashes before userinfo

Ending the authority at a backslash exposed a credential: for
`https://\user:pass@host/path`, WHATWG parsing skips every leading "/"
or "\" after a special scheme (and in scheme-relative references), so
`user:pass` is userinfo, but the new boundary stopped at the first
backslash and found no "@". The same applied to `https:\\user:pass@host`
and `https:user:pass@host`.

Start the authority after that whole slash run for non-file special
schemes and scheme-relative references. File URLs keep their "//"
boundary because they carry no userinfo. Load the adapter in tests with
fresh mitmproxy stand-ins scoped to its import instead of leaving them
in sys.modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(capture): parse URL controls and mixed network authorities

* fix(capture): distinguish native paths and URL base authorities

* fix(capture): preserve file-relative path evidence after separators

* fix(comparison): recheck historical calls-only changes (#2223)

* fix(comparison): preserve incomplete function call evidence

Keep missing, unknown, partial and unresolved call relationships unknown in function comparisons and reconstruction claims. Aggregate IDA caller and callee coverage without overwriting uncertainty.

Validate with 144 focused tests, typecheck/lint, docs checks and test discovery. Real providers were not verified; the full headless IDA session suite requires unavailable Windows native controls.

* fix(comparison): recheck historical calls-only changes

---------

Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>

* fix(android): make DEX descriptor pattern portable

* fix(ci): isolate Linux Hopper process fixture inventory

* fix(capture): remove empty URL userinfo markers

* test(capture): cover empty URL userinfo markers

* test(ci): isolate Hopper process inventory fixtures

* fix(ci): declare process inventory callback

* test(ci): isolate alternate launcher process fixture

* fix(capture): keep URLs that WHATWG rejects instead of redacting userinfo

File URLs cannot carry userinfo (an "@" in a file host fails the parse),
and a special authority whose host or port WHATWG rejects fails the whole
URL. Neither has credentials, so the bridge now keeps their source text
and byte sidecar. Checked against Node's URL over 30,000 spellings: no
under-redaction, over-redaction 20,540 -> 5,865 (remaining: empty
userinfo, IPv4 range, IDNA, non-special and baseless references).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(capture): strip trailing URL whitespace and validate IP hosts and ports

The previous host check counted trailing C0 controls and spaces, which
WHATWG strips, so `https://user:pw@host ` kept its password. Trailing
C0/space is now removed from the parse view (raw offsets are unchanged).

Special hosts now also follow the WHATWG IPv4 parser (ends-in-a-number,
hex/octal parts, range) and IPv6 literal syntax (zone IDs rejected), and
ports are bounded lexically before int(), which raised ValueError past
4300 digits on Python 3.11+ and failed the whole capture.

Node URL differential over 126,000 spellings: under-redaction 8,084 -> 0,
ValueError 2,550 -> 0; remaining special-scheme over-redaction is IDNA
and the pinned empty-userinfo case. Cost: +0.6 us per URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(capture): redact userinfo before hosts that need Unicode mapping

A non-ASCII special host may map under UTS #46 to an accepted host
(fullwidth digits, ideographic full stops, ignored code points), so
treat it as accepted instead of running the ASCII IPv4 check on it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
Co-authored-by: Lucien <164857476+dakjdakd@users.noreply.github.com>
2026-10-13 21:46:15 -07:00
morluto f6905719c3 fix(browser): bound capture work and preserve observation gaps
Own service-worker CDP registrations and retry failed teardown. Admit redaction and network metadata against remaining capture capacity, retain explicit omission reasons, and preserve source URL, redirect ancestry, caller-selected content, and historical capture values. Forward CLI cancellation through provider cleanup.

Reviewed fixes: #2263, #2637, #2767, #2845, #2859, #2863, #2872, #2874, #2876, #2879, #2955, #2957, #2961, #3033.

Validation: 25 current event/network source tests passed; earlier focused browser lanes passed 201 source tests and 176 boundary tests with two skips, plus 35 Python tests. Real browser/Windows lanes were not rerun. Playwright allocation before admission remains unbounded by these retention limits.
2026-10-14 08:31:29 +08:00