fix(python): preserve source boundaries in pyc previews (#6492)

Preserve source code-point prefix endpoints in the shared marshal UTF-8 decoder, including separately encoded surrogate values. Keep native TextDecoder spans, full validation, byte identity and the existing 80-unit budget. Update the PyInstaller consumer without changing its text contract.

Validation: 207 relevant source tests;24 affected compiled CLI/MCP/filesystem CI tests; independent CPython3.11.2 compiler/marshal oracle and real argparse;18 equivalent performance tasks; build/check passed. Codex reviewed7c0f7abc6b without major findings.

CI bypass authorized for demonstrated baseline-only failures:101 source and179 built cases reproduced on actual tested main ba9c5bc68. Cancelled shard remainder is unverified. See PR body and discussion for scope and evidence.

Fixes #6476
This commit is contained in:
xjh
2026-10-24 06:42:13 +08:00
committed by GitHub
parent 667ab5ae08
commit 1081008c9d
6 changed files with 235 additions and 21 deletions
+18
View File
@@ -1625,3 +1625,21 @@ it("leaves 3.11 alpha magics with other code layouts unsupported", () => {
parse(pythonPycFixture("py311", { magic: 3495 })).header,
).toMatchObject({ python_version: "3.11", code_layout: "py311" });
});
it.each([false, true])(
"preserves a source-encoded lone surrogate at the preview boundary (truncated=%s)",
(truncated) => {
const text = Buffer.concat([
Buffer.from("a".repeat(79)),
Buffer.from([0xed, 0xa0, 0x80]),
...(truncated ? [Buffer.from("z")] : []),
]);
const report = parse(withConsts([marshalUnicode(text)]));
expect(root(report).consts[0]).toEqual({
kind: "str",
value: "a".repeat(79) + "\ud800",
bytes: truncated ? 83 : 82,
truncated,
});
},
);