mirror of https://github.com/astral-sh/ruff
6fffde72e7
## Summary
On `main`, string lexing consists of walking through the string
character-by-character to search for the closing quote (with some
nuance: we also need to skip escaped characters, and error if we see
newlines in non-triple-quoted strings). This PR rewrites `lex_string` to
instead use `memchr` to search for the closing quote, which is
significantly faster. On my machine, at least, the `globals.py`
benchmark (which contains a lot of docstrings) gets 40% faster...
```text
lexer/numpy/globals.py time: [3.6410 µs 3.6496 µs 3.6585 µs]
thrpt: [806.53 MiB/s 808.49 MiB/s 810.41 MiB/s]
change:
time: [-40.413% -40.185% -39.984%] (p = 0.00 < 0.05)
thrpt: [+66.623% +67.181% +67.822%]
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
lexer/unicode/pypinyin.py
time: [12.422 µs 12.445 µs 12.467 µs]
thrpt: [337.03 MiB/s 337.65 MiB/s 338.27 MiB/s]
change:
time: [-9.4213% -9.1930% -8.9586%] (p = 0.00 < 0.05)
thrpt: [+9.8401% +10.124% +10.401%]
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe
lexer/pydantic/types.py time: [107.45 µs 107.50 µs 107.56 µs]
thrpt: [237.11 MiB/s 237.24 MiB/s 237.35 MiB/s]
change:
time: [-4.0108% -3.7005% -3.3787%] (p = 0.00 < 0.05)
thrpt: [+3.4968% +3.8427% +4.1784%]
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
2 (2.00%) high mild
5 (5.00%) high severe
lexer/numpy/ctypeslib.py
time: [46.123 µs 46.165 µs 46.208 µs]
thrpt: [360.36 MiB/s 360.69 MiB/s 361.01 MiB/s]
change:
time: [-19.313% -18.996% -18.710%] (p = 0.00 < 0.05)
thrpt: [+23.016% +23.451% +23.935%]
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
3 (3.00%) low mild
1 (1.00%) high mild
4 (4.00%) high severe
lexer/large/dataset.py time: [231.07 µs 231.19 µs 231.33 µs]
thrpt: [175.87 MiB/s 175.97 MiB/s 176.06 MiB/s]
change:
time: [-2.0437% -1.7663% -1.4922%] (p = 0.00 < 0.05)
thrpt: [+1.5148% +1.7981% +2.0864%]
Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
5 (5.00%) high mild
5 (5.00%) high severe
```
|
||
|---|---|---|
| .. | ||
| ruff | ||
| ruff_benchmark | ||
| ruff_cache | ||
| ruff_dev | ||
| ruff_diagnostics | ||
| ruff_formatter | ||
| ruff_index | ||
| ruff_linter | ||
| ruff_macros | ||
| ruff_notebook | ||
| ruff_python_ast | ||
| ruff_python_codegen | ||
| ruff_python_formatter | ||
| ruff_python_index | ||
| ruff_python_literal | ||
| ruff_python_parser | ||
| ruff_python_resolver | ||
| ruff_python_semantic | ||
| ruff_python_stdlib | ||
| ruff_python_trivia | ||
| ruff_shrinking | ||
| ruff_source_file | ||
| ruff_text_size | ||
| ruff_wasm | ||
| ruff_workspace | ||