Files
ruff/crates/ruff_python_trivia/src/snapshots
Charlie Marsh 86ccdcc9d9 Add support for multi-character operator tokens to SimpleTokenizer (#6563)
## Summary

Allows for proper lexing of tokens like `->`.

The main challenge is to ensure that our forward and backwards
representations are the same for cases like `===`. Specifically, we want
that to lex as `==` followed by `=` regardless of whether it's a
forwards or backwards lex. To do so, we identify the range of the
sequential characters (the full span of `===`), lex it forwards, then
return the last token.

## Test Plan

`cargo test`
2023-08-16 09:09:19 -04:00
..