konsti 2cbe1733c8 Use CommentRanges in backwards lexing (#7360) ...

## Summary

The tokenizer was split into a forward and a backwards tokenizer. The
backwards tokenizer uses the same names as the forwards ones (e.g.
`next_token`). The backwards tokenizer gets the comment ranges that we
already built to skip comments.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>

2023-09-16 03:21:45 +00:00

..

src

Use CommentRanges in backwards lexing (#7360)

2023-09-16 03:21:45 +00:00

tests

Remove parser dependency from ruff-python-ast (#6096)

2023-07-26 17:47:22 +02:00

Cargo.toml

Use CommentRanges in backwards lexing (#7360)

2023-09-16 03:21:45 +00:00