Add a dedicated token indexer for continuations and comments (#1886)

The primary motivation is that we can now robustly detect `\` continuations due to the addition of `Tok::NonLogicalNewline`. This PR generalizes the approach we took to comments (track all lines that contain any comments), and applies it to continuations too.
This commit is contained in:
Charlie Marsh
2023-01-15 01:57:31 -05:00
committed by GitHub
parent 2c644619e0
commit 3791ca721a
40 changed files with 259 additions and 114 deletions

View File

@@ -1,8 +1,10 @@
mod generator;
mod indexer;
mod locator;
mod stylist;
pub(crate) use generator::Generator;
pub(crate) use indexer::Indexer;
pub(crate) use locator::Locator;
use rustpython_parser::error::ParseError;
use rustpython_parser::parser;