Dhruv Manilawala 3b4c8fffe5 Lex Jupyter line magic with `Mode::Jupyter` (#23) ... Lex Jupyter line magic with `Mode::Jupyter` This PR adds a new token `MagicCommand`[^1] which the lexer will recognize when in `Mode::Jupyter`. The rules for the lexer is as follows: 1. Given that we are at the start of line, skip the indentation and look for [characters that represent the start of a magic command](635815e8f1/IPython/core/inputtransformer2.py (L335-L346)), determine the magic kind and capture all the characters following it as the command string. 2. If the command extends multiple lines, the lexer will skip the line continuation character (`\`) but only if it's followed by a newline (`\n` or `\r`). The reason to skip this only in case of newline is because they can occur in the command string which we should not skip: ```rust // Skip this backslash // v // !pwd \ // && ls -a | sed 's/^/\\ /' // ^^ // Don't skip these backslashes ``` 3. The parser, when in `Mode::Jupyter`, will filter these tokens before the parsing begins. There is a small caveat when the magic command is indented. In the following example, when the parser filters out magic command, it'll throw an indentation error: ```python for i in range(5): !ls # What the parser will see for i in range(5): ``` [^1]: I would prefer to have some other name as this not only represent a line magic (`%`) but also shell command (`!`), help command (`?`) and others. In original implementation, it's named as ["IPython Syntax"](635815e8f1/IPython/core/inputtransformer2.py (L332))		2023-07-18 09:24:24 +05:30
..
error.rs	numerous refactoring	2023-05-10 14:35:38 +09:00
format.rs	Add utilities to enum (#44)	2023-05-16 23:29:49 +09:00
lib.rs	Remove `fold`, `unparse`, and `location` features (#9)	2023-06-19 17:26:17 -04:00
mode.rs	Lex Jupyter line magic with `Mode::Jupyter` (#23)	2023-07-18 09:24:24 +05:30