Python
/
ruff
mirror of https://github.com/astral-sh/ruff
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
376 Commits 514 Branches 394 Tags 554 MiB
Commit Graph

42 Commits

Author SHA1 Message Date
Dhruv Manilawala 3b4c8fffe5
Lex Jupyter line magic with `Mode::Jupyter` (#23) 
Lex Jupyter line magic with `Mode::Jupyter`

This PR adds a new token `MagicCommand`[^1] which the lexer will
recognize when in `Mode::Jupyter`. The rules for the lexer is as
follows:
1. Given that we are at the start of line, skip the indentation and look
for [characters that represent the start of a magic
command](635815e8f1/IPython/core/inputtransformer2.py (L335-L346)),
determine the magic kind and capture all the characters following it as
the command string.
2. If the command extends multiple lines, the lexer will skip the line
continuation character (`\`) but only if it's followed by a newline
(`\n` or `\r`). The reason to skip this only in case of newline is
because they can occur in the command string which we should not skip:

	```rust
    //        Skip this backslash
    //        v
    //   !pwd \
    //      && ls -a | sed 's/^/\\    /'
    //                          ^^
    //                          Don't skip these backslashes
	```

3. The parser, when in `Mode::Jupyter`, will filter these tokens before
the parsing begins. There is a small caveat when the magic command is
indented. In the following example, when the parser filters out magic
command, it'll throw an indentation error:

	```python
	for i in range(5):
		!ls

	# What the parser will see
	for i in range(5):
	
	```

[^1]: I would prefer to have some other name as this not only represent
a line magic (`%`) but also shell command (`!`), help command (`?`) and
others. In original implementation, it's named as ["IPython
Syntax"](635815e8f1/IPython/core/inputtransformer2.py (L332))
 2023-07-18 09:24:24 +05:30
Charlie Marsh f0d200c8a1
Remove `fold`, `unparse`, and `location` features (#9) 2023-06-19 17:26:17 -04:00
Jeong, YunWon b2f95e2848
Fix LinearLocator \r handling (#80) 2023-06-02 22:35:53 +09:00
Jeong, YunWon 5e9e8a7589
Linear Locator (#46) 2023-06-01 13:53:31 +09:00
Jeong, YunWon ff17f6e178
Add utilities to enum (#44) 
* Add utilities to enum

* Fix unexpected pyo3 dependency propagation
 2023-05-16 23:29:49 +09:00
Jeong YunWon 3bdf8a940a Hash for ConversionFlag 2023-05-16 22:56:17 +09:00
Jeong, YunWon 9d47d3d212
specialize ConversionFlag (#42) 
* specialize ConversionFlag

* Change value of ConversionFlag to i8 and None to -1

* is_* methods to ConversionFlag
 2023-05-16 22:52:50 +09:00
Micha Reiser 192379cede
Move `range` from `Attributed` to `Node`s (#22) 
* Move `range` from `Attributed` to `Node`s

* No Attributed + custom for Range PoC

* Generate all located variants, generate enum implementations

* Implement `Copy` on simple enums

* Move `Suite` to `ranged` and `located`

* Update tests
---------

Co-authored-by: Jeong YunWon <jeong@youknowone.org>
 2023-05-15 15:08:12 +09:00
Jeong YunWon cbe4e8c5f3 Make parser location optional 2023-05-11 04:40:10 +09:00
Jeong YunWon 4dc030ba9d Vendor SourceLocation from ruff 2023-05-10 17:00:12 +09:00
Jeong YunWon a3d9d8cb14 numerous refactoring 
- Split parser core and compiler core. Fix #14
- AST int type to `u32`
- Updated asdl_rs.py and update_asdl.sh fix #6
- Use `ruff_python_ast::SourceLocation` for Python source location. Deleted our own Location.
- Renamed ast::Located to ast::Attributed to distinguish terms for TextSize and SourceLocation
- `ast::<Node>`s for TextSize located ast. `ast::located::<Node>` for Python source located ast.
- And also strictly renaming `located` to refer only python location related interfaces.
- `SourceLocator` to convert locations.
- New `source-code` features of to disable python locations when unnecessary.
- Also including fully merging https://github.com/astral-sh/RustPython/pull/4 closes #9
 2023-05-10 14:35:38 +09:00
Jeong YunWon 09a6afdd04 Adapt SourceLocation 2023-05-09 20:34:48 +09:00
Jeong YunWon a14e43e03a Separate byteoffset ast and located ast 2023-05-09 00:21:52 +09:00
Jeong YunWon f47dfca4e3 Rename compiler Location to TextSize 2023-05-08 03:38:10 +09:00
Micha Reiser 58c35ab458 Replace row/column based Location with byte-offsets. 2023-05-08 03:38:10 +09:00
Jeong YunWon 13d6e275ef Remove CompileError 2023-05-06 17:15:18 +09:00
Jeong YunWon f9b5469642 Update cspell for compiler 2023-03-16 22:39:09 +09:00
Jeong YunWon 0ea53825db Merge pull request #4608 from coolreader18/bag-deser 
Rework frozen modules and directly deserialize to CodeObject<Literal>
 2023-03-10 05:19:23 +09:00
Noa 9d6ae774f8 Rework frozen modules and directly deserialize to CodeObject<Literal> 2023-03-06 13:45:33 -06:00
Charlie Marsh 713dd2b91e Add optional serde dependency 2023-03-02 15:38:40 -05:00
Noa 969ea23d67 Address review comments 2023-03-01 21:11:59 -06:00
Noa b80bbec8e6 Custom marshal enc/decoding impl 2023-03-01 20:47:21 -06:00
Jeong YunWon e26369a34e use super::* from tests submodules 2023-02-22 21:01:39 +09:00
Jeong YunWon 66e3080173 Fix ModeParseError message 2023-02-22 20:28:03 +09:00
Noa 872b9d4765 Switch from 64-bit instruction enum to out-of-line arg values 2023-01-12 23:05:17 -06:00
harupy 2dfd053bed Implement Default for Location 2023-01-05 22:48:47 +09:00
harupy 0365752bf3 Use Self 2023-01-04 13:19:23 +09:00
harupy 6d140426c1 Add with_col_offset and with_row_offset to Location 2023-01-04 13:16:03 +09:00
Jeong YunWon 28785784b2 Fix nightly clippy warnings 2022-12-05 12:18:16 +09:00
dvermd a5b59f3c9d improve col_offset in new line and lalr 2022-10-26 21:30:51 +02:00
dvermd fa41a1e2f6 Fix ast types' _fields and use 0-based column 2022-10-27 03:09:38 +09:00
Jeong YunWon 518cf728c3 Fix nightly clippy warnings 2022-10-16 02:38:50 +09:00
Charlie Marsh 26b529f9dc Add PartialOrd to Location 2022-09-04 16:31:43 -04:00
Jeong YunWon 40690b9761 use thiserror 2022-08-23 05:08:24 +09:00
Jeong YunWon 1fd898c14c new_syntax_error working without full compiler 
Fix #4100
 2022-08-23 05:08:24 +09:00
Jeong YunWon 42b95a9a95 Move out CompileError to core as generic form 2022-08-23 01:30:00 +09:00
Jeong YunWon e8230efe1a Integrate CompileError to compiler-core::BaseError 2022-08-22 23:18:41 +09:00
Jeong YunWon 7fcc18daea integrate CodegenError to compiler-core::Error 2022-08-22 18:43:03 +09:00
Jeong YunWon 2b7bf79d29 Integrate ParseError to compiler-core::Error 2022-08-22 16:28:08 +09:00
Jeong YunWon 904fc477f1 integrate ast::Location into compilre-core::Location 2022-08-22 08:42:20 +09:00
Jeong YunWon bfac0355dc Share location between compiler crates 2022-08-22 08:42:20 +09:00
Jeong YunWon c16e650071 rustpython-bytecode -> rustpython-compiler-core 2022-08-22 08:42:20 +09:00