Commit Graph

4191 Commits

Author SHA1 Message Date
Alex Waygood cd2af3be73
[red-knot] Reduce allocations when normalizing `VendoredPath`s (#11992) 2024-06-24 13:08:01 +01:00
Micha Reiser e2e98d005c
Fix missing related settings header (#12013) 2024-06-24 12:29:10 +02:00
renovate[bot] 53a80a5c11
Update Rust crate rustc-hash to v2 (#12001) 2024-06-23 20:46:42 -04:00
ukyen 068b75cc8e
[`pyflakes`] Detect assignments that shadow definitions (`F811`) (#11961)
## Summary
This PR updates `F811` rule to include assignment as possible shadowed
binding. This will fix issue: #11828 .

## Test Plan

Add a test file, F811_30.py, which includes a redefinition after an
assignment and a verified snapshot file.
2024-06-23 13:29:32 -04:00
Denny Wong c3f61a012e
[`ruff`] Add `assert-with-print-expression` rule (#11974) (#11981)
## Summary

Addresses #11974 to add a `RUF` rule to replace `print` expressions in
`assert` statements with the inner message.

An autofix is available, but is considered unsafe as it changes
behaviour of the execution, notably:
- removal of the printout in `stdout`, and
- `AssertionError` instance containing a different message.

While the detection of the condition is a straightforward matter,
deciding how to resolve the print arguments into a string literal can be
a relatively subjective matter. The implementation of this PR chooses to
be as tolerant as possible, and will attempt to reformat any number of
`print` arguments containing single or concatenated strings or variables
into either a string literal, or a f-string if any variables or
placeholders are detected.

## Test Plan

`cargo test`.

## Examples
For ease of discussion, this is the diff for the tests:

```diff
 # Standard Case
 # Expects:
 # - single StringLiteral
-assert True, print("This print is not intentional.")
+assert True, "This print is not intentional."
 
 # Concatenated string literals
 # Expects:
 # - single StringLiteral
-assert True, print("This print" " is not intentional.")
+assert True, "This print is not intentional."
 
 # Positional arguments, string literals
 # Expects:
 # - single StringLiteral concatenated with " "
-assert True, print("This print", "is not intentional")
+assert True, "This print is not intentional"
 
 # Concatenated string literals combined with Positional arguments
 # Expects:
 # - single stringliteral concatenated with " " only between `print` and `is`
-assert True, print("This " "print", "is not intentional.")
+assert True, "This print is not intentional."
 
 # Positional arguments, string literals with a variable
 # Expects:
 # - single FString concatenated with " "
-assert True, print("This", print.__name__, "is not intentional.")
+assert True, f"This {print.__name__} is not intentional."

 # Mixed brackets string literals
 # Expects:
 # - single StringLiteral concatenated with " "
-assert True, print("This print", 'is not intentional', """and should be removed""")
+assert True, "This print is not intentional and should be removed"
 
 # Mixed brackets with other brackets inside
 # Expects:
 # - single StringLiteral concatenated with " " and escaped brackets
-assert True, print("This print", 'is not "intentional"', """and "should" be 'removed'""")
+assert True, "This print is not \"intentional\" and \"should\" be 'removed'"
 
 # Positional arguments, string literals with a separator
 # Expects:
 # - single StringLiteral concatenated with "|"
-assert True, print("This print", "is not intentional", sep="|")
+assert True, "This print|is not intentional"
 
 # Positional arguments, string literals with None as separator
 # Expects:
 # - single StringLiteral concatenated with " "
-assert True, print("This print", "is not intentional", sep=None)
+assert True, "This print is not intentional"
 
 # Positional arguments, string literals with variable as separator, needs f-string
 # Expects:
 # - single FString concatenated with "{U00A0}"
-assert True, print("This print", "is not intentional", sep=U00A0)
+assert True, f"This print{U00A0}is not intentional"
 
 # Unnecessary f-string
 # Expects:
 # - single StringLiteral
-assert True, print(f"This f-string is just a literal.")
+assert True, "This f-string is just a literal."
 
 # Positional arguments, string literals and f-strings
 # Expects:
 # - single FString concatenated with " "
-assert True, print("This print", f"is not {'intentional':s}")
+assert True, f"This print is not {'intentional':s}"
 
 # Positional arguments, string literals and f-strings with a separator
 # Expects:
 # - single FString concatenated with "|"
-assert True, print("This print", f"is not {'intentional':s}", sep="|")
+assert True, f"This print|is not {'intentional':s}"
 
 # A single f-string
 # Expects:
 # - single FString
-assert True, print(f"This print is not {'intentional':s}")
+assert True, f"This print is not {'intentional':s}"
 
 # A single f-string with a redundant separator
 # Expects:
 # - single FString
-assert True, print(f"This print is not {'intentional':s}", sep="|")
+assert True, f"This print is not {'intentional':s}"
 
 # Complex f-string with variable as separator
 # Expects:
 # - single FString concatenated with "{U00A0}", all placeholders preserved
 condition = "True is True"
 maintainer = "John Doe"
-assert True, print("Unreachable due to", condition, f", ask {maintainer} for advice", sep=U00A0)
+assert True, f"Unreachable due to{U00A0}{condition}{U00A0}, ask {maintainer} for advice"
 
 # Empty print
 # Expects:
 # - `msg` entirely removed from assertion
-assert True, print()
+assert True
 
 # Empty print with separator
 # Expects:
 # - `msg` entirely removed from assertion
-assert True, print(sep=" ")
+assert True
 
 # Custom print function that actually returns a string
 # Expects:
@@ -100,4 +100,4 @@
 # Use of `builtins.print`
 # Expects:
 # - single StringLiteral
-assert True, builtins.print("This print should be removed.")
+assert True, "This print should be removed."
```

## Known Issues

The current implementation resolves all arguments and separators of the
`print` expression into a single string, be it
`StringLiteralValue::single` or a `FStringValue::single`. This:

- potentially joins together strings well beyond the ideal character
limit for each line, and
- does not preserve multi-line strings in their original format, in
favour of a single line `"...\n...\n..."` format.

These are purely formatting issues only occurring in unusual scenarios.

Additionally, the autofix will tolerate `print` calls that were
previously invalid:

```python
assert True, print("this", "should not be allowed", sep=42)
```

This will be transformed into
```python
assert True, f"this{42}should not be allowed"
```
which some could argue is an alteration of behaviour.

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2024-06-23 16:54:55 +00:00
Gilles Peiffer 0c8b5eb17a
Clarify special control flow parameters for `PLR0917`: `too-many-positional` (#11978) 2024-06-23 11:16:09 -04:00
Alex Waygood 375d2c87b2
[red-knot] Simplify conversions from `std::path::Path` to `VendoredPath(Buf)` (#11988) 2024-06-23 15:52:26 +01:00
Alex Waygood f846fc9e07
[red-knot] Once again, add more tests asserting that the `VendoredFileSystem` and the `VERSIONS` parser work with the vendored typeshed stubs (#11987) 2024-06-23 14:57:43 +01:00
Alex Waygood 92b145e56a
[red-knot] Manually implement `Debug` for `VendoredFileSystem` (#11983) 2024-06-23 14:25:56 +01:00
Eric Nielsen 715609663a
Update PEP reference in future_rewritable_type_annotation.rs (#11985)
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

Documentation mentions:

> PEP 563 enabled the use of a number of convenient type annotations,
such as `list[str]` instead of `List[str]`

but it meant [PEP 585](https://peps.python.org/pep-0585/) instead.

[PEP 563](https://peps.python.org/pep-0563/) is the one defining `from
__future__ import annotations`.

## Test Plan

No automated test required, just verify that
https://peps.python.org/pep-0585/ is the correct reference.
2024-06-22 20:15:12 -05:00
Micha Reiser 519a278899
[red-knot] Remove itertools dependency from `ruff_db` (#11984) 2024-06-22 18:37:51 +00:00
Alex Waygood 91d091bb81
[red-knot] Use POSIX representations of paths when creating the typeshed zip file (#11982) 2024-06-22 17:54:19 +01:00
Dhruv Manilawala 81160320de
Manual impl of `Debug` on `Token` (#11958)
## Summary

I look at the token stream a lot, not specifically in the playground but
in the terminal output and it's annoying to scroll a lot to find
specific location. Most of the information is also redundant.

The final format we end up with is: `<kind> <range> (flags = ...)` e.g.,
`String 0..4 (flags = BYTE_STRING)` where the flags part is only
populated if there are any flags set.
2024-06-22 04:18:24 +00:00
Jane Lewis ad4a88657b
Remove usage of `std::path::absolute` from snapshot test (#11973) 2024-06-21 20:21:12 +01:00
Alex Waygood 611f4e5c5f
Revert "[red-knot] Add more tests asserting that the VendoredFileSystem and the `VERSIONS` parser work with the vendored typeshed stubs" (#11975) 2024-06-21 19:14:24 +00:00
Jane Lewis 791f6a1820
`ruff server`: Closing an untitled, unsaved notebook document no longer throws an error (#11942)
## Summary

Fixes #11651.
Fixes #11851.

We were double-closing a notebook document from the index, once in
`textDocument/didClose` and then in the `notebookDocument/didClose`
handler. The second time this happens, taking a snapshot fails.

I've rewritten how we handle snapshots for closing notebooks / notebook
cells so that any failure is simply logged instead of propagating
upwards. This implementation works consistently even if we don't receive
`textDocument/didClose` notifications for each specific cell, since they
get closed (and the diagnostics get cleared) in the notebook document
removal process.

## Test Plan

1. Open an untitled, unsaved notebook with the `Create: New Jupyter
Notebook` command from the VS Code command palette (`Ctrl/Cmd + Shift +
P`)
2. Without saving the document, close it.
3. No error popup should appear.
4. Run the debug command (`Ruff: print debug information`) to confirm
that there are no open documents
2024-06-21 10:53:30 -07:00
Alex Waygood 3d0230f469
[red-knot] Add more tests asserting that the VendoredFileSystem and the `VERSIONS` parser work with the vendored typeshed stubs (#11970) 2024-06-21 16:53:10 +00:00
Alex Waygood da79bac33c
[red-knot] Make the `VERSIONS` parser use `ModuleName` as its key type (#11968) 2024-06-21 15:46:45 +00:00
Alex Waygood 8de0cd6565
[red-knot] Move typeshed `VERSIONS` parser to the module resolver crate (#11967) 2024-06-21 16:41:08 +01:00
Alex Waygood 3277d031f8
[red-knot] Move the vendored typeshed stubs to the module resolver crate (#11966) 2024-06-21 13:47:54 +00:00
Alex Waygood 736a4ead14
[red-knot] Move module-resolution logic to its own crate (#11964) 2024-06-21 13:25:44 +00:00
Dhruv Manilawala 27ebff36ec
Remove `Token::is_trivia` method (#11962)
Sorry, a leftover from my rebase
2024-06-21 10:24:42 +00:00
Dhruv Manilawala 96da136e6a
Move token and error structs into related modules (#11957)
## Summary

This PR does some housekeeping into moving certain structs into related
modules. Specifically,
1. Move `LexicalError` from `lexer.rs` to `error.rs` which also contains
the `ParseError`
2. Move `Token`, `TokenFlags` and `TokenValue` from `lexer.rs` to
`token.rs`
2024-06-21 10:07:19 +00:00
Dhruv Manilawala 4667d8697c
Remove duplication around `is_trivia` functions (#11956)
## Summary

This PR removes the duplication around `is_trivia` functions.

There are two of them in the codebase:
1. In `pycodestyle`, it's for newline, indent, dedent, non-logical
newline and comment
2. In the parser, it's for non-logical newline and comment

The `TokenKind::is_trivia` method used (1) but that's not correct in
that context. So, this PR introduces a new `is_non_logical_token` helper
method for the `pycodestyle` crate and updates the
`TokenKind::is_trivia` implementation with (2).

This also means we can remove `Token::is_trivia` method and the
standalone `token_source::is_trivia` function and use the one on
`TokenKind`.

## Test Plan

`cargo insta test`
2024-06-21 10:02:40 +00:00
Will Yardley 690e94f4fb
`ruff-check`: update docs for fix_only (#11959) 2024-06-21 08:13:04 +02:00
dedebenui 9fd84e63bc
Update `trapz` and `in1d` deprecation for NPY201 (#11948) 2024-06-21 08:08:00 +02:00
Jane Lewis 3ab7a8da73
Add Jupyter Notebook document change snapshot test (#11944)
## Summary

Closes #11914.

This PR introduces a snapshot test that replays the LSP requests made
during a document formatting request, and confirms that the notebook
document is updated in the expected way.
2024-06-21 05:29:27 +00:00
Micha Reiser 927069c12f
[red-knot] Upgrade to Salsa 3.0 (#11952) 2024-06-20 20:19:16 +01:00
Jane Lewis c8ff89c73c
`ruff server`: Support the usage of tildes and environment variables in `logFile` (#11945)
## Summary

Fixes #11911.

`shellexpand` is now used on `logFile` to expand the file path, allowing
the usage of `~` and environment variables.

## Test Plan

1. Set `logFile` in either Neovim or Helix to a file path that needs
expansion, like `~/.config/helix/ruff_logs.txt`.
2. Ensure that `RUFF_TRACE` is set to `messages` or `verbose`
3. Open a Python file in Neovim/Helix
4. Confirm that a file at the path specified was created, with the
expected logs.
2024-06-20 18:51:46 +00:00
Dhruv Manilawala b54922fd73
Bump version to v0.4.10 (#11953) 2024-06-20 22:37:44 +05:30
Dhruv Manilawala 3f884b4b34
Avoid running logical line rule logic if not enabled (#11951)
## Summary

This PR updates the logical line rules entry-point function to only run
the logic if any of the rules within that group is enabled.

Although this shouldn't really give any performance improvements, it's
better not to do additional work if we can. This is also consistent with
how other rules are run.

## Test Plan

`cargo insta test`
2024-06-20 16:28:53 +00:00
Micha Reiser b456051be8
[red-knot] Add tracing to Salsa queries (#11949) 2024-06-20 13:33:41 +02:00
Micha Reiser 2dfbf118d7
[red-knot] Extract `red_knot_python_semantic` crate (#11926) 2024-06-20 13:24:24 +02:00
Dhruv Manilawala ed948eaefb
Avoid moving back the lexer for triple-quoted fstring (#11939)
## Summary

This PR avoids moving back the lexer for a triple-quoted f-string during
the re-lexing phase.

The reason this is a problem is that for a triple-quoted f-string the
newlines are part of the f-string itself, specifically they'll be part
of the `FStringMiddle` token. So, if we moved the lexer back, there
would be a `Newline` token whose range would be in between an
`FStringMiddle` token. This creates a panic in downstream usage.

fixes: #11937 

## Test Plan

Add test cases and validate the snapshots.
2024-06-20 16:27:36 +05:30
Micha Reiser 22733cb7c7
red-knot(Salsa): Types without refinements (#11899) 2024-06-20 12:49:38 +02:00
Dhruv Manilawala a26bd01be2
Avoid depth counting when detecting indentation (#11947)
## Summary

This PR avoids the `depth` counter when detecting indentation from
non-logical lines because it seems to never be used. It might have been
a leftover when the logic was added originally in #11608.

## Test Plan

`cargo insta test`
2024-06-20 10:42:35 +05:30
Dhruv Manilawala b617d90651
Update `E999` to show all syntax errors (#11900)
## Summary

This PR updates the linter to show all the parse errors as diagnostics
instead of just the first one.

Note that this doesn't affect the parse error displayed as error log
message. This will be removed in a follow-up PR.

### Breaking?

I don't think this is a breaking change even though this might give more
diagnostics. The main reason is that this shouldn't affect any users
because it'll only give additional diagnostics in the case of multiple
syntax errors.

## Test Plan

Add an integration test case which would raise more than one parse
error.
2024-06-19 13:09:54 +05:30
Dhruv Manilawala cdc7c71449
Avoid consuming trailing whitespace during re-lexing (#11933)
## Summary

This PR updates the re-lexing logic to avoid consuming the trailing
whitespace and move the lexer explicitly to the last newline character
encountered while moving backwards.

Consider the following code snippet as taken from the test case
highlighted with whitespace (`.`) and newline (`\n`) characters:
```py
# There are trailing whitespace before the newline character but those whitespaces are
# part of the comment token
f"""hello {x # comment....\n
#                     ^
y = 1\n
```

The parser is at `y` when it's trying to recover from an unclosed `{`,
so it calls into the re-lexing logic which tries to move the lexer back
to the end of the previous line. But, as it consumed all whitespaces it
moved the lexer to the location marked by `^` in the above code snippet.
But, those whitespaces are part of the comment token. This means that
the range for the two tokens were overlapping which introduced the
panic.

Note that this is only a bug when there's a comment with a trailing
whitespace otherwise it's fine to move the lexer to the whitespace
character. This is because the lexer would just skip the whitespace
otherwise. Nevertheless, this PR updates the logic to move it explicitly
to the newline character in all cases.

fixes: #11929 

## Test Plan

Add test cases and update the snapshot. Make sure that it doesn't panic
on the code snippet in the linked issue.
2024-06-19 12:14:18 +05:30
Jane Lewis ff3bf583b2
`ruff server`: Add tracing setup guide to Neovim documentation (#11884)
A follow-up to [this
suggestion](https://github.com/astral-sh/ruff/pull/11747#discussion_r1634297757)
on the tracing PR.

---------

Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
2024-06-18 13:39:41 -07:00
Adrin Jalali 2e7c3454e0
ENH copyright-notice: check in the first 4096 bytes instead of 1024 (#11927)
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary
related to https://github.com/astral-sh/ruff/issues/5306

The check right now only checks in the first 1024 bytes, and that's
really not enough when there's a docstring at the beginning of a file.

A more proper fix might be needed, which might be more complex (and I
don't have the `rust` skills to implement that). But this temporary
"fix" might enable more users to use this.

Context: We want to use this rule in
https://github.com/scikit-learn/scikit-learn/ and we got blocked because
of this hardcoded rule (which TBH took us quite a while to figure out
why it was failing since it's not documented).

## Test Plan

This is already kinda tested, modified the test for the new byte number.

<!-- How was it tested? -->
2024-06-18 11:04:34 -05:00
Alex Waygood 1d73d60bd3
[red-knot]: Add a VendoredFileSystem implementation (#11863)
Co-authored-by: Micha Reiser <micha@reiser.io>
2024-06-18 15:43:39 +00:00
Micha Reiser f666d79cd7
red-knot: Symbol table (#11860) 2024-06-18 13:10:45 +00:00
Micha Reiser 26ac805e6d
red-knot: Port module resolver to salsa (#11835) 2024-06-18 12:11:58 +00:00
Micha Reiser 98b13b9844
red-knot: Add a method to resolve a file for an arbitrary `VfsPath` (#11826)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-06-18 12:03:30 +00:00
Dhruv Manilawala 13ad24b13e
Avoid syntax errors for test cases (#11923)
## Summary

This PR removes most of the syntax errors from the test cases. This
would create noise when https://github.com/astral-sh/ruff/pull/11901 is
complete. These syntax errors are also just noise for the test itself.

## Test Plan

Update the snapshots and verify that they're still the same.
2024-06-18 17:16:27 +05:30
psychedelicious 104608b2f7
Update docs for E711, E712 (#4560) (#11859) 2024-06-18 11:20:37 +01:00
Dhruv Manilawala 1e0642fac8
Use re-lexing for normal list parsing (#11871)
## Summary

This PR is a follow-up on #11845 to add the re-lexing logic for normal
list parsing.

A normal list parsing is basically parsing elements without any
separator in between i.e., there can only be trivia tokens in between
the two elements. Currently, this is only being used for parsing
**assignment statement** and **f-string elements**. Assignment
statements cannot be in a parenthesized context, but f-string can have
curly braces so this PR is specifically for them.

I don't think this is an ideal recovery but the problem is that both
lexer and parser could add an error for f-strings. If the lexer adds an
error it'll emit an `Unknown` token instead while the parser adds the
error directly. I think we'd need to move all f-string errors to be
emitted by the parser instead. This way the parser can correctly inform
the lexer that it's out of an f-string and then the lexer can pop the
current f-string context out of the stack.

## Test Plan

Add test cases, update the snapshots, and run the fuzzer.
2024-06-18 12:14:41 +05:30
Jane Lewis c53d55a483
`ruff server`: Add tracing setup guide to Helix documentation (#11883)
A follow-up to [this
suggestion](https://github.com/astral-sh/ruff/pull/11747#discussion_r1634297757)
on the tracing PR.
2024-06-18 03:41:24 +00:00
Jane Lewis ffc98522cd
`ruff server`: Defer notebook cell deletion to avoid an error message (#11864)
## Summary

Fixes https://github.com/astral-sh/ruff-vscode/issues/496.

Cells are no longer removed from the notebook index when a notebook gets
updated, but rather when `textDocument/didClose` is called for them.
This solves an issue where their premature removal from the notebook
cell index would cause their URL to be un-queryable in the
`textDocument/didClose` handler.

## Test Plan

Create and then delete a notebook cell in VS Code. No error should
appear.
2024-06-18 03:37:40 +00:00
Dhruv Manilawala 8499abfa7f
Implement re-lexing logic for better error recovery (#11845)
## Summary

This PR implements the re-lexing logic in the parser.

This logic is only applied when recovering from an error during list
parsing. The logic is as follows:
1. During list parsing, if an unexpected token is encountered and it
detects that an outer context can understand it and thus recover from
it, it invokes the re-lexing logic in the lexer
2. This logic first checks if the lexer is in a parenthesized context
and returns if it's not. Thus, the logic is a no-op if the lexer isn't
in a parenthesized context
3. It then reduces the nesting level by 1. It shouldn't reset it to 0
because otherwise the recovery from nested list parsing will be
incorrect
4. Then, it tries to find last newline character going backwards from
the current position of the lexer. This avoids any whitespaces but if it
encounters any character other than newline or whitespace, it aborts.
5. Now, if there's a newline character, then it needs to be re-lexed in
a logical context which means that the lexer needs to emit it as a
`Newline` token instead of `NonLogicalNewline`.
6. If the re-lexing gives a different token than the current one, the
token source needs to update it's token collection to remove all the
tokens which comes after the new current position.

It turns out that the list parsing isn't that happy with the results so
it requires some re-arranging such that the following two errors are
raised correctly:
1. Expected comma
2. Recovery context error

For (1), the following scenarios needs to be considered:
* Missing comma between two elements
* Half parsed element because the grammar doesn't allow it (for example,
named expressions)

For (2), the following scenarios needs to be considered:
1. If the parser is at a comma which means that there's a missing
element otherwise the comma would've been consumed by the first `eat`
call above. And, the parser doesn't take the re-lexing route on a comma
token.
2. If it's the first element and the current token is not a comma which
means that it's an invalid element.

resolves: #11640 

## Test Plan

- [x] Update existing test snapshots and validate them
- [x] Add additional test cases specific to the re-lexing logic and
validate the snapshots
- [x] Run the fuzzer on 3000+ valid inputs
- [x] Run the fuzzer on invalid inputs
- [x] Run the parser on various open source projects
- [x] Make sure the ecosystem changes are none
2024-06-17 06:47:00 +00:00
Micha Reiser 1f654ee729
Upgrade to Rust 1.79 (#11875) 2024-06-17 07:15:10 +01:00
Dhruv Manilawala f8f0053a6c
Trim trailing whitespace in server debug message (#11895) 2024-06-17 05:46:08 +00:00
github-actions[bot] e7c4d28c5e
Sync vendored typeshed stubs (#11885) 2024-06-15 02:15:19 +01:00
Dhruv Manilawala 4f49e918a9
Bump version to v0.4.9 (#11872) 2024-06-14 20:36:22 +05:30
Dhruv Manilawala d681a45b08
Make `ruff_db` a required crate for `ruff_python_semantic` (#11874)
## Summary

This PR makes the `ruff_db` a required crate for `ruff_python_semantic`.

Refer
https://github.com/astral-sh/ruff/actions/runs/9516626143/job/26233307158?pr=11872

## Test Plan

1. `maturin sdist --out dist`
2. `tar -xf dist/ruff-0.4.8.tar.gz --directory=dist/ruff-0.4.8`
3. `pip install dist/ruff-0.4.8.tar.gz` works
2024-06-14 14:43:04 +01:00
Micha Reiser c5bc368e43
[red-knot] Improve `Vfs` and `FileSystem` documentation (#11856) 2024-06-13 11:49:27 +00:00
Micha Reiser 73370fe798
Use `starts_with('/')` instead of `is_absolute` to avoid platform specific API (#11855) 2024-06-13 12:35:31 +01:00
Micha Reiser 22b6488550
red-knot: Add directory support to `MemoryFileSystem` (#11825) 2024-06-13 07:48:28 +00:00
Micha Reiser d4dd96d1f4
red-knot: `source_text`, `line_index`, and `parsed_module` queries (#11822) 2024-06-13 07:37:02 +00:00
Micha Reiser efbf7b14b5
red-knot[salsa part 2]: Setup semantic DB and Jar (#11837)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-06-13 08:00:51 +01:00
Dhruv Manilawala 9dc226be97
Add supported commands in server capabilities (#11850)
## Summary

This PR updates the server capabilities to include the commands that
Ruff supports. This is similar to how there's a list of possible code
actions supported by the server.

I noticed this when I was trying to find whether Helix supported
workspace commands or not based on Jane's comment
(https://github.com/astral-sh/ruff/pull/11831#discussion_r1634984921)
and I found the `:lsp-workspace-command` in the editor but it didn't
show up anything in the picker.

So, I looked at the implementation in Helix
(9c479e6d2d/helix-term/src/commands/typed.rs (L1372-L1384))
which made me realize that Ruff doesn't provide this in its
capabilities. Currently, this does require `ruff` to be first in the
list of language servers in the user config but that should be resolved
by https://github.com/helix-editor/helix/pull/10176. So, the following
config should work:

```toml
[[language]]
name = "python"
# Ruff should come first until https://github.com/helix-editor/helix/pull/10176 is released
language-servers = ["ruff", "pyright"]
```

## Test Plan

1. Neovim's server capabilities output should include the supported
commands:

```
  executeCommandProvider = {                                                                                                                          
    commands = { "ruff.applyFormat", "ruff.applyAutofix", "ruff.applyOrganizeImports", "ruff.printDebugInformation" },                                
    workDoneProgress = false                                                                                                                          
  },
```

2. Helix should now display the commands to pick from when
`:lsp-workspace-command` is invoked:

<img width="832" alt="Screenshot 2024-06-13 at 08 47 14"
src="https://github.com/astral-sh/ruff/assets/67177269/09048ecd-c974-4e09-ab56-9482ff3d780b">
2024-06-13 09:32:43 +05:30
Alex Waygood bcbddac21c
Fix `Display` implementation for typeshed `VERSIONS` parser (#11848) 2024-06-12 19:56:52 +00:00
Alex Waygood 4ed3aed8d3
[red-knot] Add a parser for typeshed's VERSIONS file (#11836) 2024-06-12 11:44:45 +00:00
Dhruv Manilawala 60ea72a6bc
Add list terminator kind for error recovery (#11843)
## Summary

This PR adds a new enum to determine the kind of terminator token i.e.,
is it actually terminates the list or is it used for error recovery.

This is important because the parser should take the error recovery
route in case the terminator token is used for better error recovery.
This will then try to re-lex the token if it's the case.

I haven't updated any reference to use this new enum as otherwise it'll
update the snapshots. I plan to do that in a follow-up PR so that it's
easier to reason about.

## Test plan

`cargo insta test`
2024-06-12 08:33:26 +00:00
Dhruv Manilawala a525b4be3d
Separate terminator token for f-string elements kind (#11842)
## Summary

This PR separates the terminator token for f-string elements depending
on the context. A list of f-string element can occur either in a regular
f-string or a format spec of an f-string. The terminator token is
different depending on that context.

## Test Plan

`cargo insta test` and verify the updated snapshots.
2024-06-12 13:57:35 +05:30
Micha Reiser 93973b96cb
red-knot: `VfsFile` input ingredient and a `Vfs` (#11802) 2024-06-12 07:06:15 +00:00
Dhruv Manilawala db8f2c2d9f
Use the existing `ruff_python_trivia::is_python_whitespace` function (#11844)
## Summary

This PR re-uses the `ruff_python_trivia::is_python_whitespace` in the
lexer instead of defining its own. This was mainly to avoid circular
dependency which was resolved in #11261.
2024-06-12 05:59:19 +00:00
Carl Meyer 5c0df7a150
[red-knot] add type narrowing (#11790)
## Summary

Add Constraint nodes to flow graph, and narrow types based on that (only
`is None` and `is not None` narrowing supported for now, to prototype
the structure.)

Also add simplification of zero- and one-element unions and
intersections, and flattening of intersections.

There's a lot more normalization logic needed for unions and
intersections (as is obvious from the inferred type in the added
`narrow_none` test), but this will be non-trivial and I'd rather do it
in a separate PR.

Here's a flowchart diagram for the code in the added `narrow_none` test:

![Screenshot 2024-06-07 at 2 58
00 PM](https://github.com/astral-sh/ruff/assets/61586/5152a400-739c-41ff-8bbf-3c19d16bd083)

The top branch is for the `if` expression in the initial assignment to
`x`; that `Constraint` node would only affect the type of `flag`, which
we don't care about in this test.

The second branch is for the `if` statement, with `Constraint` node
affecting the type of `x`.

## Test Plan

Added tests.
2024-06-12 04:38:50 +00:00
Jane Lewis 7d5cf1811b
`ruff server`: Improve error message when a command is run on an unavailable document (#11823)
## Summary

Fixes #11744.

We now show a distinct popup message when we fail to get a document
snapshot during command execution. This message more clearly
communicates the issue to the user, instead of a generic "ruff
encountered an error" message.

## Test Plan

Try running `Fix all auto-fixable problems` on an incompatible file (for
example: `settings.json`). You should see the following popup message:
<img width="456" alt="Screenshot 2024-06-11 at 11 47 16 AM"
src="https://github.com/astral-sh/ruff/assets/19577865/3a28e3d7-3896-4dd0-b117-f87300dd3b68">
2024-06-11 18:50:01 +00:00
Jane Lewis 4e9d771aa0
`ruff server`: Introduce the `ruff.printDebugInformation` command (#11831)
## Summary

Closes #11715.

Introduces a new command, `ruff.printDebugInformation`. This will print
useful information about the status of the server to `stderr`.

Right now, the information shown by this command includes:
* The path to the server executable
* The version of the executable
* The text encoding being used
* The number of open documents and workspaces
* A list of registered configuration files
* The capabilities of the client

## Test Plan

First, checkout and use [the corresponding `ruff-vscode`
PR](https://github.com/astral-sh/ruff-vscode/pull/495).

Running the `Print debug information` command in VS Code should show
something like the following in the Output channel:

<img width="991" alt="Screenshot 2024-06-11 at 11 41 46 AM"
src="https://github.com/astral-sh/ruff/assets/19577865/ab93c009-bb7b-4291-b057-d44fdc6f9f86">
2024-06-11 11:42:46 -07:00
Jane Lewis 507f5c1137
`ruff server`: Tracing system now respects log level and trace level, with options to log to a file (#11747)
## Summary

Fixes #10968.
Fixes #11545.

The server's tracing system has been rewritten from the ground up. The
server now has trace level and log level settings which restrict the
tracing events and spans that get logged.

* A `logLevel` setting has been added, which lets a user set the log
level. By default, it is set to `"info"`.
* A `logFile` setting has also been added, which lets the user supply an
optional file to send tracing output (it does not have to exist as a
file yet). By default, if this is unset, tracing output will be sent to
`stderr`.
* A `$/setTrace` handler has also been added, and we also set the trace
level from the initialization options. For editors without direct
support for tracing, the environment variable `RUFF_TRACE` can override
the trace level.
* Small changes have been made to how we display tracing output. We no
longer use `tracing-tree`, and instead use
`tracing_subscriber::fmt::Layer` to format output. Thread names are now
included in traces, and I've made some adjustment to thread worker names
to be more useful.

## Test Plan

In VS Code, with `ruff.trace.server` set to its default value, no logs
from Ruff should appear.

After changing `ruff.trace.server` to either `messages` or `verbose`,
you should see log messages at `info` level or higher appear in Ruff's
output:
<img width="1005" alt="Screenshot 2024-06-10 at 10 35 04 AM"
src="https://github.com/astral-sh/ruff/assets/19577865/6050d107-9815-4bd2-96d0-e86f096a57f5">

In Helix, by default, no logs from Ruff should appear.

To set the trace level in Helix, you'll need to modify your language
configuration as follows:
```toml
[language-server.ruff]
command = "/Users/jane/astral/ruff/target/debug/ruff"
args = ["server", "--preview"]
environment = { "RUFF_TRACE" = "messages" }
```

After doing this, logs of `info` level or higher should be visible in
Helix:
<img width="1216" alt="Screenshot 2024-06-10 at 10 39 26 AM"
src="https://github.com/astral-sh/ruff/assets/19577865/8ff88692-d3f7-4fd1-941e-86fb338fcdcc">

You can use `:log-open` to quickly open the Helix log file.

In Neovim, by default, no logs from Ruff should appear.

To set the trace level in Neovim, you'll need to modify your
configuration as follows:
```lua
require('lspconfig').ruff.setup {
  cmd = {"/path/to/debug/executable", "server", "--preview"},
  cmd_env = { RUFF_TRACE = "messages" }
}
```

You should see logs appear in `:LspLog` that look like the following:
<img width="1490" alt="Screenshot 2024-06-11 at 11 24 01 AM"
src="https://github.com/astral-sh/ruff/assets/19577865/576cd5fa-03cf-477a-b879-b29a9a1200ff">

You can adjust `logLevel` and `logFile` in `settings`:
```lua
require('lspconfig').ruff.setup {
  cmd = {"/path/to/debug/executable", "server", "--preview"},
  cmd_env = { RUFF_TRACE = "messages" },
  settings = {
    logLevel = "debug",
    logFile = "your/log/file/path/log.txt"
  }
}
```

The `logLevel` and `logFile` can also be set in Helix like so:
```toml
[language-server.ruff.config.settings]
logLevel = "debug"
logFile = "your/log/file/path/log.txt"
```

Even if this log file does not exist, it should now be created and
written to after running the server:

<img width="1148" alt="Screenshot 2024-06-10 at 10 43 44 AM"
src="https://github.com/astral-sh/ruff/assets/19577865/ab533cf7-d5ac-4178-97f1-e56da17450dd">
2024-06-11 11:29:47 -07:00
Charlie Marsh 08b548626a
Avoid suggesting starmap when arguments are used outside call (#11830)
## Summary

Closes https://github.com/astral-sh/ruff/issues/11810.
2024-06-10 17:10:06 -04:00
Gilles Peiffer b3b2f57d8e
[`pylint`] Fix flag name in `too-many-public-methods` (`PLR0904`) (#11809) 2024-06-09 19:44:12 -04:00
Dhruv Manilawala 549cc1e437
Build `CommentRanges` outside the parser (#11792)
## Summary

This PR updates the parser to remove building the `CommentRanges` and
instead it'll be built by the linter and the formatter when it's
required.

For the linter, it'll be built and owned by the `Indexer` while for the
formatter it'll be built from the `Tokens` struct and passed as an
argument.

## Test Plan

`cargo insta test`
2024-06-09 09:55:17 +00:00
Philipp Thiel 7509a48eab
Adapted fix to work identical to format (#10999)
## Summary

The fix for E203 now produces the same result as ruff format in cases
where a slice ends on a colon and the closing square bracket is on the
following line.

Refers to https://github.com/astral-sh/ruff/issues/10973

## Test Plan

The minimal reproduction case in the ticket was added as test case
producing no error. Additional cases with multiple spaces or a tab
before the colon where added to make sure that the rule still finds
these.
2024-06-08 19:29:18 -04:00
Alex Waygood af821ecda1
Fix `TypeVarTuple` typo in pyupgrade rule (#11806) 2024-06-08 22:47:55 +00:00
Aleksei Latyshev ccc418cc49
[`refurb`] Implement `repeated-global` (`FURB154`) (#11187)
Implement repeated_global (FURB154) lint.
See:
- https://github.com/astral-sh/ruff/issues/1348
- [original
lint](https://github.com/dosisod/refurb/blob/master/refurb/checks/builtin/simplify_global_and_nonlocal.py)

## Test Plan
cargo test
2024-06-08 20:35:40 +00:00
aditya pillai ed947792cf
Handle non-printable characters in diff view (#11687)
Co-authored-by: Micha Reiser <micha@reiser.io>
2024-06-08 06:22:03 +00:00
Charlie Marsh ee1621b2f9
Use real file path when available in `ruff server` (#11800)
## Summary

As-is, we're using the URL path for all files, leading us to use paths
like:

```
/c%3A/Users/crmar/workspace/fastapi/tests/main.py
```

This doesn't match against per-file ignores and other patterns in Ruff
configuration.

This PR modifies the LSP to use the real file path if available, and the
virtual file path if not.

Closes https://github.com/astral-sh/ruff/issues/11751.

## Test Plan

Ran the LSP on Windows. In the FastAPI repo, added:

```toml
[tool.ruff.lint.per-file-ignores]
"tests/**/*.py" = ["F401"]
```

And verified that an unused import was ignored in `tests` after this
change, but not before.
2024-06-07 22:48:53 -07:00
Micha Reiser 32ca704956
Rename `PreorderVisitor` to `SourceOrderVisitor` (#11798)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-06-07 17:01:58 +00:00
Alex Waygood 37d8de3316
[red-knot] Include vendored typeshed stubs as a zipfile in the Ruff binary (#11779)
Co-authored-by: Micha Reiser <micha@reiser.io>
Co-authored-by: Carl Meyer <carl@astral.sh>
2024-06-07 15:00:36 +00:00
Carl Meyer 4157c8635b
[red-knot] add None type (#11788)
Add type for None.
2024-06-07 08:40:22 -06:00
Dhruv Manilawala d22f3402e1
Remove `result_like` dependency (#11793)
## Summary

This PR removes the `result-like` dependency and instead implement the
required functionality. The motivation being that `noqa.is_enabled()` is
easier to read than `noqa.into()`.

For context, I was just trying to understand the syntax error workflow
and I saw these flags which were being converted via `into`. I always
find `into` confusing because you never know what's it being converted
into unless you know the type. Later realized that it's just a boolean
flag. After removing the usages from these two flags, it turns out that
the dependency is only being used in one rule so I thought to remove
that as well.

## Test Plan

`cargo insta test`
2024-06-07 11:53:22 +05:30
Embers-of-the-Fire ea27445479
[`refurb`] Fix misbehavior of `operator.itemgetter` when getter param is a tuple (#11774) 2024-06-07 03:10:52 +00:00
Carl Meyer 540d76892f
[red-knot] remove duplicate test from bad merge (#11787)
Somehow a merge of a PR that had all-green CI duplicated this test when
it merged into main, breaking the build.
2024-06-06 22:40:19 +00:00
Carl Meyer cd101c83ae
[red-knot] condense int literals (#11784)
Display `(Literal[1] | Literal[2])` as `Literal[1, 2]`, and `(Literal[1]
| Literal[2] | OtherType)` as `(Literal[1, 2] | OtherType)`.

Fixes #11782

---------

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-06-06 16:30:40 -06:00
Carl Meyer b2fc0df6db
[red-knot] flatten unions (#11783)
Flatten union types. Fixes #11781
2024-06-06 16:13:40 -06:00
Alex Waygood 93eefb1417
[red-knot] Cleanup module-resolution logic in `module.rs` (#11777) 2024-06-06 17:33:02 +01:00
Alex Waygood 303ef02f93
[red-knot] Encapsulate module resolution logic in `module.rs` (#11767) 2024-06-06 14:31:09 +00:00
Dhruv Manilawala 1b7d08c2c9
Consider `:` to terminate parenthesized with items (#11775)
## Summary

This PR is a follow-up to this discussion
(https://github.com/astral-sh/ruff/pull/11770#discussion_r1628917209)
which adds the `:` token in the terminator set for parenthesized with
items.

The main motivation is to avoid parsing too much in speculative mode.
This is evident with the following _before_ and _after_ parsed with
items list for the following code:

```py
with (item1, item2:
    foo
```

<table>
  <tr>
    <th>Before (3 items)</th>
    <th>After (2 items)</th>
  </tr>
  <tr>
    <td>
<pre>
parsed_with_items: [
    ParsedWithItem {
        item: WithItem {
            range: 6..11,
            context_expr: Name(
                ExprName {
                    range: 6..11,
                    id: "item1",
                    ctx: Load,
                },
            ),
            optional_vars: None,
        },
        is_parenthesized: false,
    },
    ParsedWithItem {
        item: WithItem {
            range: 13..18,
            context_expr: Name(
                ExprName {
                    range: 13..18,
                    id: "item2",
                    ctx: Load,
                },
            ),
            optional_vars: None,
        },
        is_parenthesized: false,
    },
    ParsedWithItem {
        item: WithItem {
            range: 24..27,
            context_expr: Name(
                ExprName {
                    range: 24..27,
                    id: "foo",
                    ctx: Load,
                },
            ),
            optional_vars: None,
        },
        is_parenthesized: false,
    },
]
</pre>
	</td>
    <td>
<pre>
parsed_with_items: [
    ParsedWithItem {
        item: WithItem {
            range: 6..11,
            context_expr: Name(
                ExprName {
                    range: 6..11,
                    id: "item1",
                    ctx: Load,
                },
            ),
            optional_vars: None,
        },
        is_parenthesized: false,
    },
    ParsedWithItem {
        item: WithItem {
            range: 13..18,
            context_expr: Name(
                ExprName {
                    range: 13..18,
                    id: "item2",
                    ctx: Load,
                },
            ),
            optional_vars: None,
        },
        is_parenthesized: false,
    },
]
</pre>
	</td>
  </tr>
</table>

## Test Plan

`cargo insta test`
2024-06-06 18:40:44 +05:30
Carl Meyer fcaa62f0d9
[red-knot] support if-expressions in type inference and CFG (#11765) 2024-06-06 04:40:44 -06:00
Embers-of-the-Fire f144edeefa
[Bug fix] Fix rule B909's panic when checking large loop blocks (#11772) 2024-06-06 12:23:28 +02:00
Dhruv Manilawala 6c1fa1d440
Use speculative parsing for with-items (#11770)
## Summary

This PR updates the with-items parsing logic to use speculative parsing
instead.

### Existing logic

First, let's understand the previous logic:
1. The parser sees `(`, it doesn't know whether it's part of a
parenthesized with items or a parenthesized expression
2. Consider it a parenthesized with items and perform a hand-rolled
speculative parsing
3. Then, verify the assumption and if it's incorrect convert the parsed
with items into an appropriate expression which becomes part of the
first with item

Here, in (3) there are lots of edge cases which we've to deal with:
1. Trailing comma with a single element should be [converted to the
expression as
is](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2140-L2153))
2. Trailing comma with multiple elements should be [converted to a tuple
expression](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2155-L2178))
3. Limit the allowed expression based on whether it's
[(1)](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2144-L2152))
or
[(2)](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2157-L2171))
4. [Consider postfix
expressions](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2181-L2200))
after (3)
5. [Consider `if`
expressions](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2203-L2208))
after (3)
6. [Consider binary
expressions](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2210-L2228))
after (3)

Consider other cases like
* [Single generator
expression](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2020-L2035))
* [Expecting a
comma](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2122-L2130))

And, this is all possible only if we allow parsing these expressions in
the [with item parsing
logic](9b2cf569b2/crates/ruff_python_parser/src/parser/statement.rs (L2287-L2334)).

### Speculative parsing

With #11457 merged, we can simplify this logic by changing the step (3)
from above to just rewind the parser back to the `(` if our assumption
(parenthesized with-items) was incorrect and then continue parsing it
considering parenthesized expression.

This also behaves a lot similar to what a PEG parser does which is to
consider the first grammar rule and if it fails consider the second
grammar rule and so on.

resolves: #11639 

## Test Plan

- [x] Verify the updated snapshots
- [x] Run the fuzzer on around 3000 valid source code (locally)
2024-06-06 08:59:56 +00:00
Max Muoto 5a5a588a72
[`pylint`] Implement `dict-iter-missing-items` (`C0206`) (#11688)
## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

This PR implements the [consider dict
items](https://pylint.pycqa.org/en/latest/user_guide/messages/convention/consider-using-dict-items.html)
rule from Pylint. Enabling this rule flags:

```python
ORCHESTRA = {
    "violin": "strings",
    "oboe": "woodwind",
    "tuba": "brass",
    "gong": "percussion",
}


for instrument in ORCHESTRA: 
    print(f"{instrument}: {ORCHESTRA[instrument]}")

for instrument in ORCHESTRA.keys(): 
    print(f"{instrument}: {ORCHESTRA[instrument]}")

for instrument in (inline_dict := {"foo": "bar"}): 
    print(f"{instrument}: {inline_dict[instrument]}")
```

For not using `items()` to extract the value out of the dict. We ignore
the case of an assignment, as you can't modify the underlying
representation with the value in the list of tuples returned.
 

## Test Plan

<!-- How was it tested? -->

`cargo test`.

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2024-06-06 00:28:01 -04:00
Carl Meyer 084e5464fb
[red-knot] support walrus expressions in type inference (#11762)
## Summary

Add support for walrus expressions, both in expression type inference
and in symbol definition type inference.

## Test Plan

Added test.
2024-06-05 15:13:10 -06:00
Carl Meyer 31f97329c0
[red-knot] refactor Definitions out of symbol table (#11761)
## Summary

Definitions are used in symbol table and in flow graph, and aren't
inherently owned by one or the other; move them into their own
submodule.

## Test Plan

Existing tests.
2024-06-05 14:35:01 -06:00
Carl Meyer b46e9e825a
[red-knot] arithmetic on int literals (#11760)
## Summary

Add support for inferring int literal types from basic arithmetic on int
literals. Just to begin showing examples of resolving more complex
expression types, and because this will be useful in testing walrus
expressions.

## Test Plan

Added test.
2024-06-05 14:10:37 -06:00
Carl Meyer 9b2cf569b2
[red-knot] rename Definition::None to Definition::Unbound (#11758)
## Summary

After looking at this a bit, I think it does make sense to have
`Unbound` as part of the `Definition` enum; if we are modeling `Unbound`
as a type (which currently we are), then every symbol implicitly starts
each scope with a "definition" as unbound, and the cleanest way to model
that is as a real `Definition`. We should be able to handle a definition
of "unbound" anywhere we handle definitions.

But the name `None` wasn't clear enough; changing the name to `Unbound`
and adding a doc comment.

Also change `[first].into_iter()` to `std::iter::once(first)`, from
post-land code review on a prior PR.

## Test Plan

Existing tests.
2024-06-05 11:32:26 -06:00
Micha Reiser 5806bc915d
Fix formatter instability for lines only consisting of zero-width characters (#11748) 2024-06-05 17:55:14 +02:00
Micha Reiser b0b4706e2d
Red-knot: Track scopes per expression (#11754) 2024-06-05 17:53:26 +02:00