## Summary
Fixes#8191 by introducing `--exit-non-zero-on-format` to `ruff format`
which pretty much does what it says on the tin.
## Test Plan
Added a new test!
---------
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
This PR reworks `TypeInferenceBuilder::infer_type_expression()` so that
we emit diagnostics when encountering a list literal in a type
expression. The only place where a list literal is allowed in a type
expression is if it appears as the first argument to `Callable[]`, and
`Callable` is already heavily special-cased in our type-expression
parsing.
In order to ensure that list literals are _always_ allowed as the
_first_ argument to `Callabler` (but never allowed as the second, third,
etc. argument), I had to do some refactoring of our type-expression
parsing for `Callable` annotations.
## Test Plan
New mdtests added, and existing ones updated
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
fixes#15048
We want to handle more types from Type::KnownInstance
## Test Plan
Add tests for each type added explicitly in the match
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
These are just cosmetic changes, but I'm separating them out into a
standalone PR to make a branch I have stacked on top of this easier to
review
## Test Plan
Existing tests all pass
Summary
--
This PR updates `check_path` in the `ruff_linter` crate to return a
`Vec<Message>` instead of a `Vec<Diagnostic>`. The main motivation for
this is to make it easier to convert semantic syntax errors directly
into `Message`s rather than `Diagnostic`s in #16106. However, this also
has the benefit of keeping the preview check on unsupported syntax
errors in `check_path`, as suggested in
https://github.com/astral-sh/ruff/pull/16429#discussion_r1974748024.
All of the interesting changes are in the first commit. The second
commit just renames variables like `diagnostics` to `messages`, and the
third commit is a tiny import fix.
I also updated the `ExpandedMessage::location` field name, which caused
a few extra commits tidying up the playground code. I thought it was
nicely symmetric with `end_location`, but I'm happy to revert that too.
Test Plan
--
Existing tests. I also tested the playground and server manually.
## Summary
Part of #15382
This PR infers the return type `lambda` expression as `Unknown`. In the
future, it would be more useful to infer the expression type considering
the surrounding context (#16696).
## Test Plan
Update existing test cases from `@todo` to the (verified) return type.
## Summary
Previously, the `name` field was on `Parameter` which required it to be
always optional regardless of the parameter kind because a
`typing.Callable` signature does not have name for the parameters. This
is the case for positional-only parameters. This wasn't enforced at the
type level which meant that downstream usages would have to unwrap on
`name` even though it's guaranteed to be present.
This commit moves the `name` field from `Parameter` to the
`ParameterKind` variants and makes it optional only for
`ParameterKind::PositionalOnly` variant while required for all other
variants.
One change that's now required is that a `Callable` form using a gradual
form for parameter types (`...`) would have a default `args` and
`kwargs` name used for variadic and keyword-variadic parameter kind
respectively. This is also the case for invalid `Callable` type forms. I
think this is fine as names are not relevant in this context but happy
to make it optional even in variadic variants.
## Test Plan
No new tests; make sure existing tests are passing.
## Summary
Add error messages for invalid nodes in type expressions
Fixes#16816
## Test Plan
Extend annotations/invalid.md to handle these invalid AST nodes error
messages
## Summary
This PR adds a playground for Red Knot
[Screencast from 2024-08-14
10-33-54.webm](https://github.com/user-attachments/assets/ae81d85f-74a3-4ba6-bb61-4a871b622f05)
Sharing does work 😆 I just forgot to start wrangler.
It supports:
* Multiple files
* Showing the AST
* Showing the tokens
* Sharing
* Persistence to local storage
Future extensions:
* Configuration support: The `pyproject.toml` would *just* be another
file.
* Showing type information on hover
## Blockers
~~Salsa uses `catch_unwind` to break cycles, which Red Knot uses
extensively when inferring types in the standard library.
However, WASM (at least `wasm32-unknown-unknown`) doesn't support
`catch_unwind` today, so the playground always crashes when the type
inference encounters a cycle.~~
~~I created a discussion in the [salsa
zulip](https://salsa.zulipchat.com/#narrow/stream/333573-salsa-3.2E0/topic/WASM.20support)
to see if it would be possible to **not** use catch unwind to break
cycles.~~
~~[Rust tracking issue for WASM catch unwind
support](https://github.com/rust-lang/rust/issues/118168)~~
~~I tried to build the WASM with the nightly compiler option but ran
into problems because wasm-bindgen doesn't support WASM-exceptions. We
could try to write the binding code by hand.~~
~~Another alternative is to use `wasm32-unknown-emscripten` but it's
rather painful to build~~
## Summary
This PR detects the use of PEP 701 f-strings before 3.12. This one
sounded difficult and ended up being pretty easy, so I think there's a
good chance I've over-simplified things. However, from experimenting in
the Python REPL and checking with [pyright], I think this is correct.
pyright actually doesn't even flag the comment case, but Python does.
I also checked pyright's implementation for
[quotes](98dc4469cc/packages/pyright-internal/src/analyzer/checker.ts (L1379-L1398))
and
[escapes](98dc4469cc/packages/pyright-internal/src/analyzer/checker.ts (L1365-L1377))
and think I've approximated how they do it.
Python's error messages also point to the simple approach of these
characters simply not being allowed:
```pycon
Python 3.11.11 (main, Feb 12 2025, 14:51:05) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f'''multiline {
... expression # comment
... }'''
File "<stdin>", line 3
}'''
^
SyntaxError: f-string expression part cannot include '#'
>>> f'''{not a line \
... continuation}'''
File "<stdin>", line 2
continuation}'''
^
SyntaxError: f-string expression part cannot include a backslash
>>> f'hello {'world'}'
File "<stdin>", line 1
f'hello {'world'}'
^^^^^
SyntaxError: f-string: expecting '}'
```
And since escapes aren't allowed, I don't think there are any tricky
cases where nested quotes or comments can sneak in.
It's also slightly annoying that the error is repeated for every nested
quote character, but that also mirrors pyright, although they highlight
the whole nested string, which is a little nicer. However, their check
is in the analysis phase, so I don't think we have such easy access to
the quoted range, at least without adding another mini visitor.
## Test Plan
New inline tests
[pyright]:
https://pyright-play.net/?pythonVersion=3.11&strict=true&code=EYQw5gBAvBAmCWBjALgCgO4gHaygRgEoAoEaCAIgBpyiiBiCLAUwGdknYIBHAVwHt2LIgDMA5AFlwSCJhwAuCAG8IoMAG1Rs2KIC6EAL6iIxosbPmLlq5foRWiEAAcmERAAsQAJxAomnltY2wuSKogA6WKIAdABWfPBYqCAE%2BuSBVqbpWVm2iHwAtvlMWMgB2ekiolUAgq4FjgA2TAAeEMieSADWCsoV5qoaqrrGDJ5MiDz%2B8ABuLqosAIREhlXlaybrmyYMXsDw7V4AnoysyAmQ5SIhwYo3d9cheADUeKlv5O%2BpQA
## Summary
For now, `property_tests.rs` has grown larger and larger, making the
file difficult to read and maintain.
Although the code has been split, the test paths and full names remain
unchanged. There are no changes affecting test execution.
## Summary
This PR simplifies `IterationError` and `ContextManagerError` so that
they no longer "remember" what type it was that was (respectively) not
iterable or not valid as a context manager. Instead, the type that was
iterated over (or was used as a context manager) is passed back in when
calling the error struct's `report_diagnostic` method.
The motivations for this are:
- It significantly simplifies the code
- It reduces the size of these types on the stack
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This PR brings an optimization.
- `get_cached_db` no longer returns a `MutexGuard`; instead, it returns
a cloned database.
### `get_cached_db`
Previously, the `MutexGuard` was held inside the property test function
(defined in the macro), which prevented multiple property tests from
running in parallel. More specifically, the program could only test one
random test case at a time, which likely caused a significant
bottleneck.
On my local machine, running:
```
QUICKCHECK_TESTS=100000 cargo test --release -p red_knot_python_semantic -- --ignored stable
```
showed about **a 75% speedup** (from \~60s to \~15s).
Fixes#16751
## Summary
Previously, unsafe fixes were counted as "fixable" in
`Printer::write_statistics`, in contrast to the behaviour in
`Printer::write_once`. This changes the behaviour to align with
`write_once`, including them only if `--unsafe-fixes` is set.
We now also reuse `Printer::write_summary` to avoid duplicating the
logic for whether or not to report if there are hidden fixes.
## Test Plan
Existing tests modified to use an unsafe-fixable rule, and new ones
added to cover the case with `--unsafe-fixes`
These should all be minor cosmetic changes. To summarize:
* In many cases, `-` was replaced with `^` for primary annotations.
This is because, previously, whether `-` or `^` was used depended
on the severity. But in the new data model, it's based on whether
the annotation is "primary" or not. We could of course change this
in whatever way we want, but I think we should roll with this for now.
* The "secondary messages" in the old API are rendered as
sub-diagnostics. This in turn results in a small change in the output
format, since previously, the secondary messages were represented as
just another snippet. We use sub-diagnostics because that's the intended
way to enforce relative ordering between messages within a diagnostic.
* The "info:" prefix used in some annotation messages has been dropped.
We could re-add this, but I think I like it better without this prefix.
I believe those 3 cover all of the snapshot changes here.
... and switch to the new one.
We do this switch by converting the old diagnostics to a
`Diagnostic`, and then rendering that.
This does not quite emit identical output. There are some
changes. They *could* be fixed to remain the same, but the
changes aren't obviously worse to me and I think the right
way to *improve* them is to move Red Knot to the new `Diagnostic`
API.
The next commit will have the snapshot changes.
In our existing diagnostics, our message is just the diagnostic
ID, and the message goes to the annotation. In reality, the
diagnostic can have its own message distinct from the optional
messages associated with an annotation.
In order to make the outputs match, we do a small tweak here:
when the main diagnostic message is empty, we drop the colon
after the diagnostic ID.
I expect that we'll want to rejigger this output format more
in the future, but for now this was a very simple change to
preserve the status quo.
When moving over to the new renderer, I noticed that it
was emitting an extra line terminator compared to the status
quo. This removes it by turning the line terminator into a
line delimiter between diagnostics.
This cleans up how we handle calling unions of types. #16568 adding a
three-level structure for callable signatures (`Signatures`,
`CallableSignature`, and `Signature`) to handle unions and overloads.
This PR updates the bindings side to mimic that structure. What used to
be called `CallOutcome` is now `Bindings`, and represents the result of
binding actual arguments against a possible union of callables.
`CallableBinding` is the result of binding a single, possibly
overloaded, callable type. `Binding` is the result of binding a single
overload.
While we're here, this also cleans up `CallError` greatly. It was
previously extracting error information from the bindings and storing it
in the error result. It is now a simple enum, carrying no data, that's
used as a status code to talk about whether the overall binding was
successful or not. We are now more consistent about walking the binding
itself to get detailed information about _how_ the binding was
unsucessful.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR fixes a bug in the check for fully static callable type where we
would skip unannotated parameter type.
## Test Plan
Add tests using the new `CallableTypeFromFunction` special form.
## Summary
This is a re-creation of https://github.com/astral-sh/ruff/pull/16764 by
@mtshiba, which I closed meaning to immediately reopen (GitHub wasn't
updating the PR with the latest pushed changes), and which GitHub will
not allow me to reopen for some reason. Pasting the summary from that PR
below:
> From https://github.com/astral-sh/ruff/pull/16641
>
> As stated in this comment
(https://github.com/astral-sh/ruff/pull/16641#discussion_r1996153702),
the current ordering implementation for intersection types is incorrect.
So, I will introduce lexicographic ordering for intersection types.
## Test Plan
One property test stabilised (tested locally with
`QUICKCHECK_TESTS=2000000 cargo test --release -p
red_knot_python_semantic -- --ignored
types::property_tests::stable::negation_reverses_subtype_order`), and
existing mdtests that previously failed now pass.
Primarily-authored-by:
[mtshiba](https://github.com/astral-sh/ruff/commits?author=mtshiba)
---------
Co-authored-by: Shunsuke Shibayama <sbym1346@gmail.com>
## Summary
A small followup to https://github.com/astral-sh/ruff/pull/16386. We now
tell the user exactly what it was about their decorator that constituted
invalid syntax on Python <3.9, and the range now highlights the specific
sub-expression that is invalid rather than highlighting the whole
decorator
## Test Plan
Inline snapshots are updated, and new ones are added.
## Summary
Stop flagging each invocation of `django.utils.safestring.mark_safe`
(also available at, `django.utils.html.mark_safe`) as an error.
Instead, allow string literals as valid uses for `mark_safe`.
Also, update the documentation, pointing at
`django.utils.html.format_html` for dynamic content generation use
cases.
Closes#16702
## Test Plan
I verified several possible uses, but string literals, are still
flagged.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
This PR adds a new `--color` CLI option that controls whether the output
should be colorized or not.
This is implements part of
https://github.com/astral-sh/ruff/issues/16727 except that it doesn't
implement the persistent configuration support as initially proposed in
the CLI document. I realized, that having this as a persistent
configuration is somewhat awkward because we may end up writing tracing
logs **before** we loaded and resolved the settings. Arguably, it's
probably fine to color the output up to that point, but it feels like a
somewhat broken experience. That's why I decided not to add the
persistent configuration option for now.
## Test Plan
I tested this change manually by running Red Knot with `--color=always`,
`--color=never`, and `--color=auto` (or no argument) and verified that:
* The diagnostics are or aren't colored
* The tracing output is or isn't colored.
---------
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
## Summary
Another salsa upgrade.
The main motivation is to stay on a recent salsa version because there
are still a lot of breaking changes happening.
The most significant changes in this update:
* Salsa no longer derives `Debug` by default. It now requires
`interned(debug)` (or similar)
* This version ships the foundation for garbage collecting interned
values. However, this comes at the cost that queries now track which
interned values they created (or read). The micro benchmarks in the
salsa repo showed a significant perf regression. Will see if this also
visible in our benchmarks.
## Test Plan
`cargo test`
Currently the red-knot LSP server emits any log messages of level `INFO`
or higher from non-red-knot crates. This makes its output quite verbose,
because Salsa emits an `INFO` level message every time it executes a
query. I use red-knot as LSP with neovim, and this spams the log file
quite a lot.
It seems like a better default to only emit `WARN` or higher messages
from non-red-knot sources.
I confirmed that this fixes the nvim LSP log spam.
## Summary
Uses the `try_call_dunder` infrastructure for augmented assignment and
fixes the logic to work for types other than `Type::Instance(…)`. This
allows us to infer the correct type here:
```py
x = (1, 2)
x += (3, 4)
reveal_type(x) # revealed: tuple[Literal[1], Literal[2], Literal[3], Literal[4]]
```
Or in this (extremely weird) scenario:
```py
class Meta(type):
def __iadd__(cls, other: int) -> str:
return ""
class C(metaclass=Meta): ...
cls = C
cls += 1
reveal_type(cls) # revealed: str
```
Union and intersection handling could also be improved here, but I made
no attempt to do so in this PR.
## Test Plan
New MD tests
## Summary
A follow-up to https://github.com/astral-sh/ruff/pull/16705 which
documents various kinds of diagnostics that can appear when assigning to
an attribute.
## Test Plan
New snapshot tests.
## Summary
This PR implements the first part of
https://github.com/astral-sh/ruff/discussions/16440. It ensures that Red
Knot's module resolver is case sensitive on all systems.
This PR combines a few approaches:
1. It uses `canonicalize` on non-case-sensitive systems to get the real
casing of a path. This works for as long as no symlinks or mapped
network drives (the windows `E:\` is mapped to `\\server\share` thingy).
This is the same as what Pyright does
2. If 1. fails, fall back to recursively list the parent directory and
test if the path's file name matches the casing exactly as listed in by
list dir. This is the same approach as CPython takes in its module
resolver. The main downside is that it requires more syscalls because,
unlike CPython, we Red Knot needs to invalidate its caches if a file
name gets renamed (CPython assumes that the folders are immutable).
It's worth noting that the file watching test that I added that renames
`lib.py` to `Lib.py` currently doesn't pass on case-insensitive systems.
Making it pass requires some more involved changes to `Files`. I plan to
work on this next. There's the argument that landing this PR on its own
isn't worth it without this issue being addressed. I think it's still a
good step in the right direction even when some of the details on how
and where the path case sensitive comparison is implemented.
## Test plan
I added multiple integration tests (including a failing one). I tested
that the `case-sensitivity` detection works as expected on Windows,
MacOS and Linux and that the fast-paths are taken accordingly.
## Summary
This PR detects unparenthesized assignment expressions used in set
literals and comprehensions and in sequence indexes. The link to the
release notes in https://github.com/astral-sh/ruff/issues/6591 just has
this entry:
> * Assignment expressions can now be used unparenthesized within set
literals and set comprehensions, as well as in sequence indexes (but not
slices).
with no other information, so hopefully the test cases I came up with
cover all of the changes. I also tested these out in the Python REPL and
they actually worked in Python 3.9 too. I'm guessing this may be another
case that was "formally made part of the language spec in Python 3.10,
but usable -- and commonly used -- in Python >=3.9" as @AlexWaygood
added to the body of #6591 for context managers. So we may want to
change the version cutoff, but I've gone along with the release notes
for now.
## Test Plan
New inline parser tests and linter CLI tests.
We don't actually hook this up to anything in this PR, but we do
go to some trouble to granularly unit test it. The unit tests caught
plenty of bugs after I initially wrote down the implementation, so they
were very much worth it.
Closes#16506
Instead of hard-coding a specific context window,
it seemed prudent to make this configurable. That
makes it easier to test different context window
sizes as well.
I am not totally convinced that this is the right
place for this configuration. I could see the context
window size being a property of `Diagnostic` instead,
since we might want to change the context window
size based not just on some end user configuration,
but perhaps also the specific diagnostic.
But for now, I think it's fine for it to live here,
and all of the rendering logic doesn't care where
it lives. So it should be relatively easy to change
in the future.
This adds a new configuration knob to diagnostic rendering that, when
enabled, will make diagnostic rendering much more terse. Specifically,
it will guarantee that each diagnostic will only use one line.
This doesn't actually hook the concise output option up to anything.
We'll do that plumbing in the next commit.
Summary
--
This is closely related to (and stacked on)
https://github.com/astral-sh/ruff/pull/16544 and detects star
annotations in function definitions.
I initially called the variant `StarExpressionInAnnotation` to mirror
`StarExpressionInIndex`, but I realized it's not really a "star
expression" in this position and renamed it. `StarAnnotation` seems in
line with the PEP.
Test Plan
--
Two new inline tests. It looked like there was pretty good existing
coverage of this syntax, so I just added simple examples to test the
version cutoff.
## Summary
Follow-up release for Ruff v0.10 that now includes the following two
changes that we intended to ship but slipped:
* Changes to how the Python version is inferred when a `target-version`
is not specified (#16319)
* `blanket-noqa` (`PGH004`): Also detect blanked file-level noqa
comments (and not just line level comments).
## Test plan
I verified that the binary built on this branch respects the
`requires-python` setting
([logs](https://www.diffchecker.com/qyJWYi6W/), left: v0.10, right:
v0.11)
## Summary
This PR includes minor improvements to binary operation inference,
specifically for tuple concatenation.
### Before
```py
reveal_type((1, 2) + (3, 4)) # revealed: @Todo(return type of decorated function)
# If TODO is ignored, the revealed type would be `tuple[1|2|3|4, ...]`
```
The `builtins.tuple` type stub defines `__add__`, but it appears to only
work for homogeneous tuples. However, I think this limitation is not
ideal for many use cases.
### After
```py
reveal_type((1, 2) + (3, 4)) # revealed: tuple[Literal[1], Literal[2], Literal[3], Literal[4]]
```
## Test Plan
### Added
- `mdtest/binary/tuples.md`
### Affected
- `mdtest/slots.md` (a test have been moved out of the `False-Negative`
block.)
## Summary
This changeset adds proper support for assignments to attributes:
```py
obj.attr = value
```
In particular, the following new features are now available:
* We previously didn't raise any errors if you tried to assign to a
non-existing attribute `attr`. This is now fixed.
* If `type(obj).attr` is a data descriptor, we now call its `__set__`
method instead of trying to assign to the load-context type of
`obj.attr`, which can be different for data descriptors.
* An initial attempt was made to support unions and intersections, as
well as possibly-unbound situations. There are some remaining TODOs in
tests, but they only affect edge cases. Having nested diagnostics would
be one way that could help solve the remaining cases, I believe.
## Follow ups
The following things are planned as follow-ups:
- Write a test suite with snapshot diagnostics for various attribute
assignment errors
- Improve the diagnostics. An easy improvement would be to highlight the
right hand side of the assignment as a secondary span (with the rhs type
as additional information). Some other ideas are mentioned in TODO
comments in this PR.
- Improve the union/intersection/possible-unboundness handling
- Add support for calling custom `__setattr__` methods (see new false
positive in the ecosystem results)
## Ecosystem changes
Some changes are related to assignments on attributes with a custom
`__setattr__` method (see above). Since we didn't notice missing
attributes at all in store context previously, these are new.
The other changes are related to properties. We previously used their
read-context type to test the assignment. That results in weird error
messages, as we often see assignments to `self.property` and then we
think that those are instance attributes *and* descriptors, leading to
union types. Now we properly look them up on the meta type, see the
decorated function, and try to overwrite it with the new value (as we
don't understand decorators yet). Long story short: the errors are still
weird, we need to understand decorators to make them go away.
## Test Plan
New Markdown tests
Summary
--
This PR reuses a slightly modified version of the
`check_tuple_unpacking` method added for detecting unpacking in `return`
and `yield` statements to detect the same issue in the iterator clause
of `for` loops.
I ran into the same issue with a bare `for x in *rest: ...` example
(invalid even on Python 3.13) and added it as a comment on
https://github.com/astral-sh/ruff/issues/16520.
I considered just making this an additional `StarTupleKind` variant as
well, but this change was in a different version of Python, so I kept it
separate.
Test Plan
--
New inline tests.
There can be semi-cyclic inheritance patterns (e.g. recursive generics)
that are not technically inheritance cycles, but that can cause us to
hit Salsa query cycles in evaluating a type's MRO. Add fixed-point
handling to these MRO-related queries so we don't panic on these cycles.
The details of what queries we hit in what order in this case will
change as we implement support for generics, but ultimately we will
probably need cycle handling for all queries that can re-enter type
inference, otherwise we are susceptible to small changes in query
execution order causing panics.
Fixes#14333
Further reduces the panicking set of seeds in #14737
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Part of #15655
Replaced statement nodes with autogenerated ones. Reused the stuff we
introduced in #16285. Nothing except for copying the nodes to new
format.
## Test Plan
Tests run without any changes. Also moved the test that checks size of
AST nodes to `generated.rs` since all of the structs that it tests are
now there.
<!-- How was it tested? -->
## Summary
This PR stabilizes the behavior introduced in
https://github.com/astral-sh/ruff/pull/15985
The new behavior improves the inference of `str.strip` calls:
* before: The rule only considered calls on string or byte literals
(`"abcd".strip`)
* now: The rule also catches calls to `strip` on object where the type
is known to be a `str` or `bytes` (e.g. `a = "abc"; a.strip("//")`)
The new behavior shipped as part of Ruff 0.9.6 on the 10th of Feb which
is a little more than a month ago.
There have been now new issues or PRs related to the new behavior.
## Summary
This PR promotes the fix improvements for `PLR1714` that were introduced
in https://github.com/astral-sh/ruff/pull/14372/ to stable.
The improvement is that the fix now proposes to use a set if all
elements are hashable:
```
foo == "bar" or foo == "baz" or foo == "qux"
```
Gets fixed to
```py
foo in {"bar", "baz", "qux"}
```
where it previously always got fixed to a tuple.
The new fix was first released in ruff 0.8.0 (Nov last year). This is
not a breaking change. The change was preview gated only to get some
extra test coverage.
There are no open issues or PRs related to this changed fix behavior.
## Summary
This PR stabilizes the behavior changes introduced by
https://github.com/astral-sh/ruff/pull/13305 that were gated behind
preview.
The change is that `__new__` methods are now no longer flagged by
`invalid-first-argument-name-for-class-method` (`N804`) but instead by
`bad-staticmethod-argument` (`PLW0211`)
> __new__ methods are technically static methods, with cls as their
first argument. However, Ruff currently classifies them as classmethod,
which causes two issues:
## Test Plan
There have been no new issues or PRs related to `N804` or `PLW0211`
since the behavior change was released in Ruff 0.9.7 (about 3 weeks
ago).
This is a somewhat recent change but I don't think it's necessary to
leave this in preview for another 2 months. The main reason why it was
in preview
is that it is breaking, not because it is a risky change.
## Summary
This PR stabilizes the fix for `PYI018` introduced in
https://github.com/astral-sh/ruff/pull/15999/ (first released with Ruff
0.9.5 early February)
There are no known issues with the fix or open PRs.
## Summary
Deprecate `S320` because defusedxml has deprecated there `lxml` module
and `lxml` has been hardened since.
flake8-bandit has removed their implementation as well
(https://github.com/PyCQA/bandit/pull/1212).
Addresses https://github.com/astral-sh/ruff/issues/13707
## Test Plan
I verified that selecting `S320` prints a warning and fails if the
preview mode is enabled.
## Summary
This PR stabilizes the fixes improvements made in
https://github.com/astral-sh/ruff/pull/15562 (released with ruff 0.9.3
in mid January).
There's no open issue or PR related to the changed fix behavior.
This is not a breaking change. The fix was only gated behind preview to
get some more test coverage before releasing.
## Summary
This PR stabilizes the behavior change introduced in
https://github.com/astral-sh/ruff/pull/15872/
The diagnostic range is now the range of the redundant `mode` argument
where it previously was the range of the entire `open` call:
Before:
```
UP015.py:2:1: UP015 [*] Unnecessary mode argument
|
1 | open("foo", "U")
2 | open("foo", "Ur")
| ^^^^^^^^^^^^^^^^^ UP015
3 | open("foo", "Ub")
4 | open("foo", "rUb")
|
= help: Remove mode argument
```
Now:
```
UP015.py:2:13: UP015 [*] Unnecessary mode argument
|
1 | open("foo", "U")
2 | open("foo", "Ur")
| ^^^^ UP015
3 | open("foo", "Ub")
4 | open("foo", "rUb")
|
= help: Remove mode argument
```
This is a breaking change because it may require moving a `noqa` comment
onto a different line, e.g if you have
```py
open(
"foo",
"Ur",
) # noqa: UP015
```
Needs to be rewritten to
```py
open(
"foo",
"Ur", # noqa: UP015
)
```
There have been now new issues or PRs since the new preview behavior was
implemented. It first was released as part of Ruff 0.9.5 on the 5th of
Feb (a little more than a month ago)
## Test Plan
I reviewed the snapshot tests
## Summary
This PR stabilizes the preview behavior introduced in
https://github.com/astral-sh/ruff/pull/15719 to recognize all symbols
named `TYPE_CHECKING` as type-checking
checks in `if TYPE_CHECKING` conditions. This ensures compatibility with
mypy and pyright.
This PR also stabilizes the new behavior that removes `if 0:` and `if
False` to be no longer considered type checking blocks.
Since then, this syntax has been removed from the typing spec and was
only used for Python modules that don't have a `typing` module
([comment](https://github.com/astral-sh/ruff/pull/15719#issuecomment-2612787793)).
The preview behavior was first released with Ruff 0.9.5 (6th of
February), which was about a month ago. There are no open issues or PRs
for the changed behavior
## Test Plan
The snapshots for `SIM108` change because `SIM108` ignored type checking
blocks but it can no
simplify `if 0` or `if False` blocks again because they're no longer
considered type checking blocks.
The changes in the `TC005` snapshot or only due to that `if 0` and `if
False` are no longer recognized as type checking blocks
<!-- How was it tested? -->
## Summary
This PR stabilizes the preview behavior for `invalid-argument-name`
(`N803`)
to ignore argument names of functions decorated with `typing.override`
because
these methods are *out of the authors* control.
This behavior was introduced in
https://github.com/astral-sh/ruff/pull/15954
and released as part of Ruff 0.9.5 (6th of February).
There have been no new issues or PRs since this behavior change
(preview) was introduced.
## Summary
This PR stabilizes the preview behavior introduced in
https://github.com/astral-sh/ruff/pull/15905
The behavior change is that the rule now also recognizes `type(expr) is
type(None)` comparisons where `expr` isn't a name expression.
For example, the rule now detects `type(a.b) is type(None)` and suggests
rewriting the comparison to `a.b is None`.
The new behavior was introduced with Ruff 0.9.5 (6th of February), about
a month ago. There are no open issues or PRs related to this rule (or
behavior change).
## Summary
This PR stabilizes the new behavior introduced in
https://github.com/astral-sh/ruff/pull/14512 to also detect defalut
value arguemnts to `os.environ.get` that have an invalid type (not
`str`).
There's an upstream issue for this behavior change
https://github.com/pylint-dev/pylint/issues/10092 that was accepted and
a PR, but it hasn't been merged yet.
This behavior change was first shipped with Ruff 0.8.1 (Nov 22).
There has only be one PR since the new behavior was introduced but it
was unrelated to the scope increase
(https://github.com/astral-sh/ruff/pull/14841).
## Summary
This PR stabilizes the behavior change introduced in
https://github.com/astral-sh/ruff/pull/15542 to allow
for statements with an empty body in `pytest.raises` and `pytest.warns`
with statements.
This raised an error before but is now allowed:
```py
with pytest.raises(KeyError, match='unknown'):
async for _ in gpt.generate(gpt_request):
pass
```
The same applies to
```py
with pytest.raises(KeyError, match='unknown'):
async for _ in gpt.generate(gpt_request):
...
```
There have been now new issues or PRs related to PT012 or PT031 since
this behavior change was introduced in ruff 0.9.3 (January 23rd).
## Summary
This PR deprecates UP038. Using PEP 604 syntax in `isinstance` and
`issubclass` calls isn't a recommended pattern (or community agreed best
practice)
and it negatively impacts performance.
Resolves https://github.com/astral-sh/ruff/issues/7871
## Test Plan
I tested that selecting `UP038` results in a warning in no-preview mode
and an error in preview mode
Summary
--
Stabilizes TC006. The test was already in the right place.
Test Plan
--
No open issues or PRs. The last related [issue] was closed on
2025-02-09.
[issue]: https://github.com/astral-sh/ruff/issues/16037
Summary
--
Stabilizes S704, which is also being recoded from RUF035 in 0.10.
Test Plan
--
Existing tests with `PreviewMode` removed from the settings.
There was one issue closed on 2024-12-20 calling the rule noisy and
asking for a config option, but the option was added and then there were
no more issues or PRs.
Summary
--
Stabilizes DTZ901, renames the rule function to match the rule name,
removes the `preview_rules` test, and handles some nits in the docs
(mention `min` first to match the rule name too).
Test Plan
--
1 closed issue on 2024-11-12, 4 days after the rule was added. No issues
since
## Summary
Follow-up to #16677.
This change converts all unit tests (69 of them) in `noqa.rs` to use
inline snapshots instead. It extends the file by more than 1000 lines,
but the tests are now much easier to read and reason about.
## Test Plan
`cargo insta test`.
Summary
--
Stabilizes RUF041. The tests are already in the right place, and the
docs look good.
Test Plan
--
0 issues, 1 [PR] fixing nested literals and unions the day after the
rule was added. No changes since then
I wonder if the fix in that PR could be relevant for
https://github.com/astral-sh/ruff/pull/16639, where I noticed a
potential issue with `Union`. It could be unrelated, though.
[PR]: https://github.com/astral-sh/ruff/pull/14641
Summary
--
Stabilizes PTH210. Tests and docs looked good.
Test Plan
--
Mentioned in 1 open issue around Python 3.14 support (`"."` becomes a
valid suffix in 3.14). Otherwise no issues or PRs since 2024-12-12, 6
days after the rule was added.
## Summary
Follow-up to #16659.
This change adds tests for these three cases, which are (also) not
covered by existing tests:
* `# noqa: A` (lone incomplete code)
* `# noqa: A123, B` (complete codes, last one incomplete)
* `# noqa: A123B` (squashed codes, last one incomplete)
Summary
--
Stabilizes RUF051. The tests and docs looked good.
Test Plan
--
1 closed documentation issue from 4 days after the rule was added and 1
typo fix from the same day it was added, but no other issues or PRs.
Summary
--
Stabilizes LOG015. The tests and docs looked good.
Test Plan
--
1 closed documentation issue from 4 days after the rule was added, but
no other issues or PRs.
Summary
--
Stabilizes RUF048 and moves its test to the right place. The docs look
good.
Test Plan
--
0 closed or open issues. There was 1 [PR] related to an extension to the
rule, but it was closed without comment.
[PR]: https://github.com/astral-sh/ruff/pull/14701
Summary
--
Stabilizes RUF046 and moves its test to the right place. The docs look
good.
Test Plan
--
2 closed newline/whitespace issues from early January and 1 closed issue
about really being multiple rules, but otherwise no recent issues or
PRs.
# Summary
The goal of this PR is to address various issues around parsing
suppression comments by
1. Unifying the logic used to parse in-line (`# noqa`) and file-level
(`# ruff: noqa`) noqa comments
2. Recovering from certain errors and surfacing warnings in these cases
Closes#15682
Supersedes #12811
Addresses
https://github.com/astral-sh/ruff/pull/14229#discussion_r1835481018
Related: #14229 , #12809
## Summary
This PR changes the default value of
`lint.flake8-builtins.builtins-strict-checking` added in
https://github.com/astral-sh/ruff/pull/15951 from `true` to `false`.
This also allows simplifying the default option logic and removes the
dependence on preview mode.
https://github.com/astral-sh/ruff/issues/15399 was already closed by
#15951, but this change will finalize the behavior mentioned in
https://github.com/astral-sh/ruff/issues/15399#issuecomment-2587017147.
As an example, strict checking flags modules based on their last
component, so `utils/logging.py` triggers A005. Non-strict checking
checks the path to the module, so `utils/logging.py` is allowed (this is
the example and desired behavior from #15399 exactly) but a top-level
`logging.py` or `logging/__init__.py` is still disallowed.
## Test Plan
Existing tests from #15951 and #16006, with the snapshot updated in
`a005_module_shadowing_strict_default` to reflect the new default.
Summary
--
Stabilizes UP044, renames the module to match the rule name, and removes
the `PreviewMode` from the test settings.
Test Plan
--
2 closed issues in November, just after the rule was added, otherwise no
issues
Summary
--
Stabilizes SIM905 and adds a small addition to the docs. The test was
already in the right place.
Test Plan
--
No issues except 2 recent, general issues about whitespace
normalization.
## Summary
This PR stabilizes several preview-only behaviours for
`custom-typevar-for-self` (`PYI019`). Namely:
- A new, more accurate technique is now employed for detecting custom
TypeVars that are replaceable with `Self`. See
https://github.com/astral-sh/ruff/pull/15888 for details.
- The range of the diagnostic is now the full function header rather
than just the return annotation. (Previously, the rule only applied to
methods with return annotations, but this is no longer true due to the
changes in the first bullet point.)
- The fix is now available even when preview mode is not enabled.
## Test Plan
- Existing snapshots that do not have preview mode enabled are updated
- Preview-specific snapshots are removed
- I'll check the ecosystem report on this PR to verify everything's as
expected
Summary
--
Stabilizes PLC1802. The tests were already in the right place, and I
just tidied the docs a little bit.
Test Plan
--
1 issue closed 4 days after the rule was added, no other issues
Summary
--
Stabilizes PLW1507. The tests were already in the right place, and I
just tidied the docs a little bit.
Test Plan
--
1 issue from 2 weeks ago but just suggesting to mark the fix unsafe. The
shallow vs deep copy *does* change the program behavior, just usually in
a preferable way.
## Summary
Stabilizes FAST003, completing the group with FAST001 and FAST002.
## Test Plan
Last bug fix (false positive) was fixed on 2025-01-13, almost 2 months
ago.
The test case was already in the right place.
## Summary
Stabilizes C420 for the 0.10 release.
## Test Plan
No open issues or PRs (except a general issue about [string
normalization](https://github.com/astral-sh/ruff/issues/16579)). The
last (and only) false-negative bug fix was over a month ago.
The tests for this rule were already not on the `preview_rules` test, so
I just changed the `RuleGroup`. The documentation looked okay to me.
## Summary
Resolves#15368.
The following options have been renamed:
* `builtins-allowed-modules` → `allowed-modules`
* `builtins-ignorelist` → `ignorelist`
* `builtins-strict-checking` → `strict-checking`
To preserve compatibility, the old names are kept as Serde aliases.
## Test Plan
`cargo nextest run` and `cargo insta test`.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
`RUF035` has been backported into bandit as `S704` in this
[PR](https://github.com/PyCQA/bandit/pull/1225)
This moves the rule and its corresponding setting to the `flake8-bandit`
category
## Test Plan
`cargo nextest run`
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
For context, the initial implementation started out by sending a log
notification to the client to include this information in the client
channel. This is a bit ineffective because it doesn't allow the client
to display this information in a more obvious way. In addition to that,
it isn't obvious from a users perspective as to where the information is
being printed unless they actually open the output channel.
The change was to actually return this formatted string that contains
the information and let the client handle how it should display this
information. For example, in the Ruff VS Code extension we open a split
window and show this information which is similar to what rust-analyzer
does.
The notification request was kept as a precaution in case there are
users who are actually utilizing this way. If they exists, it should a
minority as it requires the user to actually dive into the code to
understand how to hook into this notification. With 0.10, we're removing
the old way as it only clobbers the output channel with a long message.
fixes: #16225
## Test Plan
Tested it out locally that the information is not being logged to the
output channel of VS Code.
## Summary
This PR stabilizies the fix for
https://github.com/astral-sh/ruff/issues/14001
We try to only make breaking formatting changes once a year. However,
the plan was to release this fix as part of Ruff 0.9 but I somehow
missed it when promoting all other formatter changes.
I think it's worth making an exception here considering that this is a
bug fix, it improves readability, and it should be rare
(very few files in a single project). Our version policy explicitly
allows breaking formatter changes in any minor release and the idea of
only making breaking formatter changes once a year is mainly to avoid
multiple releases throughout the year that introduce large formatter
changes
Closes https://github.com/astral-sh/ruff/issues/14001
## Test Plan
Updated snapshot
## Summary
Since Ruff changed to GitHub releases, tags are no longer annotated and
`git describe` no longer picks them up. Instead, it's necessary to also
search lightweight tags.
This changes fixes the `version` command to give more accurate
`last_tag`/`commits_since_last_tag` information. This only affects
development builds, as this information is not present in releases.
## Test Plan
Testing is a little tricky because this information changes on every
commit. Running manually on current `main` and my branch:
`main`:
```
# cargo run --bin ruff -- version --output-format=text
ruff 0.9.10+2547 (dd2313ab0 2025-03-12)
# cargo run --bin ruff -- version --output-format=json
{
"version": "0.9.10",
"commit_info": {
"short_commit_hash": "dd2313ab0",
"commit_hash": "dd2313ab0faea90abf66a75f1b5c388e728d9d0a",
"commit_date": "2025-03-12",
"last_tag": "v0.4.10",
"commits_since_last_tag": 2547
}
}
```
This PR:
```
# cargo run --bin ruff -- version --output-format=text
ruff 0.9.10+46 (11f39f616 2025-03-12)
# cargo run --bin ruff -- version --output-format=json
{
"version": "0.9.10",
"commit_info": {
"short_commit_hash": "11f39f616",
"commit_hash": "11f39f6166c3d7a521725b938a166659f64abb59",
"commit_date": "2025-03-12",
"last_tag": "0.9.10",
"commits_since_last_tag": 46
}
}
```
## Summary
This PR adds a new `CallableTypeFromFunction` special form to allow
extracting the abstract signature of a function literal i.e., convert a
`Type::Function` into a `Type::Callable` (`CallableType::General`).
This is done to support testing the `is_gradual_equivalent_to` type
relation specifically the case we want to make sure that a function that
has parameters with no annotations and does not have a return type
annotation is gradual equivalent to `Callable[[Any, Any, ...], Any]`
where the number of parameters should match between the function literal
and callable type.
Refer
https://github.com/astral-sh/ruff/pull/16634#discussion_r1989976692
### Bikeshedding
The name `CallableTypeFromFunction` is a bit too verbose. A possibly
alternative from Carl is `CallableTypeOf` but that would be similar to
`TypeOf` albeit with a limitation that the former only accepts function
literal types and errors on other types.
Some other alternatives:
* `FunctionSignature`
* `SignatureOf` (similar issues as `TypeOf`?)
* ...
## Test Plan
Update `type_api.md` with a new section that tests this special form,
both invalid and valid forms.
## Summary
Add support for calling custom `__getattr__` methods in case an
attribute is not otherwise found. This allows us to get rid of many
ecosystem false positives where we previously emitted errors when
accessing attributes on `argparse.Namespace`.
closes#16614
## Test Plan
* New Markdown tests
* Observed expected ecosystem changes (the changes for `arrow` also look
fine, since the `Arrow` class has a custom [`__getattr__`
here](1d70d00919/arrow/arrow.py (L802-L815)))
Pulls in the latest Salsa main branch, which supports fixpoint
iteration, and uses it to handle all query cycles.
With this, we no longer need to skip any corpus files to avoid panics.
Latest perf results show a 6% incremental and 1% cold-check regression.
This is not a "no cycles" regression, as tomllib and typeshed do trigger
some definition cycles (previously handled by our old
`infer_definition_types` fallback to `Unknown`). We don't currently have
a benchmark we can use to measure the pure no-cycles regression, though
I expect there would still be some regression; the fixpoint iteration
feature in Salsa does add some overhead even for non-cyclic queries.
I think this regression is within the reasonable range for this feature.
We can do further optimization work later, but I don't think it's the
top priority right now. So going ahead and acknowledging the regression
on CodSpeed.
Mypy primer is happy, so this doesn't regress anything on our
currently-checked projects. I expect it probably unlocks adding a number
of new projects to our ecosystem check that previously would have
panicked.
Fixes#13792Fixes#14672
## Summary
Implements attribute access on intersection types, which didn't
previously work. For example:
```py
from typing import Any
class P: ...
class Q: ...
class A:
x: P = P()
class B:
x: Any = Q()
def _(obj: A):
if isinstance(obj, B):
reveal_type(obj.x) # revealed: P & Any
```
Refers to [this comment].
[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985040363
## Test Plan
New Markdown tests
## Summary
Background - as a follow up to #16611 I noticed that there's a lot of
code duplicated between the `is_assignable_to` and `is_subtype_of`
functions and considered trying to merge them.
[A subtype and an assignable type are pretty much the
same](https://typing.python.org/en/latest/spec/concepts.html#the-assignable-to-or-consistent-subtyping-relation),
except that subtypes are by definition fully static, so I think we can
replace the whole of `is_subtype_of` with:
```
if !self.is_fully_static(db) || !target.is_fully_static(db) {
return false;
}
return self.is_assignable_to(target)
```
if we move all of the logic to is_assignable_to and delete duplicate
code. Then we can discuss if it even makes sense to have a separate
is_subtype_of function (I think the answer is yes since it's used by a
bunch of other places, but we may be able to basically rip out the
concept).
Anyways while playing with combining the functions I noticed is that the
handling of Intersections in `is_subtype_of` has a special case for two
intersections, which I didn't include in the last PR - rather I first
handled right hand intersections before left hand, which should properly
handle double intersections (hand-wavy explanation I can justify if
needed - (A & B & C) is assignable to (A & B) because the left is
assignable to both A and B, but none of A, B, or C is assignable to (A &
B)).
I took a look at what breaks if I remove the handling for double
intersections, and the reason it is needed is because is_disjoint does
not properly handle intersections with negative conditions (so instead
`is_subtype_of` basically implements the check correctly).
This PR adds support to is_disjoint for properly checking negative
branches, which also lets us simplify `is_subtype_of`, bringing it in
line with `is_assignable_to`
## Test Plan
Added a bunch of tests, most of which failed before this fix
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This is a pure restructuring of the `attributes.md` and
`descriptor_protocol.md` test suites. They have grown organically and I
didn't want to make major structural changes in my recent PR to keep the
diff clean.
## Summary
A follow up to address [this comment]:
> Similarly here, it might be a little more performant to have a single
`Type::instance()` branch with an inner match over `class.known()`
rather than having multiple branches with `if class.is_known()` guards
[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985159037
## Summary
Properly handle binary operator inference for union types.
This fixes a bug I noticed while looking at ecosystem results. The MRE
version of it is this:
```py
def sub(x: float, y: float):
# Red Knot: Operator `-` is unsupported between objects of type `int | float` and `int | float`
return x - y
```
## Test Plan
- New Markdown tests.
- Expected diff in the ecosystem checks
## Summary
Part of #15382
This PR adds the check for whether a callable type is fully static or
not.
A callable type is fully static if all of the parameter types are fully
static _and_ the return type is fully static _and_ if it does not use
the gradual form (`...`) for its parameters.
## Test Plan
Update `is_fully_static.md` with callable types.
It seems that currently this test is grouped into either fully static or
not, I think it would be useful to split them up in groups like
callable, etc. I intentionally avoided that in this PR but I'll put up a
PR for an appropriate split.
Note: I've an explicit goal of updating the property tests with the new
callable types once all relations are implemented.
## Summary
This PR closes#16248.
If the return type of the function isn't assignable to the one
specified, an `invalid-return-type` error occurs.
I thought it would be better to report this as a different kind of error
than the `invalid-assignment` error, so I defined this as a new error.
## Test Plan
All type inconsistencies in the test cases have been replaced with
appropriate ones.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
This updates the `Signature` and `CallBinding` machinery to support
multiple overloads for a callable. This is currently only used for
`KnownFunction`s that we special-case in our type inference code. It
does **_not_** yet update the semantic index builder to handle
`@overload` decorators and construct a multi-signature `Overloads`
instance for real Python functions.
While I was here, I updated many of the `try_call` special cases to use
signatures (possibly overloaded ones now) and `bind_call` to check
parameter lists. We still need some of the mutator methods on
`OverloadBinding` for the special cases where we need to update return
types based on some Rust code.
## Summary
One of the motivations in https://github.com/astral-sh/ruff/pull/16428
for panicking when the `test` or `debug_assertions` features are enabled
and a lookup of a `KnownClass` fails is that we've had some latent bugs
in our code where certain variants have been silently falling back to
`Unknown` in every typeshed lookup without us realising. But that in
itself isn't a great motivation for panicking in
`KnownClass::to_instance()`, since we can fairly easily add some tests
that assert that we don't unexpectedly fallback to `Unknown` for any
`KnownClass` variant. This PR adds those tests.
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This mostly fixes#14899
My motivation was similar to the last comment by @sharkdp there. I ran
red_knot on a codebase and the most common error was patterns like this
failing:
```
def foo(x: str): ...
x: Any = ...
if isinstance(x, str):
foo(x) # Object of type `Any & str` cannot be assigned to parameter 1 (`x`) of function `foo`; expected type `str`
```
The desired behavior is pretty much to ignore Any/Unknown when resolving
intersection assignability - `Any & str` should be assignable to `str`,
and `str` should be assignable to `str & Any`
The fix is actually very similar to the existing code in
`is_subtype_of`, we need to correctly handle intersections on either
side, while being careful to handle dynamic types as desired.
This does not fix the second test case from that issue:
```
static_assert(is_assignable_to(Intersection[Unrelated, Any], Not[tuple[Unrelated, Any]]))
```
but that's misleading because the root cause there has nothing to do
with gradual types. I added a simpler test case that also fails:
```
static_assert(is_assignable_to(Unrelated, Not[tuple[Unrelated]]))
```
This is because we don't determine that Unrelated does not subclass from
tuple so we can't rule out this relation. If that logic is improved then
this fix should also handle the case of the intersection
## Test Plan
Added a bunch of is_assignable_to tests, most of which failed before
this fix.
## Summary
Part of https://github.com/astral-sh/ruff/issues/15382
This PR adds support for inferring the `lambda` expression and return
the `CallableType`.
Currently, this is only limited to inferring the parameters and a todo
type for the return type.
For posterity, I tried using the `file_expression_type` to infer the
return type of lambda but it would always lead to cycle. The main reason
is that in `infer_parameter_definition`, the default expression is being
inferred using `file_expression_type`, which is correct, but it then
Take the following source code as an example:
```py
lambda x=1: x
```
Here's how the code will flow:
* `infer_scope_types` for the global scope
* `infer_lambda_expression`
* `infer_expression` for the default value `1`
* `file_expression_type` for the return type using the body expression.
This is because the body creates it's own scope
* `infer_scope_types` (lambda body scope)
* `infer_name_load` for the symbol `x` whose visible binding is the
lambda parameter `x`
* `infer_parameter_definition` for parameter `x`
* `file_expression_type` for the default value `1`
* `infer_scope_types` for the global scope because of the default
expression
This will then reach to `infer_definition` for the parameter `x` again
which then creates the cycle.
## Test Plan
Add tests around `lambda` expression inference.
## Summary
We currently fail to add the stubs for the `venv` stdlib module because
there is a `venv/` ignore pattern in the top-level `.gitignore` file.
## Test Plan
Ran the typeshed sync workflow manually once to see if the `venv/`
folder is now correctly added.
## Summary
Theoretically this should be slightly more performant, since the
`class.is_known()` calls each do a separate Salsa lookup, which we can
avoid if we do a single `match` on the value of `class.known()`. It also
ends up being two lines less code overall!
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
Fixes#16566, fixes#16575
The semantics of `Type::class_member` changed in
https://github.com/astral-sh/ruff/pull/16416, but the property-test
infrastructure was not updated. That means that the property tests were
panicking on the second `expect_type` call here:
0361021863/crates/red_knot_python_semantic/src/types/property_tests.rs (L151-L158)
With the somewhat unhelpful message:
```
Expected a (possibly unbound) type, not an unbound symbol
```
Applying this patch, and then running `QUICKCHECK_TESTS=1000000 cargo
test --release -p red_knot_python_semantic -- --ignored
types::property_tests::stable::equivalent_to_is_reflexive` showed
clearly that it was no longer able to find _any_ methods on _any_
classes due to the change in semantics of `Type::class_member`:
```diff
--- a/crates/red_knot_python_semantic/src/types/property_tests.rs
+++ b/crates/red_knot_python_semantic/src/types/property_tests.rs
@@ -27,7 +27,7 @@
use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
use crate::db::tests::{setup_db, TestDb};
-use crate::symbol::{builtins_symbol, known_module_symbol};
+use crate::symbol::{builtins_symbol, known_module_symbol, Symbol};
use crate::types::{
BoundMethodType, CallableType, IntersectionBuilder, KnownClass, KnownInstanceType,
SubclassOfType, TupleType, Type, UnionType,
@@ -150,10 +150,11 @@ impl Ty {
Ty::BuiltinsFunction(name) => builtins_symbol(db, name).symbol.expect_type(),
Ty::BuiltinsBoundMethod { class, method } => {
let builtins_class = builtins_symbol(db, class).symbol.expect_type();
- let function = builtins_class
- .class_member(db, method.into())
- .symbol
- .expect_type();
+ let Symbol::Type(function, ..) =
+ builtins_class.class_member(db, method.into()).symbol
+ else {
+ panic!("no method `{method}` on class `{class}`");
+ };
create_bound_method(db, function, builtins_class)
}
```
This PR updates the property-test infrastructure to use `Type::member`
rather than `Type::class_member`.
## Test Plan
- Ran `QUICKCHECK_TESTS=1000000 cargo test --release -p
red_knot_python_semantic -- --ignored types::property_tests::stable`
successfully
- Checked that there were no remaining uses of `Type::class_member` in
`property_tests.rs`
## Summary
Fixes a small nit of mine -- we are currently inconsistent in our
spelling between "metaclass" and "meta class", and between "meta type"
and "meta-type". This PR means that we consistently use "metaclass" and
"meta-type".
## Test Plan
`uvx pre-commit run -a`
## Summary
Part of https://github.com/astral-sh/ruff/issues/15382
This PR implements a general callable type that wraps around a
`Signature` and it uses that new type to represent `typing.Callable`.
It also implements `Display` support for `Callable`. The format is as:
```
([<arg name>][: <arg type>][ = <default type>], ...) -> <return type>
```
The `/` and `*` separators are added at the correct boundary for
positional-only and keyword-only parameters. Now, as `typing.Callable`
only has positional-only parameters, the rendered signature would be:
```py
Callable[[int, str], None]
# (int, str, /) -> None
```
The `/` separator represents that all the arguments are positional-only.
The relationship methods that check assignability, subtype relationship,
etc. are not yet implemented and will be done so as a follow-up.
## Test Plan
Add test cases for display support for `Signature` and various mdtest
for `typing.Callable`.
## Summary
Resolves#16365
Add support for unpacking `with` statement targets.
## Test Plan
Added some test cases, alike the ones added by #15058.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
* Attributes/method are now properly looked up on metaclasses, when
called on class objects
* We properly distinguish between data descriptors and non-data
descriptors (but we do not yet support them in store-context, i.e.
`obj.data_descr = …`)
* The descriptor protocol is now implemented in a single unified place
for instances, classes and dunder-calls. Unions and possibly-unbound
symbols are supported in all possible stages of the process by creating
union types as results.
* In general, the handling of "possibly-unbound" symbols has been
improved in a lot of places: meta-class attributes, attributes,
descriptors with possibly-unbound `__get__` methods, instance
attributes, …
* We keep track of type qualifiers in a lot more places. I anticipate
that this will be useful if we import e.g. `Final` symbols from other
modules (see relevant change to typing spec:
https://github.com/python/typing/pull/1937).
* Detection and special-casing of the `typing.Protocol` special form in
order to avoid lots of changes in the test suite due to new `@Todo`
types when looking up attributes on builtin types which have `Protocol`
in their MRO. We previously
looked up attributes in a wrong way, which is why this didn't come up
before.
closes#16367closes#15966
## Context
The way attribute lookup in `Type::member` worked before was simply
wrong (mostly my own fault). The whole instance-attribute lookup should
probably never have been integrated into `Type::member`. And the
`Type::static_member` function that I introduced in my last descriptor
PR was the wrong abstraction. It's kind of fascinating how far this
approach took us, but I am pretty confident that the new approach
proposed here is what we need to model this correctly.
There are three key pieces that are required to implement attribute
lookups:
- **`Type::class_member`**/**`Type::find_in_mro`**: The
`Type::find_in_mro` method that can look up attributes on class bodies
(and corresponding bases). This is a partial function on types, as it
can not be called on instance types like`Type::Instance(…)` or
`Type::IntLiteral(…)`. For this reason, we usually call it through
`Type::class_member`, which is essentially just
`type.to_meta_type().find_in_mro(…)` plus union/intersection handling.
- **`Type::instance_member`**: This new function is basically the
type-level equivalent to `obj.__dict__[name]` when called on
`Type::Instance(…)`. We use this to discover instance attributes such as
those that we see as declarations on class bodies or as (annotated)
assignments to `self.attr` in methods of a class.
- The implementation of the descriptor protocol. It works slightly
different for instances and for class objects, but it can be described
by the general framework:
- Call `type.class_member("attribute")` to look up "attribute" in the
MRO of the meta type of `type`. Call the resulting `Symbol` `meta_attr`
(even if it's unbound).
- Use `meta_attr.class_member("__get__")` to look up `__get__` on the
*meta type* of `meta_attr`. Call it with `__get__(meta_attr, self,
self.to_meta_type())`. If this fails (either the lookup or the call),
just proceed with `meta_attr`. Otherwise, replace `meta_attr` in the
following with the return type of `__get__`. In this step, we also probe
if a `__set__` or `__delete__` method exists and store it in
`meta_attr_kind` (can be either "data descriptor" or "normal attribute
or non-data descriptor").
- Compute a `fallback` type.
- For instances, we use `self.instance_member("attribute")`
- For class objects, we use `class_attr =
self.find_in_mro("attribute")`, and then try to invoke the descriptor
protocol on `class_attr`, i.e. we look up `__get__` on the meta type of
`class_attr` and call it with `__get__(class_attr, None, self)`. This
additional invocation of the descriptor protocol on the fallback type is
one major asymmetry in the otherwise universal descriptor protocol
implementation.
- Finally, we look at `meta_attr`, `meta_attr_kind` and `fallback`, and
handle various cases of (possible) unboundness of these symbols.
- If `meta_attr` is bound and a data descriptor, just return `meta_attr`
- If `meta_attr` is not a data descriptor, and `fallback` is bound, just
return `fallback`
- If `meta_attr` is not a data descriptor, and `fallback` is unbound,
return `meta_attr`
- Return unions of these three possibilities for partially-bound
symbols.
This allows us to handle class objects and instances within the same
framework. There is a minor additional detail where for instances, we do
not allow the fallback type (the instance attribute) to completely
shadow the non-data descriptor. We do this because we (currently) don't
want to pretend that we can statically infer that an instance attribute
is always set.
Dunder method calls can also be embedded into this framework. The only
thing that changes is that *there is no fallback type*. If a dunder
method is called on an instance, we do not fall back to instance
variables. If a dunder method is called on a class object, we only look
it up on the meta class, never on the class itself.
## Test Plan
New Markdown tests.
## Summary
This PR closes#15199.
The change I just made is to set all variables to type `Unknown` if
unpacking fails, but in some cases this may be excessive.
For example:
```py
a, b, c = "ab"
reveal_type(a) # Unknown, but it would be reasonable to think of it as LiteralString
reveal_type(c) # Unknown
```
```py
# Failed to unpack before the starred expression
(a, b, *c, d, e) = (1,)
reveal_type(a) # Unknown
reveal_type(b) # Unknown
...
# Failed to unpack after the starred expression
(a, b, *c, d, e) = (1, 2, 3)
reveal_type(a) # Unknown, but should it be Literal[1]?
reveal_type(b) # Unknown, but should it be Literal[2]?
reveal_type(c) # Todo
reveal_type(d) # Unknown
reveal_type(e) # Unknown
```
I will modify it if you think it would be better to make it a different
type than just `Unknown`.
## Test Plan
I have made appropriate modifications to the test cases affected by this
change, and also added some more test cases.
## Summary
This should give us better coverage for the unsupported syntax error
features and
increases our confidence that the formatter doesn't accidentially
introduce new unsupported
syntax errors.
A feature like this would have been very useful when working on f-string
formatting
where it took a lot of iteration to find all Python 3.11 or older
incompatibilities.
## Test Plan
I applied my changes on top of
https://github.com/astral-sh/ruff/pull/16523 and
removed the target version check in the with-statement formatting code.
As expected,
the integration tests now failed
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
If an mdtest fails, the error output will include an example command
that can be run to re-run just the failing test, e.g
```
To rerun this specific test, set the environment variable: MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute"
MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__with_sync
```
This is very helpful, but because we're printing the envvar value
surrounded in double-quotes, the bits between backticks in this example
get interpreted as a shell interpolation. When running this in zsh, for
example, I see
```console
❯ MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__with_sync
zsh: command not found: __exit__
Compiling red_knot_python_semantic v0.0.0 (/home/ericmarkmartin/Development/ruff/crates/red_knot_python_semantic)
Compiling red_knot_test v0.0.0 (/home/ericmarkmartin/Development/ruff/crates/red_knot_test)
Finished `test` profile [unoptimized + debuginfo] target(s) in 6.09s
Running tests/mdtest.rs (target/debug/deps/mdtest-149b8f9d937e36bc)
running 1 test
test mdtest__with_sync ... ok
```
[^1]
This is a minor annoyance which we can solve by using single-quotes
instead of double-quotes for this string. To do so safely, we also
escape single-quotes possibly contained within the string.
There is a [shell-quote](https://github.com/allenap/shell-quote) crate,
which seems to handle all this escaping stuff for you but fixing this
issue perfectly isn't a big deal (if there are more things to escape we
can deal with it then), so adding a new dependency (even a dev one)
seemed overkill.
[^1]: The filter does still work---it turns out that the filter
`MDTEST_TEST_FILTER="sync.md - With statements - Context manager with
non-callable attribute"` (what you get after the failed interpolation)
is still good enough
## Test Plan
<!-- How was it tested? -->
I broke the ``## Context manager with non-callable `__exit__`
attribute`` test by deleting the error assertion, then successfully ran
the new command it printed out.
Summary
--
Unlike the other syntax errors detected so far, parenthesized keyword
arguments are only allowed *before* 3.8. It sounds like they were only
accidentally allowed before that [^1].
As an aside, you get a pretty confusing error from Python for this, so
it's nice that we can catch it:
```pycon
>>> def f(**kwargs): ...
... f((a)=1)
...
File "<python-input-0>", line 2
f((a)=1)
^^^
SyntaxError: expression cannot contain assignment, perhaps you meant "=="?
>>>
```
Test Plan
--
Inline tests.
[^1]: https://github.com/python/cpython/issues/78822
Summary
--
Checks for tuple unpacking in `return` and `yield` statements before
Python 3.8, as described [here].
Test Plan
--
Inline tests.
[here]: https://github.com/python/cpython/issues/76298
## Summary
- `Never` is callable
- `Never` is iterable
- Arbitrary attributes can be accessed on `Never`
Split out from #16416 that is going to be required.
## Test Plan
Tests for all properties above.
## Summary
This came up in https://github.com/astral-sh/ruff/issues/16477
It's not obvious from the D417 rule's documentation that it only checks
docstrings
with an arguments section. Functions without such a section aren't
checked.
This PR tries to make this clearer in the documentation.
## Summary
This PR introduces a new mdtest option `system` that can either be
`in-memory` or `os`
where `in-memory` is the default.
The motivation for supporting `os` is so that we can write OS/system
specific tests
with mdtests. Specifically, I want to write mdtests for the module
resolver,
testing that module resolution is case sensitive.
## Test Plan
I tested that the case-sensitive module resolver test start failing when
setting `system = "os"`
## Summary
Python's module resolver is case sensitive.
This PR adds mdtests that assert that our module resolution is case
sensitive.
The tests currently all pass because our in memory file system is case
sensitive.
I'll add support for using the real file system to the mdtest framework
in a separate PR.
This PR also adds support for specifying extra search paths to the
mdtest framework.
## Test Plan
The tests fail when running them using the real file system.
To kick off the work of supporting generics, this adds many new
(currently failing) tests, showing the behavior we plan to support.
This is still missing a lot! Not included:
- typevar tuples
- param specs
- variance
- `Self`
But it's a good start! We can add more failing tests for those once we
tackle these.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
This is split out of https://github.com/astral-sh/ruff/pull/14029, to
reduce the size of that PR, and to validate that this "fallback type"
support in `TypeInference` doesn't come with a performance cost. It also
improves the reliability and debuggability of our current (temporary)
cycle handling.
In order to recover from a cycle, we have to be able to construct a
"default" `TypeInference` where all expressions and definitions have
some "default" type. In our current cycle handling, this "default" type
is just unknown or a todo type. With fixpoint iteration, the "default"
type will be `Type::Never`, which is the "bottom" type that fixpoint
iteration starts from.
Since it would be costly (both in space and time) to actually enumerate
all expressions and definitions in a scope, just to insert the same
default type for all of them, instead we add an optional "missing type"
fallback to `TypeInference`, which (if set) is the fallback type for any
expression or definition which doesn't have an explicit type set.
With this change, cycles can no longer result in the dreaded "Missing
key" errors looking up the type of some expression.
... with supporting types. This is meant to give us a base to work with
in terms of our new diagnostic data model. I expect the representations
to be tweaked over time, but I think this is a decent start.
I would also like to add doctest examples, but I think it's better if we
wait until an initial version of the renderer is done for that.
This puts them out of the way so that they can hopefully be removed more
easily in the (near) future, and so that they don't get in the way of
the new types. This also makes the intent of the migration a bit clearer
in the code and hopefully results in less confusion.
This trait should eventually go away, so we rename it (and supporting
types) to make room for a new concrete `Diagnostic` type.
This commit is just the rename. In the next commit, we'll move it to a
different module.
Summary
--
Another simple one, just detect type parameter lists in functions
and classes. Like pyright, we don't emit a second diagnostic for
`type` alias statements, which were also introduced in 3.12.
Test Plan
--
Inline tests.
## Summary
This PR does a small refactor to avoid double
`symbol_table(...).symbol(...)` call to check for `__slots__` and
`TYPE_CHECKING`. It merges them into a single call.
I noticed this while looking at
https://github.com/astral-sh/ruff/pull/16468.
Josh Cannon
Alex Waygood
Matthew Mckee
Brent Westbrook
Dhruv Manilawala
Micha Reiser
cake-monotone
Peter Hill
Kaxil Naik
Andrew Gallant
Douglas Creager
Mauro Fontana
Dylan
Carl Meyer
github-actions[bot]
David Peter
Shaygan Hooshyari
InSync
David Salvisberg
Peter Hill
Joey Bar
Shunsuke Shibayama
Joey Bar
Eric Mark Martin
Vasco Schiavo