## Summary
The previous `try_call_dunder_with_policy` API was a bit of a footgun
since you needed to pass `NO_INSTANCE_FALLBACK` in *addition* to other
policies that you wanted for the member lookup. Implicit calls to dunder
methods never access instance members though, so we can do this
implicitly in `try_call_dunder_with_policy`.
No functional changes.
## Summary
`Type::member_lookup_with_policy` now falls back to calling
`__getattribute__` when a member cannot be found as a second fallback
after `__getattr__`.
closes https://github.com/astral-sh/ty/issues/441
## Test Plan
Added markdown tests.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
This should address a problem that came up while working on
https://github.com/astral-sh/ruff/pull/18280. When looking up an
attribute (typically a dunder method) with the `MRO_NO_OBJECT_FALLBACK`
policy, the attribute is first looked up on the meta type. If the meta
type happens to be `type`, we go through the following branch in
`find_name_in_mro_with_policy`:
97ff015c88/crates/ty_python_semantic/src/types.rs (L2565-L2573)
The problem is that we now look up the attribute on `object` *directly*
(instead of just having `object` in the MRO). In this case,
`MRO_NO_OBJECT_FALLBACK` has no effect in `class_member_from_mro`:
c3feb8ce27/crates/ty_python_semantic/src/types/class.rs (L1081-L1082)
So instead, we need to explicitly respect the `MRO_NO_OBJECT_FALLBACK`
policy here by returning `Symbol::Unbound`.
## Test Plan
Added new Markdown tests that explain the ecosystem changes that we
observe.
## Summary
Fix a bug that involved writes to attributes on union/intersection types
that included modules as elements.
This is a prerequisite to avoid some ecosystem false positives in
https://github.com/astral-sh/ruff/pull/18312
## Test Plan
Added regression test
## Summary
This PR moves the diagnostics API for the language server out from the
request handler module to the diagnostics API module.
This is in preparation to add support for publishing diagnostics.
## Summary
Resolves https://github.com/astral-sh/ty/issues/485.
`infer_binary_intersection_type_comparison()` now checks for all
positive members before concluding that an operation is unsupported for
a given intersection type.
## Test Plan
Markdown tests.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
This is a practice I followed on previous projects. Should hopefully
further help developers who want to update the documentation.
The big downside is that it's annoying to see this *as a user of the
documentation* if you don't open the Markdown file in the browser. But
I'd argue that those files don't really follow the original Markdown
spirit anyway with all the inline HTML.
## Summary
This is something I wrote a few months ago, and continued to update from
time to time. It was mostly written for my own education. I found a few
bugs while writing it at the time (there are still one or two TODOs in
the test assertions that are probably bugs). Our other tests are fairly
comprehensive, but they are usually structured around a certain
functionality or operation (subtyping, assignability, narrowing). The
idea here was to focus on individual *types and their properties*.
closes#197 (added `JustFloat` and `JustComplex` to `ty_extensions`).
## Summary
Fix remaining `knot.toml` reference and replace it with `ty.toml`. This
change was probably still in flight while we renamed things.
## Test Plan
Added a second assertion which ensures that the config file has any
effect.
## Summary
It doesn't seem to be necessary for our generics implementation to carry
the `GenericContext` in the `ClassBase` variants. Removing it simplifies
the code, fixes many TODOs about `Generic` or `Protocol` appearing
multiple times in MROs when each should only appear at most once, and
allows us to more accurately detect runtime errors that occur due to
`Generic` or `Protocol` appearing multiple times in a class's bases.
In order to remove the `GenericContext` from the `ClassBase` variant, it
turns out to be necessary to emulate
`typing._GenericAlias.__mro_entries__`, or we end up with a large number
of false-positive `inconsistent-mro` errors. This PR therefore also does
that.
Lastly, this PR fixes the inferred MROs of PEP-695 generic classes,
which implicitly inherit from `Generic` even if they have no explicit
bases.
## Test Plan
mdtests
## Summary
Fix some issues with subtying/assignability for instances vs callables.
We need to look up dunders on the class, not the instance, and we should
limit our logic here to delegating to the type of `__call__`, so it
doesn't get out of sync with the calls we allow.
Also, we were just entirely missing assignability handling for
`__call__` implemented as anything other than a normal bound method
(though we had it for subtyping.)
A first step towards considering what else we want to change in
https://github.com/astral-sh/ty/issues/491
## Test Plan
mdtests
---------
Co-authored-by: med <medioqrity@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Division works differently in Python than in Rust. If the result is
negative and there is a remainder, the division rounds down (instead of
towards zero). The remainder needs to be adjusted to compensate so that
`(lhs // rhs) * rhs + (lhs % rhs) == lhs`.
Fixesastral-sh/ty#481.
## Summary
https://github.com/astral-sh/ty/issues/111
This PR adds support for `frozen` dataclasses. It will emit a diagnostic
with a similar message to mypy
Note: This does not include emitting a diagnostic if `__setattr__` or
`__delattr__` are defined on the object as per the
[spec](https://docs.python.org/3/library/dataclasses.html#module-contents)
## Test Plan
mdtest
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Make sure that the following definitions all lead to the same outcome
(bug originally noticed by @AlexWaygood)
```py
from typing import ClassVar
class Descriptor:
def __get__(self, instance, owner) -> int:
return 42
class C:
a: ClassVar[Descriptor]
b: Descriptor = Descriptor()
c: ClassVar[Descriptor] = Descriptor()
reveal_type(C().a) # revealed: int (previously: int | Descriptor)
reveal_type(C().b) # revealed: int
reveal_type(C().c) # revealed: int
```
## Test Plan
New Markdown tests
## Summary
I think `division-by-zero` is a low-value diagnostic in general; most
real division-by-zero errors (especially those that are less obvious to
the human eye) will occur on values typed as `int`, in which case we
don't issue the diagnostic anyway. Mypy and pyright do not emit this
diagnostic.
Currently the diagnostic is prone to false positives because a) we do
not silence it in unreachable code, and b) we do not implement narrowing
of literals from inequality checks. We will probably fix (a) regardless,
but (b) is low priority apart from division-by-zero.
I think we have many more important things to do and should not allow
false positives on a low-value diagnostic to be a distraction. Not
opposed to re-enabling this diagnostic in future when we can prioritize
reducing its false positives.
References https://github.com/astral-sh/ty/issues/443
## Test Plan
Existing tests.
## Summary
This PR updates the language server to avoid panicking when there are
multiple workspace folders passed during initialization. The server
currently picks up the first workspace folder and provides a warning and
a log message.
## Test Plan
<img width="1724" alt="Screenshot 2025-05-17 at 11 43 09"
src="https://github.com/user-attachments/assets/1a7ddbc3-198d-4191-a28f-9b69321e8f99"
/>
## Summary
Resolves [#461](https://github.com/astral-sh/ty/issues/461).
ty was hardcoded to infer `BytesLiteral` types for integer indexing into
`BytesLiteral`. It will now infer `IntLiteral` types instead.
## Test Plan
Markdown tests.
Summary
--
I thought that emitting multiple diagnostics at once would be difficult
to port to a diagnostic construction model closer to ty's
`InferContext::report_lint`, so as a first step toward that, this PR
removes `Checker::report_diagnostics`.
In many cases I was able to do some related refactoring to avoid
allocating a `Vec<Diagnostic>` at all, often by adding a `Checker` field
to a `Visitor` or by passing a `Checker` instead of a `&mut
Vec<Diagnostic>`.
In other cases, I had to fall back on something like
```rust
for diagnostic in diagnostics {
checker.report_diagnostic(diagnostic);
}
```
which I guess is a bit worse than the `extend` call in
`report_diagnostics`, but hopefully it won't make too much of a
difference.
I'm still not quite sure what to do with the remaining loop cases. The
two main use cases for collecting a sequence of diagnostics before
emitting any of them are:
1. Applying a single `Fix` to a group of diagnostics
2. Avoiding an earlier diagnostic if something goes wrong later
I was hoping we could get away with just a `DiagnosticGuard` that
reported a `Diagnostic` on drop, but I guess we will still need a
`DiagnosticGuardBuilder` that can be collected in these cases and
produce a `DiagnosticGuard` once we know we actually want the
diagnostics.
Test Plan
--
Existing tests
Closes https://github.com/astral-sh/ty/issues/453.
## Summary
Add an additional info diagnostic to `unresolved-import` check to hint
to users that they should make sure their Python environment is properly
configured for ty, linking them to the corresponding doc. This
diagnostic is only shown when an import is not relative, e.g., `import
maturin` not `import .maturin`.
## Test Plan
Updated snapshots with new info message and reran tests.
The PR add the `fix safety` section for rule `SIM110` (#15584 )
### Unsafe Fix Example
```python
def predicate(item):
global called
called += 1
if called == 1:
# after first call we change the method
def new_predicate(_): return False
globals()['predicate'] = new_predicate
return True
def foo():
for item in range(10):
if predicate(item):
return True
return False
def foo_gen():
return any(predicate(item) for item in range(10))
called = 0
print(foo()) # true – returns immediately on first call
called = 0
print(foo_gen()) # false – second call uses new `predicate`
```
### Note
I notice that
[here](46be305ad2/crates/ruff_linter/src/rules/flake8_simplify/rules/reimplemented_builtin.rs (L60))
we have two rules, `SIM110` & `SIM111`. The second one seems not anymore
active. Should I delete `SIM111`?
This implements the stopgap approach described in
https://github.com/astral-sh/ty/issues/336#issuecomment-2880532213 for
handling literal types in generic class specializations.
With this approach, we will promote any literal to its instance type,
but _only_ when inferring a generic class specialization from a
constructor call:
```py
class C[T]:
def __init__(self, x: T) -> None: ...
reveal_type(C("string")) # revealed: C[str]
```
If you specialize the class explicitly, we still use whatever type you
provide, even if it's a literal:
```py
from typing import Literal
reveal_type(C[Literal[5]](5)) # revealed: C[Literal[5]]
```
And this doesn't apply at all to generic functions:
```py
def f[T](x: T) -> T:
return x
reveal_type(f(5)) # revealed: Literal[5]
```
---
As part of making this happen, we also generalize the `TypeMapping`
machinery. This provides a way to apply a function to type, returning a
new type. Complicating matters is that for function literals, we have to
apply the mapping lazily, since the function's signature is not created
until (and if) someone calls its `signature` method. That means we have
to stash away the mappings that we want to apply to the signatures
parameter/return annotations once we do create it. This requires some
minor `Cow` shenanigans to continue working for partial specializations.
This is a follow-on to #18155. For the example raised in
https://github.com/astral-sh/ty/issues/370:
```py
import tempfile
with tempfile.TemporaryDirectory() as tmp: ...
```
the new logic would notice that both overloads of `TemporaryDirectory`
match, and combine their specializations, resulting in an inferred type
of `str | bytes`.
This PR updates the logic to match our other handling of other calls,
where we only keep the _first_ matching overload. The result for this
example then becomes `str`, matching the runtime behavior. (We still do
not implement the full [overload resolution
algorithm](https://typing.python.org/en/latest/spec/overload.html#overload-call-evaluation)
from the spec.)
## Summary
Add a new diagnostic hint if you try to use PEP 604 `X | Y` union syntax
in a non-type-expression before 3.10.
closes https://github.com/astral-sh/ty/issues/437
## Test Plan
New snapshot test
## Summary
This PR unifies the ruff `Message` enum variants for syntax errors and
rule violations into a single `Message` struct consisting of a shared
`db::Diagnostic` and some additional, optional fields used for some rule
violations.
This version of `Message` is nearly a drop-in replacement for
`ruff_diagnostics::Diagnostic`, which is the next step I have in mind
for the refactor.
I think this is also a useful checkpoint because we could possibly add
some of these optional fields to the new `Diagnostic` type. I think
we've previously discussed wanting support for `Fix`es, but the other
fields seem less relevant, so we may just need to preserve the `Message`
wrapper for a bit longer.
## Test plan
Existing tests
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
* Remove the following rules
* name
* `airflow.auth.managers.base_auth_manager.is_authorized_dataset` →
`airflow.api_fastapi.auth.managers.base_auth_manager.is_authorized_asset`
*
`airflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_dataset`
→
`airflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_asset`
* Update the following rules
* name
* `airflow.models.baseoperatorlink.BaseOperatorLink` →
`airflow.sdk.BaseOperatorLink`
* `airflow.api_connexion.security.requires_access` → "Use
`airflow.api_fastapi.core_api.security.requires_access_*` instead`"
* `airflow.api_connexion.security.requires_access_dataset`→
`airflow.api_fastapi.core_api.security.requires_access_asset`
* `airflow.notifications.basenotifier.BaseNotifier` →
`airflow.sdk.bases.notifier.BaseNotifier`
* `airflow.www.auth.has_access` → None
* `airflow.www.auth.has_access_dataset` → None
* `airflow.www.utils.get_sensitive_variables_fields`→ None
* `airflow.www.utils.should_hide_value_for_key`→ None
* class attribute
* `airflow..sensors.weekday.DayOfWeekSensor`
* `use_task_execution_day` removed
*
`airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager`
* `is_authorized_dataset`
* Add the following rules
* class attribute
* `airflow.auth.managers.base_auth_manager.BaseAuthManager` |
`airflow.providers.fab.auth_manager.fab_auth_manager.FabAuthManager`
* name
* `airflow.auth.managers.base_auth_manager.BaseAuthManager` →
`airflow.api_fastapi.auth.managers.base_auth_manager.BaseAuthManager` *
`is_authorized_dataset` → `is_authorized_asset`
* refactor
* simplify unnecessary match with if else
* rename Replacement::Name as Replacement::AttrName
## Test Plan
<!-- How was it tested? -->
The test fixtures have been revised and updated.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
In the later development of Airflow 3.0, backward compatibility was not
added for some cases. Thus, the following rules are moved back to AIR302
* airflow.hooks.subprocess.SubprocessResult →
airflow.providers.standard.hooks.subprocess.SubprocessResult
* airflow.hooks.subprocess.working_directory →
airflow.providers.standard.hooks.subprocess.working_directory
* airflow.operators.datetime.target_times_as_dates →
airflow.providers.standard.operators.datetime.target_times_as_dates
* airflow.operators.trigger_dagrun.TriggerDagRunLink →
airflow.providers.standard.operators.trigger_dagrun.TriggerDagRunLink
* airflow.sensors.external_task.ExternalTaskSensorLink →
airflow.providers.standard.sensors.external_task.ExternalDagLink (**This
one contains a minor change**)
* airflow.sensors.time_delta.WaitSensor →
airflow.providers.standard.sensors.time_delta.WaitSensor
## Test Plan
<!-- How was it tested? -->
This primarily comes up with annotated `self` parameters in
constructors:
```py
class C[T]:
def __init__(self: C[int]): ...
```
Here, we want infer a specialization of `{T = int}` for a call that hits
this overload.
Normally when inferring a specialization of a function call, typevars
appear in the parameter annotations, and not in the argument types. In
this case, this is reversed: we need to verify that the `self` argument
(`C[T]`, as we have not yet completed specialization inference) is
assignable to the parameter type `C[int]`.
To do this, we simply look for a typevar/type in both directions when
performing inference, and apply the inferred specialization to argument
types as well as parameter types before verifying assignability.
As a wrinkle, this exposed that we were not checking
subtyping/assignability for function literals correctly. Our function
literal representation includes an optional specialization that should
be applied to the signature. Before, function literals were considered
subtypes of (assignable to) each other only if they were identical Salsa
objects. Two function literals with different specializations should
still be considered subtypes of (assignable to) each other if those
specializations result in the same function signature (typically because
the function doesn't use the typevars in the specialization).
Closes https://github.com/astral-sh/ty/issues/370
Closes https://github.com/astral-sh/ty/issues/100
Closes https://github.com/astral-sh/ty/issues/258
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
My editor runs `rustfmt` on save to format Rust code, not `cargo fmt`.
With our recent bump to the Rust 2024 edition, the formatting that
`rustfmt`/`cargo fmt` applies changed. Unfortunately, `rustfmt` and
`cargo fmt` have different behaviors for determining which edition to
use when formatting: `cargo fmt` looks for the Rust edition in
`Cargo.toml`, whereas `rustfmt` looks for it in `rustfmt.toml`. As a
result, whenever I save, I have to remember to manually run `cargo fmt`
before committing/pushing.
There is an open issue asking for `rustfmt` to also look at `Cargo.toml`
when it's present (https://github.com/rust-lang/rust.vim/issues/368),
but it seems like they "closed" that issue just by bumping the default
edition (six years ago, from 2015 to 2018).
In the meantime, this PR adds a `rustfmt.toml` file with our current
Rust edition so that both invocation have the same behavior. I don't
love that this duplicates information in `Cargo.toml`, but I've added a
reminder comment there to hopefully ensure that we bump the edition in
both places three years from now.
## Summary
Support direct uses of `typing.TypeAliasType`, as in:
```py
from typing import TypeAliasType
IntOrStr = TypeAliasType("IntOrStr", int | str)
def f(x: IntOrStr) -> None:
reveal_type(x) # revealed: int | str
```
closes https://github.com/astral-sh/ty/issues/392
## Ecosystem
The new false positive here:
```diff
+ error[invalid-type-form] altair/utils/core.py:49:53: The first argument to `Callable` must be either a list of types, ParamSpec, Concatenate, or `...`
```
comes from the fact that we infer the second argument as a type
expression now. We silence false positives for PEP695 `ParamSpec`s, but
not for `P = ParamSpec("P")` inside `Callable[P, ...]`.
## Test Plan
New Markdown tests
## Summary
just a minor nit followup to
https://github.com/astral-sh/ruff/pull/18010 -- put all the
non-`Visitor` methods of `SemanticIndexBuilder` in the same impl block
rather than having multiple impl blocks
## Test Plan
`cargo build`
Summary
--
I noticed these `cfg` directives while working on diagnostics. I think
it makes more sense to apply an `insta` filter in the test instead. I
copied this filter from a CLI test for the same rule.
Test Plan
--
Existing tests, especially Windows CI on this PR
## Summary
With this PR we now detect that x is always defined in `use`:
```py
if flag and (x := number):
use(x)
```
When outside if, it's still detected as possibly not defined
```py
flag and (x := number)
# error: [possibly-unresolved-reference]
use(x)
```
In order to achieve that, I had to find a way to get access to the
flow-snapshots of the boolean expression when analyzing the flow of the
if statement. I did it by special casing the visitor of boolean
expression to return flow control information, exporting two snapshots -
`maybe_short_circuit` and `no_short_circuit`. When indexing
boolean expression itself we must assume all possible flows, but when
it's inside if statement, we can be smarter than that.
## Test Plan
Fixed existing and added new mdtests.
I went through some of mypy primer results and they look fine
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Add various attributes to `NamedTuple` classes/instances that are
available at runtime.
closes https://github.com/astral-sh/ty/issues/417
## Test Plan
New Markdown tests
The PR add the `fix safety` section for rule `SIM210` (#15584 )
It is a little cheating, as the Fix safety section is copy/pasted by
#18086 as the problem is the same.
### Unsafe Fix Example
```python
class Foo():
def __eq__(self, other):
return 0
def foo():
return True if Foo() == 0 else False
def foo_fix():
return Foo() == 0
print(foo()) # False
print(foo_fix()) # 0
```
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#18107
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Snapshot tests
<!-- How was it tested? -->
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
`ProviderReplacement::Name` was designed back when we only wanted to do
linting. Now we also want to fix the user code. It would be easier for
us to replace them with better AutoImport struct.
## Test Plan
<!-- How was it tested? -->
The test fixture has been updated as some cases can now be fixed
## Summary
The PR adds an explicit check for `"__builtins__"` during name lookup,
similar to how `"__file__"` is implemented. The inferred type is
`Any`.
closes https://github.com/astral-sh/ty/issues/393
## Test Plan
Added a markdown test for `__builtins__`.
---------
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
This makes an easy tweak to allow our diagnostics for unmatched
overloads to apply to method calls. Previously, they only worked for
function calls.
There is at least one other case worth addressing too, namely, class
literals. e.g., `type()`. We had a diagnostic snapshot test case to
track it.
Closesastral-sh/ty#274
## Summary
Model that `type[C]` is always assignable to `type`, even if `C` is not
fully static.
closes https://github.com/astral-sh/ty/issues/312
## Test Plan
* New Markdown tests
* Property tests
## Summary
This PR deletes the `DiagnosticKind` type by inlining its three fields
(`name`, `body`, and `suggestion`) into three other diagnostic types:
`Diagnostic`, `DiagnosticMessage`, and `CacheMessage`.
Instead of deferring to an internal `DiagnosticKind`, both `Diagnostic`
and `DiagnosticMessage` now have their own macro-generated `AsRule`
implementations.
This should make both https://github.com/astral-sh/ruff/pull/18051 and
another follow-up PR changing the type of `name` on `CacheMessage`
easier since its type will be able to change separately from
`Diagnostic` and `DiagnosticMessage`.
## Test Plan
Existing tests
## Summary
Resolves [#290](https://github.com/astral-sh/ty/issues/290).
All arguments, synthesized or not, are now accounted for in
`too-many-positional-arguments`'s error message.
For example, consider this example:
```python
class C:
def foo(self): ...
C().foo(1) # !!!
```
Previously, ty would say:
> Too many positional arguments to bound method foo: expected 0, got 1
After this change, it will say:
> Too many positional arguments to bound method foo: expected 1, got 2
This is what Python itself does too:
```text
Traceback (most recent call last):
File "<python-input-0>", line 3, in <module>
C().foo()
~~~~~~~^^
TypeError: C.foo() takes 0 positional arguments but 1 was given
```
## Test Plan
Markdown tests.
The PR add the `fix safety` section for rule `SIM103` (#15584 )
### Unsafe Fix Example
```python
class Foo:
def __eq__(self, other):
return 1
def foo():
if Foo() == 1:
return True
return False
def foo_fix():
return Foo() == 1
print(foo()) # True
print(foo_fix()) # 1
```
### Note
I updated the code snippet example, because I thought it was cool to
have a correct example, i.e., that I can paste inside the playground and
it works :-)
Fixes#18069
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
-->
## Summary
This PR addresses a bug in the `flake8-simplify` rule `SIM905`
(split-static-string) where `str.split(maxsplit=0)` and
`str.rsplit(maxsplit=0)` produced incorrect results for empty strings or
strings starting/ending with whitespace. The fix ensures that the
linting rule's suggested replacements now align with Python's native
behavior for these specific `maxsplit=0` scenarios.
## Test Plan
1. Added new test cases to the existing
`crates/ruff_linter/resources/test/fixtures/flake8_simplify/SIM905.py`
fixture to cover the scenarios described in issue #18069.
2. Ran `cargo test -p ruff_linter`.
3. Verified and accepted the updated snapshots for `SIM905.py` using
`cargo insta review`. The new snapshots confirm the corrected behavior
for `maxsplit=0`.
The diagnostic now includes a pointer to the implementation definition
along with each possible overload.
This doesn't include information about *why* each overload failed. But
given the emphasis on concise output (since there can be *many*
unmatched overloads), it's not totally clear how to include that
additional information.
Fixes#274
These are, after all, specific to function types. The methods on `Type`
are more like conveniences that return something when the type *happens*
to be a function. But defining them on `FunctionType` itself makes it
easy to call them when you have a `FunctionType` instead of a `Type`.
I found the previous code somewhat harder to read. Namely, a `for`
loop was being used to encode "execute zero or one times, but not
more." Which is sometimes okay, but it seemed clearer to me to use
more explicit case analysis here.
This should have no behavioral changes.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Similiar to https://github.com/astral-sh/ruff/pull/17941.
`Replacement::Name` was designed for linting only. Now, we also want to
fix the user code. It would be easier to replace it with a better
AutoImport struct whenever possible.
On the other hand, `AIR301` and `AIR311` contain attribute changes that
can still use a struct like `Replacement::Name`. To reduce the
confusion, I also updated it as `Replacement::AttrName`
Some of the original `Replacement::Name` has been replaced as
`Replacement::Message` as they're not directly mapping and the message
has now been moved to `help`
## Test Plan
<!-- How was it tested? -->
The test fixtures have been updated
The PR add the `fix safety` section for rule `RUF007` (#15584 )
It seems that the fix was always marked as unsafe #14401
## Unsafety example
This first example is a little extreme. In fact, the class `Foo`
overrides the `__getitem__` method but in a very special, way. The
difference lies in the fact that `zip(letters, letters[1:])` call the
slice `letters[1:]` which is behaving weird in this case, while
`itertools.pairwise(letters)` call just `__getitem__(0), __getitem__(1),
...` and so on.
Note that the diagnostic is emitted: [playground](https://play.ruff.rs)
I don't know if we want to mention this problem, as there is a subtile
bug in the python implementation of `Foo` which make the rule unsafe.
```python
from dataclasses import dataclass
import itertools
@dataclass
class Foo:
letters: str
def __getitem__(self, index):
return self.letters[index] + "_foo"
letters = Foo("ABCD")
zip_ = zip(letters, letters[1:])
for a, b in zip_:
print(a, b) # A_foo B, B_foo C, C_foo D, D_foo _
pair = itertools.pairwise(letters)
for a, b in pair:
print(a, b) # A_foo B_foo, B_foo C_foo, C_foo D_foo
```
This other example is much probable.
here, `itertools.pairwise` was shadowed by a costume function
[(playground)](https://play.ruff.rs)
```python
from dataclasses import dataclass
from itertools import pairwise
def pairwise(a):
return []
letters = "ABCD"
zip_ = zip(letters, letters[1:])
print([(a, b) for a, b in zip_]) # [('A', 'B'), ('B', 'C'), ('C', 'D')]
pair = pairwise(letters)
print(pair) # []
```
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#17599.
## Test Plan
Snapshot tests.
---------
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
## Summary
Dunder methods are never looked up on instances. We do this implicitly
in `try_call_dunder`, but the corresponding flag was missing in the
instance-construction code where we use `member_lookup_with_policy`
directly.
fixes https://github.com/astral-sh/ty/issues/322
## Test Plan
Added regression test.
## Summary
This PR adds cycle handling for `infer_unpack_types` based on the
analysis in astral-sh/ty#364.
Fixes: astral-sh/ty#364
## Test Plan
Add a cycle handling test for unpacking in `cycle.md`
## Summary
Add a micro-benchmark for the code pattern observed in
https://github.com/astral-sh/ty/issues/362.
This currently takes around 1 second on my machine.
## Test Plan
```bash
cargo bench -p ruff_benchmark -- 'ty_micro\[many_tuple' --sample-size 10
```
Follows on from (and depends on)
https://github.com/astral-sh/ruff/pull/18021.
This updates our function specialization inference to infer type
mappings from parameters that are generic protocols.
For now, this only works when the argument _explicitly_ implements the
protocol by listing it as a base class. (We end up using exactly the
same logic as for generic classes in #18021.) For this to work with
classes that _implicitly_ implement the protocol, we will have to check
the types of the protocol members (which we are not currently doing), so
that we can infer the specialization of the protocol that the class
implements.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Understand that `__file__` is always set and a `str` when looked up as
an implicit global from a Python file we are type checking.
## Test Plan
mdtests
## Summary
Fix the lookup of `submodule`s in cases where the `parent` module has a
self-referential import like `from parent import submodule`. This allows
us to infer proper types for many symbols where we previously inferred
`Never`. This leads to many new false (and true) positives across the
ecosystem because the fact that we previously inferred `Never` shadowed
a lot of problems. For example, we inferred `Never` for `os.path`, which
is why we now see a lot of new diagnostics related to `os.path.abspath`
and similar.
```py
import os
reveal_type(os.path) # previously: Never, now: <module 'os.path'>
```
closes https://github.com/astral-sh/ty/issues/261
closes https://github.com/astral-sh/ty/issues/307
## Ecosystem analysis
```
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ Diagnostic ID ┃ Severity ┃ Removed ┃ Added ┃ Net Change ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ call-non-callable │ error │ 1 │ 5 │ +4 │
│ call-possibly-unbound-method │ warning │ 6 │ 26 │ +20 │
│ invalid-argument-type │ error │ 26 │ 94 │ +68 │
│ invalid-assignment │ error │ 18 │ 46 │ +28 │
│ invalid-context-manager │ error │ 9 │ 4 │ -5 │
│ invalid-raise │ error │ 1 │ 1 │ 0 │
│ invalid-return-type │ error │ 3 │ 20 │ +17 │
│ invalid-super-argument │ error │ 4 │ 0 │ -4 │
│ invalid-type-form │ error │ 573 │ 0 │ -573 │
│ missing-argument │ error │ 2 │ 10 │ +8 │
│ no-matching-overload │ error │ 0 │ 715 │ +715 │
│ non-subscriptable │ error │ 0 │ 35 │ +35 │
│ not-iterable │ error │ 6 │ 7 │ +1 │
│ possibly-unbound-attribute │ warning │ 14 │ 31 │ +17 │
│ possibly-unbound-import │ warning │ 13 │ 0 │ -13 │
│ possibly-unresolved-reference │ warning │ 0 │ 8 │ +8 │
│ redundant-cast │ warning │ 1 │ 0 │ -1 │
│ too-many-positional-arguments │ error │ 2 │ 0 │ -2 │
│ unknown-argument │ error │ 2 │ 0 │ -2 │
│ unresolved-attribute │ error │ 583 │ 304 │ -279 │
│ unresolved-import │ error │ 0 │ 96 │ +96 │
│ unsupported-operator │ error │ 0 │ 17 │ +17 │
│ unused-ignore-comment │ warning │ 29 │ 2 │ -27 │
├───────────────────────────────┼──────────┼─────────┼───────┼────────────┤
│ TOTAL │ │ 1293 │ 1421 │ +128 │
└───────────────────────────────┴──────────┴─────────┴───────┴────────────┘
Analysis complete. Found 23 unique diagnostic IDs.
Total diagnostics removed: 1293
Total diagnostics added: 1421
Net change: +128
```
* We see a lot of new errors (`no-matching-overload`) related to
`os.path.dirname` and other `os.path` operations because we infer `str |
None` for `__file__`, but many projects use something like
`os.path.dirname(__file__)`.
* We also see many new `unresolved-attribute` errors related to the fact
that we now infer proper module types for some imports (e.g. `import
kornia.augmentation as K`), but we don't allow implicit imports (e.g.
accessing `K.auto.operations` without also importing `K.auto`). See
https://github.com/astral-sh/ty/issues/133.
* Many false positive `invalid-type-form` are removed because we now
infer the correct type for some type expression instead of `Never`,
which is not valid in a type annotation/expression context.
## Test Plan
Added new Markdown tests
## Summary
If the user tries to use a new builtin on an old Python version, tell
them what Python version the builtin was added on, what our inferred
Python version is for their project, and what configuration settings
they can tweak to fix the error.
## Test Plan
Snapshots and screenshots:
First take on a contributing guide for `ty`. Lots of it is copied from
the existing Ruff contribution guide.
I've put this in Ruff repo, since I think a contributing guide belongs
where the code is. I also updated the Ruff contributing guide to link to
the `ty` one.
Once this is merged, we can also add a link from the `CONTRIBUTING.md`
in ty repo (which focuses on making contributions to things that are
actually in the ty repo), to this guide.
I also updated the pull request template to mention that it might be a
ty PR, and mention the `[ty]` PR title prefix.
Feel free to update/modify/merge this PR before I'm awake tomorrow.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
Co-authored-by: David Peter <mail@david-peter.de>
Fixes: https://github.com/astral-sh/ty/issues/92
## Summary
We currently get a `invalid-argument-type` error when using
`dataclass.fields` on a dataclass, because we do not synthesize the
`__dataclass_fields__` member.
This PR fixes this diagnostic.
Note that we do not yet model the `Field` type correctly. After that is
done, we can assign a more precise `tuple[Field, ...]` type to this new
member.
## Test Plan
New mdtest.
---------
Co-authored-by: David Peter <mail@david-peter.de>
This updates our function specialization inference to infer type
mappings from parameters that are generic aliases, e.g.:
```py
def f[T](x: list[T]) -> T: ...
reveal_type(f(["a", "b"])) # revealed: str
```
Though note that we're still inferring the type of list literals as
`list[Unknown]`, so for now we actually need something like the
following in our tests:
```py
def _(x: list[str]):
reveal_type(f(x)) # revealed: str
```
We were not inducting into instance types and subclass-of types when
looking for legacy typevars, nor when apply specializations.
This addresses
https://github.com/astral-sh/ruff/pull/17832#discussion_r2081502056
```py
from __future__ import annotations
from typing import TypeVar, Any, reveal_type
S = TypeVar("S")
class Foo[T]:
def method(self, other: Foo[S]) -> Foo[T | S]: ... # type: ignore[invalid-return-type]
def f(x: Foo[Any], y: Foo[Any]):
reveal_type(x.method(y)) # revealed: `Foo[Any | S]`, but should be `Foo[Any]`
```
We were not detecting that `S` made `method` generic, since we were not
finding it when searching the function signature for legacy typevars.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
If a try-catch block guards the names, we don't raise warnings. During
this change, I discovered that some of the replacement types were
missed. Thus, I extend the fix to types other than AutoImport as well
## Test Plan
<!-- How was it tested? -->
Test fixtures are added and updated.
## Summary
Updates the `--python` flag to accept Python executables in virtual
environments. Notably, we do not query the executable and it _must_ be
in a canonical location in a virtual environment. This is pretty naive,
but solves for the trivial case of `ty check --python .venv/bin/python3`
which will be a common mistake (and `ty check --python $(which python)`)
I explored this while trying to understand Python discovery in ty in
service of https://github.com/astral-sh/ty/issues/272, I'm not attached
to it, but figure it's worth sharing.
As an alternative, we can add more variants to the
`SearchPathValidationError` and just improve the _error_ message, i.e.,
by hinting that this looks like a virtual environment and suggesting the
concrete alternative path they should provide. We'll probably want to do
that for some other cases anyway (e.g., `3.13` as described in the
linked issue)
This functionality is also briefly mentioned in
https://github.com/astral-sh/ty/issues/193
Closes https://github.com/astral-sh/ty/issues/318
## Test Plan
e.g.,
```
uv run ty check --python .venv/bin/python3
```
needs test coverage still
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
The existing implementation of RUF060 (InEmptyCollection) is not
recursive, meaning that although set([]) results in an empty collection,
the existing code fails it because set is taking an argument.
The updated implementation allows set and frozenset to take empty
collection as positional argument (which results in empty
set/frozenset).
## Test Plan
Added test cases for recursive cases + updated snapshot (see RUF060.py).
---------
Co-authored-by: Marcus Näslund <marcus.naslund@kognity.com>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Fixes#17776.
This PR also handles all other `PTH*` rules that don't support file
descriptors.
## Test Plan
<!-- How was it tested? -->
Update existing tests.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
We can only guarantee the safety of the autofix for number literals, all
other cases may change the runtime behaviour of the program or introduce
a syntax error. For the cases reported in the issue that would result in
a syntax error, I disabled the autofix.
Follow-up of #17661.
Fixes#16472.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Snapshot tests.
<!-- How was it tested? -->
Function literals have an optional specialization, which is applied to
the parameter/return type annotations lazily when the function's
signature is requested. We were previously only applying this
specialization to the final overload of an overloaded function.
This manifested most visibly for `list.__add__`, which has an overloaded
definition in the typeshed:
b398b83631/crates/ty_vendored/vendor/typeshed/stdlib/builtins.pyi (L1069-L1072)
Closes https://github.com/astral-sh/ty/issues/314
## Summary
I found this bug while working on #18041. The following code leads to
infinite recursion.
```python
from ty_extensions import is_disjoint_from, static_assert, TypeOf
class C:
@property
def prop(self) -> int:
return 1
static_assert(not is_disjoint_from(int, TypeOf[C.prop]))
```
The cause is a trivial missing binding in `is_disjoint_from`. This PR
fixes the bug and adds a test case (this is a simple fix and may not
require a new test case?).
## Test Plan
A new test case is added to
`mdtest/type_properties/is_disjoint_from.md`.
## Summary
Added version 3.14 to the script generating the `known_stdlib.rs` file.
Rebuilt the known stdlibs with latest version (2025.5.10) of [stdlibs
Python lib](https://pypi.org/project/stdlibs/) (which added support for
3.14.0b1).
_Note: Python 3.14 is now in [feature
freeze](https://peps.python.org/pep-0745/) so the modules in stdlib
should be stable._
_See also: #15506_
## Test Plan
The following command has been run. Using for tests the `compression`
module which been introduced with Python 3.14.
```sh
ruff check --no-cache --select I001 --target-version py314 --fix
```
With ruff 0.11.9:
```python
import base64
import datetime
import compression
print(base64, compression, datetime)
```
With this PR:
```python
import base64
import compression
import datetime
print(base64, compression, datetime)
```
## Summary
`KnownClass::Range`, `KnownInstanceType::Any` and `ClassBase::any()` are
no longer used or useful: all our tests pass with them removed.
`KnownModule::Abc` _is_ now used outside of tests, however, so I removed
the `#[allow(dead_code)]` branch above that variant.
## Test Plan
`cargo test -p ty_python_semantic`
Following #17991, removes some of
https://github.com/astral-sh/ruff/pull/17222 which is no longer strictly
necessary. I don't actually think it's that ugly to have around? no
strong feelings on retaining it or not.
Follow-up to https://github.com/astral-sh/ruff/pull/17991 ensuring we do
not allow detection of system environments when the origin is
`VIRTUAL_ENV` or a discovered `.venv` directory — i.e., those always
require a `pyvenv.cfg` file.
Adds test coverage for https://github.com/astral-sh/ruff/pull/17991,
which includes some minor refactoring of the virtual environment test
infrastructure.
I tried to minimize stylistic changes, but there are still a few because
I was a little confused by the setup. I could see this evolving more in
the future, as I don't think the existing model can capture all the test
coverage I'm looking for.
This adds basic support for non-virtual Python environments by accepting
a directory without a `pyvenv.cfg` which allows existing, subsequent
site-packages discovery logic to succeed. We can do better here in the
long-term, by adding more eager validation (for error messages) and
parsing the Python version from the discovered site-packages directory
(which isn't relevant yet, because we don't use the discovered Python
version from virtual environments as the default `--python-version` yet
either).
Related
- https://github.com/astral-sh/ty/issues/265
- https://github.com/astral-sh/ty/issues/193
You can review this commit by commit if it makes you happy.
I tested this manually; I think refactoring the test setup is going to
be a bit more invasive so I'll stack it on top (see
https://github.com/astral-sh/ruff/pull/17996).
```
❯ uv run ty check --python /Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/ -vv example
2025-05-09 12:06:33.685911 DEBUG Version: 0.0.0-alpha.7 (f9c4c8999 2025-05-08)
2025-05-09 12:06:33.685987 DEBUG Architecture: aarch64, OS: macos, case-sensitive: case-insensitive
2025-05-09 12:06:33.686002 DEBUG Searching for a project in '/Users/zb/workspace/ty'
2025-05-09 12:06:33.686123 DEBUG Resolving requires-python constraint: `>=3.8`
2025-05-09 12:06:33.686129 DEBUG Resolved requires-python constraint to: 3.8
2025-05-09 12:06:33.686142 DEBUG Project without `tool.ty` section: '/Users/zb/workspace/ty'
2025-05-09 12:06:33.686147 DEBUG Searching for a user-level configuration at `/Users/zb/.config/ty/ty.toml`
2025-05-09 12:06:33.686156 INFO Defaulting to python-platform `darwin`
2025-05-09 12:06:33.68636 INFO Python version: Python 3.8, platform: darwin
2025-05-09 12:06:33.686375 DEBUG Adding first-party search path '/Users/zb/workspace/ty'
2025-05-09 12:06:33.68638 DEBUG Using vendored stdlib
2025-05-09 12:06:33.686634 DEBUG Discovering site-packages paths from sys-prefix `/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none` (`--python` argument')
2025-05-09 12:06:33.686667 DEBUG Attempting to parse virtual environment metadata at '/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/pyvenv.cfg'
2025-05-09 12:06:33.686671 DEBUG Searching for site-packages directory in `sys.prefix` path `/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none`
2025-05-09 12:06:33.686702 DEBUG Resolved site-packages directories for this environment are: ["/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/lib/python3.10/site-packages"]
2025-05-09 12:06:33.686706 DEBUG Adding site-packages search path '/Users/zb/.local/share/uv/python/cpython-3.10.17-macos-aarch64-none/lib/python3.10/site-packages'
...
❯ uv run ty check --python /tmp -vv example
2025-05-09 15:36:10.819416 DEBUG Version: 0.0.0-alpha.7 (f9c4c8999 2025-05-08)
2025-05-09 15:36:10.819708 DEBUG Architecture: aarch64, OS: macos, case-sensitive: case-insensitive
2025-05-09 15:36:10.820118 DEBUG Searching for a project in '/Users/zb/workspace/ty'
2025-05-09 15:36:10.821652 DEBUG Resolving requires-python constraint: `>=3.8`
2025-05-09 15:36:10.821667 DEBUG Resolved requires-python constraint to: 3.8
2025-05-09 15:36:10.8217 DEBUG Project without `tool.ty` section: '/Users/zb/workspace/ty'
2025-05-09 15:36:10.821888 DEBUG Searching for a user-level configuration at `/Users/zb/.config/ty/ty.toml`
2025-05-09 15:36:10.822072 INFO Defaulting to python-platform `darwin`
2025-05-09 15:36:10.822439 INFO Python version: Python 3.8, platform: darwin
2025-05-09 15:36:10.822773 DEBUG Adding first-party search path '/Users/zb/workspace/ty'
2025-05-09 15:36:10.822929 DEBUG Using vendored stdlib
2025-05-09 15:36:10.829872 DEBUG Discovering site-packages paths from sys-prefix `/tmp` (`--python` argument')
2025-05-09 15:36:10.829911 DEBUG Attempting to parse virtual environment metadata at '/private/tmp/pyvenv.cfg'
2025-05-09 15:36:10.829917 DEBUG Searching for site-packages directory in `sys.prefix` path `/private/tmp`
ty failed
Cause: Invalid search path settings
Cause: Failed to discover the site-packages directory: Failed to search the `lib` directory of the Python installation at `sys.prefix` path `/private/tmp` for `site-packages`
```
## Summary
Suppress false positives for uses of PEP-695 `ParamSpec` in `Callable`
annotations:
```py
from typing_extensions import Callable
def f[**P](c: Callable[P, int]):
pass
```
addresses a comment here:
https://github.com/astral-sh/ty/issues/157#issuecomment-2859284721
## Test Plan
Adapted Markdown tests
Re: #17526
## Summary
Add integration test for semantic syntax for `IrrefutableCasePattern`,
`SingleStarredAssignment`, `WriteToDebug`, and `InvalidExpression`.
## Notes
- Following @ntBre's suggestion, I will keep the test coming in batches
like this over the next few days in separate PRs to keep the review load
per PR manageable while also not spamming too many.
- I did not add a test for `del __debug__` which is one of the examples
in `crates/ruff_python_parser/src/semantic_errors.rs:1051`.
For python version `<= 3.8` there is no error and for `>=3.9` the error
is not `WriteToDebug` but `SyntaxError: cannot delete __debug__ on
Python 3.9 (syntax was removed in 3.9)`.
- The `blacken-docs` bypass is necessary because otherwise the test does
not pass pre-commit checks; but we want to check for this faulty syntax.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
This is a test.
Summary
--
This was suggested on Discord, I hope this is roughly what we had in
mind. I took the message from the ty README, but I'm more than happy to
update it. Otherwise I just tried to mimic the appearance of the `ruff
analyze graph` warning (although I'm realizing now the whole text is
bold for ruff).
Test Plan
--
New warnings in the CLI tests. I thought this might be undesirable but
it looks like uv did the same thing
(https://github.com/astral-sh/uv/pull/6166).
This makes one very simple change: we report all call binding
errors from each union variant.
This does result in duplicate-seeming diagnostics. For example,
when two union variants are invalid for the same reason.
## Summary
Adds a simple progress bar for the `ty check` CLI command. The style is
taken from uv, and like uv the bar is always shown - for smaller
projects it is fast enough that it isn't noticeable. We could
alternatively hide it completely based on some heuristic for the number
of files, or only show it after some amount of time.
I also disabled it when `--watch` is passed, cancelling inflight checks
was leading to zombie progress bars. I think we can fix this by using
[`MultiProgress`](https://docs.rs/indicatif/latest/indicatif/struct.MultiProgress.html)
and managing all the bars globally, but I left that out for now.
Resolves https://github.com/astral-sh/ty/issues/98.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
* `airflow.models.Connection` → `airflow.sdk.Connection`
* `airflow.models.Variable` → `airflow.sdk.Variable`
## Test Plan
<!-- How was it tested? -->
The test fixtures has been updated (see the first commit for easier
review)
This does a deeper removal of the `lint:` prefix by removing the
`DiagnosticId::as_str` method and replacing it with `as_concise_str`. We
remove the associated error type and simplify the `Display` impl for
`DiagnosticId` as well.
This turned out to catch a `lint:` that was still in the diagnostic
output: the part that says why a lint is enabled.
We just set the ID on the `Message` and it just does what we want in
this case. I think I didn't do this originally because I was trying to
preserve the existing rendering? I'm not sure. I might have just missed
this method.
In a subsequent commit, we're going to start using `annotate-snippets`'s
functionality for diagnostic IDs in the rendering. As part of doing
that, I wanted to remove this special casing of an empty message. I did
that independently to see what, if anything, would change. (The changes
look fine to me. They'll be tweaked again in the next commit along with
a bunch of others.)
## Summary
Use a self-reference "marker" ~~and fixpoint iteration~~ to solve the
stack overflow problems with recursive protocols. This is not pretty and
somewhat tedious, but seems to work fine. Much better than all my
fixpoint-iteration attempts anyway.
closes https://github.com/astral-sh/ty/issues/93
## Test Plan
New Markdown tests.
## Summary
Add cycle handling for `try_metaclass` and `pep695_generic_context`
queries, as well as adjusting the cycle handling for `try_mro` to ensure
that it short-circuits on cycles and won't grow MROs indefinitely.
This reduces the number of failing fuzzer seeds from 68 to 17. The
latter count includes fuzzer seeds 120, 160, and 335, all of which
previously panicked but now either hang or are very slow; I've
temporarily skipped those seeds in the fuzzer until I can dig into that
slowness further.
This also allows us to move some more ecosystem projects from `bad.txt`
to `good.txt`, which I've done in
https://github.com/astral-sh/ruff/pull/17903
## Test Plan
Added mdtests.
@AlexWaygood pointed out that the `SliceLiteral` type variant was
originally created to handle slices before we had generics.
https://github.com/astral-sh/ruff/pull/17927#discussion_r2078115787
Now that we _do_ have generics, we can use a specialization of the
`slice` builtin type for slice literals.
This depends on https://github.com/astral-sh/ruff/pull/17956, since we
need to make sure that all typevar defaults are fully substituted when
specializing `slice`.
It's possible for a typevar to list another typevar as its default
value:
```py
class C[T, U = T]: ...
```
When specializing this class, if a type isn't provided for `U`, we would
previously use the default as-is, leaving an unspecialized `T` typevar
in the specialization. Instead, we want to use what `T` is mapped to as
the type of `U`.
```py
reveal_type(C()) # revealed: C[Unknown, Unknown]
reveal_type(C[int]()) # revealed: C[int, int]
reveal_type(C[int, str]()) # revealed: C[int, str]
```
This is especially important for the `slice` built-in type.
## Summary
This PR is a first step toward integration of the new `Diagnostic` type
into ruff. There are two main changes:
- A new `UnifiedFile` enum wrapping `File` for red-knot and a
`SourceFile` for ruff
- ruff's `Message::SyntaxError` variant is now a `Diagnostic` instead of
a `SyntaxErrorMessage`
The second of these changes was mostly just a proof of concept for the
first, and it went pretty smoothly. Converting `DiagnosticMessage`s will
be most of the work in replacing `Message` entirely.
## Test Plan
Existing tests, which show no changes.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Micha Reiser <micha@reiser.io>
Summary
--
This PR resolves both the typing-related and syntax error TODOs added in
#17563 by tracking a set of `global` bindings for each scope. As
discussed below, we avoid the additional AST traversal from ruff by
collecting `Name`s from `global` statements while building the semantic
index and emit a syntax error if the `Name` is already bound in the
current scope at the point of the `global` statement. This has the
downside of separating the error from the `SemanticSyntaxChecker`, but I
plan to explore using this approach in the `SemanticSyntaxChecker`
itself as a follow-up. It seems like this may be a better approach for
ruff as well.
Test Plan
--
Updated all of the related mdtests to remove the TODOs (and add quotes I
forgot on the messages).
There is one remaining TODO, but it requires `nonlocal` support, which
isn't even incorporated into the `SemanticSyntaxChecker` yet.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Fixes: astral-sh/ty#159
This PR adds support for using `Self` in methods.
When the type of an annotation is `TypingSelf` it is converted to a type
var based on:
https://typing.python.org/en/latest/spec/generics.html#self
I just skipped Protocols because it had more problems and the tests was
not useful.
Also I need to create a follow up PR that implicitly assumes `self`
argument has type `Self`.
In order to infer the type in the `in_type_expression` method I needed
to have scope id and semantic index available. I used the idea from
[this PR](https://github.com/astral-sh/ruff/pull/17589/files) to pass
additional context to this method.
Also I think in all places that `in_type_expression` is called we need
to have this context because `Self` can be there so I didn't split the
method into one version with context and one without.
## Test Plan
Added new tests from spec.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
Co-authored-by: Carl Meyer <carl@astral.sh>
#17897 added variance handling for legacy typevars — but they were only
being considered when checking generic aliases of the same class:
```py
class A: ...
class B(A): ...
class C[T]: ...
static_assert(is_subtype_of(C[B], C[A]))
```
and not for generic subclasses:
```py
class D[U](C[U]): ...
static_assert(is_subtype_of(D[B], C[A]))
```
Now we check those too!
Closes https://github.com/astral-sh/ty/issues/101
Fixes#17867
## Summary
The CPython parser does not allow generator expressions which are the
sole arguments in an argument list to have a trailing comma.
With this change, we start flagging such instances.
## Test Plan
Added new inline tests.
## Summary
We now expect the client to send initialization options to opt-in to
experimental (but LSP-standardized) features, like completion support.
Specifically, the client should set `"experimental.completions.enable":
true`.
Closes https://github.com/astral-sh/ty/issues/74.
## Summary
This PR adds support for the `__all__` module variable.
Reference spec:
https://typing.python.org/en/latest/spec/distributing.html#library-interface-public-and-private-symbols
This PR adds a new `dunder_all_names` query that returns a set of
`Name`s defined in the `__all__` variable of the given `File`. The query
works by implementing the `StatementVisitor` and collects all the names
by recognizing the supported idioms as mentioned in the spec. Any idiom
that's not recognized are ignored.
The current implementation is minimum to what's required for us to
remove all the false positives that this is causing. Refer to the
"Follow-ups" section below to see what we can do next. I'll a open
separate issue to keep track of them.
Closes: astral-sh/ty#106Closes: astral-sh/ty#199
### Follow-ups
* Diagnostics:
* Add warning diagnostics for unrecognized `__all__` idioms, `__all__`
containing non-string element
* Add an error diagnostic for elements that are present in `__all__` but
not defined in the module. This could lead to runtime error
* Maybe we should return `<type>` instead of `Unknown | <type>` for
`module.__all__`. For example:
https://playknot.ruff.rs/2a6fe5d7-4e16-45b1-8ec3-d79f2d4ca894
* Mark a symbol that's mentioned in `__all__` as used otherwise it could
raise (possibly in the future) "unused-name" diagnostic
Supporting diagnostics will require that we update the return type of
the query to be something other than `Option<FxHashSet<Name>>`,
something that behaves like a result and provides a way to check whether
a name exists in `__all__`, loop over elements in `__all__`, loop over
the invalid elements, etc.
## Ecosystem analysis
The following are the maximum amount of diagnostics **removed** in the
ecosystem:
* "Type <module '...'> has no attribute ..."
* `collections.abc` - 14
* `numpy` - 35534
* `numpy.ma` - 296
* `numpy.char` - 37
* `numpy.testing` - 175
* `hashlib` - 311
* `scipy.fft` - 2
* `scipy.stats` - 38
* "Module '...' has no member ..."
* `collections.abc` - 85
* `numpy` - 508
* `numpy.testing` - 741
* `hashlib` - 36
* `scipy.stats` - 68
* `scipy.interpolate` - 7
* `scipy.signal` - 5
The following modules have dynamic `__all__` definition, so `ty` assumes
that `__all__` doesn't exists in that module:
* `scipy.stats`
(95a5d6ea8b/scipy/stats/__init__.py (L665))
* `scipy.interpolate`
(95a5d6ea8b/scipy/interpolate/__init__.py (L221))
* `scipy.signal` (indirectly via
95a5d6ea8b/scipy/signal/_signal_api.py (L30))
* `numpy.testing`
(de784cd6ee/numpy/testing/__init__.py (L16-L18))
~There's this one category of **false positives** that have been added:~
Fixed the false positives by also ignoring `__all__` from a module that
uses unrecognized idioms.
<details><summary>Details about the false postivie:</summary>
<p>
The `scipy.stats` module has dynamic `__all__` and it imports a bunch of
symbols via star imports. Some of those modules have a mix of valid and
invalid `__all__` idioms. For example, in
95a5d6ea8b/scipy/stats/distributions.py (L18-L24),
2 out of 4 `__all__` idioms are invalid but currently `ty` recognizes
two of them and says that the module has a `__all__` with 5 values. This
leads to around **2055** newly added false positives of the form:
```
Type <module 'scipy.stats'> has no attribute ...
```
I think the fix here is to completely ignore `__all__`, not only if
there are invalid elements in it, but also if there are unrecognized
idioms used in the module.
</p>
</details>
## Test Plan
Add a bunch of test cases using the new `ty_extensions.dunder_all_names`
function to extract a module's `__all__` names.
Update various test cases to remove false positives around `*` imports
and re-export convention.
Add new test cases for named import behavior as `*` imports covers all
of it already (thanks Alex!).
## Summary
Fixes#17541
Before this change, in the case of overloaded functions,
`@dataclass_transform` was detected only when applied to the
implementation, not the overloads.
However, the spec also allows this decorator to be applied to any of the
overloads as well.
With this PR, we start handling `@dataclass_transform`s applied to
overloads.
## Test Plan
Fixed existing TODOs in the test suite.
## Summary
This is sort of an anticlimactic resolution to #17863, but now that we
understand what the root cause for the stack overflows was, I think it's
fine to enable running on this project. See the linked ticket for the
full analysis.
closes#17863
## Test Plan
Ran lots of times locally and never observed a crash at worker thread
stack sizes > 8 MiB.
We now track the variance of each typevar, and obey the `covariant` and
`contravariant` parameters to the legacy `TypeVar` constructor. We still
don't yet infer variance for PEP-695 typevars or for the
`infer_variance` legacy constructor parameter.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
A recursive protocol like the following would previously lead to stack
overflows when attempting to create the union type for the `P | None`
member, because `UnionBuilder` checks if element types are fully static,
and the fully-static check on `P` would in turn list all members and
check whether all of them were fully static, leading to a cycle.
```py
from __future__ import annotations
from typing import Protocol
class P(Protocol):
parent: P | None
```
Here, we make the fully-static check on protocols a salsa query and add
fixpoint iteration, starting with `true` as the initial value (assume
that the recursive protocol is fully-static). If the recursive protocol
has any non-fully-static members, we still return `false` when
re-executing the query (see newly added tests).
closes#17861
## Test Plan
Added regression test
## Summary
Resolves#15502.
`ty generate-shell-completion` now works in a similar manner to `ruff
generate-shell-completion`.
## Test Plan
Manually:
<details>
```shell
$ cargo run --package ty generate-shell-completion nushell
module completions {
# An extremely fast Python type checker.
export extern ty [
--help(-h) # Print help
--version(-V) # Print version
]
# ...
}
export use completions *
```
</details>
@AlexWaygood discovered that even though we've been propagating
specializations to _parent_ base classes correctly, we haven't been
passing them on to _grandparent_ base classes:
https://github.com/astral-sh/ruff/pull/17832#issuecomment-2854360969
```py
class Bar[T]:
x: T
class Baz[T](Bar[T]): ...
class Spam[T](Baz[T]): ...
reveal_type(Spam[int]().x) # revealed: `T`, but should be `int`
```
This PR updates the MRO machinery to apply the current specialization
when starting to iterate the MRO of each base class.
## Summary
This PR partially addresses #16418 via the following:
- `LinterSettings::unresolved_python_version` is now a `TargetVersion`,
which is a thin wrapper around an `Option<PythonVersion>`
- `Checker::target_version` now calls `TargetVersion::linter_version`
internally, which in turn uses `unwrap_or_default` to preserve the
current default behavior
- Calls to the parser now call `TargetVersion::parser_version`, which
calls `unwrap_or_else(PythonVersion::latest)`
- The `Checker`'s implementation of
`SemanticSyntaxContext::python_version` also uses
`TargetVersion::parser_version` to use `PythonVersion::latest` for
semantic errors
In short, all lint rule behavior should be unchanged, but we default to
the latest Python version for the new syntax errors, which should
minimize confusing version-related syntax errors for users without a
version configured.
## Test Plan
Existing tests, which showed no changes (except for printing default
settings).
## Summary
Introducing a new rule based on discussions in #15732 and #15729 that
checks for unnecessary in with empty collections.
I called it in_empty_collection and gave the rule number RUF060.
Rule is in preview group.
e.g.,
```
❯ uv run -q -- ty -V
ty 0.0.0-alpha.4 (08881edba 2025-05-05)
❯ uv run -q -- ty --version
ty 0.0.0-alpha.4 (08881edba 2025-05-05)
```
Previously, this just displayed `ty 0.0.0` because it didn't use our
custom version implementation. We no longer have a short version —
matching the interface in uv. We could add a variant for it, if it seems
important to people. However, I think we found it more confusing than
not over there and didn't get any complaints about the change.
Closes https://github.com/astral-sh/ty/issues/54
Extends https://github.com/astral-sh/ruff/pull/17866, using
`dist-workspace.toml` as a source of truth for versions to enable
version retrieval in distributions that are not Git repositories (i.e.,
Python source distributions and source tarballs consumed by Linux
distros).
I retain the Git tag lookup from
https://github.com/astral-sh/ruff/pull/17866 as a fallback — it seems
harmless, but we could drop it to simplify things here.
I confirmed this works from the repository as well as Python source and
binary distributions:
```
❯ uv run --refresh-package ty --reinstall-package ty -q -- ty version
ty 0.0.1-alpha.1+5 (2eadc9e61 2025-05-05)
❯ uv build
...
❯ uvx --from ty@dist/ty-0.0.0a1.tar.gz --no-cache -q -- ty version
ty 0.0.1-alpha.1
❯ uvx --from ty@dist/ty-0.0.0a1-py3-none-macosx_11_0_arm64.whl -q -- ty version
ty 0.0.1-alpha.1
```
Requires https://github.com/astral-sh/ty/pull/36
cc @Gankra and @MichaReiser for review.
Currently, `ty version` pulls its information from the Ruff repository —
but we want this to pull from the repository in the directory _above_
when Ruff is a submodule.
I tested this in the `ty` repository after tagging an arbitrary commit:
```
❯ uv run --refresh-package ty --reinstall-package ty ty version
Built ty @ file:///Users/zb/workspace/ty
Uninstalled 1 package in 2ms
Installed 1 package in 1ms
ty 0.0.0+3 (34253b1d4 2025-05-05)
```
We also use the last Git tag as the source of truth for the version,
instead of the crate version. However, we'll need a way to set the
version for releases still, as the tag is published _after_ the build.
We can either tag early (without pushing the tag to the remote), or add
another environment variable. (**Note, this approach is changed in a
follow-up. See https://github.com/astral-sh/ruff/pull/17868**)
From this repository, the version will be `unknown`:
```
❯ cargo run -q --bin ty -- version
ty unknown
```
We could add special handling like... `ty unknown (ruff@...)` but I see
that as a secondary goal.
Closes https://github.com/astral-sh/ty/issues/5
The reviewer situation in this repository is unhinged, cc @Gankra and
@MichaReiser for review.
## Summary
This fixes some false positives that showed up in the primer diff for
https://github.com/astral-sh/ruff/pull/17832
## Test Plan
new mdtests added that fail with false-positive diagnostics on `main`
## Summary
This PR fixes#17595.
## Test Plan
New test cases are added to `mdtest/narrow/conditionals/nested.md`.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
If a typevar is declared as having a default, we shouldn't require a
type to be specified for that typevar when explicitly specializing a
generic class:
```py
class WithDefault[T, U = int]: ...
reveal_type(WithDefault[str]()) # revealed: WithDefault[str, int]
```
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Fixes
https://github.com/astral-sh/ruff/pull/17832#issuecomment-2851224968. We
had a comment that we did not need to apply specializations to generic
aliases, or to the bound `self` of a bound method, because they were
already specialized. But they might be specialized with a type variable,
which _does_ need to be specialized, in the case of a "multi-step"
specialization, such as:
```py
class LinkedList[T]: ...
class C[U]:
def method(self) -> LinkedList[U]:
return LinkedList[U]()
```
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
closes#17472
This is obviously just a band-aid solution to this problem (in that you
can always make your [pathological
inputs](28994edd82/sympy/polys/numberfields/resolvent_lookup.py)
bigger and it will still crash), but I think this is not an unreasonable
change — even if we add more sophisticated solutions later. I tried
using `stacker` as suggested by @MichaReiser, and it works. But it's
unclear where exactly would be the right place to put it, and even for
the `sympy` problem, we would need to add it both in the semantic index
builder AST traversal and in type inference. Increasing the default
stack size for worker threads, as proposed here, doesn't solve the
underlying problem (that there is a hard limit), but it is more
universal in the sense that it is not specific to large binary-operator
expression chains.
To determine a reasonable stack size, I created files that look like
*right associative*:
```py
from typing import reveal_type
total = (1 + (1 + (1 + (1 + (… + 1)))))
reveal_type(total)
```
*left associative*
```py
from typing import reveal_type
total = 1 + 1 + 1 + 1 + … + 1
reveal_type(total)
```
with a variable amount of operands (`N`). I then chose the stack size
large enough to still be able to handle cases that existing type
checkers can not:
```
right
N = 20: mypy takes ~ 1min
N = 350: pyright crashes with a stack overflow (mypy fails with "too many nested parentheses")
N = 800: ty(main) infers Literal[800] instantly
N = 1000: ty(main) crashes with "thread '<unknown>' has overflowed its stack"
N = 7000: ty(this branch) infers Literal[7000] instantly
N = 8000+: ty(this branch) crashes
left
N = 300: pyright emits "Maximum parse depth exceeded; break expression into smaller sub-expressions"
total is inferred as Unknown
N = 5500: mypy crashes with "INTERNAL ERROR"
N = 2500: ty(main) infers Literal[2500] instantly
N = 3000: ty(main) crashes with "thread '<unknown>' has overflowed its stack"
N = 22000: ty(this branch) infers Literal[22000] instantly
N = 23000+: ty(this branch) crashes
```
## Test Plan
New regression test.
This fixes cycle panics in several ecosystem projects (moved to
`good.txt` in a following PR
https://github.com/astral-sh/ruff/pull/17834 because our mypy-primer job
doesn't handle it well if we move projects to `good.txt` in the same PR
that fixes `ty` to handle them), as well as in the minimal case in the
added mdtest. It also fixes a number of panicking fuzzer seeds. It
doesn't appear to cause any regression in any ecosystem project or any
fuzzer seed.
The PR add the fix safety section for rule `RUF013`
(https://github.com/astral-sh/ruff/issues/15584 )
The fix was introduced here #4831
The rule as a lot of False Negative (as it is explained in the docs of
the rule).
The main reason because the fix is unsafe is that it could change code
generation tools behaviour, as in the example here:
```python
def generate_api_docs(func):
hints = get_type_hints(func)
for param, hint in hints.items():
if is_optional_type(hint):
print(f"Parameter '{param}' is optional")
else:
print(f"Parameter '{param}' is required")
# Before fix
def create_user(name: str, roles: list[str] = None):
pass
# After fix
def create_user(name: str, roles: Optional[list[str]] = None):
pass
# Generated docs would change from "roles is required" to "roles is optional"
```
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
Re: #17526
## Summary
Add test fixtures for `AwaitOutsideAsync` and
`AsyncComprehensionOutsideAsyncFunction` errors.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
This is a test.
<!-- How was it tested? -->
Re: #17526
## Summary
Add integration tests for Python Semantic Syntax for
`InvalidStarExpression`, `DuplicateMatchKey`, and
`DuplicateMatchClassAttribute`.
## Note
- Red knot integration tests for `DuplicateMatchKey` exist already in
line 89-101.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
This is a test.
<!-- How was it tested? -->
When attempting to determine whether `import foo.bar.baz` is a known
first-party import relative to [user-provided source
paths](https://docs.astral.sh/ruff/settings/#src), when `preview` is
enabled we now check that `SRC/foo/bar/baz` is a directory or
`SRC/foo/bar/baz.py` or `SRC/foo/bar/baz.pyi` exist.
Previously, we just checked the analogous thing for `SRC/foo`, but this
can be misleading in situations with disjoint namespace packages that
share a common base name (e.g. we may be working inside the namespace
package `foo.buzz` and importing `foo.bar` from elsewhere).
Supersedes #12987Closes#12984
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#17798
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Snapshot tests
<!-- How was it tested? -->
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Skip attribute check in try catch block (`AIR301`)
## Test Plan
<!-- How was it tested? -->
update
`crates/ruff_linter/resources/test/fixtures/airflow/AIR301_names_try.py`
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Remove `airflow.utils.dag_parsing_context.get_parsing_context` from
AIR301 as it has been moved to AIR311
## Test Plan
<!-- How was it tested? -->
the test fixture was updated in the previous PR
## Summary
When entering an `infer_expression_types` cycle from
`TypeInferenceBuilder::infer_standalone_expression`, we might get back a
`TypeInference::cycle_fallback(…)` that doesn't actually contain any new
types, but instead it contains a `cycle_fallback_type` which is set to
`Some(Type::Never)`. When calling `self.extend(…)`, we therefore don't
really pull in a type for the expression we're interested in. This
caused us to panic if we tried to call `self.expression_type(…)` after
`self.extend(…)`.
The proposed fix here is to retrieve that type from the nested
`TypeInferenceBuilder` directly, which will correctly fall back to
`cycle_fallback_type`.
## Details
I minimized the second example from #17792 a bit further and used this
example for debugging:
```py
from __future__ import annotations
class C: ...
def f(arg: C):
pass
x, _ = f(1)
assert x
```
This is self-referential because when we check the assignment statement
`x, _ = f(1)`, we need to look up the signature of `f`. Since evaluation
of annotations is deferred, we look up the public type of `C` for the
`arg` parameter. The public use of `C` is visibility-constraint by "`x`"
via the `assert` statement. While evaluating this constraint, we need to
look up the type of `x`, which in turn leads us back to the `x, _ =
f(1)` definition.
The reason why this only showed up in the relatively peculiar case with
unpack assignments is the code here:
78b4c3ccf1/crates/ty_python_semantic/src/types/infer.rs (L2709-L2718)
For a non-unpack assignment like `x = f(1)`, we would not try to infer
the right-hand side eagerly. Instead, we would enter a
`infer_definition_types` cycle that handles the situation correctly. For
unpack assignments, however, we try to infer the type of `value`
(`f(1)`) and therefore enter the cycle via `standalone_expression_type
=> infer_expression_type`.
closes#17792
## Test Plan
* New regression test
* Made sure that we can now run successfully on scipy => see #17850
This PR updates the semantic model for Python 3.14 by essentially
equating "run using Python 3.14" with "uses `from __future__ import
annotations`".
While this is not technically correct under the hood, it appears to be
correct for the purposes of our semantic model. That is: from the point
of view of deciding when to parse, bind, etc. annotations, these two
contexts behave the same. More generally these contexts behave the same
unless you are performing some kind of introspection like the following:
Without future import:
```pycon
>>> from annotationlib import get_annotations,Format
>>> def foo()->Bar:...
...
>>> get_annotations(foo,format=Format.FORWARDREF)
{'return': ForwardRef('Bar')}
>>> get_annotations(foo,format=Format.STRING)
{'return': 'Bar'}
>>> get_annotations(foo,format=Format.VALUE)
Traceback (most recent call last):
[...]
NameError: name 'Bar' is not defined
>>> get_annotations(foo)
Traceback (most recent call last):
[...]
NameError: name 'Bar' is not defined
```
With future import:
```
>>> from __future__ import annotations
>>> from annotationlib import get_annotations,Format
>>> def foo()->Bar:...
...
>>> get_annotations(foo,format=Format.FORWARDREF)
{'return': 'Bar'}
>>> get_annotations(foo,format=Format.STRING)
{'return': 'Bar'}
>>> get_annotations(foo,format=Format.VALUE)
{'return': 'Bar'}
>>> get_annotations(foo)
{'return': 'Bar'}
```
(Note: the result of the last call to `get_annotations` in these
examples relies on the fact that, as of this writing, the default value
for `format` is `Format.VALUE`).
If one day we support lint rules targeting code that introspects using
the new `annotationlib`, then it is possible we will need to revisit our
approximation.
Closes#15100
## Summary
Currently red-knot does not understand `Foo` and `Bar` here as being
equivalent:
```py
from typing import Protocol
class A: ...
class B: ...
class C: ...
class Foo(Protocol):
x: A | B | C
class Bar(Protocol):
x: B | A | C
```
Nor does it understand `A | B | Foo` as being equivalent to `Bar | B |
A`. This PR fixes that.
## Test Plan
new mdtest assertions added that fail on `main`
## Summary
Currently this assertion fails on `main`, because we do not synthesize a
`__call__` attribute for Callable types:
```py
from typing import Protocol, Callable
from knot_extensions import static_assert, is_assignable_to
class Foo(Protocol):
def __call__(self, x: int, /) -> str: ...
static_assert(is_assignable_to(Callable[[int], str], Foo))
```
This PR fixes that.
See previous discussion about this in
https://github.com/astral-sh/ruff/pull/16493#discussion_r1985098508 and
https://github.com/astral-sh/ruff/pull/17682#issuecomment-2839527750
## Test Plan
Existing mdtests updated; a couple of new ones added.
This adds support for legacy generic classes, which use a
`typing.Generic` base class, or which inherit from another generic class
that has been specialized with legacy typevars.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Quick follow-on to #17788. If there is no bound `self` parameter, we can
reuse the existing `CallArgument{,Type}s`, and we can use a straight
`Vec` instead of a `VecDeque`.
## Summary
Remove mutability in parameter types for a few functions such as
`with_self` and `try_call`. I tried the `Rc`-approach with cheap cloning
[suggest
here](https://github.com/astral-sh/ruff/pull/17733#discussion_r2068722860)
first, but it turns out we need a whole stack of prepended arguments
(there can be [both `self` *and*
`cls`](3cf44e401a/crates/red_knot_python_semantic/resources/mdtest/call/constructor.md?plain=1#L113)),
and we would need the same construct not just for `CallArguments` but
also for `CallArgumentTypes`. At that point we're cloning `VecDeque`s
anyway, so the overhead of cloning the whole `VecDeque` with all
arguments didn't seem to justify the additional code complexity.
## Benchmarks
Benchmarks on tomllib, black, jinja, isort seem neutral.
## Summary
Add the ability to detect instance attribute assignments in class
methods that are generic.
This does not address the code duplication mentioned in #16928. I can
open a ticket for this after this has been merged.
closes#16928
## Test Plan
Added regression test.
This PR does the wiring necessary to respond to completion requests from
LSP clients.
As far as the actual completion results go, they are nearly about the
dumbest and simplest thing we can do: we simply return a de-duplicated
list of all identifiers from the current module.
Summary
--
Fixes#16598 by adding the `--python` flag to `ruff analyze graph`,
which adds a `PythonPath` to the `SearchPathSettings` for module
resolution. For the [albatross-virtual-workspace] example from the uv
repo, this updates the output from the initial issue:
```shell
> ruff analyze graph packages/albatross
{
"packages/albatross/check_installed_albatross.py": [
"packages/albatross/src/albatross/__init__.py"
],
"packages/albatross/src/albatross/__init__.py": []
}
```
To include both the the workspace `bird_feeder` import _and_ the
third-party `tqdm` import in the output:
```shell
> myruff analyze graph packages/albatross --python .venv
{
"packages/albatross/check_installed_albatross.py": [
"packages/albatross/src/albatross/__init__.py"
],
"packages/albatross/src/albatross/__init__.py": [
".venv/lib/python3.12/site-packages/tqdm/__init__.py",
"packages/bird-feeder/src/bird_feeder/__init__.py"
]
}
```
Note the hash in the uv link! I was temporarily very confused why my
local tests were showing an `iniconfig` import instead of `tqdm` until I
realized that the example has been updated on the uv main branch, which
I had locally.
Test Plan
--
A new integration test with a stripped down venv based on the
`albatross` example.
[albatross-virtual-workspace]:
aa629c4a54/scripts/workspaces/albatross-virtual-workspace
## Summary
Contains the same changes to the semantic type inference as
https://github.com/astral-sh/ruff/pull/17705.
Fixes#17694
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
<!-- How was it tested? -->
Snapshot tests.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
Adds preliminary support for `NamedTuple`s, including:
* No false positives when constructing a `NamedTuple` object
* Correct signature for the synthesized `__new__` method, i.e. proper
checking of constructor calls
* A patched MRO (`NamedTuple` => `tuple`), mainly to make type inference
of named attributes possible, but also to better reflect the runtime
MRO.
All of this works:
```py
from typing import NamedTuple
class Person(NamedTuple):
id: int
name: str
age: int | None = None
alice = Person(1, "Alice", 42)
alice = Person(id=1, name="Alice", age=42)
reveal_type(alice.id) # revealed: int
reveal_type(alice.name) # revealed: str
reveal_type(alice.age) # revealed: int | None
# error: [missing-argument]
Person(3)
# error: [too-many-positional-arguments]
Person(3, "Eve", 99, "extra")
# error: [invalid-argument-type]
Person(id="3", name="Eve")
```
Not included:
* type inference for index-based access.
* support for the functional `MyTuple = NamedTuple("MyTuple", […])`
syntax
## Test Plan
New Markdown tests
## Ecosystem analysis
```
Diagnostic Analysis Report
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━┓
┃ Diagnostic ID ┃ Severity ┃ Removed ┃ Added ┃ Net Change ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━┩
│ lint:call-non-callable │ error │ 0 │ 3 │ +3 │
│ lint:call-possibly-unbound-method │ warning │ 0 │ 4 │ +4 │
│ lint:invalid-argument-type │ error │ 0 │ 72 │ +72 │
│ lint:invalid-context-manager │ error │ 0 │ 2 │ +2 │
│ lint:invalid-return-type │ error │ 0 │ 2 │ +2 │
│ lint:missing-argument │ error │ 0 │ 46 │ +46 │
│ lint:no-matching-overload │ error │ 19121 │ 0 │ -19121 │
│ lint:not-iterable │ error │ 0 │ 6 │ +6 │
│ lint:possibly-unbound-attribute │ warning │ 13 │ 32 │ +19 │
│ lint:redundant-cast │ warning │ 0 │ 1 │ +1 │
│ lint:unresolved-attribute │ error │ 0 │ 10 │ +10 │
│ lint:unsupported-operator │ error │ 3 │ 9 │ +6 │
│ lint:unused-ignore-comment │ warning │ 15 │ 4 │ -11 │
├───────────────────────────────────┼──────────┼─────────┼───────┼────────────┤
│ TOTAL │ │ 19152 │ 191 │ -18961 │
└───────────────────────────────────┴──────────┴─────────┴───────┴────────────┘
Analysis complete. Found 13 unique diagnostic IDs.
Total diagnostics removed: 19152
Total diagnostics added: 191
Net change: -18961
```
I uploaded the ecosystem full diff (ignoring the 19k
`no-matching-overload` diagnostics)
[here](https://shark.fish/diff-namedtuple.html).
* There are some new `missing-argument` false positives which come from
the fact that named tuples are often created using unpacking as in
`MyNamedTuple(*fields)`, which we do not understand yet.
* There are some new `unresolved-attribute` false positives, because
methods like `_replace` are not available.
* Lots of the `invalid-argument-type` diagnostics look like true
positives
---------
Co-authored-by: Douglas Creager <dcreager@dcreager.net>
## Summary
This PR updates the existing overload matching methods to return an
iterator of all the matched overloads instead.
This would be useful once the overload call evaluation algorithm is
implemented which should provide an accurate picture of all the matched
overloads. The return type would then be picked from either the only
matched overload or the first overload from the ones that are matched.
In an earlier version of this PR, it tried to check if using an
intersection of return types from the matched overload would help reduce
the false positives but that's not enough. [This
comment](https://github.com/astral-sh/ruff/pull/17618#issuecomment-2842891696)
keep the ecosystem analysis for that change for prosperity.
> [!NOTE]
>
> The best way to review this PR is by hiding the whitespace changes
because there are two instances where a large match expression is
indented to be inside a loop over matching overlods
>
> <img width="1207" alt="Screenshot 2025-04-28 at 15 12 16"
src="https://github.com/user-attachments/assets/e06cbfa4-04fa-435f-84ef-4e5c3c5626d1"
/>
## Test Plan
Make sure existing test cases are unaffected and no ecosystem changes.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This is not yet fixing anything as the names are not changed, but it
lays down the foundation for fixing.
## Test Plan
<!-- How was it tested? -->
the existing test fixture should already cover this change
## Summary
Part of #17412
Starred expressions cannot be used as values in assignment expressions.
Add a new semantic syntax error to catch such instances.
Note that we already have
`ParseErrorType::InvalidStarredExpressionUsage` to catch some starred
expression errors during parsing, but that does not cover top level
assignment expressions.
## Test Plan
- Added new inline tests for the new rule
- Found some examples marked as "valid" in existing tests (`_ = *data`),
which are not really valid (per this new rule) and updated them
- There was an existing inline test - `assign_stmt_invalid_value_expr`
which had instances of `*` expression which would be deemed invalid by
this new rule. Converted these to tuples, so that they do not trigger
this new rule.
## Summary
Model the lookup of `__new__` without going through
`Type::try_call_dunder`. The `__new__` method is only looked up on the
constructed type itself, not on the meta-type.
This now removes ~930 false positives across the ecosystem (vs 255 for
https://github.com/astral-sh/ruff/pull/17662). It introduces 30 new
false positives related to the construction of enums via something like
`Color = enum.Enum("Color", ["RED", "GREEN"])`. This is expected,
because we don't handle custom metaclass `__call__` methods. The fact
that we previously didn't emit diagnostics there was a coincidence (we
incorrectly called `EnumMeta.__new__`, and since we don't fully
understand its signature, that happened to work with `str`, `list`
arguments).
closes#17462
## Test Plan
Regression test
## Summary
Part of #15383.
As per the spec
(https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions):
For `@staticmethod` and `@classmethod`:
> If one overload signature is decorated with `@staticmethod` or
`@classmethod`, all overload signatures must be similarly decorated. The
implementation, if present, must also have a consistent decorator. Type
checkers should report an error if these conditions are not met.
For `@final` and `@override`:
> If a `@final` or `@override` decorator is supplied for a function with
overloads, the decorator should be applied only to the overload
implementation if it is present. If an overload implementation isn’t
present (for example, in a stub file), the `@final` or `@override`
decorator should be applied only to the first overload. Type checkers
should enforce these rules and generate an error when they are violated.
If a `@final` or `@override` decorator follows these rules, a type
checker should treat the decorator as if it is present on all overloads.
## Test Plan
Update existing tests; add snapshots.
## Summary
As mentioned in the spec
(https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions),
part of #15383:
> The `@overload`-decorated definitions must be followed by an overload
implementation, which does not include an `@overload` decorator. Type
checkers should report an error or warning if an implementation is
missing. Overload definitions within stub files, protocols, and on
abstract methods within abstract base classes are exempt from this
check.
## Test Plan
Remove TODOs from the test; create one diagnostic snapshot.
Re: #17526
## Summary
Adds tests to red knot and `linter.rs` for the semantic syntax.
Specifically add tests for `ReboundComprehensionVariable`,
`DuplicateTypeParameter`, and `MultipleCaseAssignment`.
Refactor the `test_async_comprehension_in_sync_comprehension` →
`test_semantic_error` to be more general for all semantic syntax test
cases.
## Test Plan
This is a test.
## Question
I'm happy to contribute more tests the coming days.
Should that happen here or should we merge this PR such that the
refactor `test_async_comprehension_in_sync_comprehension` →
`test_semantic_error` is available on main and others can chime in, too?
## Summary
Part of #15383, this PR adds the core infrastructure to check for
invalid overloads and adds a diagnostic to raise if there are < 2
overloads for a given definition.
### Design notes
The requirements to check the overloads are:
* Requires `FunctionType` which has the `to_overloaded` method
* The `FunctionType` **should** be for the function that is either the
implementation or the last overload if the implementation doesn't exists
* Avoid checking any `FunctionType` that are part of an overload chain
* Consider visibility constraints
This required a couple of iteration to make sure all of the above
requirements are fulfilled.
#### 1. Use a set to deduplicate
The logic would first collect all the `FunctionType` that are part of
the overload chain except for the implementation or the last overload if
the implementation doesn't exists. Then, when iterating over all the
function declarations within the scope, we'd avoid checking these
functions. But, this approach would fail to consider visibility
constraints as certain overloads _can_ be behind a version check. Those
aren't part of the overload chain but those aren't a separate overload
chain either.
<details><summary>Implementation:</summary>
<p>
```rs
fn check_overloaded_functions(&mut self) {
let function_definitions = || {
self.types
.declarations
.iter()
.filter_map(|(definition, ty)| {
// Filter out function literals that result from anything other than a function
// definition e.g., imports.
if let DefinitionKind::Function(function) = definition.kind(self.db()) {
ty.inner_type()
.into_function_literal()
.map(|ty| (ty, definition.symbol(self.db()), function.node()))
} else {
None
}
})
};
// A set of all the functions that are part of an overloaded function definition except for
// the implementation function and the last overload in case the implementation doesn't
// exists. This allows us to collect all the function definitions that needs to be skipped
// when checking for invalid overload usages.
let mut overloads: HashSet<FunctionType<'db>> = HashSet::default();
for (function, _) in function_definitions() {
let Some(overloaded) = function.to_overloaded(self.db()) else {
continue;
};
if overloaded.implementation.is_some() {
overloads.extend(overloaded.overloads.iter().copied());
} else if let Some((_, previous_overloads)) = overloaded.overloads.split_last() {
overloads.extend(previous_overloads.iter().copied());
}
}
for (function, function_node) in function_definitions() {
let Some(overloaded) = function.to_overloaded(self.db()) else {
continue;
};
if overloads.contains(&function) {
continue;
}
// At this point, the `function` variable is either the implementation function or the
// last overloaded function if the implementation doesn't exists.
if overloaded.overloads.len() < 2 {
if let Some(builder) = self
.context
.report_lint(&INVALID_OVERLOAD, &function_node.name)
{
let mut diagnostic = builder.into_diagnostic(format_args!(
"Function `{}` requires at least two overloads",
&function_node.name
));
if let Some(first_overload) = overloaded.overloads.first() {
diagnostic.annotate(
self.context
.secondary(first_overload.focus_range(self.db()))
.message(format_args!("Only one overload defined here")),
);
}
}
}
}
}
```
</p>
</details>
#### 2. Define a `predecessor` query
The `predecessor` query would return the previous `FunctionType` for the
given `FunctionType` i.e., the current logic would be extracted to be a
query instead. This could then be used to make sure that we're checking
the entire overload chain once. The way this would've been implemented
is to have a `to_overloaded` implementation which would take the root of
the overload chain instead of the leaf. But, this would require updates
to the use-def map to somehow be able to return the _following_
functions for a given definition.
#### 3. Create a successor link
This is what Pyrefly uses, we'd create a forward link between two
functions that are involved in an overload chain. This means that for a
given function, we can get the successor function. This could be used to
find the _leaf_ of the overload chain which can then be used with the
`to_overloaded` method to get the entire overload chain. But, this would
also require updating the use-def map to be able to "see" the
_following_ function.
### Implementation
This leads us to the final implementation that this PR implements which
is to consider the overloaded functions using:
* Collect all the **function symbols** that are defined **and** called
within the same file. This could potentially be an overloaded function
* Use the public bindings to get the leaf of the overload chain and use
that to get the entire overload chain via `to_overloaded` and perform
the check
This has a limitation that in case a function redefines an overload,
then that overload will not be checked. For example:
```py
from typing import overload
@overload
def f() -> None: ...
@overload
def f(x: int) -> int: ...
# The above overload will not be checked as the below function with the same name
# shadows it
def f(*args: int) -> int: ...
```
## Test Plan
Update existing mdtest and add snapshot diagnostics.
## Summary
@sharkdp and I realised in our 1:1 this morning that our control flow
for `assert` statements isn't quite accurate at the moment. Namely, for
something like this:
```py
def _(x: int | None):
assert x is None, reveal_type(x)
```
we currently reveal `None` for `x` here, but this is incorrect. In
actual fact, the `msg` expression of an `assert` statement (the
expression after the comma) will only be evaluated if the test (`x is
None`) evaluates to `False`. As such, we should be adding a constraint
of `~None` to `x` in the `msg` expression, which should simplify the
inferred type of `x` to `int` in that context (`(int | None) & ~None` ->
`int`).
## Test Plan
Mdtests added.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
We were previously recording wrong reachability constraints for negative
branches. Instead of `[cond] AND (NOT [True])` below, we were recording
`[cond] AND (NOT ([cond] AND [True]))`, i.e. we were negating not just
the last predicate, but the `AND`-ed reachability constraint from last
clause. With this fix, we now record the correct constraints for the
example from #17723:
```py
def _(cond: bool):
if cond:
# reachability: [cond]
if True:
# reachability: [cond] AND [True]
pass
else:
# reachability: [cond] AND (NOT [True])
x
```
closes#17723
## Test Plan
* Regression test.
* Verified the ecosystem changes
## Summary
Part of #15383, this PR adds `is_equivalent_to` support for overloaded
callables.
This is mainly done by delegating it to the subtyping check in that two
types A and B are considered equivalent if A is a subtype of B and B is
a subtype of A.
## Test Plan
Add test cases for overloaded callables in `is_equivalent_to.md`
## Summary
Includes minor changes to the semantic type inference to help detect the
return type of function call.
Fixes#17691
## Test Plan
Snapshot tests
## Summary
Subtyping was already modeled, but assignability also needs an explicit
branch. Removes 921 ecosystem false positives.
## Test Plan
New Markdown tests.
We are currently representing type variables using a `KnownInstance`
variant, which wraps a `TypeVarInstance` that contains the information
about the typevar (name, bounds, constraints, default type). We were
previously only constructing that type for PEP 695 typevars. This PR
constructs that type for legacy typevars as well.
It also detects functions that are generic because they use legacy
typevars in their parameter list. With the existing logic for inferring
specializations of function calls (#17301), that means that we are
correctly detecting that the definition of `reveal_type` in the typeshed
is generic, and inferring the correct specialization of `_T` for each
call site.
This does not yet handle legacy generic classes; that will come in a
follow-on PR.
A small PR that just updates the various settings/configurations to
allow Python 3.14. At the moment selecting that target version will
have no impact compared to Python 3.13 - except that a warning
is emitted if the user does so with `preview` disabled.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Apply auto fixes to cases where the names have changed in Airflow 3 in
AIR302 and split the huge test cases into different test cases based on
proivder
## Test Plan
<!-- How was it tested? -->
the test cases has been split into multiple for easier checking
Summary
--
This PR resolves https://github.com/astral-sh/ruff/issues/9761 by adding
a linter configuration option to disable
`typing_extensions` imports. As mentioned [here], it would be ideal if
we could
detect whether or not `typing_extensions` is available as a dependency
automatically, but this seems like a much easier fix in the meantime.
The default for the new option, `typing-extensions`, is `true`,
preserving the current behavior. Setting it to `false` will bail out of
the new
`Checker::typing_importer` method, which has been refactored from the
`Checker::import_from_typing` method in
https://github.com/astral-sh/ruff/pull/17340),
with `None`, which is then handled specially by each rule that calls it.
I considered some alternatives to a config option, such as checking if
`typing_extensions` has been imported or checking for a `TYPE_CHECKING`
block we could use, but I think defaulting to allowing
`typing_extensions` imports and allowing the user to disable this with
an option is both simple to implement and pretty intuitive.
[here]:
https://github.com/astral-sh/ruff/issues/9761#issuecomment-2790492853
Test Plan
--
New linter tests exercising several combinations of Python versions and
the new config option for PYI019. I also added tests for the other
affected rules, but only in the case where the new config option is
enabled. The rules' existing tests also cover the default case.
This is done in what appears to be the same way as Ruff: we get the CWD,
strip the prefix from the path if possible, and use that. If stripping
the prefix fails, then we print the full path as-is.
Fixes#17233
## Summary
Removes ~850 diagnostics related to assignability of callable types,
where the callable-being-assigned-to has a "Todo signature", which
should probably accept any left hand side callable/signature.
This PR promotes the fix applicability of [readlines-in-for
(FURB129)](https://docs.astral.sh/ruff/rules/readlines-in-for/#readlines-in-for-furb129)
to always safe.
In the original PR (https://github.com/astral-sh/ruff/pull/9880), the
author marked the rule as unsafe because Ruff's type inference couldn't
quite guarantee that we had an `IOBase` object in hand. Some false
positives were recorded in the test fixture. However, before the PR was
merged, Charlie added the necessary type inference and the false
positives went away.
According to the [Python
documentation](https://docs.python.org/3/library/io.html#io.IOBase), I
believe this fix is safe for any proper implementation of `IOBase`:
>[IOBase](https://docs.python.org/3/library/io.html#io.IOBase) (and its
subclasses) supports the iterator protocol, meaning that an
[IOBase](https://docs.python.org/3/library/io.html#io.IOBase) object can
be iterated over yielding the lines in a stream. Lines are defined
slightly differently depending on whether the stream is a binary stream
(yielding bytes), or a text stream (yielding character strings). See
[readline()](https://docs.python.org/3/library/io.html#io.IOBase.readline)
below.
and then in the [documentation for
`readlines`](https://docs.python.org/3/library/io.html#io.IOBase.readlines):
>Read and return a list of lines from the stream. hint can be specified
to control the number of lines read: no more lines will be read if the
total size (in bytes/characters) of all lines so far exceeds hint. [...]
>Note that it’s already possible to iterate on file objects using for
line in file: ... without calling file.readlines().
I believe that a careful reading of our [versioning
policy](https://docs.astral.sh/ruff/versioning/#version-changes)
requires that this change be deferred to a minor release - but please
correct me if I'm wrong!
This PR collects all behavior gated under preview into a new module
`ruff_linter::preview` that exposes functions like
`is_my_new_feature_enabled` - just as is done in the formatter crate.
## Summary
Do not emit errors when defining `TypedDict`s:
```py
from typing_extensions import TypedDict
# No error here
class Person(TypedDict):
name: str
age: int | None
# No error for this alternative syntax
Message = TypedDict("Message", {"id": int, "content": str})
```
## Ecosystem analysis
* Removes ~ 450 false positives for `TypedDict` definitions.
* Changes a few diagnostic messages.
* Adds a few (< 10) false positives, for example:
```diff
+ error[lint:unresolved-attribute]
/tmp/mypy_primer/projects/hydra-zen/src/hydra_zen/structured_configs/_utils.py:262:5:
Type `Literal[DataclassOptions]` has no attribute `__required_keys__`
+ error[lint:unresolved-attribute]
/tmp/mypy_primer/projects/hydra-zen/src/hydra_zen/structured_configs/_utils.py:262:42:
Type `Literal[DataclassOptions]` has no attribute `__optional_keys__`
```
* New true positive
4f8263cd7f/corporate/lib/remote_billing_util.py (L155-L157)
```diff
+ error[lint:invalid-assignment]
/tmp/mypy_primer/projects/zulip/corporate/lib/remote_billing_util.py:155:5:
Object of type `RemoteBillingIdentityDict | LegacyServerIdentityDict |
None` is not assignable to `LegacyServerIdentityDict | None`
```
## Test Plan
New Markdown tests
The PR add the `fix safety` section for rule `RUF027` (#15584 ).
Actually, I have an example of a false positive. Should I include it in
the` fix safety` section?
---------
Co-authored-by: Dylan <dylwil3@gmail.com>
The PR add the fix safety section for rule `FLY002` (#15584 )
The motivation for the content of the fix safety section is given by the
following example
```python
foo = 1
bar = [2, 3]
try:
result_join = " ".join((foo, bar))
print(f"Join result: {result_join}")
except TypeError as e:
print(f"Join error: {e}")
```
which print `Join error: sequence item 0: expected str instance, int
found`
But after the fix is applied, we have
```python
foo = 1
bar = [2, 3]
try:
result_join = f"{foo} {bar}"
print(f"Join result: {result_join}")
except TypeError as e:
print(f"Join error: {e}")
```
which print `Join result: 1 [2, 3]`
---------
Co-authored-by: dylwil3 <dylwil3@gmail.com>
## Summary
This PR add the `fix safety` section for rule `ASYNC116` in
`long_sleep_not_forever.rs` for #15584
---------
Co-authored-by: dylwil3 <dylwil3@gmail.com>
## Summary
I remember we discussed about adding this as a property tests so here I
am.
## Test Plan
```console
❯ QUICKCHECK_TESTS=10000000 cargo test --locked --release --package red_knot_python_semantic -- --ignored types::property_tests::stable::bottom_callable_is_subtype_of_all_fully_static_callable
Finished `release` profile [optimized] target(s) in 0.10s
Running unittests src/lib.rs (target/release/deps/red_knot_python_semantic-e41596ca2dbd0e98)
running 1 test
test types::property_tests::stable::bottom_callable_is_subtype_of_all_fully_static_callable ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 233 filtered out; finished in 30.91s
```
As discussed today, this is needed to handle legacy generic classes
without having to infer the types of the class's explicit bases eagerly
at class construction time. Pulling this out into a separate PR so
there's a smaller diff to review.
This also makes our representation of generic classes and functions more
consistent — before, we had separate Rust types and enum variants for
generic/non-generic classes, but a single type for generic functions.
Now we each a single (respective) type for each.
There were very few places we were differentiation between generic and
non-generic _class literals_, and these are handled now by calling the
(salsa cached) `generic_context` _accessor function_.
Note that _`ClassType`_ is still an enum with distinct variants for
non-generic classes and specialized generic classes.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Add "airflow.operators.python.get_current_context" →
"airflow.sdk.get_current_context" rule
## Test Plan
<!-- How was it tested? -->
the test fixture has been updated accordingly
## Summary
Even though the original suggestion works, they've been removed in later
version and is no longer the best practices.
e.g., many sql realted operators have been removed and are now suggested
to use SQLExecuteQueryOperator instead
## Test Plan
The existing test fixtures have been updated
Previously, we could iterate over files in an unspecified order (via
`HashSet` iteration) and we could accumulate diagnostics from files in
an unspecified order (via parallelism).
Here, we change the status quo so that diagnostics collected from files
are sorted after checking is complete. For now, we sort by severity
(with higher severity diagnostics appearing first) and then by
diagnostic ID to give a stable ordering.
I'm not sure if this is the best ordering.
## Summary
After https://github.com/astral-sh/ruff/pull/17620 (which this PR is
based on), I was looking at other call sites of `Type::into_class_type`,
and I began to feel that _all_ of them were currently buggy due to
silently skipping unspecialized generic class literal types (though in
some cases the bug hadn't shown up yet because we don't understand
legacy generic classes from typeshed), and in every case they would be
better off if an unspecialized generic class literal were implicitly
specialized with the default specialization (which is the usual Python
typing semantics for an unspecialized reference to a generic class),
instead of silently skipped.
So I changed the method to implicitly apply the default specialization,
and added a test that previously failed for detecting metaclasses on an
unspecialized generic base.
I also renamed the method to `to_class_type`, because I feel we have a
strong naming convention where `Type::into_foo` is always a trivial
`const fn` that simply returns `Some()` if the type is of variant `Foo`
and `None` otherwise. Even the existing method (with it handling both
`GenericAlias` and `ClassLiteral`, and distinguishing kinds of
`ClassLiteral`) was stretching this convention, and the new version
definitely breaks that envelope.
## Test Plan
Added a test that failed before this PR.
## Summary
The `ClassLiteralType::inheritance_cycle` method is intended to detect
inheritance cycles that would result in cyclic MROs, emit a diagnostic,
and skip actually trying to create the cyclic MRO, falling back to an
"error" MRO instead with just `Unknown` and `object`.
This method didn't work properly for generic classes. It used
`fully_static_explicit_bases`, which filter-maps `explicit_bases` over
`Type::into_class_type`, which returns `None` for an unspecialized
generic class literal. So in a case like `class C[T](C): ...`, because
the explicit base is an unspecialized generic, we just skipped it, and
failed to detect the class as cyclically defined.
Instead, iterate directly over all `explicit_bases`, and explicitly
handle both the specialized (`GenericAlias`) and unspecialized
(`ClassLiteral`) cases, so that we check all bases and correctly detect
cyclic inheritance.
## Test Plan
Added mdtests.
Summary
--
While going through the syntax errors in [this comment], I was surprised
to see the error `name 'x' is assigned to before global declaration`,
which corresponds to [load-before-global-declaration (PLE0118)] and has
also been reimplemented as a syntax error (#17135). However, it looks
like neither of the implementations consider `global` declarations in
the top-level module scope, which is a syntax error in CPython:
```python
# try.py
x = None
global x
```
```shell
> python -m compileall -f try.py
Compiling 'try.py'...
*** File "try.py", line 2
global x
^^^^^^^^
SyntaxError: name 'x' is assigned to before global declaration
```
I'm not sure this is the best or most elegant solution, but it was a
quick fix that passed all of our tests.
Test Plan
--
New PLE0118 test case.
[this comment]:
https://github.com/astral-sh/ruff/issues/7633#issuecomment-1740424031
[load-before-global-declaration (PLE0118)]:
https://docs.astral.sh/ruff/rules/load-before-global-declaration/#load-before-global-declaration-ple0118
## Summary
Tracked structs have some issues with fixpoint iteration in Salsa, and
there's not actually any need for this to be tracked, it should be
interned like most of our type structs.
The removed comment was probably never correct (in that we could have
disambiguated sufficiently), and is definitely not relevant now that
`TypeVarInstance` also holds its `Definition`.
## Test Plan
Existing tests.
## Summary
This PR adds special-casing for `@final` and `@override` decorator for a
similar reason as https://github.com/astral-sh/ruff/pull/17591 to
support the invalid overload check.
Both `final` and `override` are identity functions which can be removed
once `TypeVar` support is added.
## Summary
As promised, this just adds a TODO comment to document something we
discussed today that should probably be improved at some point, but
isn't a priority right now (since it's an issue that in practice would
only affect generic classes with both `__init__` and `__new__` methods,
where some typevar is bound to `Unknown` in one and to some other type
in another.)
## Summary
Part of #17412
Add a new compile-time syntax error for detecting `nonlocal`
declarations at a module level.
## Test Plan
- Added new inline tests for the syntax error
- Updated existing tests for `nonlocal` statement parsing to be inside a
function scope
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
## Summary
While adding semantic error support to red-knot, I noticed duplicate
diagnostics for code like this:
```py
# error: [invalid-syntax] "cannot use an asynchronous comprehension outside of an asynchronous function on Python 3.9 (syntax was added in 3.11)"
# error: [invalid-syntax] "`asynchronous comprehension` outside of an asynchronous function"
[reveal_type(x) async for x in AsyncIterable()]
```
Beyond the duplication, the first error message doesn't make much sense
because this syntax is _not_ allowed on Python 3.11 either.
To fix this, this PR renames the
`async-comprehension-outside-async-function` semantic syntax error to
`async-comprehension-in-sync-comprehension` and fixes the rule to avoid
applying outside of sync comprehensions at all.
## Test Plan
New linter test demonstrating the false positive. The mdtests from my red-knot
PR also reflect this change.
## Summary
This PR updates the `to_overloaded` method to use an iterative approach
instead of a recursive one.
Refer to
https://github.com/astral-sh/ruff/pull/17585#discussion_r2056804587 for
context.
The main benefit here is that it avoids calling the `to_overloaded`
function in a recursive manner which is a salsa query. So, this is a bit
hand wavy but we should also see less memory used because the cache will
only contain a single entry which should be the entire overload chain.
Previously, the recursive approach would mean that each of the function
involved in an overload chain would have a cache entry. This reduce in
memory shouldn't be too much and I haven't looked at the actual data for
it.
## Test Plan
Existing test cases should pass.
This mostly only improves things for incorrect arguments and for an
incorrect return type. It doesn't do much to improve the case where
`__bool__` isn't callable and leaves the union/other cases untouched
completely.
I picked this one because, at first glance, this _looked_ like a lower
hanging fruit. The conceptual improvement here is pretty
straight-forward: add annotations for relevant data. But it took me a
bit to figure out how to connect all of the pieces.
I wanted to use this method in other places, so I moved it
to what appears to be a God-type. I also made it slightly
more versatile: callers can ask for the entire parameter list
by omitting a specific parameter index.
## Summary
Historically we have avoided narrowing on `==` tests because in many
cases it's unsound, since subclasses of a type could compare equal to
who-knows-what. But there are a lot of types (literals and unions of
them, as well as some known instances like `None` -- single-valued
types) whose `__eq__` behavior we know, and which we can safely narrow
away based on equality comparisons.
This PR implements equality narrowing in the cases where it is sound.
The most elegant way to do this (and the way that is most in-line with
our approach up until now) would be to introduce new Type variants
`NeverEqualTo[...]` and `AlwaysEqualTo[...]`, and then implement all
type relations for those variants, narrow by intersection, and let union
and intersection simplification sort it all out. This is analogous to
our existing handling for `AlwaysFalse` and `AlwaysTrue`.
But I'm reluctant to add new `Type` variants for this, mostly because
they could end up un-simplified in some types and make types even more
complex. So let's try this approach, where we handle more of the
narrowing logic as a special case.
## Test Plan
Updated and added tests.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Another follow-up to the unions-of-large-literals optimization. Restore
the behavior that e.g. `Literal[""] | ~Literal[""]` collapses to
`object`.
## Test Plan
Added mdtests.
## Summary
This is required because otherwise the inferred type is not going to be
`Type::FunctionLiteral` but a todo type because we don't recognize
`TypeVar` yet:
```py
_FuncT = TypeVar("_FuncT", bound=Callable[..., Any])
def abstractmethod(funcobj: _FuncT) -> _FuncT: ...
```
This is mainly required to raise diagnostic when only some (and not all)
`@overload`-ed functions are decorated with `@abstractmethod`.
## Summary
This PR adds a new method `FunctionType::to_overloaded` which converts a
`FunctionType` into an `OverloadedFunction` which contains all the
`@overload`-ed `FunctionType` and the implementation `FunctionType` if
it exists.
There's a big caveat here (it's the way overloads work) which is that
this method can only "see" all the overloads that comes _before_ itself.
Consider the following example:
```py
from typing import overload
@overload
def foo() -> None: ...
@overload
def foo(x: int) -> int: ...
def foo(x: int | None) -> int | None:
return x
```
Here, when the `to_overloaded` method is invoked on the
1. first `foo` definition, it would only contain a single overload which
is itself and no implementation.
2. second `foo` definition, it would contain both overloads and still no
implementation
3. third `foo` definition, it would contain both overloads and the
implementation which is itself
### Usages
This method will be used in the logic for checking invalid overload
usages. It can also be used for #17541.
## Test Plan
Make sure that existing tests pass.
## Summary
This is a first step toward `global` support in red-knot (#15385). I
went through all the matches for `global` in the `mypy/test-data`
directory, but I didn't find anything too interesting that wasn't
already covered by @carljm's suggestions on Discord. I still pulled in a
couple of cases for a little extra variety. I also included a section
from the
[PLE0118](https://docs.astral.sh/ruff/rules/load-before-global-declaration/)
tests in ruff that will become syntax errors once #17463 is merged and
we handle `global` statements.
I don't think I figured out how to use `@Todo` properly, so please let
me know if I need to fix that. I hope this is a good start to the test
suite otherwise.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Status
--
This is a pretty minor change, but it was breaking a red-knot mdtest
until #17463 landed. Now this should close#11934 as the last syntax
error being tracked there!
Summary
--
Moves `Parser::validate_parameters` to
`SemanticSyntaxChecker::duplicate_parameter_name`.
Test Plan
--
Existing tests, with `## Errors` replaced with `## Semantic Syntax
Errors`.
We now handle generic constructor methods on generic classes correctly:
```py
class C[T]:
def __init__[S](self, t: T, s: S): ...
x = C(1, "str")
```
Here, constructing `C` requires us to infer a specialization for the
generic contexts of `C` and `__init__` at the same time.
At first I thought I would need to track the full stack of nested
generic contexts here (since the `[S]` context is nested within the
`[T]` context). But I think this is the only way that we might need to
specialize more than one generic context at once — in all other cases, a
containing generic context must be specialized before we get to a nested
one, and so we can just special-case this.
While we're here, we also construct the generic context for a generic
function lazily, when its signature is accessed, instead of eagerly when
inferring the function body.
## Summary
Model assignability of class instances with a `__call__` method to
`Callable` types. This should solve some false positives related to
`functools.partial` (yes, 1098 fewer diagnostics!).
Reference:
https://github.com/astral-sh/ruff/issues/17343#issuecomment-2824618483
## Test Plan
New Markdown tests.
## Summary
Many symbols in typeshed are defined without being declared. For
example:
```pyi
# builtins:
IOError = OSError
# types
LambdaType = FunctionType
NotImplementedType = _NotImplementedType
# typing
Text = str
# random
uniform = _inst.uniform
# optparse
make_option = Option
# all over the place:
_T = TypeVar("_T")
```
Here, we introduce a change that skips widening the public type of these
symbols (by unioning with `Unknown`).
fixes#17032
## Ecosystem analysis
This is difficult to analyze in detail, but I went over most changes and
it looks very favorable to me overall. The diff on the overall numbers
is:
```
errors: 1287 -> 859 (reduction by 428)
warnings: 45 -> 59 (increase by 14)
```
### Removed false positives
`invalid-base` examples:
```diff
- error[lint:invalid-base] /tmp/mypy_primer/projects/pip/src/pip/_vendor/rich/console.py:548:27: Invalid class base with type `Unknown | Literal[_local]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
- error[lint:invalid-base] /tmp/mypy_primer/projects/tornado/tornado/iostream.py:84:25: Invalid class base with type `Unknown | Literal[OSError]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
- error[lint:invalid-base] /tmp/mypy_primer/projects/mitmproxy/test/conftest.py:35:40: Invalid class base with type `Unknown | Literal[_UnixDefaultEventLoopPolicy]` (all bases must be a class, `Any`, `Unknown` or `Todo`)
```
`invalid-exception-caught` examples:
```diff
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/cloud-init/cloudinit/cmd/status.py:334:16: Cannot catch object of type `Literal[ProcessExecutionError]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
- error[lint:invalid-exception-caught] /tmp/mypy_primer/projects/jinja/src/jinja2/loaders.py:537:16: Cannot catch object of type `Literal[TemplateNotFound]` in an exception handler (must be a `BaseException` subclass or a tuple of `BaseException` subclasses)
```
`unresolved-reference` examples
7a0265d36e/cloudinit/handlers/jinja_template.py (L120-L123)
(we now understand the `isinstance` narrowing)
```diff
- error[lint:unresolved-attribute] /tmp/mypy_primer/projects/cloud-init/cloudinit/handlers/jinja_template.py:123:16: Type `Exception` has no attribute `errno`
```
`unknown-argument` examples
https://github.com/hauntsaninja/boostedblob/blob/master/boostedblob/request.py#L53
```diff
- error[lint:unknown-argument] /tmp/mypy_primer/projects/boostedblob/boostedblob/request.py:53:17: Argument `connect` does not match any known parameter of bound method `__init__`
```
`unknown-argument`
There are a lot of `__init__`-related changes because we now understand
[`@attr.s`](3d42a6978a/src/attr/__init__.pyi (L387))
as a `@dataclass_transform` annotated symbol. For example:
```diff
- error[lint:unknown-argument] /tmp/mypy_primer/projects/attrs/tests/test_hooks.py:72:18: Argument `x` does not match any known parameter of bound method `__init__`
```
### New false positives
This can happen if a symbol that previously was inferred as `X |
Unknown` was assigned-to, but we don't yet understand the assignability
to `X`:
https://github.com/strawberry-graphql/strawberry/blob/main/strawberry/exceptions/handler.py#L90
```diff
+ error[lint:invalid-assignment] /tmp/mypy_primer/projects/strawberry/strawberry/exceptions/handler.py:90:9: Object of type `def strawberry_threading_exception_handler(args: tuple[type[BaseException], BaseException | None, TracebackType | None, Thread | None]) -> None` is not assignable to attribute `excepthook` of type `(_ExceptHookArgs, /) -> Any`
```
### New true positives
6bbb5519fe/tests/tracer/test_span.py (L714)
```diff
+ error[lint:invalid-argument-type] /tmp/mypy_primer/projects/dd-trace-py/tests/tracer/test_span.py:714:33: Argument to this function is incorrect: Expected `str`, found `Literal[b"\xf0\x9f\xa4\x94"]`
```
### Changed diagnostics
A lot of changed diagnostics because we now show `@Todo(Support for
`typing.TypeVar` instances in type expressions)` instead of `Unknown`
for all kinds of symbols that used a `_T = TypeVar("_T")` as a type. One
prominent example is the `list.__getitem__` method:
`builtins.pyi`:
```pyi
_T = TypeVar("_T") # previously `TypeVar | Unknown`, now just `TypeVar`
# …
class list(MutableSequence[_T]):
# …
@overload
def __getitem__(self, i: SupportsIndex, /) -> _T: ...
# …
```
which causes this change in diagnostics:
```py
xs = [1, 2]
reveal_type(xs[0]) # previously `Unknown`, now `@Todo(Support for `typing.TypeVar` instances in type expressions)`
```
## Test Plan
Updated Markdown tests
## Summary
Apply auto fixes to cases where the names have changed in Airflow 3
## Test Plan
Add `AIR301_names_fix.py` and `AIR301_provider_names_fix.py` test fixtures
This pull request fixes https://github.com/astral-sh/ruff/issues/17014
changes this
```python
from __future__ import annotations
flag1 = True
flag2 = True
if flag1 == True or flag2 == True:
pass
if flag1 == False and flag2 == False:
pass
flag3 = True
if flag1 == flag3 and (flag2 == False or flag3 == True): # Should become: if flag1==flag3 and (not flag2 or flag3)
pass
if flag1 == True and (flag2 == False or not flag3 == True): # Should become: if flag1 and (not flag2 or not flag3)
pass
if flag1 != True and (flag2 != False or not flag3 == True): # Should become: if not flag1 and (flag2 or not flag3)
pass
flag = True
while flag == True: # Should become: while flag
flag = False
flag = True
x = 5
if flag == True and x > 0: # Should become: if flag and x > 0
print("ok")
flag = True
result = "yes" if flag == True else "no" # Should become: result = "yes" if flag else "no"
x = flag == True < 5
x = (flag == True) == False < 5
```
to this
```python
from __future__ import annotations
flag1 = True
flag2 = True
if flag1 or flag2:
pass
if not flag1 and not flag2:
pass
flag3 = True
if flag1 == flag3 and (not flag2 or flag3): # Should become: if flag1 == flag3 and (not flag2 or flag3)
pass
if flag1 and (not flag2 or not flag3): # Should become: if flag1 and (not flag2 or not flag3)
pass
if not flag1 and (flag2 or not flag3): # Should become: if not flag1 and (flag2 or not flag3)
pass
flag = True
while flag: # Should become: while flag
flag = False
flag = True
x = 5
if flag and x > 0: # Should become: if flag and x > 0
print("ok")
flag = True
result = "yes" if flag else "no" # Should become: result = "yes" if flag else "no"
x = flag is True < 5
x = (flag) is False < 5
```
---------
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Summary
--
This PR extends semantic syntax error detection to red-knot. The main
changes here are:
1. Adding `SemanticSyntaxChecker` and `Vec<SemanticSyntaxError>` fields
to the `SemanticIndexBuilder`
2. Calling `SemanticSyntaxChecker::visit_stmt` and `visit_expr` in the
`SemanticIndexBuilder`'s `visit_stmt` and `visit_expr` methods
3. Implementing `SemanticSyntaxContext` for `SemanticIndexBuilder`
4. Adding new mdtests to test the context implementation and show
diagnostics
(3) is definitely the trickiest and required (I think) a minor addition
to the `SemanticIndexBuilder`. I tried to look around for existing code
performing the necessary checks, but I definitely could have missed
something or misused the existing code even when I found it.
There's still one TODO around `global` statement handling. I don't think
there's an existing way to look this up, but I'm happy to work on that
here or in a separate PR. This currently only affects detection of one
error (`LoadBeforeGlobalDeclaration` or
[PLE0118](https://docs.astral.sh/ruff/rules/load-before-global-declaration/)
in ruff), so it's not too big of a problem even if we leave the TODO.
Test Plan
--
New mdtests, as well as new errors for existing mdtests
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Allow (instances of) subclasses of `Any` and `Unknown` to be assignable
to (instances of) other classes, unless they are final. This allows us
to get rid of ~1000 false positives, mostly when mock-objects like
`unittest.mock.MagicMock` are assigned to various targets.
## Test Plan
Adapted and new Markdown tests.
## Summary
mypy_primer changes included here:
ebaa9fd27b..4c22d192a4
- Add strawberry as a `good.txt` project (was previously included in our
fork)
- Print Red Knot compilation errors to stderr (thanks @MichaReiser)
## Summary
We currently emit a diagnostic for code like the following:
```py
from typing import Any
# error: Invalid class base with type `GenericAlias` (all bases must be a class, `Any`, `Unknown` or `Todo`)
class C(tuple[Any, ...]): ...
```
The changeset here silences this diagnostic by recognizing instances of
`GenericAlias` in `ClassBase::try_from_type`, and inferring a `@Todo`
type for them. This is a change in preparation for #17557, because `C`
previously had `Unknown` in its MRO …
```py
reveal_type(C.__mro__) # tuple[Literal[C], Unknown, Literal[object]]
```
… which would cause us to think that `C` is assignable to everything.
The changeset also removes some false positive `invalid-base`
diagnostics across the ecosystem.
## Test Plan
Updated Markdown tests.
## Summary
Add parentheses to multi-element intersections, when displayed in a
context that's otherwise potentially ambiguous.
## Test Plan
Update mdtest files
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
#17451 was incomplete. `AlwaysFalsy` and `AlwaysTruthy` are not the only
two types that are super-types of some literals (of a given kind) and
not others. That set also includes intersections containing
`AlwaysTruthy` or `AlwaysFalsy`, and intersections containing literal
types of the same kind. Cover these cases as well.
Fixes#17478.
## Test Plan
Added mdtests.
`QUICKCHECK_TESTS=1000000 cargo test -p red_knot_python_semantic --
--ignored types::property_tests::stable` failed on both
`all_fully_static_type_pairs_are_subtypes_of_their_union` and
`all_type_pairs_are_assignable_to_their_union` prior to this PR, passes
after it.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
I gave up trying to do this one lint at a time and just (mostly)
mechanically translated this entire file in one go.
Generally the messages stay the same (with most moving from an
annotation message to the diagnostic's main message). I added a couple
of `info` sub-diagnostics where it seemed to be the obvious intent.
This finishes the migration for the `INVALID_ASSIGNMENT` lint.
Notice how I'm steadily losing steam in terms of actually improving the
diagnostics. This change is more mechanical, because taking the time to
revamp every diagnostic is a ton of effort. Probably future migrations
will be similar unless there are easy pickings.
We mostly keep things the same here, but the message has been moved from
the annotation to the diagnostic's top-line message. I think this is
perhaps a little worse, but some bigger improvements could be made here.
Indeed, we could perhaps even add a "fix" here.
This moves all INVALID_ASSIGNMENT lints related to unpacking over to the new
diagnostic model.
While we're here, we improve the diagnostic a bit by adding a secondary
annotation covering where the value is. We also split apart the original
singular message into one message for the diagnostic and the "expected
versus got" into annotation messages.
This tests the diagnostic rendering of a case that wasn't previously
covered by snapshots: when unpacking fails because there are too few
values, but where the left hand side can tolerate "N or more." In the
code, this is a distinct diagnostic, so we capture it here.
(Sorry about the diff here, but it made sense to rename the other
sections and that changes the name of the snapshot file.)
I believe this was an artifact of an older iteration of the diagnostic
reporting API. But this is strictly not necessary now, and indeed, might
even be annoying. It is okay, but perhaps looks a little odd, to do
`builder.into_diagnostic("...")` if you don't want to add anything else
to the diagnostic.
I suspect this will be used pretty frequently (I wanted it
immediately). And more practically, this avoids needing to
import `Annotation` to create it.
## Summary
I ran red-knot on every project in mypy-primer. I moved every project
where red-knot ran to completion (fast enough, and mypy-primer could
handle its output) into `good.txt`, so it will run in our CI.
The remaining projects I left listed in `bad.txt`, with a comment
summarizing the failure mode (a few don't fail, they are just slow -- on
a debug build, at least -- or output too many diagnostics for
mypy-primer to handle.)
We will now run CI on 109 projects; 34 are left in `bad.txt`.
## Test Plan
CI on this PR!
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Takes the `good.txt` changes from #17474, and removes the following
projects:
- arrow (not part of mypy_primer upstream)
- freqtrade, hydpy, ibis, pandera, xarray (saw panics locally, all
related to try_metaclass cycles)
Increases the mypy_primer CI run time to ~4 min.
## Test Plan
Three successful CI runs.
## Summary
* Add initial support for `typing.dataclass_transform`
* Support decorating a function decorator with `@dataclass_transform(…)`
(used by `attrs`, `strawberry`)
* Support decorating a metaclass with `@dataclass_transform(…)` (used by
`pydantic`, but doesn't work yet, because we don't seem to model
`__new__` calls correctly?)
* *No* support yet for decorating base classes with
`@dataclass_transform(…)`. I haven't figured out how this even supposed
to work. And haven't seen it being used.
* Add `strawberry` as an ecosystem project, as it makes heavy use of
`@dataclass_transform`
## Test Plan
New Markdown tests
This is an implementation of the discussion from #16719.
This change will allow list function calls to be replaced with
comprehensions:
```python
result = list()
for i in range(3):
result.append(i + 1)
# becomes
result = [i + 1 for i in range(3)]
```
I added a new test to `PERF401.py` to verify that this fix will now work
for `list()`.
## Summary
This PR is a follow-up to #16852.
Instance variables bound in comprehensions are recorded, allowing type
inference to work correctly.
This required adding support for unpacking in comprehension which
resolves https://github.com/astral-sh/ruff/issues/15369.
## Test Plan
One TODO in `mdtest/attributes.md` is now resolved, and some new test
cases are added.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
If two types are gradually-equivalent, that means they share the same
set of possible materializations. There's no need to keep two such types
in the same union or intersection; we should simplify them.
Fixes https://github.com/astral-sh/ruff/issues/17465
The one downside here is that now we will simplify e.g. `Unknown |
Todo(...)` to just `Unknown`, if `Unknown` was added to the union first.
This is correct from a type perspective (they are equivalent types), but
it can mean we lose visibility into part of the cause for the type
inferring as unknown. I think this is OK, but if we think it's important
to avoid this, I can add a special case to try to preserve `Todo` over
`Unknown`, if we see them both in the same union or intersection.
## Test Plan
Added and updated mdtests.
## Summary
The long line of projects in `mypy_primer.yaml` is hard to work with
when adding projects or checking whether they are currently run. Use a
one-per-line text file instead.
## Test Plan
Ecosystem check on this PR.
## Summary
add fix safety section to replace_stdout_stderr and
super_call_with_parameters, for #15584
I checked the behavior and found that these two files could only
potentially delete the appended comments, so I submitted them as a PR.
The PR fixes#16457 .
Specifically, `FURB161` is marked safe, but the rule generates safe
fixes only in specific cases. Therefore, we attempt to mark the fix as
unsafe when we are not in one of these cases.
For instances, the fix is marked as aunsafe just in case of strings (as
pointed out in the issue). Let me know if I should change something.
---------
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
Member lookup can be cyclic, with type inference of implicit members. A
sample case is shown in the added mdtest.
There's no clear way to handle such cases other than to fixpoint-iterate
the cycle.
Fixes#17457.
## Test Plan
Added test.
## Summary
This change adds an auto-fix for manual dict comprehensions. It also
copies many of the improvements from #13919 (and associated PRs fixing
issues with it), and moves some of the utility functions from
`manual_list_comprehension.rs` into a separate `helpers.rs` to be used
in both.
## Test Plan
I added a preview test case to showcase the new fix and added a test
case in `PERF403.py` to make sure lines with semicolons function. I
didn't yet make similar tests to the ones I added earlier to
`PERF401.py`, but the logic is the same, so it might be good to add
those to make sure they work.
You can now use subscript expressions in a type expression to explicitly
specialize generic classes, just like you could already do in value
expressions.
This still does not implement bidirectional checking, so a type
annotation on an assignment does not influence how we infer a
specialization for a (not explicitly specialized) constructor call. You
might get an `invalid-assignment` error if (a) we cannot infer a class
specialization from the constructor call (in which case you end up e.g.
trying to assign `C[Unknown]` to `C[int]`) or if (b) we can infer a
specialization, but it doesn't match the annotation.
Closes https://github.com/astral-sh/ruff/issues/17432
## Summary
There was some narrowing constraints not covered from the previous PR
```py
def _(x: object):
if (type(y := x)) is bool:
reveal_type(y) # revealed: bool
```
Also, refactored a bit
## Test Plan
Update type_api.md
In #17403 I added a comment asserting that all same-kind literal types
share all the same super-types. This is true, with two notable
exceptions: the types `AlwaysTruthy` and `AlwaysFalsy`. These two types
are super-types of some literal types within a given kind and not
others: `Literal[0]`, `Literal[""]`, and `Literal[b""]` inhabit
`AlwaysFalsy`, while other literals inhabit `AlwaysTruthy`.
This PR updates the literal-unions optimization to handle these types
correctly.
Fixes https://github.com/astral-sh/ruff/issues/17447
Verified locally that `QUICKCHECK_TESTS=100000 cargo test -p
red_knot_python_semantic -- --ignored types::property_tests::stable` now
passes again.
## Summary
Fixes#17147.
This was landed in #17149 and then reverted in #17335 because it caused
cycle panics in checking pybind11. #17456 fixed the cause of that panic.
## Test Plan
Add new narrow/assert.md test file
Co-authored-by: Matthew Mckee <matthewmckee04@yahoo.co.uk>
## Summary
We were over-conflating the conditions for deferred name resolution.
`from __future__ import annotations` defers annotations, but not class
bases. In stub files, class bases are also deferred. Modeling this
correctly also reduces likelihood of cycles in Python files using `from
__future__ import annotations` (since deferred resolution is inherently
cycle-prone). The same cycles are still possible in `.pyi` files, but
much less likely, since typically there isn't anything in a `pyi` file
that would cause an early return from a scope, or otherwise cause
visibility constraints to persist to end of scope. Usually there is only
code at module global scope and class scope, which can't have `return`
statements, and `raise` or `assert` statements in a stub file would be
very strange. (Technically according to the spec we'd be within our
rights to just forbid a whole bunch of syntax outright in a stub file,
but I kinda like minimizing unnecessary differences between the handling
of Python files and stub files.)
## Test Plan
Added mdtests.
## Summary
Part of #15383, this PR adds support for overloaded callables.
Typing spec: https://typing.python.org/en/latest/spec/overload.html
Specifically, it does the following:
1. Update the `FunctionType::signature` method to return signatures from
a possibly overloaded callable using a new `FunctionSignature` enum
2. Update `CallableType` to accommodate overloaded callable by updating
the inner type to `Box<[Signature]>`
3. Update the relation methods on `CallableType` with logic specific to
overloads
4. Update the display of callable type to display a list of signatures
enclosed by parenthesis
5. Update `CallableTypeOf` special form to recognize overloaded callable
6. Update subtyping, assignability and fully static check to account for
callables (equivalence is planned to be done as a follow-up)
For (2), it is required to be done in this PR because otherwise I'd need
to add some workaround for `into_callable_type` and I though it would be
best to include it in here.
For (2), another possible design would be convert `CallableType` in an
enum with two variants `CallableType::Single` and
`CallableType::Overload` but I decided to go with `Box<[Signature]>` for
now to (a) mirror it to be equivalent to `overload` field on
`CallableSignature` and (b) to avoid any refactor in this PR. This could
be done in a follow-up to better split the two kind of callables.
### Design
There were two main candidates on how to represent the overloaded
definition:
1. To include it in the existing infrastructure which is what this PR is
doing by recognizing all the signatures within the
`FunctionType::signature` method
2. To create a new `Overload` type variant
<details><summary>For context, this is what I had in mind with the new
type variant:</summary>
<p>
```rs
pub enum Type {
FunctionLiteral(FunctionType),
Overload(OverloadType),
BoundMethod(BoundMethodType),
...
}
pub struct OverloadType {
// FunctionLiteral or BoundMethod
overloads: Box<[Type]>,
// FunctionLiteral or BoundMethod
implementation: Option<Type>
}
pub struct BoundMethodType {
kind: BoundMethodKind,
self_instance: Type,
}
pub enum BoundMethodKind {
Function(FunctionType),
Overload(OverloadType),
}
```
</p>
</details>
The main reasons to choose (1) are the simplicity in the implementation,
reusing the existing infrastructure, avoiding any complications that the
new type variant has specifically around the different variants between
function and methods which would require the overload type to use `Type`
instead.
### Implementation
The core logic is how to collect all the overloaded functions. The way
this is done in this PR is by recording a **use** on the `Identifier`
node that represents the function name in the use-def map. This is then
used to fetch the previous symbol using the same name. This way the
signatures are going to be propagated from top to bottom (from first
overload to the final overload or the implementation) with each function
/ method. For example:
```py
from typing import overload
@overload
def foo(x: int) -> int: ...
@overload
def foo(x: str) -> str: ...
def foo(x: int | str) -> int | str:
return x
```
Here, each definition of `foo` knows about all the signatures that comes
before itself. So, the first overload would only see itself, the second
would see the first and itself and so on until the implementation or the
final overload.
This approach required some updates specifically recognizing
`Identifier` node to record the function use because it doesn't use
`ExprName`.
## Test Plan
Update existing test cases which were limited by the overload support
and add test cases for the following cases:
* Valid overloads as functions, methods, generics, version specific
* Invalid overloads as stated in
https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions
(implementation will be done in a follow-up)
* Various relation: fully static, subtyping, and assignability (others
in a follow-up)
## Ecosystem changes
_WIP_
After going through the ecosystem changes (there are a lot!), here's
what I've found:
We need assignability check between a callable type and a class literal
because a lot of builtins are defined as classes in typeshed whose
constructor method is overloaded e.g., `map`, `sorted`, `list.sort`,
`max`, `min` with the `key` parameter, `collections.abc.defaultdict`,
etc. (https://github.com/astral-sh/ruff/issues/17343). This makes up
most of the ecosystem diff **roughly 70 diagnostics**. For example:
```py
from collections import defaultdict
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(int)
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(list)
class Foo:
def __init__(self, x: int):
self.x = x
# red-knot: No overload of function `__new__` matches arguments [lint:no-matching-overload]
map(Foo, ["a", "b", "c"])
```
Duplicate diagnostics in unpacking
(https://github.com/astral-sh/ruff/issues/16514) has **~16
diagnostics**.
Support for the `callable` builtin which requires `TypeIs` support. This
is **5 diagnostics**. For example:
```py
from typing import Any
def _(x: Any | None) -> None:
if callable(x):
# red-knot: `Any | None`
# Pyright: `(...) -> object`
# mypy: `Any`
# pyrefly: `(...) -> object`
reveal_type(x)
```
Narrowing on `assert` which has **11 diagnostics**. This is being worked
on in https://github.com/astral-sh/ruff/pull/17345. For example:
```py
import re
match = re.search("", "")
assert match
match.group() # error: [possibly-unbound-attribute]
```
Others:
* `Self`: 2
* Type aliases: 6
* Generics: 3
* Protocols: 13
* Unpacking in comprehension: 1
(https://github.com/astral-sh/ruff/pull/17396)
## Performance
Refer to
https://github.com/astral-sh/ruff/pull/17366#issuecomment-2814053046.
## Summary
Add more narrowing analysis for match statements:
* add narrowing constraints from guard expressions
* add negated constraints from previous predicates and guards to
subsequent cases
This PR doesn't address that guards can mutate your subject, and so
theoretically invalidate some of these narrowing constraints that you've
previously accumulated. Some prior art on this issue [here][mutable
guards].
[mutable guards]:
https://www.irif.fr/~scherer/research/mutable-patterns/mutable-patterns-mlworkshop2024-abstract.pdf
## Test Plan
Add some new tests, and update some existing ones
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#14866Fixes#17437
## Test Plan
Update mdtests in `narrow/`
## Summary
Prevent overcommit by using max 4 threads as intended.
Unintuitively, `.max()` returns the maximum value of `self` and the
argument (not limiting to the argument). To limit the value to 4, one
needs to use `.min()`.
https://doc.rust-lang.org/std/cmp/trait.Ord.html#method.max
## Summary
This PR extends version-related syntax error detection to red-knot. The
main changes here are:
1. Passing `ParseOptions` specifying a `PythonVersion` to parser calls
2. Adding a `python_version` method to the `Db` trait to make this
possible
3. Converting `UnsupportedSyntaxError`s to `Diagnostic`s
4. Updating existing mdtests to avoid unrelated syntax errors
My initial draft of (1) and (2) in #16090 instead tried passing a
`PythonVersion` down to every parser call, but @MichaReiser suggested
the `Db` approach instead
[here](https://github.com/astral-sh/ruff/pull/16090#discussion_r1969198407),
and I think it turned out much nicer.
All of the new `python_version` methods look like this:
```rust
fn python_version(&self) -> ruff_python_ast::PythonVersion {
Program::get(self).python_version(self)
}
```
with the exception of the `TestDb` in `ruff_db`, which hard-codes
`PythonVersion::latest()`.
## Test Plan
Existing mdtests, plus a new mdtest to see at least one of the new
diagnostics.
add fix safety section to docs for #15584, I'm new to ruff and not sure
if the content of this PR is correct, but I hope it can be helpful.
---------
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
part of: #15655
I tried generating the source order function using code generation. I
tried a simple approach, but it is not enough to generate all of them
this way.
There is one good thing, that most of the implementations are fine with
this. We only have a few that are not. So one benefit of this PR could
be it eliminates a lot of the code, hence changing the AST structure
will only leave a few places to be fixed.
The `source_order` field determines if a node requires a source order
implementation. If it’s empty it means source order does not visit
anything.
Initially I didn’t want to repeat the field names. But I found two
things:
- `ExprIf` statement unlike other statements does not have the fields
defined in source order. This and also some fields do not need to be
included in the visit. So we just need a way to determine order, and
determine presence.
- Relying on the fields sounds more complicated to me. Maybe another
solution is to add a new attribute `order` to each field? I'm open to
suggestions.
But anyway, except for the `ExprIf` we don't need to write the field
names in order. Just knowing what fields must be visited are enough.
Some nodes had a more complex visitor:
`ExprCompare` required zipping two fields.
`ExprBoolOp` required a match over the fields.
`FstringValue` required a match, I created a new walk_ function that
does the match. and used it in code generation. I don’t think this
provides real value. Because I mostly moved the code from one file to
another. I was tried it as an option. I prefer to leave it in the code
as before.
Some visitors visit a slice of items. Others visit a single element. I
put a check on this in code generation to see if the field requires a
for loop or not. I think better approach is to have a consistent style.
So we can by default loop over any field that is a sequence.
For field types `StringLiteralValue` and `BytesLiteralValue` the types
are not a sequence in toml definition. But they implement `iter` so they
are iterated over. So the code generation does not properly identify
this. So in the code I'm checking for their types.
## Test Plan
All the tests should pass without any changes.
I checked the generated code to make sure it's the same as old code. I'm
not sure if there's a test for the source order visitor.
## Summary
This changeset allows us to generate the signature of synthesized
`__init__` functions in dataclasses by analyzing the fields on the class
(and its superclasses). There are certain things that I have not yet
attempted to model in this PR, like `kw_only`,
[`dataclasses.KW_ONLY`](https://docs.python.org/3/library/dataclasses.html#dataclasses.KW_ONLY)
or functionality around
[`dataclasses.field`](https://docs.python.org/3/library/dataclasses.html#dataclasses.field).
ticket: https://github.com/astral-sh/ruff/issues/16651
## Ecosystem analysis
These two seem to depend on missing features in generics (see [relevant
code
here](9898ccbb78/tests/core/test_generics.py (L54))):
> ```diff
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:24:
Argument `x` does not match any known parameter
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:38:
Argument `y` does not match any known parameter
> ```
These two are true positives. See [relevant code
here](9898ccbb78/tests/core/test_config.py (L154-L161)).
> ```diff
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:161:24:
Argument to this function is incorrect: Expected `int`, found
`Literal["test"]`
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:172:24:
Argument to this function is incorrect: Expected `int | float`, found
`Literal["test"]`
> ```
This one depends on `**` unpacking of dictionaries, which we don't
support yet:
> ```diff
> + error[lint:missing-argument]
/tmp/mypy_primer/projects/mypy_primer/mypy_primer/globals.py:218:11: No
arguments provided for required parameters `new`, `old`, `repo`,
`type_checker`, `mypyc_compile_level`, `custom_typeshed_repo`,
`new_typeshed`, `old_typeshed`, `new_prepend_path`, `old_prepend_path`,
`additional_flags`, `project_selector`, `known_dependency_selector`,
`local_project`, `expected_success`, `project_date`, `shard_index`,
`num_shards`, `output`, `old_success`, `coverage`, `bisect`,
`bisect_output`, `validate_expected_success`,
`measure_project_runtimes`, `concurrency`, `base_dir`, `debug`, `clear`
> ```
## Test Plan
New Markdown tests.
## Summary
Support dataclasses with `order=True`:
```py
@dataclass(order=True)
class WithOrder:
x: int
WithOrder(1) < WithOrder(2) # no error
```
Also adds some additional tests to `dataclasses.md`.
ticket: #16651
## Test Plan
New Markdown tests
This PR adds **_very_** basic inference of generic typevars at call
sites. It does not bring in a full unification algorithm, and there are
a few TODOs in the test suite that are not discharged by this. But it
handles a good number of useful cases! And the PR does not add anything
that would go away with a more sophisticated constraint solver.
In short, we just look for typevars in the formal parameters, and assume
that the inferred type of the corresponding argument is what that
typevar should map to. If a typevar appears more than once, we union
together the corresponding argument types.
Cases we are not yet handling:
- We are not widening literals.
- We are not recursing into parameters that are themselves generic
aliases.
- We are not being very clever with parameters that are union types.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This is similar to https://github.com/astral-sh/ruff/pull/17095, it adds
assignability check for bound methods to callables.
## Test Plan
Add test cases to for assignability; specifically it uses gradual types
because otherwise it would just delegate to `is_subtype_of`.
## Summary
closes#16615
This PR includes:
- Introduces a new type: `Type::BoundSuper`
- Implements member lookup for `Type::BoundSuper`, resolving attributes
by traversing the MRO starting from the specified class
- Adds support for inferring appropriate arguments (`pivot_class` and
`owner`) for `super()` when it is used without arguments
When `super(..)` appears in code, it can be inferred into one of the
following:
- `Type::Unknown`: when a runtime error would occur (e.g. calling
`super()` out of method scope, or when parameter validation inside
`super` fails)
- `KnownClass::Super::to_instance()`: when the result is an *unbound
super object* or when a dynamic type is used as parameters (MRO
traversing is meaningless)
- `Type::BoundSuper`: the common case, representing a properly
constructed `super` instance that is ready for MRO traversal and
attribute resolution
### Terminology
Python defines the terms *bound super object* and *unbound super
object*.
An **unbound super object** is created when `super` is called with only
one argument (e.g.
`super(A)`). This object may later be bound via the `super.__get__`
method. However, this form is rarely used in practice.
A **bound super object** is created either by calling
`super(pivot_class, owner)` or by using the implicit form `super()`,
where both arguments are inferred from the context. This is the most
common usage.
### Follow-ups
- Add diagnostics for `super()` calls that would result in runtime
errors (marked as TODO)
- Add property tests for `Type::BoundSuper`
## Test Plan
- Added `mdtest/class/super.md`
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
* Extend the following AIR311 rules
* `airflow.io.path.ObjectStoragePath` → `airflow.sdk.ObjectStoragePath`
* `airflow.io.storage.attach` → `airflow.sdk.io.attach`
* `airflow.models.dag.DAG` → `airflow.sdk.DAG`
* `airflow.models.DAG` → `airflow.sdk.DAG`
* `airflow.decorators.dag` → `airflow.sdk.dag`
* `airflow.decorators.task` → `airflow.sdk.task`
* `airflow.decorators.task_group` → `airflow.sdk.task_group`
* `airflow.decorators.setup` → `airflow.sdk.setup`
* `airflow.decorators.teardown` → `airflow.sdk.teardown`
## Test Plan
<!-- How was it tested? -->
The test case has been added to the button of the existing test
fixtures, confirmed to be correct and later reorgnaized
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
As discussed in
https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129,
we're to separate suggested changes from required changes.
The following symbols have been moved to AIR311 from AIR301. They still
work in Airflow 3.0, but they're suggested to be changed as they're
expected to be removed in a future version.
* arguments
* `airflow..DAG | dag`
* `sla_miss_callback`
* operators
* `sla`
* name
* `airflow.Dataset] | [airflow.datasets.Dataset` → `airflow.sdk.Asset`
* `airflow.datasets, rest @ ..`
* `DatasetAlias` → `airflow.sdk.AssetAlias`
* `DatasetAll` → `airflow.sdk.AssetAll`
* `DatasetAny` → `airflow.sdk.AssetAny`
* `expand_alias_to_datasets` → `airflow.sdk.expand_alias_to_assets`
* `metadata.Metadata` → `airflow.sdk.Metadata`
<!--airflow.models.baseoperator-->
* `airflow.models.baseoperator.chain` → `airflow.sdk.chain`
* `airflow.models.baseoperator.chain_linear` →
`airflow.sdk.chain_linear`
* `airflow.models.baseoperator.cross_downstream` →
`airflow.sdk.cross_downstream`
* `airflow.models.baseoperatorlink.BaseOperatorLink` →
`airflow.sdk.definitions.baseoperatorlink.BaseOperatorLink`
* `airflow.timetables, rest @ ..`
* `datasets.DatasetOrTimeSchedule` → *
`airflow.timetables.assets.AssetOrTimeSchedule`
* `airflow.utils, rest @ ..`
<!--airflow.utils.dag_parsing_context-->
* `dag_parsing_context.get_parsing_context` →
`airflow.sdk.get_parsing_context`
## Test Plan
<!-- How was it tested? -->
The test fixture has been updated acccordingly
## Summary
Until we optimize our full union/intersection representation to
efficiently handle large numbers of same-kind literal types "as a
block", set a fairly low limit on the size of unions of literals.
We will want to increase this limit once we've made the broader
efficiency improvement (tracked in
https://github.com/astral-sh/ruff/issues/17420).
## Test Plan
`cargo bench --bench red_knot`
## Summary
Now that we've made the large-unions benchmark fast, let's make it slow
again!
This adds a following operation (checking `len`) on the large union,
which is slow, even though building the large union is now fast. (This
is also observed in a real-world code sample.) It's slow because for
every element of the union, we fetch its `__len__` method and check it
for compatibility with `Sized`.
We can make this fast by extending the grouped-types approach, as
discussed in https://github.com/astral-sh/ruff/pull/17403, so that we
can do this `__len__` operation (which is identical for every literal
string) just once for all literal strings, instead of once per literal
string type in the union.
Until we do that, we can make this acceptably fast again for now by
setting a lowish limit on union size, which we can increase in the
future when we make it fast. This is what I'll do in the next PR.
## Test Plan
`cargo bench --bench red_knot`
## Summary
Special-case literal types in `UnionBuilder` to speed up building large
unions of literals.
This optimization is extremely effective at speeding up building even a
very large union (it improves the large-unions benchmark by 41x!). The
problem we can run into is that it is easy to then run into another
operation on the very large union (for instance, narrowing may add it to
an intersection, which then distributes it over the intersection) which
is still slow.
I think it is possible to avoid this by extending this optimized
"grouped" representation throughout not just `UnionBuilder`, but all of
our union and intersection representations. I have some work in this
direction, but rather than spending more time on it right now, I'd
rather just land this much, along with a limit on the size of these
unions (to avoid building really big unions quickly and then hitting
issues where they are used.)
## Test Plan
Existing tests and benchmarks.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Fixes incorrect negated type eq and ne assertions in
infer_binary_intersection_type_comparison
fixes#17360
## Test Plan
Remove and update some now incorrect tests
This reworks the assignability/subtyping relations a bit to handle
typevars better:
1. For the most part, types are not assignable to typevars, since
there's no guarantee what type the typevar will be specialized to.
2. An intersection is an exception, if it contains the typevar itself as
one of the positive elements. This should fall out from the other
clauses automatically, since a typevar is assignable to itself, and an
intersection is assignable to something if any positive element is
assignable to that something.
3. Constrained typevars are an exception, since they must be specialized
to _exactly_ one of the constraints, not to a _subtype_ of a constraint.
If a type is assignable to every constraint, then the type is also
assignable to the constrained typevar.
We already had a special case for (3), but the ordering of it relative
to the intersection clauses meant we weren't catching (2) correctly. To
fix this, we keep the special case for (3), but fall through to the
other match arms for non-constrained typevars and if the special case
isn't true for a constrained typevar.
Closes https://github.com/astral-sh/ruff/issues/17364
## Summary
This PR fixes a bug in the lexer specifically around line continuation
character at end of file.
The reason this was occurring is because the lexer wouldn't check for
EOL _after_ consuming the escaped newline but only if the EOL was right
after the line continuation character.
fixes: #17398
## Test Plan
Add tests for the scenarios where this should occur mainly (a) when the
state is `AfterNewline` and (b) when the state is `Other`.
## Summary
closes#17215
This PR adds regression tests for the following cycled queries:
- all_narrowing_constraints_for_expression
- all_negative_narrowing_constraints_for_expression
The following test files are included:
-
`red_knot_project/resources/test/corpus/cycle_narrowing_constraints.py`
-
`red_knot_project/resources/test/corpus/cycle_negative_narrowing_constraints.py`
These test names don't follow the existing naming convention based on
Cinder.
However, I’ve chosen these names to clearly reflect the regression
cases.
Let me know if you’d prefer to align more closely with the existing
Cinder-based style.
## Test Plan
```sh
git checkout 1a6a10b30
cargo test --package red_knot_project -- corpus
```
We weren't really using `chrono` for anything other than getting the
current time and formatting it for logs.
Unfortunately, this doesn't quite get us to a point where `chrono`
can be removed. From what I can tell, we're still bringing it via
[`tracing-subscriber`](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/)
and
[`quick-junit`](https://docs.rs/quick-junit/latest/quick_junit/).
`tracing-subscriber` does have an
[issue open about Jiff](https://github.com/tokio-rs/tracing/discussions/3128),
but there's no movement on it.
Normally I'd suggest holding off on this since it doesn't get us all of
the way there and it would be better to avoid bringing in two datetime
libraries, but we are, it appears, already there. In particular,
`env_logger` brings in Jiff. So this PR doesn't really make anything
worse, but it does bring us closer to an all-Jiff world.
## Summary
Add very early support for dataclasses. This is mostly to make sure that
we do not emit false positives on dataclass construction, but it also
lies some foundations for future extensions.
This seems like a good initial step to merge to me, as it basically
removes all false positives on dataclass constructor calls. This allows
us to use the ecosystem checks for making sure we don't introduce new
false positives as we continue to work on dataclasses.
## Ecosystem analysis
I re-ran the mypy_primer evaluation of [the `__init__`
PR](https://github.com/astral-sh/ruff/pull/16512) locally with our
current mypy_primer version and project selection. It introduced 1597
new diagnostics. Filtering those by searching for `__init__` and
rejecting those that contain `invalid-argument-type` (those could not
possibly be solved by this PR) leaves 1281 diagnostics. The current
version of this PR removes 1171 diagnostics, which leaves 110
unaccounted for. I extracted the lint + file path for all of these
diagnostics and generated a diff (of diffs), to see which
`__init__`-diagnostics remain. I looked at a subset of these: There are
a lot of `SomeClass(*args)` calls where we don't understand the
unpacking yet (this is not even related to `__init__`). Some others are
related to `NamedTuple`, which we also don't support yet. And then there
are some errors related to `@attrs.define`-decorated classes, which
would probably require support for `dataclass_transform`, which I made
no attempt to include in this PR.
## Test Plan
New Markdown tests.
## Summary
* Partial #17238
* Flyby from discord discussion - `todo_type!` now statically checks for
no parens in the message to avoid issues between debug & release build
tests
## Test Plan
many mdtests are changing
## Summary
This PR moves all the relation methods from `CallableType` to
`Signature`.
The main reason for this is that `Signature` is going to be the common
denominator between normal and overloaded callables and the core logic
to check a certain relationship is going to just require the information
that would exists on `Signature`. For example, to check whether an
overloaded callable is a subtype of a normal callable, we need to check
whether _every_ overloaded signature is a subtype of the normal
callable's signature. This "every" logic would become part of the
`CallableType` and the core logic of checking the subtyping would exists
on `Signature`.
Summary
--
This PR implements detecting the use of `await` expressions outside of
async functions. This is a reimplementation of
[await-outside-async
(PLE1142)](https://docs.astral.sh/ruff/rules/await-outside-async/) as a
semantic syntax error.
Despite the rule name, PLE1142 also applies to `async for` and `async
with`, so these are covered here too.
Test Plan
--
Existing PLE1142 tests.
I also deleted more code from the `SemanticSyntaxCheckerVisitor` to
avoid changes in other parser tests.
## Summary
Allows us to establish that two literals do not have a subtype
relationship with each other, without having to fallback to a typeshed
Instance type, which is comparatively slow.
Improves the performance of the many-string-literals union benchmark by
5x.
## Test Plan
`cargo test -p red_knot_python_semantic` and `cargo bench --bench
red_knot`.
## Summary
Add a benchmark for a large-union case that currently has exponential
blow-up in execution time.
## Test Plan
`cargo bench --bench red_knot`
## Summary
This is mainly a follow-up from
https://github.com/astral-sh/ruff/pull/17357 to use the
`concise_message` method for the red-knot server which I noticed
recently while testing the overload implementation.
This commit shuffles the reporting API around a little bit such that a
range is required, up front, when reporting a lint diagnostic. This in
turn enables us to make suppression checking eager.
In order to avoid callers needing to provide the range twice, we create
a primary annotation *without* a message inside the `Diagnostic`
encapsulated by the guard. We do this instead of requiring the message
up front because we're concerned about API complexity and the effort
involved in creating the message.
In order to provide a means of attaching a message to the primary
annotation, we expose a convenience API on `LintDiagnosticGuard` for
setting the message. This isn't generally possible for a `Diagnostic`,
but since a `LintDiagnosticGuard` knows how the `Diagnostic` was
constructed, we can offer this API correctly.
It strikes me that it might be easy to forget to attach a primary
annotation message, btu I think this the "least" bad failure mode. And
in particular, it should be somewhat obvious that it's missing once one
adds a snapshot test for how the diagnostic renders.
Otherwise, this API gives us the ability to eagerly check whether a
diagnostic should be reported with nearly minimal information. It also
shouldn't have any footguns since it guarantees that the primary
annotation is tied to the file in the typing context. And it keeps
things pretty simple: callers only need to provide what is actually
strictly necessary to make a diagnostic.
This will enable us to provide an API on `LintDiagnosticGuard` for
setting the primary annotation message. It will require an `unwrap()`,
but due to how `LintDiagnosticGuard` will build a `Diagnostic`, this
`unwrap()` will be guaranteed to succeed. (And it won't bubble out to
every user of `LintDiagnosticGuard`.)
This is the payoff from removing a bit of indirection. The types still
exist, but now callers don't need to do builder -> reporter ->
diagnostic. They can just conceptually think of it as builder ->
diagnostic.
We're going to make the guards deref to `Diagnostic` in order to remove
a layer of indirection in the reporter API. (Well, technically the layer
is not removed since the types still exist, but in actual _usage_ the
layer will be removed. We'll see how it shakes out in the next commit.)
This expands the set of types accepted for diagnostic messages. Instead
of only `std::fmt::Display` impls, we now *also* accept a concrete
`DiagnosticMessage`.
This will be useful to avoid unnecessary copies of every single
diagnostic message created via `InferContext::report_lint`. (I'll call
out how this helps in a subsequent commit.)
## Summary
Infer precise Boolean literal types for `str.startswith` calls where the
instance and the prefix are both string literals. This allows us to
understand `sys.platform.startswith(…)` branches.
## Test Plan
New Markdown tests
Summary
--
This PR reimplements [yield-outside-function
(F704)](https://docs.astral.sh/ruff/rules/yield-outside-function/) as a
semantic syntax error. Despite the name, this rule covers `yield from`
and `await` in addition to `yield`.
Test Plan
--
New linter tests, along with the existing F704 test.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
Summary
--
This PR replaces uses of version-dependent imports from `typing` or
`typing_extensions` with a centralized `Checker::import_from_typing`
method.
The idea here is to make the fix for #9761 (whatever it ends up being)
applicable to all of the rules performing similar checks.
Test Plan
--
Existing tests for the affected rules.
## Summary
For silencing `invalid-type-form` diagnostics in unreachable code, we
use the same approach that we use before and check the reachability that
we already record.
For silencing `invalid-bases`, we simply check if the type of the base
is `Never`. If so, we silence the diagnostic with the argument that the
class construction would never happen.
## Test Plan
Updated Markdown tests.
## Summary
Similar to what we did for `unresolved-reference` and
`unresolved-attribute`, we now also silence `unresolved-import`
diagnostics if the corresponding `import` statement is unreachable.
This addresses the (already closed) issue #17049.
## Test Plan
Adapted Markdown tests.
I added this accessor because tests want it, but we can also use it in
other places internally. It's a little nicer because it does the
`as_deref()` for you.
This finally completes the deletion of all old diagnostic types.
We do this by migrating the second (and last) use of secondary
diagnostic messages: to highlight the return type of a function
definition when its return value is inconsistent with the type.
Like the last diagnostic, we do actually change the message here a bit.
We don't need a sub-diagnostic here, and we can instead just add a
secondary annotation to highlight the return type.
This is the first use of the new `lint()` reporter.
I somewhat skipped a step here and also modified the actual diagnostic
message itself. The snapshots should tell the story.
We couldn't do this before because we had no way of differentiating
between "message for the diagnostic as a whole" and "message for a
specific code annotation." Now we can, so we can write more precise
messages based on the assumption that users are also seeing the code
snippet.
The downside here is that the actual message text can become quite vague
in the absence of the code snippet. This occurs, for example, with
concise diagnostic formatting. It's unclear if we should do anything
about it. I don't really see a way to make it better that doesn't
involve creating diagnostics with messages for each mode, which I think
would be a major PITA.
The upside is that this code gets a bit simpler, and we very
specifically avoid doing extra work if this specific lint is disabled.
This required a bit of surgery in the diagnostic matching and more
faffing about using a "concise" message from a diagnostic instead of
only printing the "primary" message.
In the new diagnostic data model, we really should have a main
diagnostic message *and* a primary span (with an optional message
attached to it) for every diagnostic.
In this commit, I try to make this true for the "revealed type"
diagnostic. Instead of the annotation saying both "revealed type is"
and also the revealed type itself, the annotation is now just the
revealed type and the main diagnostic message is "Revealed type."
I expect this may be controversial. I'm open to doing something
different. I tried to avoid redundancy, but maybe this is a special case
where we want the redundancy. I'm honestly not sure. I do *like* how it
looks with this commit, but I'm not working with Red Knot's type
checking daily, so my opinion doesn't count for much.
This did also require some tweaking to concise diagnostic formatting in
order to preserve the essential information.
This commit doesn't update every relevant snapshot. Just a few. I split
the rest out into the next commit.
... and replace it with use of `report()`.
Interestingly, this is the only instance of `report_diagnostic` used
directly, and thus anticipated to be the only instance of using
`report()`. If this ends up being a true single use method, we could
make it less generic and tailored specifically to "reveal type."
Two other things to note:
I left the "primary message" as empty. This avoids changing snapshots.
I address this in a subsequent commit.
The creation of a diagnostic here is a bit verbose/annoying. Certainly
more so than it was. This is somewhat expected since our diagnostic
model is more expressive and because we don't have a proc macro. I
avoided creating helpers for this case since there's only one use of
`report()`. But I expect to create helpers for the `lint()` case.
This is a surgical change that adds new `report()` and `lint()`
APIs to `InferContext`. These are intended to replace the existing
`report_*` APIs.
The comments should explain what these reporters are meant to do. For
the most part, this is "just" shuffling some code around. The actual
logic for determining whether a lint *should* be reported or not remains
unchanged and we don't make any changes to how a `Diagnostic` is
actually constructed (yet).
I initially tried to just use `LintReporter` and `DiagnosticReporter`
without the builder types, since I perceive the builder types to be an
annoying additional layer. But I found it also exceedingly annoying to
have to construct and provide the diagnostic message before you even
know if you are going to build the diagnostic. I also felt like this
could result in potentially unnecessary and costly querying in some
cases, although this is somewhat hand wavy. So I overall felt like the
builder route was the way to go. If the builders end up being super
annoying, we can probably add convenience APIs for common patterns to
paper over them.
## Summary
Basically just repeat the same thing that we did for
`unresolved-reference`, but now for attribute expressions.
We now also handle the case where the unresolved attribute (or the
unresolved reference) diagnostic originates from a stringified type
annotation.
And I made the evaluation of reachability constraints lazy (will only be
evaluated right before we are about to emit a diagnostic).
## Test Plan
New Markdown tests for stringified annotations.
I merged #17149 without checking the ecosystem results, and it still
caused a cycle panic in pybind11. Reverting for now until I fix that, so
we don't lose the ecosystem signal on other PRs.
This causes spurious query cycles.
This PR also includes an update to Salsa, which gives us db events on
cycle iteration, so we can write tests asserting the absence of a cycle.
## Summary
Track the reachability of nested scopes within their parent scopes. We
use this as an additional requirement for emitting
`unresolved-reference` diagnostics (and in the future,
`unresolved-attribute` and `unresolved-import`). This means that we only
emit `unresolved-reference` for a given use of a symbol if the use
itself is reachable (within its own scope), *and if the scope itself is
reachable*. For example, no diagnostic should be emitted for the use of
`x` here:
```py
if False:
x = 1
def f():
print(x) # this use of `x` is reachable inside the `f` scope,
# but the whole `f` scope is not reachable.
```
There are probably more fine-grained ways of solving this problem, but
they require a more sophisticated understanding of nested scopes (see
#15777, in particular
https://github.com/astral-sh/ruff/issues/15777#issuecomment-2788950267).
But it doesn't seem completely unreasonable to silence *this specific
kind of error* in unreachable scopes.
## Test Plan
Observed changes in reachability tests and ecosystem.
## Summary
Update Salsa to pull in https://github.com/salsa-rs/salsa/pull/788 which
fixes the, by now, famous *access to field whilst the value is being
initialized*.
This PR also re-enables all tests that previously triggered the panic.
## Test Plan
`cargo test`
## Summary
There is a new official URL for the typing documentation:
https://typing.python.org/
Change all https://typing.readthedocs.io/ links to use the new sub
domain, which is slightly shorter and looks more official.
## Test Plan
Tested to see if each and every new URL is accessible. I noticed that
some links go to https://typing.python.org/en/latest/source/stubs.html
which seems to be outdated, but that is a separate issue. The same page
shows up for the old URL.
## Summary
Based on the discussion in
https://github.com/astral-sh/ruff/pull/17298#discussion_r2033975460, we
decided to move the scope handling out of the `SemanticSyntaxChecker`
and into the `SemanticSyntaxContext` trait. This PR implements that
refactor by:
- Reverting all of the `Checkpoint` and `in_async_context` code in the
`SemanticSyntaxChecker`
- Adding four new methods to the `SemanticSyntaxContext` trait
- `in_async_context`: matches `SemanticModel::in_async_context` and only
detects the nearest enclosing function
- `in_sync_comprehension`: uses the new `is_async` tracking on
`Generator` scopes to detect any enclosing sync comprehension
- `in_module_scope`: reports whether we're at the top-level scope
- `in_notebook`: reports whether we're in a Jupyter notebook
- In-lining the `TestContext` directly into the
`SemanticSyntaxCheckerVisitor`
- This allows modifying the context as the visitor traverses the AST,
which wasn't possible before
One potential question here is "why not add a single method returning a
`Scope` or `Scopes` to the context?" The main reason is that the `Scope`
type is defined in the `ruff_python_semantic` crate, which is not
currently a dependency of the parser. It also doesn't appear to be used
in red-knot. So it seemed best to use these more granular methods
instead of trying to access `Scope` in `ruff_python_parser` (and
red-knot).
## Test Plan
Existing parser and linter tests.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
attribute check was missing in the previous implementation
e.g.
```python
from airflow.api.auth.backend import basic_auth
basic_auth.auth_current_user
```
This PR adds this kind of check.
## Test Plan
<!-- How was it tested? -->
The test case has been added to the button of the existing test
fixtures, confirmed to be correct and later reorgnaized
Summary
--
This PR extends the documentation of the `LoadBeforeGlobalDeclaration`
check to specify the behavior on versions of Python before 3.13. Namely,
on Python 3.12, the `else` clause of a `try` statement is visited before
the `except` handlers:
```pycon
Python 3.12.9 (main, Feb 12 2025, 14:50:50) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 10
>>> def g():
... try:
... 1 / 0
... except:
... a = 1
... else:
... global a
...
>>> def f():
... try:
... pass
... except:
... global a
... else:
... print(a)
...
File "<stdin>", line 5
SyntaxError: name 'a' is used prior to global declaration
```
The order is swapped on 3.13 (see
[CPython#111123](https://github.com/python/cpython/issues/111123)):
```pycon
Python 3.13.2 (main, Feb 5 2025, 08:05:21) [GCC 14.2.1 20250128] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> a = 10
... def g():
... try:
... 1 / 0
... except:
... a = 1
... else:
... global a
...
File "<python-input-0>", line 8
global a
^^^^^^^^
SyntaxError: name 'a' is assigned to before global declaration
>>> def f():
... try:
... pass
... except:
... global a
... else:
... print(a)
...
>>>
```
The current implementation of PLE0118 is correct for 3.13 but not 3.12:
[playground](https://play.ruff.rs/d7467ea6-f546-4a76-828f-8e6b800694c9)
(it flags the first case regardless of Python version).
We decided to maintain this incorrect diagnostic for Python versions
before 3.13 because the pre-3.13 behavior is very unintuitive and
confirmed to be a bug, although the bug fix was not backported to
earlier versions. This can lead to false positives and false negatives
for pre-3.13 code, but we also expect that to be very rare, as
demonstrated by the ecosystem check (before the version-dependent check
was reverted here).
Test Plan
--
N/a
This PR lets you explicitly specialize a generic class using a subscript
expression. It introduces three new Rust types for representing classes:
- `NonGenericClass`
- `GenericClass` (not specialized)
- `GenericAlias` (specialized)
and two enum wrappers:
- `ClassType` (a non-generic class or generic alias, represents a class
_type_ at runtime)
- `ClassLiteralType` (a non-generic class or generic class, represents a
class body in the AST)
We also add internal support for specializing callables, in particular
function literals. (That is, the internal `Type` representation now
attaches an optional specialization to a function literal.) This is used
in this PR for the methods of a generic class, but should also give us
most of what we need for specializing generic _functions_ (which this PR
does not yet tackle).
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
* Simplify match conditions in AIR301
* Fix
* `airflow.datasets.manager.DatasetManager` →
`airflow.assets.manager.AssetManager`
* `airflow.www.auth.has_access_dataset` →
`airflow.www.auth.has_access_dataset`
## Test Plan
<!-- How was it tested? -->
The test fixture has been updated accordingly
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
As discussed in
https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129,
we're to separate suggested changes from required changes.
The following symbols has been moved to AIR312 from AIR302. They still
work in Airflow 3.0, but they're suggested to be changed as they're
expected to be removed in future version
```python
from airflow.hooks.filesystem import FSHook
from airflow.hooks.package_index import PackageIndexHook
from airflow.hooks.subprocess import (SubprocessHook, SubprocessResult, working_directory)
from airflow.operators.bash import BashOperator
from airflow.operators.datetime import BranchDateTimeOperator, target_times_as_dates
from airflow.operators.trigger_dagrun import TriggerDagRunLink, TriggerDagRunOperator
from airflow.operators.empty import EmptyOperator
from airflow.operators.latest_only import LatestOnlyOperator
from airflow.operators.python import (BranchPythonOperator, PythonOperator, PythonVirtualenvOperator, ShortCircuitOperator)
from airflow.operators.weekday import BranchDayOfWeekOperator
from airflow.sensors.date_time import DateTimeSensor, DateTimeSensorAsync
from airflow.sensors.external_task import ExternalTaskMarker, ExternalTaskSensor, ExternalTaskSensorLink
from airflow.sensors.filesystem import FileSensor
from airflow.sensors.time_sensor import TimeSensor, TimeSensorAsync
from airflow.sensors.time_delta import TimeDeltaSensor, TimeDeltaSensorAsync, WaitSensor
from airflow.sensors.weekday import DayOfWeekSensor
from airflow.triggers.external_task import DagStateTrigger, WorkflowTrigger
from airflow.triggers.file import FileTrigger
from airflow.triggers.temporal import DateTimeTrigger, TimeDeltaTrigger
```
## Test Plan
<!-- How was it tested? -->
The test fixture has been updated acccordingly
---------
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
## Summary
As discussed in https://github.com/astral-sh/ruff/issues/16983 and
"mitigate" said issue for the alpha.
This PR changes the default for `PythonPlatform` to be the current
platform rather than `all`.
I'm not sure if we should be as sophisticated as supporting `ios` and
`android` as defaults but it was easy...
## Test Plan
Updated Markdown tests.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
This is a new test case that I don't know how to handle yet. It leads to
many false positives in `rich/tests/test_win32_console.py`, which does
something like:
```py
if sys.platform == "win32":
from windows_only_module import some_symbol
some_other_symbol = 1
def some_test_case():
use(some_symbol) # Red Knot: unresolved-reference
use(some_other_symbol) # Red Knot: unresolved-reference
```
Also adds a test for using unreachable symbols in type annotations or as
class bases.
## Summary
* Addresses #16511 for simple cases where only `__init__` method is
bound on class or doesn't exist at all.
* fixes a bug with argument counting in bound method diagnostics
Caveats:
* No handling of `__new__` or modified `__call__` on metaclass.
* This leads to a couple of false positive errors in tests
## Test Plan
- A couple new cases in mdtests
- cargo nextest run -p red_knot_python_semantic --no-fail-fast
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
This fix closes#16868
I noticed the issue is assigned, but the assignee appears to be actively
working on another pull request. I hope that’s okay!
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
As of Python 3.11.1, `enum.auto()` can be used in multiple assignments.
This pattern should not trigger non-unique-enums check.
Reference: [Python docs on
enum.auto()](https://docs.python.org/3/library/enum.html#enum.auto)
This fix updates the check logic to skip enum variant statements where
the right-hand side is a tuple containing a call to `enum.auto()`.
## Test Plan
<!-- How was it tested? -->
The added test case uses the example from the original issue. It
previously triggered a false positive, but now passes successfully.
Summary
--
Detect async comprehensions nested in sync comprehensions in async
functions before Python 3.11, when this was [changed].
The actual logic of this rule is very straightforward, but properly
tracking the async scopes took a bit of work. An alternative to the
current approach is to offload the `in_async_context` check into the
`SemanticSyntaxContext` trait, but that actually required much more
extensive changes to the `TestContext` and also to ruff's semantic
model, as you can see in the changes up to
31554b473507034735bd410760fde6341d54a050. This version has the benefit
of mostly centralizing the state tracking in `SemanticSyntaxChecker`,
although there was some subtlety around deferred function body traversal
that made the changes to `Checker` more intrusive too (hence the new
linter test).
The `Checkpoint` struct/system is obviously overkill for now since it's
only tracking a single `bool`, but I thought it might be more useful
later.
[changed]: https://github.com/python/cpython/issues/77527
Test Plan
--
New inline tests and a new linter integration test.
## Summary
### Improvement
Expand the following moved module into individual symbols.
* airflow.triggers.temporal
* airflow.triggers.file
* airflow.triggers.external_task
* airflow.hooks.subprocess
* airflow.hooks.package_index
* airflow.hooks.filesystem
* airflow.sensors.weekday
* airflow.sensors.time_delta
* airflow.sensors.time_sensor
* airflow.sensors.date_time
* airflow.operators.weekday
* airflow.operators.datetime
* airflow.operators.bash
This removes `Replacement::ImportPathMoved`.
## Fix
During the expansion, the following paths were also fixed
* airflow.sensors.s3_key_sensor.S3KeySensor →
airflow.providers.amazon.aws.sensors.S3KeySensor
* airflow.operators.sql.SQLThresholdCheckOperator →
airflow.providers.common.sql.operators.sql.SQLThresholdCheckOperator
* airflow.hooks.druid_hook.DruidDbApiHook →
airflow.providers.apache.druid.hooks.druid.DruidDbApiHook
* airflow.hooks.druid_hook.DruidHook →
airflow.providers.apache.druid.hooks.druid.DruidHook
* airflow.kubernetes.pod_generator.extend_object_field →
airflow.providers.cncf.kubernetes.pod_generator.extend_object_field
* airflow.kubernetes.pod_launcher.PodLauncher →
airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodLauncher
* airflow.kubernetes.pod_launcher.PodStatus →
airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodStatus
* airflow.kubernetes.pod_generator.PodDefaults →
airflow.providers.cncf.kubernetes.pod_generator.PodDefaults
* airflow.kubernetes.pod_launcher_deprecated.PodDefaults →
airflow.providers.cncf.kubernetes.pod_launcher_deprecated.PodDefaults
### Refactor
As many symbols are moved into the same module,
`SourceModuleMovedToProvider` is introduced for grouping similar logic
## Test Plan
Summary
--
This PR extends the checks in #17101 and #17282 to annotated assignments
after Python 3.13.
Currently stacked on #17282 to include `await`.
Test Plan
--
New inline tests. These are simpler than the other cases because there's
no place to put generics.
Summary
--
This PR extends the changes in #17101 to include `await` in the same
positions.
I also renamed the `valid_annotation_function` test to include `_py313`
and explicitly passed a Python version to contrast it with the `_py314`
version.
Test Plan
--
New test cases added to existing files.
## Summary
We already have partial "support" for `assert_never`, because it is
annotated as
```pyi
def assert_never(arg: Never, /) -> Never: ...
```
in typeshed. So we already emit a `invalid-argument-type` diagnostic if
the argument type to `assert_never` is not assignable to `Never`.
That is not enough, however. Gradual types like `Any`, `Unknown`,
`@Todo(…)` or `Any & int` can be assignable to `Never`. Which means that
we didn't issue any diagnostic in those cases.
Also, it seems like `assert_never` deserves a dedicated diagnostic
message, not just a generic "invalid argument type" error.
## Test Plan
New Markdown tests.
This fix closes#17026
## Summary
The check for the `PytestRaisesTooBroad` rule is now skipped if there is
a second positional argument present, which means `pytest.raises` is
used as a function.
## Test Plan
Tested on the example from the issue, which now passes the check.
```Python3
pytest.raises(Exception, func, *func_args, **func_kwargs).match("error message")
```
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Resolves#17289.
After this change, Red Knot will no longer show types on hover for
`None`, `...`, `True`, `False`, numbers, strings (but not f-strings),
and bytes literals.
## Test Plan
Unit tests.
## Summary
This implements a new approach to silencing `unresolved-reference`
diagnostics by keeping track of the reachability of each use of a
symbol. The changes merged in
https://github.com/astral-sh/ruff/pull/17169 are still needed for the
"Use of variable in nested function" test case, but that could also be
solved in another way eventually (see
https://github.com/astral-sh/ruff/issues/15777). We can use the same
technique to silence `unresolved-import` and `unresolved-attribute`
false-positives, but I think this could be merged in isolation.
## Test Plan
New Markdown tests, ecosystem tests
## Summary
The priority latency-sensitive is reserved for actions that need to run
immediately because they would otherwise block the user's action. An
example of this is a format request. VS code blocks the editor until the
save action is complete. That's why formatting a document is very
sensitive to delays and it's important that we always have a worker
thread available to run a format request *immediately*. Another example
are code completions, where it's important that they appear immediately
when the user types.
On the other hand, showing diagnostics, hover, or inlay hints has high
priority but users are used that the editor takes a few ms to compute
the overlay.
Computing this information can also be expensive (e.g. find all
references), blocking the worker for quiet some time (a few 100ms).
That's why it's important
that those requests don't clog the sensitive worker threads.
Closes#17042
## Summary
This PR fixes the issue outlined in #17042 where RUF100 (unused-noqa)
fails to detect unused file-level noqa directives (`# ruff: noqa` or `#
ruff: noqa: {code}`).
The issue stems from two underlying causes:
1. For blanket file-level directives (`# ruff: noqa`), there's a
circular dependency: the directive exempts all rules including RUF100
itself, which prevents checking for usage. This isn't changed by this
PR. I would argue it is intendend behavior - a blanket `# ruff: noqa`
directive should exempt all rules including RUF100 itself.
2. For code-specific file-level directives (e.g. `# ruff: noqa: F841`),
the handling was missing in the `check_noqa` function. This is added in
this PR.
## Notes
- For file-level directives, the `matches` array is pre-populated with
the specified codes during parsing, unlike line-level directives which
only populate their `matches` array when actually suppressing
diagnostics. This difference requires the somewhat clunky handling of
both cases. I would appreciate guidance on a cleaner design :)
- A more fundamental solution would be to change how file-level
directives initialize the `matches` array in
`FileNoqaDirectives::extract()`, but that requires more substantial
changes as it breaks existing functionality. I suspect discussions in
#16483 are relevant for this.
## Test Plan
- Local verification
- Added a test case and fixture
## Summary
Some of the migration rules has been changed during Airflow 3
development. The following are new AIR302 rules. Corresponding AIR301
has also been removed.
* airflow.sensors.external_task_sensor.ExternalTaskMarker →
airflow.providers.standard.sensors.external_task.ExternalTaskMarker
* airflow.sensors.external_task_sensor.ExternalTaskSensor →
airflow.providers.standard.sensors.external_task.ExternalTaskSensor
* airflow.sensors.external_task_sensor.ExternalTaskSensorLink →
airflow.providers.standard.sensors.external_task.ExternalTaskSensorLink
* airflow.sensors.time_delta_sensor.TimeDeltaSensor →
airflow.providers.standard.sensors.time_delta.TimeDeltaSensor
* airflow.operators.dagrun_operator.TriggerDagRunLink →
airflow.providers.standard.operators.trigger_dagrun.TriggerDagRunLink
* airflow.operators.dagrun_operator.TriggerDagRunOperator →
airflow.providers.standard.operators.trigger_dagrun.TriggerDagRunOperator
* airflow.operators.python_operator.BranchPythonOperator →
airflow.providers.standard.operators.python.BranchPythonOperator
* airflow.operators.python_operator.PythonOperator →
airflow.providers.standard.operators.python.PythonOperator
* airflow.operators.python_operator.PythonVirtualenvOperator →
airflow.providers.standard.operators.python.PythonVirtualenvOperator
* airflow.operators.python_operator.ShortCircuitOperator →
airflow.providers.standard.operators.python.ShortCircuitOperator
* airflow.operators.latest_only_operator.LatestOnlyOperator →
airflow.providers.standard.operators.latest_only.LatestOnlyOperator
* airflow.sensors.date_time_sensor.DateTimeSensor →
airflow.providers.standard.sensors.DateTimeSensor
* airflow.operators.email_operator.EmailOperator →
airflow.providers.smtp.operators.smtp.EmailOperator
* airflow.operators.email.EmailOperator →
airflow.providers.smtp.operators.smtp.EmailOperator
* airflow.operators.bash.BashOperator →
airflow.providers.standard.operators.bash.BashOperator
* airflow.operators.EmptyOperator →
airflow.providers.standard.operators.empty.EmptyOperator
closes: https://github.com/astral-sh/ruff/issues/17103
## Test Plan
The test fixture has been updated and checked after each change and
later reorganized in the latest commit
## Summary
This PR adds support for stub packages, except for partial stub packages
(a stub package is always considered non-partial).
I read the specification at
[typing.python.org/en/latest/spec/distributing.html#stub-only-packages](https://typing.python.org/en/latest/spec/distributing.html#stub-only-packages)
but I found it lacking some details, especially on how to handle
namespace packages or when the regular and stub packages disagree on
whether they're namespace packages. I tried to document my decisions in
the mdtests where the specification isn't clear and compared the
behavior to Pyright.
Mypy seems to only support stub packages in the venv folder. At least,
it never picked up my stub packages otherwise. I decided not to spend
too much time fighting mypyp, which is why I focused the comparison
around Pyright
Closes https://github.com/astral-sh/ruff/issues/16612
## Test plan
Added mdtests
## Summary
Some more edge cases that I thought of while working on integrating
knowledge of statically known branches into the `*`-import machinery
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
Use more local `expect(dead_code)` suppressions instead of a global
`allow(dead_code)` in `lib.rs`.
Remove some methods that are either easy to add later, are less likely
to be needed for red knot, or it's unclear if we'd add it the same way
as in ruff.
## Summary
This PR does the following things:
- Fixes the `python` configuration setting for mdtest (added in
https://github.com/astral-sh/ruff/pull/17221) so that it expects a path
pointing to a venv's `sys.prefix` variable rather than the a path
pointing to the venv's `site-packages` subdirectory. This brings the
`python` setting in mdtest in sync with our CLI `--python` flag.
- Tweaks mdtest so that it automatically creates a valid `pyvenv.cfg`
file for you if you don't specify one. This makes it much more ergonomic
to write an mdtest with a custom `python` setting: red-knot will reject
a `python` setting that points to a directory that doesn't have a
`pyvenv.cfg` file in it
- Tweaks mdtest so that it doesn't check a custom `pyvenv.cfg` as Python
source code if you _do_ add a custom `pyvenv.cfg` file for your mock
virtual environment in an mdtest. (You get a lot of diagnostics about
Python syntax errors in the `pyvenv.cfg` file, otherwise!)
- Rewrites the test added in
https://github.com/astral-sh/ruff/pull/17178 as an mdtest, and deletes
the original test that was added in that PR
## Test Plan
I verified that the new mdtest fails if I revert the changes to
`resolver.rs` that were added in
https://github.com/astral-sh/ruff/pull/17178
## Summary
This PR extends the mdtest options to allow setting the
`environment.python` option.
## Test Plan
I let @AlexWaygood write a test and he'll tell me if it works 😆
## Summary
This PR fixes the cycle issue that was causing problems in the `support
super` PR.
### Affected queries
- `all_narrowing_constraints_for_expression`
- `all_negative_narrowing_constraints_for_expression`
--
Additionally, `bidict` and `werkzeug` have been added to the
project-selection list in `mypy_primer`.
This PR also addresses the panics that occurred while analyzing those
packages:
- `bidict`: panic triggered by
`all_narrowing_constraints_for_expression`
- `werkzeug`: panic triggered by
`all_negative_narrowing_constraints_for_expression`
I think the mypy-primer results for this PR can serve as sufficient test
:)
Summary
--
This PR fixes the issue pointed out by @JelleZijlstra in
https://github.com/astral-sh/ruff/pull/17101#issuecomment-2777480204.
Namely, I conflated two very different errors from CPython:
```pycon
>>> def m[T](x: (yield from 1)): ...
File "<python-input-310>", line 1
def m[T](x: (yield from 1)): ...
^^^^^^^^^^^^
SyntaxError: yield expression cannot be used within the definition of a generic
>>> def m(x: (yield from 1)): ...
File "<python-input-311>", line 1
def m(x: (yield from 1)): ...
^^^^^^^^^^^^
SyntaxError: 'yield from' outside function
>>> def outer():
... def m(x: (yield from 1)): ...
...
>>>
```
I thought the second error was the same as the first, but `yield` (and
`yield from`) is actually valid in this position when inside a function
scope. The same is true for base classes, as pointed out in the original
comment.
We don't currently raise an error for `yield` outside of a function, but
that should be handled separately.
On the upside, this had the benefit of removing the
`InvalidExpressionPosition::BaseClass` variant and the
`allow_named_expr` field from the visitor because they were both no
longer used.
Test Plan
--
Updated inline tests.
Fixes: #17196
## Summary
Skipping these nodes for malformed type expressions would lead to
incorrect semantic state, which can in turn mean we emit false positives
for rules like `unused-variable`(`F841`)
## Test Plan
`cargo nextest run`
## Summary
This is a follow up to the goto type definition PR. Specifically, that
we want to avoid exposing too many semantic model internals publicly.
I want to get some feedback on the approach taken. I think it goes into
the right direction but I'm not super happy with it.
The basic idea is that we add a `Type::definition` method which does the
"goto type definition". The parts that I think make it awkward:
* We can't directly return `Definition` because we don't create a
`Definition` for modules (but we could?). Although I think it makes
sense to possibly have a more public wrapper type anyway?
* It doesn't handle unions and intersections. Mainly because not all
elements in an intersection may have a definition and we only want to
show a navigation target for intersections if there's only a single
positive element (besides maybe `Unknown`).
An alternative design or an addition to this design is to introduce a
`SemanticAnalysis(Db)` struct that has methods like
`type_definition(&self, type)` which explicitly exposes the methods we
want. I don't feel comfortable design this API yet because it's unclear
how fine granular it has to be (and if it is very fine granular,
directly using `Type` might be better after all)
## Test Plan
`cargo test`
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
Closes#17084
## Summary
This PR adds a new rule (RUF102) to detect and fix invalid rule codes in
`noqa` comments.
Invalid rule codes in `noqa` directives serve no purpose and may
indicate outdated code suppressions.
This extends the previous behaviour originating from
`crates/ruff_linter/src/noqa.rs` which would only emit a warnigs.
With this rule a `--fix` is available.
The rule:
1. Analyzes all `noqa` directives to identify invalid rule codes
2. Provides autofix functionality to:
- Remove the entire comment if all codes are invalid
- Remove only the invalid codes when mixed with valid codes
3. Preserves original comment formatting and whitespace where possible
Example cases:
- `# noqa: XYZ111` → Remove entire comment (keep empty line)
- `# noqa: XYZ222, XYZ333` → Remove entire comment (keep empty line)
- `# noqa: F401, INVALID123` → Keep only valid codes (`# noqa: F401`)
## Test Plan
- Added tests in
`crates/ruff_linter/resources/test/fixtures/ruff/RUF102.py` covering
different example cases.
<!-- How was it tested? -->
## Notes
- This does not handle cases where parsing fails. E.g. `# noqa:
NON_EXISTENT, ANOTHER_INVALID` causes a `LexicalError` and the
diagnostic is not propagated and we cannot handle the diagnostic. I am
also unsure what proper `fix` handling would be and making the user
aware we don't understand the codes is probably the best bet.
- The rule is added to the Preview rule group as it's a new addition
## Questions
- Should we remove the warnings, now that we have a rule?
- Is the current fix behavior appropriate for all cases, particularly
the handling of whitespace and line deletions?
- I'm new to the codebase; let me know if there are rule utilities which
could have used but didn't.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
For two non-disjoint types `P` and `Q`, the simplification of `(P | Q) &
~Q` is not `P`, but `P & ~Q`. In other words, the non-empty set `P & Q`
is also excluded from the type.
The same applies for a constrained typevar `[T: (P, Q)]`: `T & ~Q`
should simplify to `P & ~Q`, not just `P`.
Implementing this is actually purely a matter of removing code from the
constrained typevar simplification logic; we just need to not bother
removing the negations. If the negations are actually redundant (because
the constraint types are disjoint), normal intersection simplification
will already eliminate them (as shown in the added test.)
Summary
--
This PR detects the use of invalid syntax in annotation scopes,
including
`yield` and `yield from` expressions and named expressions. I combined a
few
different types of CPython errors here, but I think the resulting error
messages
still make sense and are even preferable to what CPython gives. For
example, we
report `yield expression cannot be used in a type annotation` for both
of these:
```pycon
>>> def f[T](x: (yield 1)): ...
File "<python-input-26>", line 1
def f[T](x: (yield 1)): ...
^^^^^^^
SyntaxError: yield expression cannot be used within the definition of a generic
>>> def foo() -> (yield x): ...
File "<python-input-28>", line 1
def foo() -> (yield x): ...
^^^^^^^
SyntaxError: 'yield' outside function
```
Fixes https://github.com/astral-sh/ruff/issues/11118.
Test Plan
--
New inline tests, along with some updates to existing tests.
Summary
--
Detects duplicate attributes in a `match` class pattern:
```python
match x:
case Class(x=1, x=2): ...
```
which are more analogous to the similar check for mapping patterns than
to the
multiple assignments rule.
I also realized that both this and the mapping check would only work on
top-level patterns, despite the possibility that they can be nested
inside other
patterns:
```python
match x:
case [{"x": 1, "x": 2}]: ... # false negative in the old version
```
and moved these checks into the recursive pattern visitor instead.
I also tidied up some of the names like the `multiple_case_assignment`
function
and the `MultipleCaseAssignmentVisitor`, which are now doing more than
checking
for multiple assignments.
Test Plan
--
New inline tests for both classes and mappings.
Summary
--
Fixes#17181. The cases being tested with multiple *keys* being equal
are actually a slightly different error, more like the error for
`MatchMapping` than like the other multiple assignment errors:
```pycon
>>> match x:
... case Class(x=x, x=x): ...
...
File "<python-input-249>", line 2
case Class(x=x, x=x): ...
^
SyntaxError: attribute name repeated in class pattern: x
>>> match x:
... case {"x": 1, "x": 2}: ...
...
File "<python-input-251>", line 2
case {"x": 1, "x": 2}: ...
^^^^^^^^^^^^^^^^
SyntaxError: mapping pattern checks duplicate key ('x')
>>> match x:
... case [x, x]: ...
...
File "<python-input-252>", line 2
case [x, x]: ...
^
SyntaxError: multiple assignments to name 'x' in pattern
```
This PR just stops the false positive reported in the issue, but I will
quickly follow it up with a new rule (or possibly combined with the
mapping rule) catching the repeated attributes separately.
Test Plan
--
New inline `test_ok` and updating the `test_err` cases to have duplicate
values instead of keys.
## Summary
It turns out that `a.` isn't a list format supported by rustdoc. I
changed the documentation to use `1.`, `2.` instead.
## Test Plan
`cargo clippy`
This adds a new `Type` variant for holding an instance of a typevar
inside of a generic function or class. We don't handle specializing the
typevars yet, but this should implement most of the typing rules for
inside the generic function/class, where we don't know yet which
specific type the typevar will be specialized to.
This PR does _not_ yet handle the constraint that multiple occurrences
of the typevar must be specialized to the _same_ time. (There is an
existing test case for this in `generics/functions.md` which is still
marked as TODO.)
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
I decided to disable the new
[`needless_continue`](https://rust-lang.github.io/rust-clippy/master/index.html#needless_continue)
rule because I often found the explicit `continue` more readable over an
empty block or having to invert the condition of an other branch.
## Test Plan
`cargo test`
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
This PR changes the inferred type for symbols in unreachable sections of
code to `Never` (instead of reporting them as unbound), in order to
silence false positive diagnostics. See the lengthy comment in the code
for further details.
## Test Plan
- Updated Markdown tests.
- Manually verified a couple of ecosystem diagnostic changes.
## Summary
If a package in `site-packages` had this directory structure:
```py
# bar/__init__.py
from .a import A
# bar/a.py
class A: ...
```
then we would fail to resolve the `from .a import A` import _if_ (as is
usually the case!) the `site-packages` search path was located inside a
`.venv` directory that was a subdirectory of the project's first-party
search path. The reason for this is a bug in `file_to_module` in the
module resolver. In this loop, we would identify that
`/project_root/.venv/lib/python3.13/site-packages/foo/__init__.py` can
be turned into a path relative to the first-party search path
(`/project_root`):
6e2b8f9696/crates/red_knot_python_semantic/src/module_resolver/resolver.rs (L101-L110)
but we'd then try to turn the relative path
(.venv/lib/python3.13/site-packages/foo/__init__.py`) into a module
path, realise that it wasn't a valid module path... and therefore
immediately `break` out of the loop before trying any other search paths
(such as the `site-packages` search path).
This bug was originally reported on Discord by @MatthewMckee4.
## Test Plan
I added a unit test for `file_to_module` in `resolver.rs`, and an
integration test that shows we can now resolve the import correctly in
`infer.rs`.
Summary
--
Detects duplicate literals in `match` mapping keys.
This PR also adds a `source` method to `SemanticSyntaxContext` to
display the duplicated key in the error message by slicing out its
range.
Test Plan
--
New inline tests.
Fixes#17164. Simply checking whether one type is gradually equivalent
to another is too simplistic here: `Any` is gradually equivalent to
`Todo`, but we should permit users to cast from `Todo` or `Unknown` to
`Any` without complaining about it. This changes our logic so that we
only complain about redundant casts if:
- the two types are exactly equal (when normalized) OR they are
equivalent (we'll still complain about `Any -> Any` casts, and about
`Any | str | int` -> `str | int | Any` casts, since their normalized
forms are exactly equal, even though the type is not fully static -- and
therefore does not participate in equivalence relations)
- AND the casted type does not contain `Todo`
We add support for `return` and `raise` statements in the control flow
graph: we simply add an edge to the terminal block, push the statements
to the current block, and proceed.
This implementation will have to be modified somewhat once we add
support for `try` statements - then we will need to check whether to
_defer_ the jump. But for now this will do!
Also in this PR: We fix the `unreachable` diagnostic range so that it
lumps together consecutive unreachable blocks.
## Summary
The existing signature for `str` calls had various problems, one of
which I noticed while looking at some ecosystem projects (`scrapy`,
added as a project to mypy_primer in this PR).
## Test Plan
- New tests for `str(…)` calls.
- Observed reduction of false positives in ecosystem checks
## Summary
A callable type is disjoint from other literal types. For example,
`Type::StringLiteral` must be an instance of exactly `str`, not a
subclass of `str`, and `str` is not callable. The same applies to other
literal types.
This should hopefully fix#17144, I couldn't produce any failures after
running property tests multiple times.
## Test Plan
Add test cases for disjointness check between callable and other literal
types.
Run property tests multiple times.
## Summary
Python `**` works differently to Rust `**`!
## Test Plan
Added an mdtest for various edge cases, and checked in the Python REPL
that we infer the correct type in all the new cases tested.
## Summary
This PR adds a CI job that causes GitHub to add annotations to a PR diff
when mdtest assertions fail. For example:
<details>
<summary>Screenshot</summary>
</details>
## Motivation
Debugging mdtest failures locally is currently a really nice experience:
- Errors are displayed with pretty colours, which makes them much more
readable
- If you run the test from inside an IDE, you can CTRL-click on a path
and jump directly to the line that had the failing assertion
- If you use
[`mdtest.py`](https://github.com/astral-sh/ruff/blob/main/crates/red_knot_python_semantic/mdtest.py),
you don't even need to recompile anything after changing an assertion in
an mdtest, amd the test results instantly live-update with each change
to the MarkDown file
Debugging mdtest failures in CI is much more unpleasant, however.
Sometimes an error message is just
> [static-assert-error] Argument evaluates to `False`
...which doesn't tell you very much unless you navigate to the line in
question that has the failing mdtest assertion. The line in question
might not even be touched by the PR, and even if it is, it can be hard
to find the line if the PR touches many files. Unlike locally, you can't
click on the error and jump straight to the line that contains the
failing assertion. You also don't get colourised output in CI
(https://github.com/astral-sh/ruff/issues/13939).
GitHub PR annotations should make it really easy to debug why mdtests
are failing on PRs, making PR review much easier.
## Test Plan
I opened a PR to my fork
[here](https://github.com/AlexWaygood/ruff/pull/11/files) with some
bogus changes to an mdtest to show what it looks like when there are
failures in CI and this job has been added. Scroll down to
`crates/red_knot_python_semantic/resources/mdtest/type_properties/is_equivalent_to.md`
on the "files changed" tab for that PR to see the annotations.
## Summary
Fixes https://github.com/astral-sh/ruff/issues/17058.
Equivalent callable types were not understood as equivalent when they
appeared nested inside unions and intersections. This PR fixes that by
ensuring that `Callable` elements nested inside unions, intersections
and tuples have their representations normalized before one union type
is compared with another for equivalence, or before one intersection
type is compared with another for equivalence.
The normalizations applied to a `Callable` type are:
- the type of the default value is stripped from all parameters (only
whether the parameter _has_ a default value is relevant to whether one
`Callable` type is equivalent to another)
- The names of the parameters are stripped from positional-only
parameters, variadic parameters and keyword-variadic parameters
- Unions and intersections that are present (top-level or nested) inside
parameter annotations or return annotations are normalized.
Adding a `CallableType::normalized()` method also allows us to simplify
the implementation of `CallableType::is_equivalent_to()`.
### Should these normalizations be done eagerly as part of a
`CallableType` constructor?
I considered this. It's something that we could still consider doing in
the future; this PR doesn't rule it out as a possibility. However, I
didn't pursue it for now, for several reasons:
1. Our current `Display` implementation doesn't handle well the
possibility that a parameter might not have a name or an annotated type.
Callable types with parameters like this would be displayed as follows:
```py
(, ,) -> None: ...
```
That's fixable! It could easily become something like `(Unknown,
Unknown) -> None: ...`. But it also illustrates that we probably want to
retain the parameter names when displaying the signature of a `lambda`
function if you're hovering over a reference to the lambda in an IDE.
Currently we don't have a `LambdaType` struct for representing `lambda`
functions; if we wanted to eagerly normalize signatures when creating
`CallableType`s, we'd probably have to add a `LambdaType` struct so that
we would retain the full signature of a `lambda` function, rather than
representing it as an eagerly simplified `CallableType`.
2. In order to ensure that it's impossible to create `CallableType`s
without the parameters being normalized, I'd either have to create an
alternative `SimplifiedSignature` struct (which would duplicate a lot of
code), or move `CallableType` to a new module so that the only way of
constructing a `CallableType` instance would be via a constructor method
that performs the normalizations eagerly on the callable's signature.
Again, this isn't a dealbreaker, and I think it's still an option, but
it would be a lot of churn, and it didn't seem necessary for now. Doing
it this way, at least to start with, felt like it would create a diff
that's easier to review and felt like it would create fewer merge
conflicts for others.
## Test Plan
- Added a regression mdtest for
https://github.com/astral-sh/ruff/issues/17058
- Ran `QUICKCHECK_TESTS=1000000 cargo test --release -p
red_knot_python_semantic -- --ignored types::property_tests::stable`
## Summary
Add an initial set of tests that will eventually document our behavior
around unreachable code. In the last section of this suite, I argue why
we should never type check unreachable sections and never emit any
diagnostics in these sections.
## Summary
Following up the discussion in
https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766548545,
we're to reorganize airflow rules. Before this discussion happens, we
combine required changes and suggested changes in to one single error
code.
This PR first rename the original error code to the new error code as we
discussed. We will gradually extract suggested changes out of AIR301 and
AIR302 to AIR311 and AIR312 in the following PRs
## Test Plan
Except for file, error code rename, the test case should work as it used
to be.
I initially split the lifetime out into three distinct lifetimes on
near-instinct because I moved the struct into the public API. But
because they are all shared borrows, and because there are no other APIs
on `DisplayDiagnostic` to access individual fields (and probably never
will be), it's probably fine to just specify one lifetime. Because of
subtyping, the one lifetime will be the shorter of the three.
There's also the point that `ruff_db` isn't _really_ a public API, since
it isn't a library that others depend on. So my instinct is probably a
bit off there.
## Summary
With this PR, we emit a diagnostic for this case where
previously didn't:
```py
from typing import Literal
def f(m: int, n: Literal[-1, 0, 1]):
# error: [division-by-zero] "Cannot divide object of type `int` by zero"
return m / n
```
## Test Plan
New Markdown test
## Summary
Add autofix infrastructure to `AIR302` name checks and use this logic to
fix`"airflow", "api_connexion", "security", "requires_access_dataset"`, `"airflow", "Dataset"` and `"airflow",
"datasets", "Dataset"`
## Test Plan
The existing test fixture reflects the update
## Summary
Closes#17112. Allows passing in string and list-of-strings literals
into `subprocess.run` (and related) calls without marking them as
untrusted input:
```py
import subprocess
subprocess.run("true")
# "instant" named expressions are also allowed
subprocess.run(c := "ls")
```
## Test Plan
Added test cases covering new behavior, passed with `cargo nextest run`.
## Summary
* ``airflow.auth.managers.base_auth_manager.is_authorized_dataset`` has
been moved to
``airflow.api_fastapi.auth.managers.base_auth_manager.is_authorized_asset``
in Airflow 3.0
* ``airflow.auth.managers.models.resource_details.DatasetDetails`` has
been moved to
``airflow.api_fastapi.auth.managers.models.resource_details.AssetDetails``
in Airflow 3.0
* Dag arguments `default_view` and `orientation` has been removed in
Airflow 3.0
* `airflow.models.baseoperatorlink.BaseOperatorLink` has been moved to
`airflow.sdk.definitions.baseoperatorlink.BaseOperatorLink` in Airflow
3.0
* ``airflow.notifications.basenotifier.BaseNotifier`` has been moved to
``airflow.sdk.BaseNotifier`` in Airflow 3.0
* ``airflow.utils.log.secrets_masker`` has been moved to
``airflow.sdk.execution_time.secrets_masker`` in Airflow 3.0
* ``airflow...DAG.allow_future_exec_dates`` has been removed in Airflow
3.0
* `airflow.utils.db.create_session` has een removed in Airflow 3.0
* `airflow.sensors.base_sensor_operator.BaseSensorOperator` has been
moved to `airflow.sdk.bases.sensor.BaseSensorOperator` removed Airflow
3.0
* `airflow.utils.file.TemporaryDirectory` has been removed in Airflow
3.0 and can be replaced by `tempfile.TemporaryDirectory`
* `airflow.utils.file.mkdirs` has been removed in Airflow 3.0 and can be
replaced by `pathlib.Path({path}).mkdir`
## Test Plan
Test fixture has been added for these changes
## Summary
Unlike other AIR3XX rules, this best practice can be applied to Airflow
1 and Airflow 2 as well. Thus, we think it might make sense for use to
move it to AIR002 so that the first number of the error align to Airflow
version as possible to reduce confusion
## Test Plan
the test fixture has been updated
It was already using this approach internally, so this is "just" a
matter of rejiggering the public API of `Diagnostic`.
We were previously writing directly to a `std::io::Write` since it was
thought that this worked better with the linear typing fakery. Namely,
it increased confidence that the diagnostic rendering was actually
written somewhere useful, instead of just being converted to a string
that could potentially get lost.
For reasons discussed in #17130, the linear type fakery was removed.
And so there is less of a reason to require a `std::io::Write`
implementation for diagnostic rendering. Indeed, this would sometimes
result in `unwrap()` calls when one wants to convert to a `String`.
## Summary
This PR adds Goto type definition to the playground, using the same
infrastructure as the LSP.
The main *challenge* with implementing this feature was that the editor
can now participate in which tab is open.
## Known limitations
The same as for the LSP. Most notably, navigating to types defined in
typeshed isn't supported.
## Test Plan
https://github.com/user-attachments/assets/22dad7c8-7ac7-463f-b066-5d5b2c45d1fe
This just adds an extra blank line. I think these tests were written
against the new renderer before it was used by Red Knot's `main`
function. Once I did that, I saw that it was missing a blank line, and
so I added it to match the status quo. But that means these snapshots
have become stale. So this commit updates them.
I put this in its own commit in case all of the information removed here
was controversial. But it *looks* stale to me. At the very least,
`TypeCheckDiagnostic` no longer exists, so that would need to be fixed.
And it doesn't really make sense to me (at this point) to make
`Diagnostic` a Salsa struct, particularly since we are keen on using it
in Ruff (at some point).
We do keep around `OldSecondaryDiagnosticMessage`, since that's part of
the Red Knot `InferContext` API. But it's a rather simple type, and
we'll be able to delete it entirely once `InferContext` exposes the new
`Diagnostic` type directly.
Since we aren't consuming `OldSecondaryDiagnosticMessage` any more, we
can now accept a slice instead of a vec. (Thanks Clippy.)
This replaces things like `TypeCheckDiagnostic` with the new Diagnostic`
type.
This is a "surgical" replacement where we retain the existing API of
of diagnostic reporting such that _most_ of Red Knot doesn't need to be
changed to support this update. But it will enable us to start using the
new diagnostic renderer and to delete the old renderer. It also paves
the path for exposing the new `Diagnostic` data model to the broader Red
Knot codebase.
Previously, this was only available in the old renderer.
To avoid regressions, we just copy it to the new renderer.
We don't bother with DRY because the old renderer will be
deleted very soon.
Now that we don't need to update the `printed` flag, this can just be an
immutable borrow.
(Arguably this should have been an immutable borrow even initially, but
I didn't want to introduce interior mutability without a more compelling
justification.)
The switch to `Arc` was done because Salsa sometimes requires cloning a
`Diagnostic` (or something that contains a `Diagnostic`). And so it
probably makes sense to make this cheap.
Since `Diagnostic` exposes a mutable API, we adopt "clone on write"
semantics. Although, it's more like, "clone on write when the `Arc` has
more than one reference." In the common case of creating a `Diagnostic`
and then immediately mutating it, no additional copies should be made
over the status quo.
We also drop the linear type fakery. Its interaction with Salsa is
somewhat awkward, and it has been suggested that there will be points
where diagnostics will be dropped unceremoniously without an opportunity
to tag them as having been ignored. Moreover, this machinery was added
out of "good sense" and isn't actually motivated by real world problems
with accidentally ignoring diagnostics. So that makes it easier, I
think, to just kick this out entirely instead of trying to find a way to
make it work.
This is temporary to scaffold the refactor.
The main idea is that we want to take the `InferContext` API,
*as it is*, and migrate that to the new diagnostic data model
*internally*. Then we can rip out the old stuff and iterate
on the API.
I did this mostly because it wasn't buying us much, and I'm
trying to simplify the public API of the types I'd like to
refactor in order to make the refactor simpler.
If we really want something like this, we can re-add it
later.
I removed this to see how much code was depending internally on the
`&[Arc<TypeCheckDiagnostic>]` representation. Thankfully, it was just
one place. So I just removed the `Deref` impl in favor of adding an
explicit `iter` method.
In general, I think using `Deref` for things like this is _somewhat_ of
an abuse. The tip-off is if there are `&self` or `&mut self` methods on
the type, then it's probably not a good candidate for `Deref`.
Summary
--
This PR reimplements
[load-before-global-declaration
(PLE0118)](https://docs.astral.sh/ruff/rules/load-before-global-declaration/)
as a semantic syntax error.
I added a `global` method to the `SemanticSyntaxContext` trait to make
this very easy, at least in ruff. Does red-knot have something similar?
If this approach will also work in red-knot, I think some of the other
PLE rules are also compile-time errors in CPython, PLE0117 in
particular. 0115 and 0116 also mention `SyntaxError`s in their docs, but
I haven't confirmed them in the REPL yet.
Test Plan
--
Existing linter tests for PLE0118. I think this actually can't be tested
very easily in an inline test because the `TestContext` doesn't have a
real way to track globals.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
Summary
--
Fixes https://github.com/astral-sh/ruff/issues/16520 by flagging single,
starred expressions in `return`, `yield`, and
`for` statements.
I thought `yield from` would also be included here, but that error is
emitted by
the CPython parser:
```pycon
>>> ast.parse("def f(): yield from *x")
Traceback (most recent call last):
File "<python-input-214>", line 1, in <module>
ast.parse("def f(): yield from *x")
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.13/ast.py", line 54, in parse
return compile(source, filename, mode, flags,
_feature_version=feature_version, optimize=optimize)
File "<unknown>", line 1
def f(): yield from *x
^
SyntaxError: invalid syntax
```
And we also already catch it in our parser.
Test Plan
--
New inline tests and updates to existing tests.
## Summary
Implement basic *Goto type definition* support for Red Knot's LSP.
This PR also builds the foundation for other LSP operations. E.g., Goto
definition, hover, etc., should be able to reuse some, if not most,
logic introduced in this PR.
The basic steps of resolving the type definitions are:
1. Find the closest token for the cursor offset. This is a bit more
subtle than I first anticipated because the cursor could be positioned
right between the callee and the `(` in `call(test)`, in which case we
want to resolve the type for `call`.
2. Find the node with the minimal range that fully encloses the token
found in 1. I somewhat suspect that 1 and 2 could be done at the same
time but it complicated things because we also need to compute the spine
(ancestor chain) for the node and there's no guarantee that the found
nodes have the same ancestors
3. Reduce the node found in 2. to a node that is a valid goto target.
This may require traversing upwards to e.g. find the closest expression.
4. Resolve the type for the goto target
5. Resolve the location for the type, return it to the LSP
## Design decisions
The current implementation navigates to the inferred type. I think this
is what we want because it means that it correctly accounts for
narrowing (in which case we want to go to the narrowed type because
that's the value's type at the given position). However, it does have
the downside that Goto type definition doesn't work whenever we infer `T
& Unknown` because intersection types aren't supported. I'm not sure
what to do about this specific case, other than maybe ignoring `Unkown`
in Goto type definition if the type is an intersection?
## Known limitations
* Types defined in the vendored typeshed aren't supported because the
client can't open files from the red knot binary (we can either
implement our own file protocol and handler OR extract the typeshed
files and point there). See
https://github.com/astral-sh/ruff/issues/17041
* Red Knot only exposes an API to get types for expressions and
definitions. However, there are many other nodes with identifiers that
can have a type (e.g. go to type of a globals statement, match patterns,
...). We can add support for those in separate PRs (after we figure out
how to query the types from the semantic model). See
https://github.com/astral-sh/ruff/issues/17113
* We should have a higher-level API for the LSP that doesn't directly
call semantic queries. I intentionally decided not to design that API
just yet.
## Test plan
https://github.com/user-attachments/assets/fa077297-a42d-4ec8-b71f-90c0802b4edb
Goto type definition on a union
<img width="1215" alt="Screenshot 2025-04-01 at 13 02 55"
src="https://github.com/user-attachments/assets/689cabcc-4a86-4a18-b14a-c56f56868085"
/>
Note: I recorded this using a custom typeshed path so that navigating to
builtins works.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
from https://github.com/astral-sh/ruff/pull/17034#discussion_r2024222525
This is a simple PR to fix the invalid behavior of `NotImplemented` on
Python >=3.10.
## Test Plan
I think it would be better if we could run mdtest across multiple Python
versions in GitHub Actions.
<!-- How was it tested? -->
---------
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
## Summary
Add support for decorators on function as well as support
for properties by adding special handling for `@property` and `@<name of
property>.setter`/`.getter` decorators.
closes https://github.com/astral-sh/ruff/issues/16987
## Ecosystem results
- ✔️ A lot of false positives are fixed by our new
understanding of properties
- 🔴 A bunch of new false positives (typically
`possibly-unbound-attribute` or `invalid-argument-type`) occur because
we currently do not perform type narrowing on attributes. And with the
new understanding of properties, this becomes even more relevant. In
many cases, the narrowing occurs through an assertion, so this is also
something that we need to implement to get rid of these false positives.
- 🔴 A few new false positives occur because we do not
understand generics, and therefore some calls to custom setters fail.
- 🔴 Similarly, some false positives occur because we do not
understand protocols yet.
- ✔️ Seems like a true positive to me. [The
setter](e624d8edfa/src/packaging/specifiers.py (L752-L754))
only accepts `bools`, but `None` is assigned in [this
line](e624d8edfa/tests/test_specifiers.py (L688)).
```
+ error[lint:invalid-assignment]
/tmp/mypy_primer/projects/packaging/tests/test_specifiers.py:688:9:
Invalid assignment to data descriptor attribute `prereleases` on type
`SpecifierSet` with custom `__set__` method
```
- ✔️ This is arguable also a true positive. The setter
[here](0c6c75644f/rich/table.py (L359-L363))
returns `Table`, but typeshed wants [setters to return
`None`](bf8d2a9912/stdlib/builtins.pyi (L1298)).
```
+ error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/rich/rich/table.py:359:5: Object of type
`Literal[padding]` cannot be assigned to parameter 2 (`fset`) of bound
method `setter`; expected type `(Any, Any, /) -> None`
```
## Follow ups
- Fix the `@no_type_check` regression
- Implement class decorators
## Test Plan
New Markdown test suites for decorators and properties.
## Summary
Adds import `numpy.typing as npt` to `default in
flake8-import-conventions.aliases`
Resolves#17028
## Test Plan
Manually ran local ruff on the altered fixture and also ran `cargo test`
## Summary
Part of #15382, this PR adds property tests for callable types.
Specifically, this PR updates the property tests to generate an
arbitrary signature for a general callable type which includes:
* Arbitrary combination of parameter kinds in the correct order
* Arbitrary number of parameters
* Arbitrary optional types for annotation and return type
* Arbitrary parameter names (no duplicate names), optional for
positional-only parameters
## Test Plan
```
QUICKCHECK_TESTS=100000 cargo test -p red_knot_python_semantic -- --ignored types::property_tests::stable
```
Also, the commands in CI:
d72b4100a3/.github/workflows/daily_property_tests.yaml (L47-L52)
## Summary
Part of #15382, this PR adds support for disjointness between two
callable types. They are never disjoint because there exists a callable
type that's a subtype of all other callable types:
```py
(*args: object, **kwargs: object) -> Never
```
The `Never` is a subtype of every fully static type thus a callable type
that has the return type of `Never` means that it is a subtype of every
return type.
## Test Plan
Add test cases related to mixed parameter kinds, gradual form (`...`)
and `Never` type.
## Summary
Currently our `Type::Callable` wraps a four-variant `CallableType` enum.
But as time has gone on, I think we've found that the four variants in
`CallableType` are really more different to each other than they are
similar to each other:
- `GeneralCallableType` is a structural type describing all callable
types with a certain signature, but the other three types are "literal
types", more similar to the `FunctionLiteral` variant
- `GeneralCallableType` is not a singleton or a single-valued type, but
the other three are all single-valued types
(`WrapperDescriptorDunderGet` is even a singleton type)
- `GeneralCallableType` has (or should have) ambiguous truthiness, but
all possible inhabitants of the other three types are always truthy.
- As a structural type, `GeneralCallableType` can contain inner unions
and intersections that must be sorted in some contexts in our internal
model, but this is not true for the other three variants.
This PR flattens `Type::Callable` into four distinct `Type::` variants.
In the process, it fixes a number of latent bugs that were concealed by
the current architecture but are laid bare by the refactor. Unit tests
for these bugs are included in the PR.
## Summary
Currently if I run `uv run crates/red_knot_python_semantic/mdtest.py`
from the Ruff repo root, I get this output:
```
~/dev/ruff (main)⚡ % uv run crates/red_knot_python_semantic/mdtest.py
Ready to watch for changes...
```
...And I then have to make some spurious whitespace changes or something
to a test file in order to get the script to actually run mdtest. This
PR changes mdtest.py so that it does an initial run of all mdtests when
you invoke the script, and _then_ starts watching for changes in test
files/Rust code.
## Summary
This PR fixes a bug in callable subtyping to consider both the
positional and keyword form of the standard parameter in the supertype
when matching against variadic, keyword-only and keyword-variadic
parameter in the subtype.
This is done by collecting the unmatched standard parameters and then
checking them against the keyword-only / keyword-variadic parameters
after the positional loop.
## Test Plan
Add test cases.
## Summary
There are quite a few places we infer `Todo` types currently, and some
of them are nested somewhat deeply in type expressions. These can cause
spurious issues for the new `redundant-cast` diagnostics. We fixed all
the false positives we saw in the mypy_primer report before merging
https://github.com/astral-sh/ruff/pull/17100, but I think there are
still lots of places where we'd emit false positives due to this check
-- we currently don't run on that many projects at all in our
mypy_primer check:
d0c8eaa092/.github/workflows/mypy_primer.yaml (L71)
This PR fixes some more false positives from this diagnostic by making
the `Type::contains_todo()` method more expansive.
## Test Plan
I added a regression test which causes us to emit a spurious diagnostic
on `main`, but does not with this PR.
## Summary
In https://github.com/python/typeshed/pull/13520 the typeshed definition
of `typing.Any` was changed from `Any = object()` to `class Any: ...`.
Our automated typeshed updater pulled down this change in
https://github.com/astral-sh/ruff/pull/17106, with the consequence that
we no longer understand `Any`, which is... not good.
This PR gives us the ability to understand `Any` defined as a class
instead of `object()`. It doesn't remove our ability to understand the
old form. Perhaps at some point we'll want to remove it, but for now we
may as well support both old and new typeshed?
This also directly patches typeshed to use the new form of `Any`; this
is purely to work around our tests that no known class is inferred as
`Unknown`, which otherwise fail with the old typeshed and the changes in
this PR. (All other tests pass.) This patch to typeshed will shortly be
subsumed by https://github.com/astral-sh/ruff/pull/17106 anyway.
## Test Plan
Without the typeshed change in this PR, all tests pass except for the
two `known_class_doesnt_fallback_to_unknown_unexpectedly_*` tests (so we
still support the old form of defining `Any`). With the typeshed change
in this PR, all tests pass, so we now support the new form in a way that
is indistinguishable to our test suite from the old form. And
indistinguishable to the ecosystem check: after rebasing
https://github.com/astral-sh/ruff/pull/17106 on this PR, there's zero
ecosystem impact.
## Summary
I don't remember exactly when we made `Identifier` a node but it is now
considered a node (it implements `AnyNodeRef`, it has a range). However,
we never updated
the `SourceOrderVisitor` to visit identifiers because we never had a use
case for it and visiting new nodes can change how the formatter
associates comments (breaking change!).
This PR updates the `SourceOrderVisitor` to visit identifiers and
changes the formatter comment visitor to skip identifiers (updating the
visitor might be desired because it could help simplifying some comment
placement logic but this is out of scope for this PR).
## Test Plan
Tests, updated snapshot tests
## Summary
I noticed we were inferring `Todo` as the declared type for annotations
such as `x: tuple[list[int], list[int]]`. This PR reworks our annotation
parsing so that we instead infer `tuple[Todo, Todo]` for this
annotation, which is quite a bit more precise.
## Test Plan
Existing mdtest updated.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
Closes#16903
## Summary
Check if the current working directory exist. If not, provide an error
instead of panicking.
Fixed a stale comment in `resolve_default_files`.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
I added a test to the `resolve_files.rs`.
Manual testing follows steps of #16903 :
- Terminal 1
```bash
mkdir tmp
cd tmp
```
- Terminal 2
```bash
rm -rf tmp
```
- Terminal 1
```bash
ruff check
```
## Open Issues / Questions to Reviewer
All tests pass when executed with `cargo nextest run`.
However, with `cargo test` the parallelization makes the other tests
fail as we change the `pwd`.
Serial execution with `cargo test` seems to require [another dependency
or some
workarounds](https://stackoverflow.com/questions/51694017/how-can-i-avoid-running-some-tests-in-parallel).
Do you think an additional dependency or test complexity is worth
testing this small edge case, do you have another implementation idea,
or should i rather remove the test?
---
P.S.: I'm currently participating in a batch at the [Recurse
Center](https://www.recurse.com/) and would love to contribute more for
the next six weeks to improve my Rust. Let me know if you're open to
mentoring/reviewing and/or if you have specific areas where help would
be most valued.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
This PR contains the scaffolding for a new control flow graph
implementation, along with its application to the `unreachable` rule. At
the moment, the implementation is a maximal over-approximation: no
control flow is modeled and all statements are counted as reachable.
With each additional statement type we support, this approximation will
improve.
So this PR just contains:
- A `ControlFlowGraph` struct and builder
- Support for printing the flow graph as a Mermaid graph
- Snapshot tests for the actual graphs
- (a very bad!) reimplementation of `unreachable` using the new structs
- Snapshot tests for `unreachable`
# Instructions for Viewing Mermaid snapshots
Unfortunately I don't know how to convince GitHub to render the Mermaid
graphs in the snapshots. However, you can view these locally in VSCode
if you install an extension that supports Mermaid graphs in Markdown,
and then add this to your `settings.json`:
```json
"files.associations": {
"*.md.snap": "markdown",
}
```
## Summary
A few smaller editor improvements that felt worth pulling out of my
other feature PRs:
* Load the `Editor` lazily: This allows splitting the entire monaco
javascript into a separate async bundle, drastically reducing the size
of the `index.js`
* Fix the name of `to_range` and `text_range` to the more idiomatic js
names `toRange` and `textRange`
* Use one indexed values for `Position::line` and `Position::column`,
which is the same as monaco (reduces the need for `+1` and `-1`
operations spread all over the place)
* Preserve the editor state when navigating between tabs. This ensures
that selections are preserved even when switching between tabs.
* Stop the default handling of the `Enter` key press event when renaming
a file because it resulted in adding a newline in the editor
## Summary
This PR adds a new but so far empty and unused `red_knot_ide` crate.
This new crate's purpose is to implement IDE-specific functionality,
such as go to definition, hover, completion, etc., which are used by
both the LSP and the playground.
The crate itself doesn't depend on `lsptypes`. The idea is that the
facade crates (e.g., `red_knot_server`) convert external to internal
types.
Not only allows this to share the logic between server and playground,
it also ensures that the core functionality is easier to test because it
can be tested without needing a full LSP.
## Test Plan
`cargo build`
## Summary
Following up from earlier discussion on Discord, this PR adds logic to
flag casts as redundant when the inferred type of the expression is the
same as the target type. It should follow the semantics from
[mypy](https://github.com/python/mypy/pull/1705).
Example:
```python
def f() -> int:
return 10
# error: [redundant-cast] "Value is already of type `int`"
cast(int, f())
```
## Summary
Part of #13694
Seems there a bit more to cover regarding `in` and other types, but i
can cover them in different PRs
## Test Plan
Add `in.md` file in narrowing conditionals folder
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Rewrites the virtual env discovery to:
* Only use of `System` APIs, this ensures that the discovery will also
work when using a memory file system (testing or WASM)
* Don't traverse ancestor directories. We're not convinced that this is
necessary. Let's wait until someone shows us a use case where it is
needed
* Start from the project root and not from the current working
directory. This ensures that Red Knot picks up the right venv even when
using `knot --project ../other-dir`
## Test Plan
Existing tests, @ntBre tested that the `file_watching` tests no longer
pick up his virtual env in a parent directory
## Summary
In preparation for #17017, where we will need them to suppress new false
positives (once we understand the `ParamSpec.args`/`ParamSpec.kwargs`
properties).
## Test Plan
Tested on branch #17017
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
* Combine AIR302 matches
* Found a few errors. Will be fixed in another PR
## Test Plan
<!-- How was it tested? -->
This PR does not change anything. The existing testing fixture should
work as it used to be
## Summary
Disallow empty `todo_type!()`s without a custom message. They can lead
to spurious diffs in `mypy_primer` where the only thing that's changed
is the file/line information.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
* The following paths are wrong
* `airflow.providers.amazon.auth_manager.avp.entities` should be
`airflow.providers.amazon.aws.auth_manager.avp.entities`
* `["airflow", "datasets", "manager", "dataset_manager"]` should be
fixed as `airflow.assets.manager` but not
`airflow.assets.manager.asset_manager`
* `["airflow", "datasets.manager", "DatasetManager"]` should be `
["airflow", "datasets", "manager", "DatasetManager"]` instead
## Test Plan
<!-- How was it tested? -->
the test fixture is updated accordingly
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Improve AIR302 test cases
## Test Plan
<!-- How was it tested? -->
test fixtures have been updated accordingly
## Summary
A quick fix for how union/intersection member search ins performed in
Knot.
## Test Plan
* Added a dunder method call test for Union, which exhibits the error
* Also added an intersection error, but it is not triggering currently
due to `call` logic not being fully implemented for intersections.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Closes#16661
This PR includes two changes:
- `NotImplementedType` is now a member of `KnownClass`
- We skip `is_assignable_to` checks for `NotImplemented` when checking
return types
### Limitation
```py
def f(cond: bool) -> int:
return 1 if cond else NotImplemented
```
The implementation covers cases where `NotImplemented` appears inside a
`Union`.
However, for more complex types (ex. `Intersection`) it will not worked.
In my opinion, supporting such complexity is unnecessary at this point.
## Test Plan
Two `mdtest` files were updated:
- `mdtest/function/return_type.md`
- `mdtest/type_properties/is_singleton.md`
To test `KnownClass`, run:
```bash
cargo test -p red_knot_python_semantic -- types::class::
```
Summary
--
Detects starred assignment targets outside of tuples and lists like `*a
= (1,)`.
This PR only considers assignment statements. I also checked annotated
assigment statements, but these give a separate error that we already
catch, so I think they're okay not to consider:
```pycon
>>> *a: list[int] = []
File "<python-input-72>", line 1
*a: list[int] = []
^
SyntaxError: invalid syntax
```
Fixes#13759
Test Plan
--
New inline tests, plus a new `SemanticSyntaxError` for an existing
parser test. I also removed a now-invalid case from an otherwise-valid
test fixture.
The new semantic error leads to two errors for the case below:
```python
*foo() = 42
```
but this matches [pyright] too.
[pyright]: https://pyright-play.net/?code=FQMw9mAUCUAEC8sAsAmAUEA
Summary
--
Detect setting or deleting `__debug__`. Assigning to `__debug__` was a
`SyntaxError` on the earliest version I tested (3.8). Deleting
`__debug__` was made a `SyntaxError` in [BPO 45000], which said it was
resolved in Python 3.10. However, `del __debug__` was also a runtime
error (`NameError`) when I tested in Python 3.9.6, so I thought it was
worth including 3.9 in this check.
I don't think it was ever a *good* idea to try `del __debug__`, so I
think there's also an argument for not making this version-dependent at
all. That would only simplify the implementation very slightly, though.
[BPO 45000]: https://github.com/python/cpython/issues/89163
Test Plan
--
New inline tests. This also required adding a `PythonVersion` field to
the `TestContext` that could be taken from the inline `ParseOptions` and
making the version field on the options accessible.
## Summary
This PR adds `as_<group>` methods to `AnyNodeRef` to e.g. convert an
`AnyNodeRef` to an `ExprRef`.
I need this for go to definition where the fallback is to test if
`AnyNodeRef` is an expression and then call `inferred_type` (listing
this mapping at every call site where we need to convert `AnyNodeRef` to
an `ExprRef` is a bit painful ;))
Split out from https://github.com/astral-sh/ruff/pull/16901
## Test Plan
`cargo test`
## Summary
We renamed the `PreorderVisitor` to `SourceOrderVisitor` a long time ago
but it seems that we missed to rename the `visit_preorder` functions to
`visit_source_order`.
This PR renames `visit_preorder` to `visit_source_order`
## Test Plan
`cargo test`
## Summary
This PR refactors the common logic for unpacking in assignment, for loops, and with items.
## Test Plan
Make sure existing tests pass.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#16744
Code from
bbf4f830b5/crates/uv-python/src/virtualenv.rs (L124-L144)
## Test Plan
Manual testing
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
From #16861, and the continuation of #16915.
This PR fixes the incorrect behavior of
`TypeInferenceBuilder::infer_name_load` in eager nested scopes.
And this PR closes#16341.
## Test Plan
New test cases are added in `annotations/deferred.md`.
## Summary
Part of #13694
Narrow in or-patterns by taking the type union of the type constraints
in each disjunct pattern.
## Test Plan
Add new tests to narrow/match.md
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Part of #13694
The implementation here was suspiciously straightforward so please lmk
if I missed something
Also some drive-by changes to DRY things up a bit
## Test Plan
Add new tests to narrow/match.md
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR adds some branches so that we infer `Todo` types for attribute
access on instances of `super()` and subtypes of `type[Enum]`. It reduces
false positives in the short term until we implement full support for
these features.
## Test Plan
New mdtests added + mypy_primer report
## Summary
Fixes https://github.com/astral-sh/ruff/issues/17018
## Test Plan
I renamed a python file to `knot.toml` and verified that there are no
diagnostics. Renaming back the file to `*.py` brings back the
diagnostics
## Summary
Resolves#16950 and [a 1.5-year-old TODO
comment](8d16a5c8c9/crates/ruff/src/diagnostics.rs (L380)).
After this change, a `pyproject.toml` will be linted the same as any
Python files would when passed via stdin.
## Test Plan
Integration tests.
## Summary
Mainly for partially fixing #16953
## Test Plan
Update is_subtype tests. And should maybe do these checks for many other
types (is subtype of object but object is not subtype)
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
Summary
--
This PR detects multiple assignments to the same name in `case` patterns
by recursively visiting each pattern.
Test Plan
--
New inline tests.
Summary
--
Detects irrefutable `match` cases before the final case using a modified
version
of the existing `Pattern::is_irrefutable` method from the AST crate. The
modified method helps to retrieve a more precise diagnostic range to
match what
Python 3.13 shows in the REPL.
Test Plan
--
New inline tests, as well as some updates to existing tests that had
irrefutable
patterns before the last block.
## Summary
`std::time::now` isn't available on `wasm32-unknown-unknown` but it is
used by `FileTime::now`.
This PR replaces the usages of `FileTime::now` with a target specific
helper function that we already had in the memory file system.
Fixes https://github.com/astral-sh/ruff/issues/16966
## Test Plan
Tested that the playground no longer crash when adding an extra-path
## Summary
Further work towards https://github.com/astral-sh/ruff/issues/14169.
We currently panic on encountering cyclic `*` imports. This is easily
fixed using fixpoint iteration.
## Test Plan
Added a test that panics on `main`, but passes with this PR
## Summary
As mentioned in
https://github.com/astral-sh/ruff/pull/16698#discussion_r2004920075,
part of #15382, this PR updates the `is_gradual_equivalent_to`
implementation between callable types to be similar to
`is_equivalent_to` and checks other attributes of parameters like name,
optionality, and parameter kind.
## Test Plan
Expand the existing test cases to consider other properties but not all
similar to how the tests are structured for subtyping and assignability.
## Summary
This PR adds initial support for `*` imports to red-knot. The approach
is to implement a standalone query, called from semantic indexing, that
visits the module referenced by the `*` import and collects all
global-scope public names that will be imported by the `*` import. The
`SemanticIndexBuilder` then adds separate definitions for each of these
names, all keyed to the same `ast::Alias` node that represents the `*`
import.
There are many pieces of `*`-import semantics that are still yet to be
done, even with this PR:
- This PR does not attempt to implement any of the semantics to do with
`__all__`. (If a module defines `__all__`, then only the symbols
included in `__all__` are imported, _not_ all public global-scope
symbols.
- With the logic implemented in this PR as it currently stands, we
sometimes incorrectly consider a symbol bound even though it is defined
in a branch that is statically known to be dead code, e.g. (assuming the
target Python version is set to 3.11):
```py
# a.py
import sys
if sys.version_info < (3, 10):
class Foo: ...
```
```py
# b.py
from a import *
print(Foo) # this is unbound at runtime on 3.11,
# but we currently consider it bound with the logic in this PR
```
Implementing these features is important, but is for now deferred to
followup PRs.
Many thanks to @ntBre, who contributed to this PR in a pairing session
on Friday!
## Test Plan
Assertions in existing mdtests are adjusted, and several new ones are
added.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
combine similar case condition in AIR302
## Test Plan
<!-- How was it tested? -->
nothing should be changed. existing test case should already cover it
Summary
--
Fixes#16943 by checking if the tuple is not parenthesized before
emitting an error.
Test Plan
--
New inline test based on the initial report
## Summary
Here I fix the last English spelling errors I could find in the repo.
Again, I am trying not to touch variable/function names, or anything
that might be misspelled in the API. The goal is to make this PR safe
and easy to merge.
## Test Plan
I have run all the unit tests. Though, again, all of the changes I make
here are to docs and docstrings. I make no code changes, which I believe
should greatly mitigate the testing concerns.
## Summary
Resolves#16895.
`abstractmethod` is now a `KnownFunction`. When a function is decorated
by `abstractmethod` or when the parent class inherits directly from
`Protocol`, `invalid-return-type` won't be emitted for that function.
## Test Plan
Markdown tests.
---------
Co-authored-by: Carl Meyer <carl@oddbird.net>
## Summary
Fixes#16912
Create a new type `DisplayMaybeParenthesizedType` that is now used in
Union and Intersection display
## Test Plan
Update callable annotations
## Summary
From #16861
This PR fixes the incorrect `ClassDef` handling of
`SemanticIndexBuilder::visit_stmt`, which fixes some of the incorrect
behavior of referencing the class itself in the class scope (a complete
fix requires a different fix, which will be done in the another PR).
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
This is a cleanup PR. I am fixing various English language spelling
errors. This is mostly in docs and docstrings.
## Test Plan
The usual CI tests were run. I tried to build the docs (though I had
some troubles there). The testing needs here are, I trust, very low
impact. (Though I would happily test more.)
## Summary
Log the origin of the sys path prefix. This should help with debugging
if someone doesn't understand
why Red Knot picks up a certain venv.
## Test Plan
Ran the CLI and tested that it logs the origin
## Summary
Part of #15382, this PR adds support for calling a variable that's
annotated with `typing.Callable`.
## Test Plan
Add test cases in a new `call/annotation.md` file.
## Summary
Part of #15382
This PR adds support for checking the assignability of two general
callable types.
This is built on top of #16804 by including the gradual parameters check
and accepting a function that performs the check between the two types.
## Test Plan
Update `is_assignable_to.md` with callable types section.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
When callables are displayed in unions, like:
```py
from typing import Callable
def foo(x: Callable[[], int] | None):
# red-knot: Revealed type is `() -> int | None` [revealed-type]
reveal_type(x)
```
This leaves the type rather ambiguous, to fix this we can add
parenthesis to callable type in union
Fixes#16893
## Test Plan
Update callable annotations tests
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#16898
## Test Plan
Update test for lang mismatch panic
Summary
--
Detects duplicate type parameter names in function definitions, class
definitions, and type alias statements.
I also boxed the `type_params` field on `StmtTypeAlias` to make it
easier to
`match` with functions and classes. (That's the reason for the red-knot
code
owner review requests, sorry!)
Test Plan
--
New `ruff_python_syntax_errors` unit tests.
Fixes#11119.
## Summary
This PR implements the "greeter" approach for checking the AST for
syntax errors emitted by the CPython compiler. It introduces two main
infrastructural changes to support all of the compile-time errors:
1. Adds a new `semantic_errors` module to the parser crate with public
`SemanticSyntaxChecker` and `SemanticSyntaxError` types
2. Embeds a `SemanticSyntaxChecker` in the `ruff_linter::Checker` for
checking these errors in ruff
As a proof of concept, it also implements detection of two syntax
errors:
1. A reimplementation of
[`late-future-import`](https://docs.astral.sh/ruff/rules/late-future-import/)
(`F404`)
2. Detection of rebound comprehension iteration variables
(https://github.com/astral-sh/ruff/issues/14395)
## Test plan
Existing F404 tests, new inline tests in the `ruff_python_parser` crate,
and a linter CLI test showing an example of the `Message` output.
I also tested in VS Code, where `preview = false` and turning off syntax
errors both disable the new errors:
And on the playground, where `preview = false` also disables the errors:
Fixes#14395
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
I am one of the core developers of Airflow and working on the
"airflow.sdk"
package, and this updates the recommended replacments to the correct
user-facing imports.[^1]
cc @Lee-W @uranusjr
[^1]:
33f0f1d639/task-sdk/src/airflow/sdk/__init__.py (L68-L93)
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Hope and pray? 😉
I'm sure there are some snapshot files I'm supposed to fix first.
<!-- How was it tested? -->
## Summary
This PR removes false-positive diagnostics for `*` imports. Currently we
always emit a diagnostic for these statements unless the module we're
importing from has a symbol named `"*"` in its symbol table for the
global scope. (And if we were doing everything correctly, no module ever
would have a symbol named `"*"` in its global scope!)
The fix here is sort-of hacky and won't be what we'll want to do
long-term. However, I think it's useful to do this as a first step
since:
- It significantly reduces false positives when running on code that
uses `*` imports
- It "resets" the tests to a cleaner state with many fewer TODOs, making
it easier to see what the hard work is that's still to be done.
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This PR adds a suite of tests for wildcard (`*`) imports. The tests
nearly all fail for now, and those that don't, ahem, pass for the wrong
reasons...
I've tried to add TODO comments in all instances for places where we are
currently inferring the incorrect thing, incorrectly emitting a
diagnostic, or emitting a diagnostic with a bad error message.
## Test Plan
`cargo test -p red_knot_python_semantic`
This breaks up call binding into two phases:
- **_Matching parameters_** just looks at the names and kinds
(positional/keyword) of each formal and actual parameters, and matches
them up. Most of the current call binding errors happen during this
phase.
- Once we have matched up formal and actual parameters, we can **_infer
types_** of each actual parameter, and **_check_** that each one is
assignable to the corresponding formal parameter type.
As part of this, we add information to each formal parameter about
whether it is a type form or not. Once [PEP
747](https://peps.python.org/pep-0747/) is finalized, we can hook that
up to this internal type form representation. This replaces the
`ParameterExpectations` type, which did the same thing in a more ad hoc
way.
While we're here, we add a new fluent API for building `Parameter`s,
which makes our signature constructors a bit nicer to read. We also
eliminate a TODO where we were consuming types from the argument list
instead of the bound parameter list when evaluating our special-case
known functions.
Closes#15460
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
This change continues to resolve#16071 (and continues the work started
in #16162). Specifically, this PR changes the code in the parser so that
it uses the `OperatorPrecedence` struct from `ruff_python_ast` instead
of its own version. This is part of an effort to get rid of the
redundant definitions of `OperatorPrecedence` throughout the codebase.
Note that this PR only makes this change for `ruff_python_parser` -- we
still want to make a similar change for the formatter (namely the
`OperatorPrecedence` defined in the expression part of the formatter,
the pattern one is different). I separated the work to keep the PRs
small and easily reviewable.
## Test Plan
Because this is an internal change, I didn't add any additional tests.
Existing tests do pass.
## Summary
Part of #15382
This PR adds support for checking the subtype relationship between the
two callable types.
The main source of reference used for implementation is
https://typing.python.org/en/latest/spec/callables.html#assignability-rules-for-callables.
The implementation is split into two phases:
1. Check all the positional parameters which includes positional-only,
standard (positional or keyword) and variadic kind
2. Collect all the keywords in a `HashMap` to do the keyword parameters
check via name lookup
For (1), there's a helper struct which is similar to `.zip_longest`
(from `itertools`) except that it allows control over one of the
iterator as that's required when processing a variadic parameter. This
is required because positional parameters needs to be checked as per
their position between the two callable types. The struct also keeps
track of the current iteration element because when the loop is exited
(to move on to the phase 2) the current iteration element would be
carried over to the phase 2 check.
This struct is internal to the `is_subtype_of` method as I don't think
it makes sense to expose it outside. It also allows me to use "self" and
"other" suffixed field names as that's only relevant in that context.
## Test Plan
Add extensive tests in markdown.
Converted all of the code snippets from
https://typing.python.org/en/latest/spec/callables.html#assignability-rules-for-callables
to use `knot_extensions.is_subtype_of` and verified the result.
## Summary
This PR checks whether two callable types are equivalent or not.
This is required because for an equivalence relationship, the default
value does not necessarily need to be the same but if the parameter in
one of the callable has a default value then the corresponding parameter
in the other callable should also have a default value. This is the main
reason a manual implementation is required.
And, as per https://typing.python.org/en/latest/spec/callables.html#id4,
the default _type_ doesn't participate in a subtype relationship, only
the optionality (required or not) participates. This means that the
following two callable types are equivalent:
```py
def f1(a: int = 1) -> None: ...
def f2(a: int = 2) -> None: ...
```
Additionally, the name of positional-only, variadic and keyword-variadic
are not required to be the same for an equivalence relation.
A potential solution to avoid the manual implementation would be to only
store whether a parameter has a default value or not but the type is
currently required to check for assignability.
## Test plan
Add tests for callable types in `is_equivalent_to.md`
## Summary
Catch some Instances, but raise type error for the rest of them
Fixes#16851
## Test Plan
Extend invalid.md in annotations
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Currently for something like `X = typing.Tuple[str, str]`, we infer the
value of `X` as `object`. That's because `Tuple` (like many of the
symbols in the typing module) is annotated as a `_SpecialForm` instance
in typeshed's stubs:
23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L215)
and we don't understand implicit type aliases yet, and the stub for
`_SpecialForm.__getitem__` says it always returns `object`:
23382f5f8c/crates/red_knot_vendored/vendor/typeshed/stdlib/typing.pyi (L198-L200)
We have existing false positives in our test suite due to this:
23382f5f8c/crates/red_knot_python_semantic/resources/mdtest/annotations/annotated.md?plain=1#L76-L78
and it's causing _many_ new false positives in #16872, which tries to
make our annotation-expression parsing stricter in some ways.
This PR therefore adds some small special casing for `KnownInstanceType`
variants that fallback to `_SpecialForm`, so that these false positives
can be avoided.
## Test Plan
Existing mdtest altered.
Cc. @MatthewMckee4
Summary
--
Fixes#16874. I previously emitted a syntax error when starred
annotations were _allowed_ rather than when they were actually used.
This caused false positives for any starred parameter name because these
are allowed to have starred annotations but not required to. The fix is
to check if the annotation is actually starred after parsing it.
Test Plan
--
New inline parser tests derived from the initial report and more
examples from the comments, although I think the first case should cover
them all.
## Summary
From #16641
The previous PR attempted to fix the errors presented in this PR, but as
discussed in the conversation, it was concluded that the approach was
undesirable and that further work would be needed to fix the errors with
a correct general solution.
In this PR, I instead add the test cases from the previous PR as TODOs,
as a starting point for future work.
## Test Plan
---------
Co-authored-by: Carl Meyer <carl@oddbird.net>
## Summary
Fixes#16744
Allows the cli to find a virtual environment from the VIRTUAL_ENV
environment variable if no `--python` is set
## Test Plan
Manual testing, of:
- Virtual environments explicitly activated using `source .venv/bin/activate`
- Virtual environments implicilty activated via `uv run`
- Broken virtual environments with no `pyvenv.cfg` file
## Summary
This change follows up on the bug-fix requested in #16747 --
`ruff_python_ast::OperatorPrecedence` had an enum variant, `BitXorOr`,
which which gave the same precedence to the `|` and `^` operators. This
goes against [Python's documentation for operator
precedence](https://docs.python.org/3/reference/expressions.html#operator-precedence),
so this PR changes the code so that it's correct.
This is part of the overall effort to unify redundant definitions of
`OperatorPrecedence` throughout the codebase (#16071)
## Test Plan
Because this is an internal change, I only ran existing tests to ensure
nothing was broken.
The single flag `has_syntax_error` on `LinterResult` is replaced with
two (private) flags: `has_valid_syntax` and
`has_no_unsupported_syntax_errors`, which record whether there are
`ParseError`s or `UnsupportedSyntaxError`s, respectively. Only the
former is used to prevent a `FixAll` action.
An attempt has been made to make consistent the usage of the phrases
"valid syntax" (which seems to be used to refer only to _parser_ errors)
and "syntax error" (which refers to both _parser_ errors and
version-specific syntax errors).
Closes#16841
## Summary
Fixes#8191 by introducing `--exit-non-zero-on-format` to `ruff format`
which pretty much does what it says on the tin.
## Test Plan
Added a new test!
---------
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
This PR reworks `TypeInferenceBuilder::infer_type_expression()` so that
we emit diagnostics when encountering a list literal in a type
expression. The only place where a list literal is allowed in a type
expression is if it appears as the first argument to `Callable[]`, and
`Callable` is already heavily special-cased in our type-expression
parsing.
In order to ensure that list literals are _always_ allowed as the
_first_ argument to `Callabler` (but never allowed as the second, third,
etc. argument), I had to do some refactoring of our type-expression
parsing for `Callable` annotations.
## Test Plan
New mdtests added, and existing ones updated
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
fixes#15048
We want to handle more types from Type::KnownInstance
## Test Plan
Add tests for each type added explicitly in the match
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
These are just cosmetic changes, but I'm separating them out into a
standalone PR to make a branch I have stacked on top of this easier to
review
## Test Plan
Existing tests all pass
Summary
--
This PR updates `check_path` in the `ruff_linter` crate to return a
`Vec<Message>` instead of a `Vec<Diagnostic>`. The main motivation for
this is to make it easier to convert semantic syntax errors directly
into `Message`s rather than `Diagnostic`s in #16106. However, this also
has the benefit of keeping the preview check on unsupported syntax
errors in `check_path`, as suggested in
https://github.com/astral-sh/ruff/pull/16429#discussion_r1974748024.
All of the interesting changes are in the first commit. The second
commit just renames variables like `diagnostics` to `messages`, and the
third commit is a tiny import fix.
I also updated the `ExpandedMessage::location` field name, which caused
a few extra commits tidying up the playground code. I thought it was
nicely symmetric with `end_location`, but I'm happy to revert that too.
Test Plan
--
Existing tests. I also tested the playground and server manually.
## Summary
Part of #15382
This PR infers the return type `lambda` expression as `Unknown`. In the
future, it would be more useful to infer the expression type considering
the surrounding context (#16696).
## Test Plan
Update existing test cases from `@todo` to the (verified) return type.
## Summary
Previously, the `name` field was on `Parameter` which required it to be
always optional regardless of the parameter kind because a
`typing.Callable` signature does not have name for the parameters. This
is the case for positional-only parameters. This wasn't enforced at the
type level which meant that downstream usages would have to unwrap on
`name` even though it's guaranteed to be present.
This commit moves the `name` field from `Parameter` to the
`ParameterKind` variants and makes it optional only for
`ParameterKind::PositionalOnly` variant while required for all other
variants.
One change that's now required is that a `Callable` form using a gradual
form for parameter types (`...`) would have a default `args` and
`kwargs` name used for variadic and keyword-variadic parameter kind
respectively. This is also the case for invalid `Callable` type forms. I
think this is fine as names are not relevant in this context but happy
to make it optional even in variadic variants.
## Test Plan
No new tests; make sure existing tests are passing.
## Summary
Add error messages for invalid nodes in type expressions
Fixes#16816
## Test Plan
Extend annotations/invalid.md to handle these invalid AST nodes error
messages
## Summary
This PR adds a playground for Red Knot
[Screencast from 2024-08-14
10-33-54.webm](https://github.com/user-attachments/assets/ae81d85f-74a3-4ba6-bb61-4a871b622f05)
Sharing does work 😆 I just forgot to start wrangler.
It supports:
* Multiple files
* Showing the AST
* Showing the tokens
* Sharing
* Persistence to local storage
Future extensions:
* Configuration support: The `pyproject.toml` would *just* be another
file.
* Showing type information on hover
## Blockers
~~Salsa uses `catch_unwind` to break cycles, which Red Knot uses
extensively when inferring types in the standard library.
However, WASM (at least `wasm32-unknown-unknown`) doesn't support
`catch_unwind` today, so the playground always crashes when the type
inference encounters a cycle.~~
~~I created a discussion in the [salsa
zulip](https://salsa.zulipchat.com/#narrow/stream/333573-salsa-3.2E0/topic/WASM.20support)
to see if it would be possible to **not** use catch unwind to break
cycles.~~
~~[Rust tracking issue for WASM catch unwind
support](https://github.com/rust-lang/rust/issues/118168)~~
~~I tried to build the WASM with the nightly compiler option but ran
into problems because wasm-bindgen doesn't support WASM-exceptions. We
could try to write the binding code by hand.~~
~~Another alternative is to use `wasm32-unknown-emscripten` but it's
rather painful to build~~
## Summary
This PR detects the use of PEP 701 f-strings before 3.12. This one
sounded difficult and ended up being pretty easy, so I think there's a
good chance I've over-simplified things. However, from experimenting in
the Python REPL and checking with [pyright], I think this is correct.
pyright actually doesn't even flag the comment case, but Python does.
I also checked pyright's implementation for
[quotes](98dc4469cc/packages/pyright-internal/src/analyzer/checker.ts (L1379-L1398))
and
[escapes](98dc4469cc/packages/pyright-internal/src/analyzer/checker.ts (L1365-L1377))
and think I've approximated how they do it.
Python's error messages also point to the simple approach of these
characters simply not being allowed:
```pycon
Python 3.11.11 (main, Feb 12 2025, 14:51:05) [Clang 19.1.6 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> f'''multiline {
... expression # comment
... }'''
File "<stdin>", line 3
}'''
^
SyntaxError: f-string expression part cannot include '#'
>>> f'''{not a line \
... continuation}'''
File "<stdin>", line 2
continuation}'''
^
SyntaxError: f-string expression part cannot include a backslash
>>> f'hello {'world'}'
File "<stdin>", line 1
f'hello {'world'}'
^^^^^
SyntaxError: f-string: expecting '}'
```
And since escapes aren't allowed, I don't think there are any tricky
cases where nested quotes or comments can sneak in.
It's also slightly annoying that the error is repeated for every nested
quote character, but that also mirrors pyright, although they highlight
the whole nested string, which is a little nicer. However, their check
is in the analysis phase, so I don't think we have such easy access to
the quoted range, at least without adding another mini visitor.
## Test Plan
New inline tests
[pyright]:
https://pyright-play.net/?pythonVersion=3.11&strict=true&code=EYQw5gBAvBAmCWBjALgCgO4gHaygRgEoAoEaCAIgBpyiiBiCLAUwGdknYIBHAVwHt2LIgDMA5AFlwSCJhwAuCAG8IoMAG1Rs2KIC6EAL6iIxosbPmLlq5foRWiEAAcmERAAsQAJxAomnltY2wuSKogA6WKIAdABWfPBYqCAE%2BuSBVqbpWVm2iHwAtvlMWMgB2ekiolUAgq4FjgA2TAAeEMieSADWCsoV5qoaqrrGDJ5MiDz%2B8ABuLqosAIREhlXlaybrmyYMXsDw7V4AnoysyAmQ5SIhwYo3d9cheADUeKlv5O%2BpQA
## Summary
For now, `property_tests.rs` has grown larger and larger, making the
file difficult to read and maintain.
Although the code has been split, the test paths and full names remain
unchanged. There are no changes affecting test execution.
## Summary
This PR simplifies `IterationError` and `ContextManagerError` so that
they no longer "remember" what type it was that was (respectively) not
iterable or not valid as a context manager. Instead, the type that was
iterated over (or was used as a context manager) is passed back in when
calling the error struct's `report_diagnostic` method.
The motivations for this are:
- It significantly simplifies the code
- It reduces the size of these types on the stack
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This PR brings an optimization.
- `get_cached_db` no longer returns a `MutexGuard`; instead, it returns
a cloned database.
### `get_cached_db`
Previously, the `MutexGuard` was held inside the property test function
(defined in the macro), which prevented multiple property tests from
running in parallel. More specifically, the program could only test one
random test case at a time, which likely caused a significant
bottleneck.
On my local machine, running:
```
QUICKCHECK_TESTS=100000 cargo test --release -p red_knot_python_semantic -- --ignored stable
```
showed about **a 75% speedup** (from \~60s to \~15s).
Fixes#16751
## Summary
Previously, unsafe fixes were counted as "fixable" in
`Printer::write_statistics`, in contrast to the behaviour in
`Printer::write_once`. This changes the behaviour to align with
`write_once`, including them only if `--unsafe-fixes` is set.
We now also reuse `Printer::write_summary` to avoid duplicating the
logic for whether or not to report if there are hidden fixes.
## Test Plan
Existing tests modified to use an unsafe-fixable rule, and new ones
added to cover the case with `--unsafe-fixes`
These should all be minor cosmetic changes. To summarize:
* In many cases, `-` was replaced with `^` for primary annotations.
This is because, previously, whether `-` or `^` was used depended
on the severity. But in the new data model, it's based on whether
the annotation is "primary" or not. We could of course change this
in whatever way we want, but I think we should roll with this for now.
* The "secondary messages" in the old API are rendered as
sub-diagnostics. This in turn results in a small change in the output
format, since previously, the secondary messages were represented as
just another snippet. We use sub-diagnostics because that's the intended
way to enforce relative ordering between messages within a diagnostic.
* The "info:" prefix used in some annotation messages has been dropped.
We could re-add this, but I think I like it better without this prefix.
I believe those 3 cover all of the snapshot changes here.
... and switch to the new one.
We do this switch by converting the old diagnostics to a
`Diagnostic`, and then rendering that.
This does not quite emit identical output. There are some
changes. They *could* be fixed to remain the same, but the
changes aren't obviously worse to me and I think the right
way to *improve* them is to move Red Knot to the new `Diagnostic`
API.
The next commit will have the snapshot changes.
In our existing diagnostics, our message is just the diagnostic
ID, and the message goes to the annotation. In reality, the
diagnostic can have its own message distinct from the optional
messages associated with an annotation.
In order to make the outputs match, we do a small tweak here:
when the main diagnostic message is empty, we drop the colon
after the diagnostic ID.
I expect that we'll want to rejigger this output format more
in the future, but for now this was a very simple change to
preserve the status quo.
When moving over to the new renderer, I noticed that it
was emitting an extra line terminator compared to the status
quo. This removes it by turning the line terminator into a
line delimiter between diagnostics.
This cleans up how we handle calling unions of types. #16568 adding a
three-level structure for callable signatures (`Signatures`,
`CallableSignature`, and `Signature`) to handle unions and overloads.
This PR updates the bindings side to mimic that structure. What used to
be called `CallOutcome` is now `Bindings`, and represents the result of
binding actual arguments against a possible union of callables.
`CallableBinding` is the result of binding a single, possibly
overloaded, callable type. `Binding` is the result of binding a single
overload.
While we're here, this also cleans up `CallError` greatly. It was
previously extracting error information from the bindings and storing it
in the error result. It is now a simple enum, carrying no data, that's
used as a status code to talk about whether the overall binding was
successful or not. We are now more consistent about walking the binding
itself to get detailed information about _how_ the binding was
unsucessful.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR fixes a bug in the check for fully static callable type where we
would skip unannotated parameter type.
## Test Plan
Add tests using the new `CallableTypeFromFunction` special form.
## Summary
This is a re-creation of https://github.com/astral-sh/ruff/pull/16764 by
@mtshiba, which I closed meaning to immediately reopen (GitHub wasn't
updating the PR with the latest pushed changes), and which GitHub will
not allow me to reopen for some reason. Pasting the summary from that PR
below:
> From https://github.com/astral-sh/ruff/pull/16641
>
> As stated in this comment
(https://github.com/astral-sh/ruff/pull/16641#discussion_r1996153702),
the current ordering implementation for intersection types is incorrect.
So, I will introduce lexicographic ordering for intersection types.
## Test Plan
One property test stabilised (tested locally with
`QUICKCHECK_TESTS=2000000 cargo test --release -p
red_knot_python_semantic -- --ignored
types::property_tests::stable::negation_reverses_subtype_order`), and
existing mdtests that previously failed now pass.
Primarily-authored-by:
[mtshiba](https://github.com/astral-sh/ruff/commits?author=mtshiba)
---------
Co-authored-by: Shunsuke Shibayama <sbym1346@gmail.com>
## Summary
A small followup to https://github.com/astral-sh/ruff/pull/16386. We now
tell the user exactly what it was about their decorator that constituted
invalid syntax on Python <3.9, and the range now highlights the specific
sub-expression that is invalid rather than highlighting the whole
decorator
## Test Plan
Inline snapshots are updated, and new ones are added.
## Summary
Stop flagging each invocation of `django.utils.safestring.mark_safe`
(also available at, `django.utils.html.mark_safe`) as an error.
Instead, allow string literals as valid uses for `mark_safe`.
Also, update the documentation, pointing at
`django.utils.html.format_html` for dynamic content generation use
cases.
Closes#16702
## Test Plan
I verified several possible uses, but string literals, are still
flagged.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
This PR adds a new `--color` CLI option that controls whether the output
should be colorized or not.
This is implements part of
https://github.com/astral-sh/ruff/issues/16727 except that it doesn't
implement the persistent configuration support as initially proposed in
the CLI document. I realized, that having this as a persistent
configuration is somewhat awkward because we may end up writing tracing
logs **before** we loaded and resolved the settings. Arguably, it's
probably fine to color the output up to that point, but it feels like a
somewhat broken experience. That's why I decided not to add the
persistent configuration option for now.
## Test Plan
I tested this change manually by running Red Knot with `--color=always`,
`--color=never`, and `--color=auto` (or no argument) and verified that:
* The diagnostics are or aren't colored
* The tracing output is or isn't colored.
---------
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
## Summary
Another salsa upgrade.
The main motivation is to stay on a recent salsa version because there
are still a lot of breaking changes happening.
The most significant changes in this update:
* Salsa no longer derives `Debug` by default. It now requires
`interned(debug)` (or similar)
* This version ships the foundation for garbage collecting interned
values. However, this comes at the cost that queries now track which
interned values they created (or read). The micro benchmarks in the
salsa repo showed a significant perf regression. Will see if this also
visible in our benchmarks.
## Test Plan
`cargo test`
Currently the red-knot LSP server emits any log messages of level `INFO`
or higher from non-red-knot crates. This makes its output quite verbose,
because Salsa emits an `INFO` level message every time it executes a
query. I use red-knot as LSP with neovim, and this spams the log file
quite a lot.
It seems like a better default to only emit `WARN` or higher messages
from non-red-knot sources.
I confirmed that this fixes the nvim LSP log spam.
## Summary
Uses the `try_call_dunder` infrastructure for augmented assignment and
fixes the logic to work for types other than `Type::Instance(…)`. This
allows us to infer the correct type here:
```py
x = (1, 2)
x += (3, 4)
reveal_type(x) # revealed: tuple[Literal[1], Literal[2], Literal[3], Literal[4]]
```
Or in this (extremely weird) scenario:
```py
class Meta(type):
def __iadd__(cls, other: int) -> str:
return ""
class C(metaclass=Meta): ...
cls = C
cls += 1
reveal_type(cls) # revealed: str
```
Union and intersection handling could also be improved here, but I made
no attempt to do so in this PR.
## Test Plan
New MD tests
## Summary
A follow-up to https://github.com/astral-sh/ruff/pull/16705 which
documents various kinds of diagnostics that can appear when assigning to
an attribute.
## Test Plan
New snapshot tests.
## Summary
This PR implements the first part of
https://github.com/astral-sh/ruff/discussions/16440. It ensures that Red
Knot's module resolver is case sensitive on all systems.
This PR combines a few approaches:
1. It uses `canonicalize` on non-case-sensitive systems to get the real
casing of a path. This works for as long as no symlinks or mapped
network drives (the windows `E:\` is mapped to `\\server\share` thingy).
This is the same as what Pyright does
2. If 1. fails, fall back to recursively list the parent directory and
test if the path's file name matches the casing exactly as listed in by
list dir. This is the same approach as CPython takes in its module
resolver. The main downside is that it requires more syscalls because,
unlike CPython, we Red Knot needs to invalidate its caches if a file
name gets renamed (CPython assumes that the folders are immutable).
It's worth noting that the file watching test that I added that renames
`lib.py` to `Lib.py` currently doesn't pass on case-insensitive systems.
Making it pass requires some more involved changes to `Files`. I plan to
work on this next. There's the argument that landing this PR on its own
isn't worth it without this issue being addressed. I think it's still a
good step in the right direction even when some of the details on how
and where the path case sensitive comparison is implemented.
## Test plan
I added multiple integration tests (including a failing one). I tested
that the `case-sensitivity` detection works as expected on Windows,
MacOS and Linux and that the fast-paths are taken accordingly.
## Summary
This PR detects unparenthesized assignment expressions used in set
literals and comprehensions and in sequence indexes. The link to the
release notes in https://github.com/astral-sh/ruff/issues/6591 just has
this entry:
> * Assignment expressions can now be used unparenthesized within set
literals and set comprehensions, as well as in sequence indexes (but not
slices).
with no other information, so hopefully the test cases I came up with
cover all of the changes. I also tested these out in the Python REPL and
they actually worked in Python 3.9 too. I'm guessing this may be another
case that was "formally made part of the language spec in Python 3.10,
but usable -- and commonly used -- in Python >=3.9" as @AlexWaygood
added to the body of #6591 for context managers. So we may want to
change the version cutoff, but I've gone along with the release notes
for now.
## Test Plan
New inline parser tests and linter CLI tests.
We don't actually hook this up to anything in this PR, but we do
go to some trouble to granularly unit test it. The unit tests caught
plenty of bugs after I initially wrote down the implementation, so they
were very much worth it.
Closes#16506
Instead of hard-coding a specific context window,
it seemed prudent to make this configurable. That
makes it easier to test different context window
sizes as well.
I am not totally convinced that this is the right
place for this configuration. I could see the context
window size being a property of `Diagnostic` instead,
since we might want to change the context window
size based not just on some end user configuration,
but perhaps also the specific diagnostic.
But for now, I think it's fine for it to live here,
and all of the rendering logic doesn't care where
it lives. So it should be relatively easy to change
in the future.
This adds a new configuration knob to diagnostic rendering that, when
enabled, will make diagnostic rendering much more terse. Specifically,
it will guarantee that each diagnostic will only use one line.
This doesn't actually hook the concise output option up to anything.
We'll do that plumbing in the next commit.
Summary
--
This is closely related to (and stacked on)
https://github.com/astral-sh/ruff/pull/16544 and detects star
annotations in function definitions.
I initially called the variant `StarExpressionInAnnotation` to mirror
`StarExpressionInIndex`, but I realized it's not really a "star
expression" in this position and renamed it. `StarAnnotation` seems in
line with the PEP.
Test Plan
--
Two new inline tests. It looked like there was pretty good existing
coverage of this syntax, so I just added simple examples to test the
version cutoff.
## Summary
Follow-up release for Ruff v0.10 that now includes the following two
changes that we intended to ship but slipped:
* Changes to how the Python version is inferred when a `target-version`
is not specified (#16319)
* `blanket-noqa` (`PGH004`): Also detect blanked file-level noqa
comments (and not just line level comments).
## Test plan
I verified that the binary built on this branch respects the
`requires-python` setting
([logs](https://www.diffchecker.com/qyJWYi6W/), left: v0.10, right:
v0.11)
## Summary
This PR includes minor improvements to binary operation inference,
specifically for tuple concatenation.
### Before
```py
reveal_type((1, 2) + (3, 4)) # revealed: @Todo(return type of decorated function)
# If TODO is ignored, the revealed type would be `tuple[1|2|3|4, ...]`
```
The `builtins.tuple` type stub defines `__add__`, but it appears to only
work for homogeneous tuples. However, I think this limitation is not
ideal for many use cases.
### After
```py
reveal_type((1, 2) + (3, 4)) # revealed: tuple[Literal[1], Literal[2], Literal[3], Literal[4]]
```
## Test Plan
### Added
- `mdtest/binary/tuples.md`
### Affected
- `mdtest/slots.md` (a test have been moved out of the `False-Negative`
block.)
## Summary
This changeset adds proper support for assignments to attributes:
```py
obj.attr = value
```
In particular, the following new features are now available:
* We previously didn't raise any errors if you tried to assign to a
non-existing attribute `attr`. This is now fixed.
* If `type(obj).attr` is a data descriptor, we now call its `__set__`
method instead of trying to assign to the load-context type of
`obj.attr`, which can be different for data descriptors.
* An initial attempt was made to support unions and intersections, as
well as possibly-unbound situations. There are some remaining TODOs in
tests, but they only affect edge cases. Having nested diagnostics would
be one way that could help solve the remaining cases, I believe.
## Follow ups
The following things are planned as follow-ups:
- Write a test suite with snapshot diagnostics for various attribute
assignment errors
- Improve the diagnostics. An easy improvement would be to highlight the
right hand side of the assignment as a secondary span (with the rhs type
as additional information). Some other ideas are mentioned in TODO
comments in this PR.
- Improve the union/intersection/possible-unboundness handling
- Add support for calling custom `__setattr__` methods (see new false
positive in the ecosystem results)
## Ecosystem changes
Some changes are related to assignments on attributes with a custom
`__setattr__` method (see above). Since we didn't notice missing
attributes at all in store context previously, these are new.
The other changes are related to properties. We previously used their
read-context type to test the assignment. That results in weird error
messages, as we often see assignments to `self.property` and then we
think that those are instance attributes *and* descriptors, leading to
union types. Now we properly look them up on the meta type, see the
decorated function, and try to overwrite it with the new value (as we
don't understand decorators yet). Long story short: the errors are still
weird, we need to understand decorators to make them go away.
## Test Plan
New Markdown tests
Summary
--
This PR reuses a slightly modified version of the
`check_tuple_unpacking` method added for detecting unpacking in `return`
and `yield` statements to detect the same issue in the iterator clause
of `for` loops.
I ran into the same issue with a bare `for x in *rest: ...` example
(invalid even on Python 3.13) and added it as a comment on
https://github.com/astral-sh/ruff/issues/16520.
I considered just making this an additional `StarTupleKind` variant as
well, but this change was in a different version of Python, so I kept it
separate.
Test Plan
--
New inline tests.
There can be semi-cyclic inheritance patterns (e.g. recursive generics)
that are not technically inheritance cycles, but that can cause us to
hit Salsa query cycles in evaluating a type's MRO. Add fixed-point
handling to these MRO-related queries so we don't panic on these cycles.
The details of what queries we hit in what order in this case will
change as we implement support for generics, but ultimately we will
probably need cycle handling for all queries that can re-enter type
inference, otherwise we are susceptible to small changes in query
execution order causing panics.
Fixes#14333
Further reduces the panicking set of seeds in #14737
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Part of #15655
Replaced statement nodes with autogenerated ones. Reused the stuff we
introduced in #16285. Nothing except for copying the nodes to new
format.
## Test Plan
Tests run without any changes. Also moved the test that checks size of
AST nodes to `generated.rs` since all of the structs that it tests are
now there.
<!-- How was it tested? -->
## Summary
This PR stabilizes the behavior introduced in
https://github.com/astral-sh/ruff/pull/15985
The new behavior improves the inference of `str.strip` calls:
* before: The rule only considered calls on string or byte literals
(`"abcd".strip`)
* now: The rule also catches calls to `strip` on object where the type
is known to be a `str` or `bytes` (e.g. `a = "abc"; a.strip("//")`)
The new behavior shipped as part of Ruff 0.9.6 on the 10th of Feb which
is a little more than a month ago.
There have been now new issues or PRs related to the new behavior.
## Summary
This PR promotes the fix improvements for `PLR1714` that were introduced
in https://github.com/astral-sh/ruff/pull/14372/ to stable.
The improvement is that the fix now proposes to use a set if all
elements are hashable:
```
foo == "bar" or foo == "baz" or foo == "qux"
```
Gets fixed to
```py
foo in {"bar", "baz", "qux"}
```
where it previously always got fixed to a tuple.
The new fix was first released in ruff 0.8.0 (Nov last year). This is
not a breaking change. The change was preview gated only to get some
extra test coverage.
There are no open issues or PRs related to this changed fix behavior.
## Summary
This PR stabilizes the behavior changes introduced by
https://github.com/astral-sh/ruff/pull/13305 that were gated behind
preview.
The change is that `__new__` methods are now no longer flagged by
`invalid-first-argument-name-for-class-method` (`N804`) but instead by
`bad-staticmethod-argument` (`PLW0211`)
> __new__ methods are technically static methods, with cls as their
first argument. However, Ruff currently classifies them as classmethod,
which causes two issues:
## Test Plan
There have been no new issues or PRs related to `N804` or `PLW0211`
since the behavior change was released in Ruff 0.9.7 (about 3 weeks
ago).
This is a somewhat recent change but I don't think it's necessary to
leave this in preview for another 2 months. The main reason why it was
in preview
is that it is breaking, not because it is a risky change.
## Summary
This PR stabilizes the fix for `PYI018` introduced in
https://github.com/astral-sh/ruff/pull/15999/ (first released with Ruff
0.9.5 early February)
There are no known issues with the fix or open PRs.
## Summary
Deprecate `S320` because defusedxml has deprecated there `lxml` module
and `lxml` has been hardened since.
flake8-bandit has removed their implementation as well
(https://github.com/PyCQA/bandit/pull/1212).
Addresses https://github.com/astral-sh/ruff/issues/13707
## Test Plan
I verified that selecting `S320` prints a warning and fails if the
preview mode is enabled.
## Summary
This PR stabilizes the fixes improvements made in
https://github.com/astral-sh/ruff/pull/15562 (released with ruff 0.9.3
in mid January).
There's no open issue or PR related to the changed fix behavior.
This is not a breaking change. The fix was only gated behind preview to
get some more test coverage before releasing.
## Summary
This PR stabilizes the behavior change introduced in
https://github.com/astral-sh/ruff/pull/15872/
The diagnostic range is now the range of the redundant `mode` argument
where it previously was the range of the entire `open` call:
Before:
```
UP015.py:2:1: UP015 [*] Unnecessary mode argument
|
1 | open("foo", "U")
2 | open("foo", "Ur")
| ^^^^^^^^^^^^^^^^^ UP015
3 | open("foo", "Ub")
4 | open("foo", "rUb")
|
= help: Remove mode argument
```
Now:
```
UP015.py:2:13: UP015 [*] Unnecessary mode argument
|
1 | open("foo", "U")
2 | open("foo", "Ur")
| ^^^^ UP015
3 | open("foo", "Ub")
4 | open("foo", "rUb")
|
= help: Remove mode argument
```
This is a breaking change because it may require moving a `noqa` comment
onto a different line, e.g if you have
```py
open(
"foo",
"Ur",
) # noqa: UP015
```
Needs to be rewritten to
```py
open(
"foo",
"Ur", # noqa: UP015
)
```
There have been now new issues or PRs since the new preview behavior was
implemented. It first was released as part of Ruff 0.9.5 on the 5th of
Feb (a little more than a month ago)
## Test Plan
I reviewed the snapshot tests
## Summary
This PR stabilizes the preview behavior introduced in
https://github.com/astral-sh/ruff/pull/15719 to recognize all symbols
named `TYPE_CHECKING` as type-checking
checks in `if TYPE_CHECKING` conditions. This ensures compatibility with
mypy and pyright.
This PR also stabilizes the new behavior that removes `if 0:` and `if
False` to be no longer considered type checking blocks.
Since then, this syntax has been removed from the typing spec and was
only used for Python modules that don't have a `typing` module
([comment](https://github.com/astral-sh/ruff/pull/15719#issuecomment-2612787793)).
The preview behavior was first released with Ruff 0.9.5 (6th of
February), which was about a month ago. There are no open issues or PRs
for the changed behavior
## Test Plan
The snapshots for `SIM108` change because `SIM108` ignored type checking
blocks but it can no
simplify `if 0` or `if False` blocks again because they're no longer
considered type checking blocks.
The changes in the `TC005` snapshot or only due to that `if 0` and `if
False` are no longer recognized as type checking blocks
<!-- How was it tested? -->
## Summary
This PR stabilizes the preview behavior for `invalid-argument-name`
(`N803`)
to ignore argument names of functions decorated with `typing.override`
because
these methods are *out of the authors* control.
This behavior was introduced in
https://github.com/astral-sh/ruff/pull/15954
and released as part of Ruff 0.9.5 (6th of February).
There have been no new issues or PRs since this behavior change
(preview) was introduced.
## Summary
This PR stabilizes the preview behavior introduced in
https://github.com/astral-sh/ruff/pull/15905
The behavior change is that the rule now also recognizes `type(expr) is
type(None)` comparisons where `expr` isn't a name expression.
For example, the rule now detects `type(a.b) is type(None)` and suggests
rewriting the comparison to `a.b is None`.
The new behavior was introduced with Ruff 0.9.5 (6th of February), about
a month ago. There are no open issues or PRs related to this rule (or
behavior change).
## Summary
This PR stabilizes the new behavior introduced in
https://github.com/astral-sh/ruff/pull/14512 to also detect defalut
value arguemnts to `os.environ.get` that have an invalid type (not
`str`).
There's an upstream issue for this behavior change
https://github.com/pylint-dev/pylint/issues/10092 that was accepted and
a PR, but it hasn't been merged yet.
This behavior change was first shipped with Ruff 0.8.1 (Nov 22).
There has only be one PR since the new behavior was introduced but it
was unrelated to the scope increase
(https://github.com/astral-sh/ruff/pull/14841).
## Summary
This PR stabilizes the behavior change introduced in
https://github.com/astral-sh/ruff/pull/15542 to allow
for statements with an empty body in `pytest.raises` and `pytest.warns`
with statements.
This raised an error before but is now allowed:
```py
with pytest.raises(KeyError, match='unknown'):
async for _ in gpt.generate(gpt_request):
pass
```
The same applies to
```py
with pytest.raises(KeyError, match='unknown'):
async for _ in gpt.generate(gpt_request):
...
```
There have been now new issues or PRs related to PT012 or PT031 since
this behavior change was introduced in ruff 0.9.3 (January 23rd).
## Summary
This PR deprecates UP038. Using PEP 604 syntax in `isinstance` and
`issubclass` calls isn't a recommended pattern (or community agreed best
practice)
and it negatively impacts performance.
Resolves https://github.com/astral-sh/ruff/issues/7871
## Test Plan
I tested that selecting `UP038` results in a warning in no-preview mode
and an error in preview mode
Summary
--
Stabilizes TC006. The test was already in the right place.
Test Plan
--
No open issues or PRs. The last related [issue] was closed on
2025-02-09.
[issue]: https://github.com/astral-sh/ruff/issues/16037
Summary
--
Stabilizes S704, which is also being recoded from RUF035 in 0.10.
Test Plan
--
Existing tests with `PreviewMode` removed from the settings.
There was one issue closed on 2024-12-20 calling the rule noisy and
asking for a config option, but the option was added and then there were
no more issues or PRs.
Summary
--
Stabilizes DTZ901, renames the rule function to match the rule name,
removes the `preview_rules` test, and handles some nits in the docs
(mention `min` first to match the rule name too).
Test Plan
--
1 closed issue on 2024-11-12, 4 days after the rule was added. No issues
since
## Summary
Follow-up to #16677.
This change converts all unit tests (69 of them) in `noqa.rs` to use
inline snapshots instead. It extends the file by more than 1000 lines,
but the tests are now much easier to read and reason about.
## Test Plan
`cargo insta test`.
Summary
--
Stabilizes RUF041. The tests are already in the right place, and the
docs look good.
Test Plan
--
0 issues, 1 [PR] fixing nested literals and unions the day after the
rule was added. No changes since then
I wonder if the fix in that PR could be relevant for
https://github.com/astral-sh/ruff/pull/16639, where I noticed a
potential issue with `Union`. It could be unrelated, though.
[PR]: https://github.com/astral-sh/ruff/pull/14641
Summary
--
Stabilizes PTH210. Tests and docs looked good.
Test Plan
--
Mentioned in 1 open issue around Python 3.14 support (`"."` becomes a
valid suffix in 3.14). Otherwise no issues or PRs since 2024-12-12, 6
days after the rule was added.
## Summary
Follow-up to #16659.
This change adds tests for these three cases, which are (also) not
covered by existing tests:
* `# noqa: A` (lone incomplete code)
* `# noqa: A123, B` (complete codes, last one incomplete)
* `# noqa: A123B` (squashed codes, last one incomplete)
Summary
--
Stabilizes RUF051. The tests and docs looked good.
Test Plan
--
1 closed documentation issue from 4 days after the rule was added and 1
typo fix from the same day it was added, but no other issues or PRs.
Summary
--
Stabilizes LOG015. The tests and docs looked good.
Test Plan
--
1 closed documentation issue from 4 days after the rule was added, but
no other issues or PRs.
Summary
--
Stabilizes RUF048 and moves its test to the right place. The docs look
good.
Test Plan
--
0 closed or open issues. There was 1 [PR] related to an extension to the
rule, but it was closed without comment.
[PR]: https://github.com/astral-sh/ruff/pull/14701
Summary
--
Stabilizes RUF046 and moves its test to the right place. The docs look
good.
Test Plan
--
2 closed newline/whitespace issues from early January and 1 closed issue
about really being multiple rules, but otherwise no recent issues or
PRs.
# Summary
The goal of this PR is to address various issues around parsing
suppression comments by
1. Unifying the logic used to parse in-line (`# noqa`) and file-level
(`# ruff: noqa`) noqa comments
2. Recovering from certain errors and surfacing warnings in these cases
Closes#15682
Supersedes #12811
Addresses
https://github.com/astral-sh/ruff/pull/14229#discussion_r1835481018
Related: #14229 , #12809
## Summary
This PR changes the default value of
`lint.flake8-builtins.builtins-strict-checking` added in
https://github.com/astral-sh/ruff/pull/15951 from `true` to `false`.
This also allows simplifying the default option logic and removes the
dependence on preview mode.
https://github.com/astral-sh/ruff/issues/15399 was already closed by
#15951, but this change will finalize the behavior mentioned in
https://github.com/astral-sh/ruff/issues/15399#issuecomment-2587017147.
As an example, strict checking flags modules based on their last
component, so `utils/logging.py` triggers A005. Non-strict checking
checks the path to the module, so `utils/logging.py` is allowed (this is
the example and desired behavior from #15399 exactly) but a top-level
`logging.py` or `logging/__init__.py` is still disallowed.
## Test Plan
Existing tests from #15951 and #16006, with the snapshot updated in
`a005_module_shadowing_strict_default` to reflect the new default.
Summary
--
Stabilizes UP044, renames the module to match the rule name, and removes
the `PreviewMode` from the test settings.
Test Plan
--
2 closed issues in November, just after the rule was added, otherwise no
issues
Summary
--
Stabilizes SIM905 and adds a small addition to the docs. The test was
already in the right place.
Test Plan
--
No issues except 2 recent, general issues about whitespace
normalization.
## Summary
This PR stabilizes several preview-only behaviours for
`custom-typevar-for-self` (`PYI019`). Namely:
- A new, more accurate technique is now employed for detecting custom
TypeVars that are replaceable with `Self`. See
https://github.com/astral-sh/ruff/pull/15888 for details.
- The range of the diagnostic is now the full function header rather
than just the return annotation. (Previously, the rule only applied to
methods with return annotations, but this is no longer true due to the
changes in the first bullet point.)
- The fix is now available even when preview mode is not enabled.
## Test Plan
- Existing snapshots that do not have preview mode enabled are updated
- Preview-specific snapshots are removed
- I'll check the ecosystem report on this PR to verify everything's as
expected
Summary
--
Stabilizes PLC1802. The tests were already in the right place, and I
just tidied the docs a little bit.
Test Plan
--
1 issue closed 4 days after the rule was added, no other issues
Summary
--
Stabilizes PLW1507. The tests were already in the right place, and I
just tidied the docs a little bit.
Test Plan
--
1 issue from 2 weeks ago but just suggesting to mark the fix unsafe. The
shallow vs deep copy *does* change the program behavior, just usually in
a preferable way.
## Summary
Stabilizes FAST003, completing the group with FAST001 and FAST002.
## Test Plan
Last bug fix (false positive) was fixed on 2025-01-13, almost 2 months
ago.
The test case was already in the right place.
## Summary
Stabilizes C420 for the 0.10 release.
## Test Plan
No open issues or PRs (except a general issue about [string
normalization](https://github.com/astral-sh/ruff/issues/16579)). The
last (and only) false-negative bug fix was over a month ago.
The tests for this rule were already not on the `preview_rules` test, so
I just changed the `RuleGroup`. The documentation looked okay to me.
## Summary
Resolves#15368.
The following options have been renamed:
* `builtins-allowed-modules` → `allowed-modules`
* `builtins-ignorelist` → `ignorelist`
* `builtins-strict-checking` → `strict-checking`
To preserve compatibility, the old names are kept as Serde aliases.
## Test Plan
`cargo nextest run` and `cargo insta test`.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
`RUF035` has been backported into bandit as `S704` in this
[PR](https://github.com/PyCQA/bandit/pull/1225)
This moves the rule and its corresponding setting to the `flake8-bandit`
category
## Test Plan
`cargo nextest run`
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
For context, the initial implementation started out by sending a log
notification to the client to include this information in the client
channel. This is a bit ineffective because it doesn't allow the client
to display this information in a more obvious way. In addition to that,
it isn't obvious from a users perspective as to where the information is
being printed unless they actually open the output channel.
The change was to actually return this formatted string that contains
the information and let the client handle how it should display this
information. For example, in the Ruff VS Code extension we open a split
window and show this information which is similar to what rust-analyzer
does.
The notification request was kept as a precaution in case there are
users who are actually utilizing this way. If they exists, it should a
minority as it requires the user to actually dive into the code to
understand how to hook into this notification. With 0.10, we're removing
the old way as it only clobbers the output channel with a long message.
fixes: #16225
## Test Plan
Tested it out locally that the information is not being logged to the
output channel of VS Code.
## Summary
This PR stabilizies the fix for
https://github.com/astral-sh/ruff/issues/14001
We try to only make breaking formatting changes once a year. However,
the plan was to release this fix as part of Ruff 0.9 but I somehow
missed it when promoting all other formatter changes.
I think it's worth making an exception here considering that this is a
bug fix, it improves readability, and it should be rare
(very few files in a single project). Our version policy explicitly
allows breaking formatter changes in any minor release and the idea of
only making breaking formatter changes once a year is mainly to avoid
multiple releases throughout the year that introduce large formatter
changes
Closes https://github.com/astral-sh/ruff/issues/14001
## Test Plan
Updated snapshot
## Summary
Since Ruff changed to GitHub releases, tags are no longer annotated and
`git describe` no longer picks them up. Instead, it's necessary to also
search lightweight tags.
This changes fixes the `version` command to give more accurate
`last_tag`/`commits_since_last_tag` information. This only affects
development builds, as this information is not present in releases.
## Test Plan
Testing is a little tricky because this information changes on every
commit. Running manually on current `main` and my branch:
`main`:
```
# cargo run --bin ruff -- version --output-format=text
ruff 0.9.10+2547 (dd2313ab0 2025-03-12)
# cargo run --bin ruff -- version --output-format=json
{
"version": "0.9.10",
"commit_info": {
"short_commit_hash": "dd2313ab0",
"commit_hash": "dd2313ab0faea90abf66a75f1b5c388e728d9d0a",
"commit_date": "2025-03-12",
"last_tag": "v0.4.10",
"commits_since_last_tag": 2547
}
}
```
This PR:
```
# cargo run --bin ruff -- version --output-format=text
ruff 0.9.10+46 (11f39f616 2025-03-12)
# cargo run --bin ruff -- version --output-format=json
{
"version": "0.9.10",
"commit_info": {
"short_commit_hash": "11f39f616",
"commit_hash": "11f39f6166c3d7a521725b938a166659f64abb59",
"commit_date": "2025-03-12",
"last_tag": "0.9.10",
"commits_since_last_tag": 46
}
}
```
## Summary
This PR adds a new `CallableTypeFromFunction` special form to allow
extracting the abstract signature of a function literal i.e., convert a
`Type::Function` into a `Type::Callable` (`CallableType::General`).
This is done to support testing the `is_gradual_equivalent_to` type
relation specifically the case we want to make sure that a function that
has parameters with no annotations and does not have a return type
annotation is gradual equivalent to `Callable[[Any, Any, ...], Any]`
where the number of parameters should match between the function literal
and callable type.
Refer
https://github.com/astral-sh/ruff/pull/16634#discussion_r1989976692
### Bikeshedding
The name `CallableTypeFromFunction` is a bit too verbose. A possibly
alternative from Carl is `CallableTypeOf` but that would be similar to
`TypeOf` albeit with a limitation that the former only accepts function
literal types and errors on other types.
Some other alternatives:
* `FunctionSignature`
* `SignatureOf` (similar issues as `TypeOf`?)
* ...
## Test Plan
Update `type_api.md` with a new section that tests this special form,
both invalid and valid forms.
## Summary
Add support for calling custom `__getattr__` methods in case an
attribute is not otherwise found. This allows us to get rid of many
ecosystem false positives where we previously emitted errors when
accessing attributes on `argparse.Namespace`.
closes#16614
## Test Plan
* New Markdown tests
* Observed expected ecosystem changes (the changes for `arrow` also look
fine, since the `Arrow` class has a custom [`__getattr__`
here](1d70d00919/arrow/arrow.py (L802-L815)))
Pulls in the latest Salsa main branch, which supports fixpoint
iteration, and uses it to handle all query cycles.
With this, we no longer need to skip any corpus files to avoid panics.
Latest perf results show a 6% incremental and 1% cold-check regression.
This is not a "no cycles" regression, as tomllib and typeshed do trigger
some definition cycles (previously handled by our old
`infer_definition_types` fallback to `Unknown`). We don't currently have
a benchmark we can use to measure the pure no-cycles regression, though
I expect there would still be some regression; the fixpoint iteration
feature in Salsa does add some overhead even for non-cyclic queries.
I think this regression is within the reasonable range for this feature.
We can do further optimization work later, but I don't think it's the
top priority right now. So going ahead and acknowledging the regression
on CodSpeed.
Mypy primer is happy, so this doesn't regress anything on our
currently-checked projects. I expect it probably unlocks adding a number
of new projects to our ecosystem check that previously would have
panicked.
Fixes#13792Fixes#14672
## Summary
Implements attribute access on intersection types, which didn't
previously work. For example:
```py
from typing import Any
class P: ...
class Q: ...
class A:
x: P = P()
class B:
x: Any = Q()
def _(obj: A):
if isinstance(obj, B):
reveal_type(obj.x) # revealed: P & Any
```
Refers to [this comment].
[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985040363
## Test Plan
New Markdown tests
## Summary
Background - as a follow up to #16611 I noticed that there's a lot of
code duplicated between the `is_assignable_to` and `is_subtype_of`
functions and considered trying to merge them.
[A subtype and an assignable type are pretty much the
same](https://typing.python.org/en/latest/spec/concepts.html#the-assignable-to-or-consistent-subtyping-relation),
except that subtypes are by definition fully static, so I think we can
replace the whole of `is_subtype_of` with:
```
if !self.is_fully_static(db) || !target.is_fully_static(db) {
return false;
}
return self.is_assignable_to(target)
```
if we move all of the logic to is_assignable_to and delete duplicate
code. Then we can discuss if it even makes sense to have a separate
is_subtype_of function (I think the answer is yes since it's used by a
bunch of other places, but we may be able to basically rip out the
concept).
Anyways while playing with combining the functions I noticed is that the
handling of Intersections in `is_subtype_of` has a special case for two
intersections, which I didn't include in the last PR - rather I first
handled right hand intersections before left hand, which should properly
handle double intersections (hand-wavy explanation I can justify if
needed - (A & B & C) is assignable to (A & B) because the left is
assignable to both A and B, but none of A, B, or C is assignable to (A &
B)).
I took a look at what breaks if I remove the handling for double
intersections, and the reason it is needed is because is_disjoint does
not properly handle intersections with negative conditions (so instead
`is_subtype_of` basically implements the check correctly).
This PR adds support to is_disjoint for properly checking negative
branches, which also lets us simplify `is_subtype_of`, bringing it in
line with `is_assignable_to`
## Test Plan
Added a bunch of tests, most of which failed before this fix
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This is a pure restructuring of the `attributes.md` and
`descriptor_protocol.md` test suites. They have grown organically and I
didn't want to make major structural changes in my recent PR to keep the
diff clean.
## Summary
A follow up to address [this comment]:
> Similarly here, it might be a little more performant to have a single
`Type::instance()` branch with an inner match over `class.known()`
rather than having multiple branches with `if class.is_known()` guards
[this comment]:
https://github.com/astral-sh/ruff/pull/16416#discussion_r1985159037
## Summary
Properly handle binary operator inference for union types.
This fixes a bug I noticed while looking at ecosystem results. The MRE
version of it is this:
```py
def sub(x: float, y: float):
# Red Knot: Operator `-` is unsupported between objects of type `int | float` and `int | float`
return x - y
```
## Test Plan
- New Markdown tests.
- Expected diff in the ecosystem checks
## Summary
Part of #15382
This PR adds the check for whether a callable type is fully static or
not.
A callable type is fully static if all of the parameter types are fully
static _and_ the return type is fully static _and_ if it does not use
the gradual form (`...`) for its parameters.
## Test Plan
Update `is_fully_static.md` with callable types.
It seems that currently this test is grouped into either fully static or
not, I think it would be useful to split them up in groups like
callable, etc. I intentionally avoided that in this PR but I'll put up a
PR for an appropriate split.
Note: I've an explicit goal of updating the property tests with the new
callable types once all relations are implemented.
## Summary
This PR closes#16248.
If the return type of the function isn't assignable to the one
specified, an `invalid-return-type` error occurs.
I thought it would be better to report this as a different kind of error
than the `invalid-assignment` error, so I defined this as a new error.
## Test Plan
All type inconsistencies in the test cases have been replaced with
appropriate ones.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
This updates the `Signature` and `CallBinding` machinery to support
multiple overloads for a callable. This is currently only used for
`KnownFunction`s that we special-case in our type inference code. It
does **_not_** yet update the semantic index builder to handle
`@overload` decorators and construct a multi-signature `Overloads`
instance for real Python functions.
While I was here, I updated many of the `try_call` special cases to use
signatures (possibly overloaded ones now) and `bind_call` to check
parameter lists. We still need some of the mutator methods on
`OverloadBinding` for the special cases where we need to update return
types based on some Rust code.
## Summary
One of the motivations in https://github.com/astral-sh/ruff/pull/16428
for panicking when the `test` or `debug_assertions` features are enabled
and a lookup of a `KnownClass` fails is that we've had some latent bugs
in our code where certain variants have been silently falling back to
`Unknown` in every typeshed lookup without us realising. But that in
itself isn't a great motivation for panicking in
`KnownClass::to_instance()`, since we can fairly easily add some tests
that assert that we don't unexpectedly fallback to `Unknown` for any
`KnownClass` variant. This PR adds those tests.
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This mostly fixes#14899
My motivation was similar to the last comment by @sharkdp there. I ran
red_knot on a codebase and the most common error was patterns like this
failing:
```
def foo(x: str): ...
x: Any = ...
if isinstance(x, str):
foo(x) # Object of type `Any & str` cannot be assigned to parameter 1 (`x`) of function `foo`; expected type `str`
```
The desired behavior is pretty much to ignore Any/Unknown when resolving
intersection assignability - `Any & str` should be assignable to `str`,
and `str` should be assignable to `str & Any`
The fix is actually very similar to the existing code in
`is_subtype_of`, we need to correctly handle intersections on either
side, while being careful to handle dynamic types as desired.
This does not fix the second test case from that issue:
```
static_assert(is_assignable_to(Intersection[Unrelated, Any], Not[tuple[Unrelated, Any]]))
```
but that's misleading because the root cause there has nothing to do
with gradual types. I added a simpler test case that also fails:
```
static_assert(is_assignable_to(Unrelated, Not[tuple[Unrelated]]))
```
This is because we don't determine that Unrelated does not subclass from
tuple so we can't rule out this relation. If that logic is improved then
this fix should also handle the case of the intersection
## Test Plan
Added a bunch of is_assignable_to tests, most of which failed before
this fix.
## Summary
Part of https://github.com/astral-sh/ruff/issues/15382
This PR adds support for inferring the `lambda` expression and return
the `CallableType`.
Currently, this is only limited to inferring the parameters and a todo
type for the return type.
For posterity, I tried using the `file_expression_type` to infer the
return type of lambda but it would always lead to cycle. The main reason
is that in `infer_parameter_definition`, the default expression is being
inferred using `file_expression_type`, which is correct, but it then
Take the following source code as an example:
```py
lambda x=1: x
```
Here's how the code will flow:
* `infer_scope_types` for the global scope
* `infer_lambda_expression`
* `infer_expression` for the default value `1`
* `file_expression_type` for the return type using the body expression.
This is because the body creates it's own scope
* `infer_scope_types` (lambda body scope)
* `infer_name_load` for the symbol `x` whose visible binding is the
lambda parameter `x`
* `infer_parameter_definition` for parameter `x`
* `file_expression_type` for the default value `1`
* `infer_scope_types` for the global scope because of the default
expression
This will then reach to `infer_definition` for the parameter `x` again
which then creates the cycle.
## Test Plan
Add tests around `lambda` expression inference.
## Summary
We currently fail to add the stubs for the `venv` stdlib module because
there is a `venv/` ignore pattern in the top-level `.gitignore` file.
## Test Plan
Ran the typeshed sync workflow manually once to see if the `venv/`
folder is now correctly added.
## Summary
Theoretically this should be slightly more performant, since the
`class.is_known()` calls each do a separate Salsa lookup, which we can
avoid if we do a single `match` on the value of `class.known()`. It also
ends up being two lines less code overall!
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
Fixes#16566, fixes#16575
The semantics of `Type::class_member` changed in
https://github.com/astral-sh/ruff/pull/16416, but the property-test
infrastructure was not updated. That means that the property tests were
panicking on the second `expect_type` call here:
0361021863/crates/red_knot_python_semantic/src/types/property_tests.rs (L151-L158)
With the somewhat unhelpful message:
```
Expected a (possibly unbound) type, not an unbound symbol
```
Applying this patch, and then running `QUICKCHECK_TESTS=1000000 cargo
test --release -p red_knot_python_semantic -- --ignored
types::property_tests::stable::equivalent_to_is_reflexive` showed
clearly that it was no longer able to find _any_ methods on _any_
classes due to the change in semantics of `Type::class_member`:
```diff
--- a/crates/red_knot_python_semantic/src/types/property_tests.rs
+++ b/crates/red_knot_python_semantic/src/types/property_tests.rs
@@ -27,7 +27,7 @@
use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
use crate::db::tests::{setup_db, TestDb};
-use crate::symbol::{builtins_symbol, known_module_symbol};
+use crate::symbol::{builtins_symbol, known_module_symbol, Symbol};
use crate::types::{
BoundMethodType, CallableType, IntersectionBuilder, KnownClass, KnownInstanceType,
SubclassOfType, TupleType, Type, UnionType,
@@ -150,10 +150,11 @@ impl Ty {
Ty::BuiltinsFunction(name) => builtins_symbol(db, name).symbol.expect_type(),
Ty::BuiltinsBoundMethod { class, method } => {
let builtins_class = builtins_symbol(db, class).symbol.expect_type();
- let function = builtins_class
- .class_member(db, method.into())
- .symbol
- .expect_type();
+ let Symbol::Type(function, ..) =
+ builtins_class.class_member(db, method.into()).symbol
+ else {
+ panic!("no method `{method}` on class `{class}`");
+ };
create_bound_method(db, function, builtins_class)
}
```
This PR updates the property-test infrastructure to use `Type::member`
rather than `Type::class_member`.
## Test Plan
- Ran `QUICKCHECK_TESTS=1000000 cargo test --release -p
red_knot_python_semantic -- --ignored types::property_tests::stable`
successfully
- Checked that there were no remaining uses of `Type::class_member` in
`property_tests.rs`
## Summary
Fixes a small nit of mine -- we are currently inconsistent in our
spelling between "metaclass" and "meta class", and between "meta type"
and "meta-type". This PR means that we consistently use "metaclass" and
"meta-type".
## Test Plan
`uvx pre-commit run -a`
## Summary
Part of https://github.com/astral-sh/ruff/issues/15382
This PR implements a general callable type that wraps around a
`Signature` and it uses that new type to represent `typing.Callable`.
It also implements `Display` support for `Callable`. The format is as:
```
([<arg name>][: <arg type>][ = <default type>], ...) -> <return type>
```
The `/` and `*` separators are added at the correct boundary for
positional-only and keyword-only parameters. Now, as `typing.Callable`
only has positional-only parameters, the rendered signature would be:
```py
Callable[[int, str], None]
# (int, str, /) -> None
```
The `/` separator represents that all the arguments are positional-only.
The relationship methods that check assignability, subtype relationship,
etc. are not yet implemented and will be done so as a follow-up.
## Test Plan
Add test cases for display support for `Signature` and various mdtest
for `typing.Callable`.
## Summary
Resolves#16365
Add support for unpacking `with` statement targets.
## Test Plan
Added some test cases, alike the ones added by #15058.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
* Attributes/method are now properly looked up on metaclasses, when
called on class objects
* We properly distinguish between data descriptors and non-data
descriptors (but we do not yet support them in store-context, i.e.
`obj.data_descr = …`)
* The descriptor protocol is now implemented in a single unified place
for instances, classes and dunder-calls. Unions and possibly-unbound
symbols are supported in all possible stages of the process by creating
union types as results.
* In general, the handling of "possibly-unbound" symbols has been
improved in a lot of places: meta-class attributes, attributes,
descriptors with possibly-unbound `__get__` methods, instance
attributes, …
* We keep track of type qualifiers in a lot more places. I anticipate
that this will be useful if we import e.g. `Final` symbols from other
modules (see relevant change to typing spec:
https://github.com/python/typing/pull/1937).
* Detection and special-casing of the `typing.Protocol` special form in
order to avoid lots of changes in the test suite due to new `@Todo`
types when looking up attributes on builtin types which have `Protocol`
in their MRO. We previously
looked up attributes in a wrong way, which is why this didn't come up
before.
closes#16367closes#15966
## Context
The way attribute lookup in `Type::member` worked before was simply
wrong (mostly my own fault). The whole instance-attribute lookup should
probably never have been integrated into `Type::member`. And the
`Type::static_member` function that I introduced in my last descriptor
PR was the wrong abstraction. It's kind of fascinating how far this
approach took us, but I am pretty confident that the new approach
proposed here is what we need to model this correctly.
There are three key pieces that are required to implement attribute
lookups:
- **`Type::class_member`**/**`Type::find_in_mro`**: The
`Type::find_in_mro` method that can look up attributes on class bodies
(and corresponding bases). This is a partial function on types, as it
can not be called on instance types like`Type::Instance(…)` or
`Type::IntLiteral(…)`. For this reason, we usually call it through
`Type::class_member`, which is essentially just
`type.to_meta_type().find_in_mro(…)` plus union/intersection handling.
- **`Type::instance_member`**: This new function is basically the
type-level equivalent to `obj.__dict__[name]` when called on
`Type::Instance(…)`. We use this to discover instance attributes such as
those that we see as declarations on class bodies or as (annotated)
assignments to `self.attr` in methods of a class.
- The implementation of the descriptor protocol. It works slightly
different for instances and for class objects, but it can be described
by the general framework:
- Call `type.class_member("attribute")` to look up "attribute" in the
MRO of the meta type of `type`. Call the resulting `Symbol` `meta_attr`
(even if it's unbound).
- Use `meta_attr.class_member("__get__")` to look up `__get__` on the
*meta type* of `meta_attr`. Call it with `__get__(meta_attr, self,
self.to_meta_type())`. If this fails (either the lookup or the call),
just proceed with `meta_attr`. Otherwise, replace `meta_attr` in the
following with the return type of `__get__`. In this step, we also probe
if a `__set__` or `__delete__` method exists and store it in
`meta_attr_kind` (can be either "data descriptor" or "normal attribute
or non-data descriptor").
- Compute a `fallback` type.
- For instances, we use `self.instance_member("attribute")`
- For class objects, we use `class_attr =
self.find_in_mro("attribute")`, and then try to invoke the descriptor
protocol on `class_attr`, i.e. we look up `__get__` on the meta type of
`class_attr` and call it with `__get__(class_attr, None, self)`. This
additional invocation of the descriptor protocol on the fallback type is
one major asymmetry in the otherwise universal descriptor protocol
implementation.
- Finally, we look at `meta_attr`, `meta_attr_kind` and `fallback`, and
handle various cases of (possible) unboundness of these symbols.
- If `meta_attr` is bound and a data descriptor, just return `meta_attr`
- If `meta_attr` is not a data descriptor, and `fallback` is bound, just
return `fallback`
- If `meta_attr` is not a data descriptor, and `fallback` is unbound,
return `meta_attr`
- Return unions of these three possibilities for partially-bound
symbols.
This allows us to handle class objects and instances within the same
framework. There is a minor additional detail where for instances, we do
not allow the fallback type (the instance attribute) to completely
shadow the non-data descriptor. We do this because we (currently) don't
want to pretend that we can statically infer that an instance attribute
is always set.
Dunder method calls can also be embedded into this framework. The only
thing that changes is that *there is no fallback type*. If a dunder
method is called on an instance, we do not fall back to instance
variables. If a dunder method is called on a class object, we only look
it up on the meta class, never on the class itself.
## Test Plan
New Markdown tests.
## Summary
This PR closes#15199.
The change I just made is to set all variables to type `Unknown` if
unpacking fails, but in some cases this may be excessive.
For example:
```py
a, b, c = "ab"
reveal_type(a) # Unknown, but it would be reasonable to think of it as LiteralString
reveal_type(c) # Unknown
```
```py
# Failed to unpack before the starred expression
(a, b, *c, d, e) = (1,)
reveal_type(a) # Unknown
reveal_type(b) # Unknown
...
# Failed to unpack after the starred expression
(a, b, *c, d, e) = (1, 2, 3)
reveal_type(a) # Unknown, but should it be Literal[1]?
reveal_type(b) # Unknown, but should it be Literal[2]?
reveal_type(c) # Todo
reveal_type(d) # Unknown
reveal_type(e) # Unknown
```
I will modify it if you think it would be better to make it a different
type than just `Unknown`.
## Test Plan
I have made appropriate modifications to the test cases affected by this
change, and also added some more test cases.
## Summary
This should give us better coverage for the unsupported syntax error
features and
increases our confidence that the formatter doesn't accidentially
introduce new unsupported
syntax errors.
A feature like this would have been very useful when working on f-string
formatting
where it took a lot of iteration to find all Python 3.11 or older
incompatibilities.
## Test Plan
I applied my changes on top of
https://github.com/astral-sh/ruff/pull/16523 and
removed the target version check in the with-statement formatting code.
As expected,
the integration tests now failed
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
If an mdtest fails, the error output will include an example command
that can be run to re-run just the failing test, e.g
```
To rerun this specific test, set the environment variable: MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute"
MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__with_sync
```
This is very helpful, but because we're printing the envvar value
surrounded in double-quotes, the bits between backticks in this example
get interpreted as a shell interpolation. When running this in zsh, for
example, I see
```console
❯ MDTEST_TEST_FILTER="sync.md - With statements - Context manager with non-callable `__exit__` attribute" cargo test -p red_knot_python_semantic --test mdtest -- mdtest__with_sync
zsh: command not found: __exit__
Compiling red_knot_python_semantic v0.0.0 (/home/ericmarkmartin/Development/ruff/crates/red_knot_python_semantic)
Compiling red_knot_test v0.0.0 (/home/ericmarkmartin/Development/ruff/crates/red_knot_test)
Finished `test` profile [unoptimized + debuginfo] target(s) in 6.09s
Running tests/mdtest.rs (target/debug/deps/mdtest-149b8f9d937e36bc)
running 1 test
test mdtest__with_sync ... ok
```
[^1]
This is a minor annoyance which we can solve by using single-quotes
instead of double-quotes for this string. To do so safely, we also
escape single-quotes possibly contained within the string.
There is a [shell-quote](https://github.com/allenap/shell-quote) crate,
which seems to handle all this escaping stuff for you but fixing this
issue perfectly isn't a big deal (if there are more things to escape we
can deal with it then), so adding a new dependency (even a dev one)
seemed overkill.
[^1]: The filter does still work---it turns out that the filter
`MDTEST_TEST_FILTER="sync.md - With statements - Context manager with
non-callable attribute"` (what you get after the failed interpolation)
is still good enough
## Test Plan
<!-- How was it tested? -->
I broke the ``## Context manager with non-callable `__exit__`
attribute`` test by deleting the error assertion, then successfully ran
the new command it printed out.
Summary
--
Unlike the other syntax errors detected so far, parenthesized keyword
arguments are only allowed *before* 3.8. It sounds like they were only
accidentally allowed before that [^1].
As an aside, you get a pretty confusing error from Python for this, so
it's nice that we can catch it:
```pycon
>>> def f(**kwargs): ...
... f((a)=1)
...
File "<python-input-0>", line 2
f((a)=1)
^^^
SyntaxError: expression cannot contain assignment, perhaps you meant "=="?
>>>
```
Test Plan
--
Inline tests.
[^1]: https://github.com/python/cpython/issues/78822
Summary
--
Checks for tuple unpacking in `return` and `yield` statements before
Python 3.8, as described [here].
Test Plan
--
Inline tests.
[here]: https://github.com/python/cpython/issues/76298
## Summary
- `Never` is callable
- `Never` is iterable
- Arbitrary attributes can be accessed on `Never`
Split out from #16416 that is going to be required.
## Test Plan
Tests for all properties above.
## Summary
This came up in https://github.com/astral-sh/ruff/issues/16477
It's not obvious from the D417 rule's documentation that it only checks
docstrings
with an arguments section. Functions without such a section aren't
checked.
This PR tries to make this clearer in the documentation.
## Summary
This PR introduces a new mdtest option `system` that can either be
`in-memory` or `os`
where `in-memory` is the default.
The motivation for supporting `os` is so that we can write OS/system
specific tests
with mdtests. Specifically, I want to write mdtests for the module
resolver,
testing that module resolution is case sensitive.
## Test Plan
I tested that the case-sensitive module resolver test start failing when
setting `system = "os"`
## Summary
Python's module resolver is case sensitive.
This PR adds mdtests that assert that our module resolution is case
sensitive.
The tests currently all pass because our in memory file system is case
sensitive.
I'll add support for using the real file system to the mdtest framework
in a separate PR.
This PR also adds support for specifying extra search paths to the
mdtest framework.
## Test Plan
The tests fail when running them using the real file system.
To kick off the work of supporting generics, this adds many new
(currently failing) tests, showing the behavior we plan to support.
This is still missing a lot! Not included:
- typevar tuples
- param specs
- variance
- `Self`
But it's a good start! We can add more failing tests for those once we
tackle these.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
This is split out of https://github.com/astral-sh/ruff/pull/14029, to
reduce the size of that PR, and to validate that this "fallback type"
support in `TypeInference` doesn't come with a performance cost. It also
improves the reliability and debuggability of our current (temporary)
cycle handling.
In order to recover from a cycle, we have to be able to construct a
"default" `TypeInference` where all expressions and definitions have
some "default" type. In our current cycle handling, this "default" type
is just unknown or a todo type. With fixpoint iteration, the "default"
type will be `Type::Never`, which is the "bottom" type that fixpoint
iteration starts from.
Since it would be costly (both in space and time) to actually enumerate
all expressions and definitions in a scope, just to insert the same
default type for all of them, instead we add an optional "missing type"
fallback to `TypeInference`, which (if set) is the fallback type for any
expression or definition which doesn't have an explicit type set.
With this change, cycles can no longer result in the dreaded "Missing
key" errors looking up the type of some expression.
... with supporting types. This is meant to give us a base to work with
in terms of our new diagnostic data model. I expect the representations
to be tweaked over time, but I think this is a decent start.
I would also like to add doctest examples, but I think it's better if we
wait until an initial version of the renderer is done for that.
This puts them out of the way so that they can hopefully be removed more
easily in the (near) future, and so that they don't get in the way of
the new types. This also makes the intent of the migration a bit clearer
in the code and hopefully results in less confusion.
This trait should eventually go away, so we rename it (and supporting
types) to make room for a new concrete `Diagnostic` type.
This commit is just the rename. In the next commit, we'll move it to a
different module.
Summary
--
Another simple one, just detect type parameter lists in functions
and classes. Like pyright, we don't emit a second diagnostic for
`type` alias statements, which were also introduced in 3.12.
Test Plan
--
Inline tests.
## Summary
This PR does a small refactor to avoid double
`symbol_table(...).symbol(...)` call to check for `__slots__` and
`TYPE_CHECKING`. It merges them into a single call.
I noticed this while looking at
https://github.com/astral-sh/ruff/pull/16468.
## Summary
This PR adds more features to #16468.
* Adds a new error rule `invalid-type-checking-constant`, which occurs
when we try to assign a value other than `False` to a user-defined
`TYPE_CHECKING` variable (it is possible to assign `...` in a stub
file).
* Allows annotated assignment to `TYPE_CHECKING`. Only types that
`False` can be assigned to are allowed. However, the type of
`TYPE_CHECKING` will be inferred to be `Literal[True]` regardless of
what the type is specified.
## Test plan
I ran the tests with `cargo test -p red_knot_python_semantic` and
confirmed that all tests passed.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Fixes https://github.com/astral-sh/ruff/issues/16476fixes: #11453
We format notebooks cell by cell. That means, that offsets in parse
errors are relative
to the cell and not the entire document. We didn't account for this fact
when emitting syntax errors for notebooks in the formatter.
This PR ensures that we correctly offset parse errors by the cell
location.
## Test Plan
Added test (it panicked before)
Summary
--
Detects the presence of a [PEP 696] type parameter default before Python
3.13.
Test Plan
--
New inline parser tests for type aliases, generic functions and generic
classes.
[PEP 696]: https://peps.python.org/pep-0696/#grammar-changes
Summary
--
This is a follow-up to #16446 to fix the diagnostic range to point to
the `*` like `pyright` does
(https://github.com/astral-sh/ruff/pull/16446#discussion_r1976900643).
Storing the range in the `ExceptClauseKind::Star` variant feels slightly
awkward, but we don't store the star itself anywhere on the
`ExceptHandler`. And we can't just take `ExceptHandler.start() +
"except".text_len()` because this code appears to be valid:
```python
try: ...
except * Error: ...
```
Test Plan
--
Existing tests.
## Summary
This PR closes#15722.
The change is that if the variable `TYPE_CHECKING` is defined/imported,
the type of the variable is interpreted as `Literal[True]` regardless of
what the value is.
This is compatible with the behavior of other type checkers (e.g. mypy,
pyright).
## Test Plan
I ran the tests with `cargo test -p red_knot_python_semantic` and
confirmed that all tests passed.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Summary
--
This is a follow up addressing the comments on #16425. As @dhruvmanila
pointed out, the naming is a bit tricky. I went with `has_no_errors` to
try to differentiate it from `is_valid`. It actually ends up negated in
most uses, so it would be more convenient to have `has_any_errors` or
`has_errors`, but I thought it would sound too much like the opposite of
`is_valid` in that case. I'm definitely open to suggestions here.
Test Plan
--
Existing tests.
## Summary
Resolves#16445.
`UP028` is now no longer always fixable: it will not offer a fix when at
least one `ExprName` target is bound to either a `global` or a
`nonlocal` declaration.
## Test Plan
`cargo nextest run` and `cargo insta test`.
## Summary
Fixes#9381. This PR fixes errors like
```
Cause: error parsing glob '/Users/me/project/{{cookiecutter.project_dirname}}/__pycache__': nested alternate groups are not allowed
```
caused by glob special characters in filenames like
`{{cookiecutter.project_dirname}}`. When the user is matching that
directory exactly, they can use the workaround given by
https://github.com/astral-sh/ruff/issues/7959#issuecomment-1764751734,
but that doesn't work for a nested config file with relative paths. For
example, the directory tree in the reproduction repo linked
[here](https://github.com/astral-sh/ruff/issues/9381#issuecomment-2677696408):
```
.
├── README.md
├── hello.py
├── pyproject.toml
├── uv.lock
└── {{cookiecutter.repo_name}}
├── main.py
├── pyproject.toml
└── tests
└── maintest.py
```
where the inner `pyproject.toml` contains a relative glob:
```toml
[tool.ruff.lint.per-file-ignores]
"tests/*" = ["F811"]
```
## Test Plan
A new CLI test in both the linter and formatter. The formatter test may
not be necessary because I didn't have to modify any additional code to
pass it, but the original report mentioned both `check` and `format`, so
I wanted to be sure both were fixed.
The PR addresses issue #16396 .
Specifically:
- If the exit statement contains a code keyword argument, it is
converted into a positional argument.
- If retrieving the code from the exit statement is not possible, a
violation is raised without suggesting a fix.
---------
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
## Summary
This PR adds support for an optional list of paths that should be
checked to `knot check`.
E.g. to only check the `src` directory
```sh
knot check src
```
The default is to check all files in the project but users can reduce
the included files by specifying one or multiple optional paths.
The main two challenges with adding this feature were:
* We now need to show an error when one of the provided paths doesn't
exist. That's why this PR now collects errors from the project file
indexing phase and adds them to the output diagnostics. The diagnostic
looks similar to ruffs (see CLI test)
* The CLI should pick up new files added to included folders. For
example, `knot check src --watch` should pick up new files that are
added to the `src` folder. This requires that we now filter the files
before adding them to the project. This is a good first step to
supporting `include` and `exclude`.
The PR makes two simplifications:
1. I didn't test the changes with case-insensitive file systems. We may
need to do some extra path normalization to support those well. See
https://github.com/astral-sh/ruff/issues/16400
2. Ideally, we'd accumulate the IO errors from the initial indexing
phase and subsequent incremental indexing operations. For example, we
should preserve the IO diagnostic for a non existing `test.py` if it was
specified as an explicit CLI argument until the file gets created and we
should show it again when the file gets deleted. However, this is
somewhat complicated because we'd need to track which files we revisited
(or were removed because the entire directory is gone). I considered
this too low a priority as it's worth dealing with right now.
The implementation doesn't support symlinks within the project but that
is the same as Ruff and is unchanged from before this PR.
Closes https://github.com/astral-sh/ruff/issues/14193
## Test Plan
Added CLI and file watching integration tests. Manually testing.
Split from F841 following discussion in #8884.
Fixes#8884.
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Add a new rule for unused assignments in tuples. Remove similar behavior
from F841.
## Test Plan
Adapt F841 tests and move them over to the new rule.
<!-- How was it tested? -->
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
This PR is the first in a series derived from
https://github.com/astral-sh/ruff/pull/16308, each of which add support
for detecting one version-related syntax error from
https://github.com/astral-sh/ruff/issues/6591. This one should be
the largest because it also includes the addition of the
`Parser::add_unsupported_syntax_error` method
Otherwise I think the general structure will be the same for each syntax
error:
* Detecting the error in the parser
* Inline parser tests for the new error
* New ruff CLI tests for the new error
## Test Plan
As noted above, there are new inline parser tests, as well as new ruff
CLI
tests. Once https://github.com/astral-sh/ruff/pull/16379 is resolved,
there should also be new mdtests for red-knot,
but this PR does not currently include those.
Regardless of whether #16408 and #16311 pan out, this part is worth
pulling out as a separate PR.
Before, you had to define a new `IndexVec` index type for each type of
association list you wanted to create. Now there's a single index type
that's internal to the alist implementation, and you use `List<K, V>` to
store a handle to a particular list.
This also adds some property tests for the alist implementation.
## Summary
This PR adds support for a pragma-style header for inline parser tests
containing JSON-serialized `ParseOptions`. For example,
```python
# parse_options: { "target-version": "3.9" }
match 2:
case 1:
pass
```
The line must start with `# parse_options: ` and then the rest of the
(trimmed) line is deserialized into `ParseOptions` used for parsing the
the test.
## Test Plan
Existing inline tests, plus two new inline tests for
`match-before-py310`.
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
## Summary
As mentioned in
https://github.com/astral-sh/ruff/pull/16296#discussion_r1967047387
This PR updates the client settings resolver to notify the user if there
are any errors in the config using a very basic approach. In addition,
each error related to specific settings are logged.
This isn't the best approach because it can log the same message
multiple times when both workspace and global settings are provided and
they both are the same. This is the case for a single workspace VS Code
instance.
I do have some ideas on how to improve this and will explore them during
my free time (low priority):
* Avoid resolving the global settings multiple times as they're static
* Include the source of the setting (workspace or global?)
* Maybe use a struct (`ResolvedClientSettings` +
`Vec<ClientSettingsResolverError>`) instead to make unit testing easier
## Test Plan
Using:
```jsonc
{
"ruff.logLevel": "debug",
// Invalid settings
"ruff.configuration": "$RANDOM",
"ruff.lint.select": ["RUF000", "I001"],
"ruff.lint.extendSelect": ["B001", "B002"],
"ruff.lint.ignore": ["I999", "F401"]
}
```
The error logs:
```
2025-02-27 12:30:04.318736000 ERROR Failed to load settings from `configuration`: error looking key 'RANDOM' up: environment variable not found
2025-02-27 12:30:04.319196000 ERROR Failed to load settings from `configuration`: error looking key 'RANDOM' up: environment variable not found
2025-02-27 12:30:04.320549000 ERROR Unknown rule selectors found in `lint.select`: ["RUF000"]
2025-02-27 12:30:04.320669000 ERROR Unknown rule selectors found in `lint.extendSelect`: ["B001"]
2025-02-27 12:30:04.320764000 ERROR Unknown rule selectors found in `lint.ignore`: ["I999"]
```
Notification preview:
<img width="470" alt="Screenshot 2025-02-27 at 12 29 06 PM"
src="https://github.com/user-attachments/assets/61f41d5c-2558-46b3-a1ed-82114fd8ec22"
/>
## Summary
Closes: https://github.com/astral-sh/ruff/issues/16267
This change skips building the `index` in RuffSettingsIndex when the
configuration preference, in the editor settings, is set to
`editorOnly`. This is appropriate due to the fact that the indexes will
go unused as long as the configuration preference persists.
## Test Plan
I have tested this in VSCode and can confirm that we skip indexing when
`editorOnly` is set. Upon switching back to `editorFirst` or
`filesystemFirst` we index the settings as normal.
I don't seen any unit tests for setting indexing at the moment, but I am
happy to give it a shot if that is something we want.
We currently keep two separate pieces of state regarding the current
loop on `SemanticIndexBuilder`. One is an enum simply reflecting whether
we are currently inside a loop, and the other is the saved flow states
for `break` statements found in the current loop.
For adding loopy control flow, I'll need to add some additional loop
state (`continue` states, for example). Prepare for this by
consolidating our existing loop state into a single struct and
simplifying the API for pushing and popping a loop.
This is purely a refactor, so tests are not changed.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Resolves#16374.
`PLW0177` now also reports the pattern of a case branch if it is an
attribute access whose qualified name is that of either `np.nan` or
`math.nan`.
As the rule is in preview, the changes are not preview-gated.
## Test Plan
`cargo nextest run` and `cargo insta test`.
Minor follow-up to https://github.com/astral-sh/ruff/pull/16161
This `not_callable` flag wasn't functional, because it could never be
`false`. It was initialized to `true` and then only ever updated with
`|=`, which can never make it `false`.
Add a test that exercises the case where it _should_ be `false` (all of
the union elements are callable) but `bindings` is also empty (all union
elements have binding errors). Before this PR, the added test wrongly
emits a diagnostic that the union `Literal[f1] | Literal[f2]` is not
callable.
And add a test where a union call results in one binding error and one
not-callable error, where we currently give the wrong result (we show
only the binding error), with a TODO.
Also add TODO comments in a couple other tests where ideally we'd report
more than just one error out of a union call.
Also update the flag name to `all_errors_not_callable` to more clearly
indicate the semantics of the flag.
## Summary
Currently, the log messages emitted by the server includes multiple
information which isn't really required most of the time.
Here's the current format:
```
0.000755625s DEBUG main ruff_server::session::index::ruff_settings: Indexing settings for workspace: /Users/dhruv/playground/ruff
0.016334666s DEBUG ThreadId(10) ruff_server::session::index::ruff_settings: Ignored path via `exclude`: /Users/dhruv/playground/ruff/.vscode
0.019954541s INFO main ruff_server::session::index: Registering workspace: /Users/dhruv/playground/ruff
0.020160416s TRACE ruff:main notification{method="textDocument/didOpen"}: ruff_server::server::api: enter
0.020209625s TRACE ruff:worker:0 request{id=1 method="textDocument/diagnostic"}: ruff_server::server::api: enter
0.020228166s DEBUG ruff:worker:0 request{id=1 method="textDocument/diagnostic"}: ruff_server::resolve: Included path via `include`: /Users/dhruv/playground/ruff/lsp/test.py
0.020359833s INFO ruff:main ruff_server::server: Configuration file watcher successfully registered
```
This PR updates the following:
* Uses current timestamp (same as red-knot) for all log levels instead
of the uptime value
* Includes the target and thread names only at the trace level
What this means is that the message is reduced to only important
information at DEBUG level:
```
2025-02-26 11:35:02.198375000 DEBUG Indexing settings for workspace: /Users/dhruv/playground/ruff
2025-02-26 11:35:02.209933000 DEBUG Ignored path via `exclude`: /Users/dhruv/playground/ruff/.vscode
2025-02-26 11:35:02.217165000 INFO Registering workspace: /Users/dhruv/playground/ruff
2025-02-26 11:35:02.217631000 DEBUG Included path via `include`: /Users/dhruv/playground/ruff/lsp/test.py
2025-02-26 11:35:02.217684000 INFO Configuration file watcher successfully registered
```
while still showing the other information (thread names and target) at
trace level:
```
2025-02-26 11:35:27.819617000 DEBUG main ruff_server::session::index::ruff_settings: Indexing settings for workspace: /Users/dhruv/playground/ruff
2025-02-26 11:35:27.830500000 DEBUG ThreadId(11) ruff_server::session::index::ruff_settings: Ignored path via `exclude`: /Users/dhruv/playground/ruff/.vscode
2025-02-26 11:35:27.837212000 INFO main ruff_server::session::index: Registering workspace: /Users/dhruv/playground/ruff
2025-02-26 11:35:27.837714000 TRACE ruff:main notification{method="textDocument/didOpen"}: ruff_server::server::api: enter
2025-02-26 11:35:27.838019000 INFO ruff:main ruff_server::server: Configuration file watcher successfully registered
2025-02-26 11:35:27.838084000 TRACE ruff:worker:1 request{id=1 method="textDocument/diagnostic"}: ruff_server::server::api: enter
2025-02-26 11:35:27.838205000 DEBUG ruff:worker:1 request{id=1 method="textDocument/diagnostic"}: ruff_server::resolve: Included path via `include`: /Users/dhruv/playground/ruff/lsp/test.py
```
## Summary
[Internal design
document](https://www.notion.so/astral-sh/In-editor-settings-19e48797e1ca807fa8c2c91b689d9070?pvs=4)
This PR expands `ruff.configuration` to allow inline configuration
directly in the editor. For example:
```json
{
"ruff.configuration": {
"line-length": 100,
"lint": {
"unfixable": ["F401"],
"flake8-tidy-imports": {
"banned-api": {
"typing.TypedDict": {
"msg": "Use `typing_extensions.TypedDict` instead"
}
}
}
},
"format": {
"quote-style": "single"
}
}
}
```
This means that now `ruff.configuration` accepts either a path to
configuration file or the raw config itself. It's _mostly_ similar to
`--config` with one difference that's highlighted in the following
section. So, it can be said that the format of `ruff.configuration` when
provided the config map is same as the one on the [playground] [^1].
## Limitations
<details><summary><b>Casing (<code>kebab-case</code> v/s/
<code>camelCase</code>)</b></summary>
<p>
The config keys needs to be in `kebab-case` instead of `camelCase` which
is being used for other settings in the editor.
This could be a bit confusing. For example, the `line-length` option can
be set directly via an editor setting or can be configured via
`ruff.configuration`:
```json
{
"ruff.configuration": {
"line-length": 100
},
"ruff.lineLength": 120
}
```
#### Possible solution
We could use feature flag with [conditional
compilation](https://doc.rust-lang.org/reference/conditional-compilation.html#the-cfg_attr-attribute)
to indicate that when used in `ruff_server`, we need the `Options`
fields to be renamed as `camelCase` while for other crates it needs to
be renamed as `kebab-case`. But, this might not work very easily because
it will require wrapping the `Options` struct and create two structs in
which we'll have to add `#[cfg_attr(...)]` because otherwise `serde`
will complain:
```
error: duplicate serde attribute `rename_all`
--> crates/ruff_workspace/src/options.rs:43:38
|
43 | #[cfg_attr(feature = "editor", serde(rename_all = "camelCase"))]
| ^^^^^^^^^^
```
</p>
</details>
<details><summary><b>Nesting (flat v/s nested keys)</b></summary>
<p>
This is the major difference between `--config` flag on the command-line
v/s `ruff.configuration` and it makes it such that `ruff.configuration`
has same value format as [playground] [^1].
The config keys needs to be split up into keys which can result in
nested structure instead of flat structure:
So, the following **won't work**:
```json
{
"ruff.configuration": {
"format.quote-style": "single",
"lint.flake8-tidy-imports.banned-api.\"typing.TypedDict\".msg": "Use `typing_extensions.TypedDict` instead"
}
}
```
But, instead it would need to be split up like the following:
```json
{
"ruff.configuration": {
"format": {
"quote-style": "single"
},
"lint": {
"flake8-tidy-imports": {
"banned-api": {
"typing.TypedDict": {
"msg": "Use `typing_extensions.TypedDict` instead"
}
}
}
}
}
}
```
#### Possible solution (1)
The way we could solve this and make it same as `--config` would be to
add a manual logic of converting the JSON map into an equivalent TOML
string which would be then parsed into `Options`.
So, the following JSON map:
```json
{ "lint.flake8-tidy-imports": { "banned-api": {"\"typing.TypedDict\".msg": "Use typing_extensions.TypedDict instead"}}}
```
would need to be converted into the following TOML string:
```toml
lint.flake8-tidy-imports = { banned-api = { "typing.TypedDict".msg = "Use typing_extensions.TypedDict instead" } }
```
by recursively convering `"key": value` into `key = value` which is to
remove the quotes from key and replacing `:` with `=`.
#### Possible solution (2)
Another would be to just accept `Map<String, String>` strictly and
convert it into `key = value` and then parse it as a TOML string. This
would also match `--config` but quotes might become a nuisance because
JSON only allows double quotes and so it'll require escaping any inner
quotes or use single quotes.
</p>
</details>
## Test Plan
### VS Code
**Requires https://github.com/astral-sh/ruff-vscode/pull/702**
**`settings.json`**:
```json
{
"ruff.lint.extendSelect": ["TID"],
"ruff.configuration": {
"line-length": 50,
"format": {
"quote-style": "single"
},
"lint": {
"unfixable": ["F401"],
"flake8-tidy-imports": {
"banned-api": {
"typing.TypedDict": {
"msg": "Use `typing_extensions.TypedDict` instead"
}
}
}
}
}
}
```
Following video showcases me doing the following:
1. Check diagnostics that it includes `TID`
2. Run `Ruff: Fix all auto-fixable problems` to test `unfixable`
3. Run `Format: Document` to test `line-length` and `quote-style`
https://github.com/user-attachments/assets/0a38176f-3fb0-4960-a213-73b2ea5b1180
### Neovim
**`init.lua`**:
```lua
require('lspconfig').ruff.setup {
init_options = {
settings = {
lint = {
extendSelect = { 'TID' },
},
configuration = {
['line-length'] = 50,
format = {
['quote-style'] = 'single',
},
lint = {
unfixable = { 'F401' },
['flake8-tidy-imports'] = {
['banned-api'] = {
['typing.TypedDict'] = {
msg = 'Use typing_extensions.TypedDict instead',
},
},
},
},
},
},
},
}
```
Same steps as in the VS Code test:
https://github.com/user-attachments/assets/cfe49a9b-9a89-43d7-94f2-7f565d6e3c9d
## Documentation Preview
https://github.com/user-attachments/assets/e0062f58-6ec8-4e01-889d-fac76fd8b3c7
[playground]: https://play.ruff.rs
[^1]: This has one advantage that the value can be copy-pasted directly
into the playground
## Summary
This PR builds on the changes in #16220 to pass a target Python version
to the parser. It also adds the `Parser::unsupported_syntax_errors` field, which
collects version-related syntax errors while parsing. These syntax
errors are then turned into `Message`s in ruff (in preview mode).
This PR only detects one syntax error (`match` statement before Python
3.10), but it has been pretty quick to extend to several other simple
errors (see #16308 for example).
## Test Plan
The current tests are CLI tests in the linter crate, but these could be
supplemented with inline parser tests after #16357.
I also tested the display of these syntax errors in VS Code:
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
In https://github.com/astral-sh/ruff/pull/16306#discussion_r1966290700,
@carljm pointed out that #16306 introduced a terminology problem, with
too many things called a "constraint". This is a follow-up PR that
renames `Constraint` to `Predicate` to hopefully clear things up a bit.
So now we have that:
- a _predicate_ is a Python expression that might influence type
inference
- a _narrowing constraint_ is a list of predicates that constraint the
type of a binding that is visible at a use
- a _visibility constraint_ is a ternary formula of predicates that
define whether a binding is visible or a statement is reachable
This is a pure renaming, with no behavioral changes.
## Summary
Model dunder-calls correctly (and in one single place), by implementing
this behavior (using `__getitem__` as an example).
```py
def getitem_desugared(obj: object, key: object) -> object:
getitem_callable = find_in_mro(type(obj), "__getitem__")
if hasattr(getitem_callable, "__get__"):
getitem_callable = getitem_callable.__get__(obj, type(obj))
return getitem_callable(key)
```
See the new `calls/dunder.md` test suite for more information. The new
behavior also needs much fewer lines of code (the diff is positive due
to new tests).
## Test Plan
New tests; fix TODOs in existing tests.
This PR adds an implementation of [association
lists](https://en.wikipedia.org/wiki/Association_list), and uses them to
replace the previous `BitSet`/`SmallVec` representation for narrowing
constraints.
An association list is a linked list of key/value pairs. We additionally
guarantee that the elements of an association list are sorted (by their
keys), and that they do not contain any entries with duplicate keys.
Association lists have fallen out of favor in recent decades, since you
often need operations that are inefficient on them. In particular,
looking up a random element by index is O(n), just like a linked list;
and looking up an element by key is also O(n), since you must do a
linear scan of the list to find the matching element. Luckily we don't
need either of those operations for narrowing constraints!
The typical implementation also suffers from poor cache locality and
high memory allocation overhead, since individual list cells are
typically allocated separately from the heap. We solve that last problem
by storing the cells of an association list in an `IndexVec` arena.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
I am working on a project that uses ruff linters' docs to generate a
fine-tuning dataset for LLMs.
To achieve this, I first ran the command `ruff rule --all
--output-format json` to retrieve all the rules. Then, I parsed the
explanation field to get these 3 consistent sections:
- `Why is this bad?`
- `What it does`
- `Example`
However, during the initial processing, I noticed that the markdown
headings are not that consistent. For instance:
- In most cases, `Use instead` appears as a normal paragraph within the
`Example` section, but in the file
`crates/ruff_linter/src/rules/flake8_bandit/rules/django_extra.rs` it is
a level-2 heading
- The heading "What it does**?**" is used in some places, while others
consistently use "What it does"
- There are 831 `Example` headings and 65 `Examples`. But all of them
only have one example case
This PR normalized these across all rules.
## Test Plan
CI are passed.
## Summary
Add a diagnostic if a pure instance variable is accessed on a class object. For example
```py
class C:
instance_only: str
def __init__(self):
self.instance_only = "a"
# error: Attribute `instance_only` can only be accessed on instances, not on the class object `Literal[C]` itself.
C.instance_only
```
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
This PR is another step in preparing to detect syntax errors in the
parser. It introduces the new `per-file-target-version` top-level
configuration option, which holds a mapping of compiled glob patterns to
Python versions. I intend to use the
`LinterSettings::resolve_target_version` method here to pass to the
parser:
f50849aeef/crates/ruff_linter/src/linter.rs (L491-L493)
## Test Plan
I added two new CLI tests to show that the `per-file-target-version` is
respected in both the formatter and the linter.
## Summary
Add support for `@classmethod`s.
```py
class C:
@classmethod
def f(cls, x: int) -> str:
return "a"
reveal_type(C.f(1)) # revealed: str
```
## Test Plan
New Markdown tests
## Summary
* Existing example did not include RawSQL() call like it should
* Also clarify the example a bit to make it clearer that the code is not
secure
## Test Plan
N/A, only documentation updated
## Summary
I spotted a minor mistake in my descriptor protocol implementation where
`C.descriptor` would pass the meta type (`type`) of the type of `C`
(`Literal[C]`) as the owner argument to `__get__`, instead of passing
`Literal[C]` directly.
## Test Plan
New test.
Two related changes. For context:
1. We were maintaining two separate arenas of `Constraint`s in each
use-def map. One was used for narrowing constraints, and the other for
visibility constraints. The visibility constraint arena was interned,
ensuring that we always used the same ID for any particular
`Constraint`. The narrowing constraint arena was not interned.
2. The TDD code relies on _all_ TDD nodes being interned and reduced.
This is an important requirement for TDDs to be a canonical form, which
allows us to use a single int comparison to test for "always true/false"
and to compare two TDDs for equivalence. But we also need to support an
individual `Constraint` having multiple values in a TDD evaluation (e.g.
to handle a `while` condition having different values the first time
it's evaluated vs later times). Previously, we handled that by
introducing a "copy" number, which was only there as a disambiguator, to
allow an interned, deduplicated constraint ID to appear in the TDD
formula multiple times.
A better way to handle (2) is to not intern the constraints in the
visibility constraint arena! The caller now gets to decide: if they add
a `Constraint` to the arena more than once, they get distinct
`ScopedConstraintId`s — which the TDD code will treat as distinct
variables, allowing them to take on different values in the ternary
function.
With that in place, we can then consolidate on a single (non-interned)
arena, which is shared for both narrowing and visibility constraints.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Resolves 3/4 requests in #16217:
- ✅ Remove not special methods: `__cmp__`, `__div__`, `__nonzero__`, and
`__unicode__`.
- ✅ Add special methods: `__next__`, `__buffer__`, `__class_getitem__`,
`__mro_entries__`, `__release_buffer__`, and `__subclasshook__`.
- ✅ Support positional-only arguments.
- ❌ Add support for module functions `__dir__` and `__getattr__`. As
mentioned in the issue the check is scoped for methods rather than
module functions. I am hesitant to expand the scope of this check
without a discussion.
## Test Plan
- Manually confirmed each example file from the issue functioned as
expected.
- Ran cargo nextest to ensure `unexpected_special_method_signature` test
still passed.
Fixes#16217.
## Summary
This is just a small refactor to move workspace related structs and impl
out from `server.rs` where `Server` is defined and into a new
`workspace.rs`.
## Summary
This PR achieves the following:
* Add support for checking method calls, and inferring return types from
method calls. For example:
```py
reveal_type("abcde".find("abc")) # revealed: int
reveal_type("foo".encode(encoding="utf-8")) # revealed: bytes
"abcde".find(123) # error: [invalid-argument-type]
class C:
def f(self) -> int:
pass
reveal_type(C.f) # revealed: <function `f`>
reveal_type(C().f) # revealed: <bound method: `f` of `C`>
C.f() # error: [missing-argument]
reveal_type(C().f()) # revealed: int
```
* Implement the descriptor protocol, i.e. properly call the `__get__`
method when a descriptor object is accessed through a class object or an
instance of a class. For example:
```py
from typing import Literal
class Ten:
def __get__(self, instance: object, owner: type | None = None) ->
Literal[10]:
return 10
class C:
ten: Ten = Ten()
reveal_type(C.ten) # revealed: Literal[10]
reveal_type(C().ten) # revealed: Literal[10]
```
* Add support for member lookup on intersection types.
* Support type inference for `inspect.getattr_static(obj, attr)` calls.
This was mostly used as a debugging tool during development, but seems
more generally useful. It can be used to bypass the descriptor protocol.
For the example above:
```py
from inspect import getattr_static
reveal_type(getattr_static(C, "ten")) # revealed: Ten
```
* Add a new `Type::Callable(…)` variant with the following sub-variants:
* `Type::Callable(CallableType::BoundMethod(…))` — represents bound
method objects, e.g. `C().f` above
* `Type::Callable(CallableType::MethodWrapperDunderGet(…))` — represents
`f.__get__` where `f` is a function
* `Type::Callable(WrapperDescriptorDunderGet)` — represents
`FunctionType.__get__`
* Add new known classes:
* `types.MethodType`
* `types.MethodWrapperType`
* `types.WrapperDescriptorType`
* `builtins.range`
## Performance analysis
On this branch, we do more work. We need to do more call checking, since
we now check all method calls. We also need to do ~twice as many member
lookups, because we need to check if a `__get__` attribute exists on
accessed members.
A brief analysis on `tomllib` shows that we now call `Type::call` 1780
times, compared to 612 calls before.
## Limitations
* Data descriptors are not yet supported, i.e. we do not infer correct
types for descriptor attribute accesses in `Store` context and do not
check writes to descriptor attributes. I felt like this was something
that could be split out as a follow-up without risking a major
architectural change.
* We currently distinguish between `Type::member` (with descriptor
protocol) and `Type::static_member` (without descriptor protocol). The
former corresponds to `obj.attr`, the latter corresponds to
`getattr_static(obj, "attr")`. However, to model some details correctly,
we would also need to distinguish between a static member lookup *with*
and *without* instance variables. The lookup without instance variables
corresponds to `find_name_in_mro`
[here](https://docs.python.org/3/howto/descriptor.html#invocation-from-an-instance).
We currently approximate both using `member_static`, which leads to two
open TODOs. Changing this would be a larger refactoring of
`Type::own_instance_member`, so I chose to leave it out of this PR.
## Test Plan
* New `call/methods.md` test suite for method calls
* New tests in `descriptor_protocol.md`
* New `call/getattr_static.md` test suite for `inspect.getattr_static`
* Various updated tests
## Summary
This avoids looking up `__bool__` on class `bool` for every
`Type::Instance(bool).bool()` call. 1% performance win on cold cache, 4%
win on incremental performance.
This updates the `SymbolBindings` and `SymbolDeclarations` types to use
a single smallvec of live bindings/declarations, instead of splitting
that out into separate containers for each field.
I'm seeing an 11-13% `cargo bench` performance improvement with this
locally (for both cold and incremental). I'm interested to see if
Codspeed agrees!
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
A minor cleanup that breaks up a `HashMap` of an enum into separate
`HashMap`s for each variant. (These separate fields were already how
this cache was being described in the big comment at the top of the
file!)
This is a small tweak to avoid adding the callable `Type` on the error
value itself. Namely, it's always available regardless of the error, and
it's easy to pass it down explicitly to the diagnostic generating code.
It's likely that the other `CallBindingError` variants will also want
the callable `Type` to improve diagnostics too. This way, we don't have
to duplicate the `Type` on each variant. It's just available to all of
them.
Ref https://github.com/astral-sh/ruff/pull/16239#discussion_r1962352646
## Summary
Follow up on the discussion
[here](https://github.com/astral-sh/ruff/pull/16121#discussion_r1962973298).
Replace builtin classes with custom placeholder names, which should
hopefully make the tests a bit easier to understand.
I carefully renamed things one after the other, to make sure that there
is no functional change in the tests.
## Summary
This PR should help in
https://github.com/astral-sh/ruff-vscode/issues/676.
There are two issues that this is trying to fix all related to the way
shutdown should happen as per the protocol:
1. After the server handled the [shutdown
request](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#shutdown)
and while waiting for the exit notification:
> If a server receives requests after a shutdown request those requests
should error with `InvalidRequest`.
But, we raised an error and exited. This PR fixes it by entering a loop
which responds to any request during this period with `InvalidRequest`
2. If the server received an [exit
notification](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#exit)
but the shutdown request was never received, the server handled that by
logging and exiting with success but as per the spec:
> The server should exit with success code 0 if the shutdown request has
been received before; otherwise with error code 1.
So, this PR fixes that as well by raising an error in this case.
## Test Plan
I'm not sure how to go about testing this without using a mock server.
## Summary
This is part of the preparation for detecting syntax errors in the
parser from https://github.com/astral-sh/ruff/pull/16090/. As suggested
in [this
comment](https://github.com/astral-sh/ruff/pull/16090/#discussion_r1953084509),
I started working on a `ParseOptions` struct that could be stored in the
parser. For this initial refactor, I only made it hold the existing
`Mode` option, but for syntax errors, we will also need it to have a
`PythonVersion`. For that use case, I'm picturing something like a
`ParseOptions::with_python_version` method, so you can extend the
current calls to something like
```rust
ParseOptions::from(mode).with_python_version(settings.target_version)
```
But I thought it was worth adding `ParseOptions` alone without changing
any other behavior first.
Most of the diff is just updating call sites taking `Mode` to take
`ParseOptions::from(Mode)` or those taking `PySourceType`s to take
`ParseOptions::from(PySourceType)`. The interesting changes are in the
new `parser/options.rs` file and smaller parts of `parser/mod.rs` and
`ruff_python_parser/src/lib.rs`.
## Test Plan
Existing tests, this should not change any behavior.
We now resolve references in "eager" scopes correctly — using the
bindings and declarations that are visible at the point where the eager
scope is created, not the "public" type of the symbol (typically the
bindings visible at the end of the scope).
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
This uses the refactoring and support for secondary diagnostic messages
to improve the diagnostic for "invalid argument type." The main
improvement here is that we show where the function being called is
defined, and annotate the span corresponding to the invalid parameter.
This is a small little hack to make the `Diagnostic` trait
capable of supporting attaching multiple spans.
This design should be considered transient. This was just the
quickest way that I could see to pass multiple spans through from
the type checker to the diagnostic renderer.
This commit has no behavioral changes.
This refactor moves the logic for turning a `D: Diagnostic` into
an `annotate_snippets::Message` into its own types. This would
ideally just be a function or something, but the `annotate-snippets`
types want borrowed data, and sometimes we need to produce owned
data. So we gather everything we need into our own types and then
spit it back out in the format that `annotate-snippets` wants.
This factor was motivated by wanting to render multiple snippets.
The logic for generating a code frame is complicated enough that
it's worth splitting out so that we can reuse it for other spans.
(Note that one should consider this prototype-level code. It is
unlikely to survive for long.)
It seems nothing is using it, and I'm not sure if it makes semantic
sense. Particularly if we want to support multiple ranges. One could
make an argument that this ought to correspond to the "primary"
range (which we should have), but I think such a concept is better
expressed as an explicit routine if possible.
## Summary
Resolves#15979.
The file explains what Red Knot is (a type checker), what state it is in
(not yet ready for user testing), what its goals ("extremely fast") and
non-goals (not a drop-in replacement for other type checkers) are as
well as what the crates contain.
## Test Plan
None.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Move class attribute (property, methods, variables) related cases in
AIR302_names to AIR302_class_attribute
## Test Plan
No functionality change. Test fixture is reogranized
Fixes false negative when slice bound uses length of string literal.
We were meant to check the following, for example. Given:
```python
text[:bound] if text.endswith(suffix) else text
```
We want to know whether:
- `suffix` is a string literal and `bound` is a number literal
- `suffix` is an expression and `bound` is
exactly `-len(suffix)` (as AST nodes, prior to evaluation.)
The issue is that negative number literals like `-10` are stored as
unary operators applied to a number literal in the AST. So when `suffix`
was a string literal but `bound` was `-len(suffix)` we were getting
caught in the match arm where `bound` needed to be a number. This is now
fixed with a guard.
Closes#16231
## Summary
This PR updates the formatter and linter to use the `PythonVersion`
struct from the `ruff_python_ast` crate internally. While this doesn't
remove the need for the `linter::PythonVersion` enum, it does remove the
`formatter::PythonVersion` enum and limits the use in the linter to
deserializing from CLI arguments and config files and moves most of the
remaining methods to the `ast::PythonVersion` struct.
## Test Plan
Existing tests, with some inputs and outputs updated to reflect the new
(de)serialization format. I think these are test-specific and shouldn't
affect any external (de)serialization.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
This PR updates the `ruff.printDebugInformation` command to return the
info as string in the response. Currently, we send a `window/logMessage`
request with the info but that has the disadvantage that it's not
visible to the user directly.
What `rust-analyzer` does with it's `rust-analyzer/status` request which
returns it as a string which then the client can just display it in a
separate window. This is what I'm thinking of doing as well.
Other editors can also benefit from it by directly opening a temporary
file with this information that the user can see directly.
There are couple of options here:
1. Keep using the command, keep the log request and return the string
2. Keep using the command, remove the log request and return the string
3. Create a new request similar to `rust-analyzer/status` which returns
a string
This PR implements (1) but I'd want to move towards (2) and remove the
log request completely. We haven't advertised it as such so this would
only require updating the VS Code extension to handle it by opening a
new document with the debug content.
## Test plan
For VS Code, refer to https://github.com/astral-sh/ruff-vscode/pull/694.
For Neovim, one could do:
```lua
local function execute_ruff_command(command)
local client = vim.lsp.get_clients({
bufnr = vim.api.nvim_get_current_buf(),
name = name,
method = 'workspace/executeCommand',
})[1]
if not client then
return
end
client.request('workspace/executeCommand', {
command = command,
arguments = {
{ uri = vim.uri_from_bufnr(0) }
},
function(err, result)
if err then
-- log error
return
end
vim.print(result)
-- Or, open a new window with the `result` content
end
}
```
## Summary
Separate ImportPathMoved and ProviderName to avoid misusing (AIR303)
## Test Plan
only code arrangement is updated. existing test fixture should be not be
changed
## Summary
Related to https://github.com/astral-sh/ruff-vscode/pull/686, this PR
ignores handling source code actions for notebooks which are not
prefixed with `notebook`.
The main motivation is that the native server does not actually handle
it well which results in gibberish code. There's some context about this
in
https://github.com/astral-sh/ruff-vscode/issues/680#issuecomment-2647490812
and the following comments.
closes: https://github.com/astral-sh/ruff-vscode/issues/680
## Test Plan
Running a notebook with the following does nothing except log the
message:
```json
"notebook.codeActionsOnSave": {
"source.organizeImports.ruff": "explicit",
},
```
while, including the `notebook` code actions does make the edit (as
usual):
```json
"notebook.codeActionsOnSave": {
"notebook.source.organizeImports.ruff": "explicit"
},
```
## Summary
This PR does the following:
* Moves the following from `types.rs` in `symbol.rs`:
* `symbol`
* `global_symbol`
* `imported_symbol`
* `symbol_from_bindings`
* `symbol_from_declarations`
* `SymbolAndQualifiers`
* `SymbolFromDeclarationsResult`
* Moves the following from `stdlib.rs` in `symbol.rs` and removes
`stdlib.rs`:
* `known_module_symbol`
* `builtins_symbol`
* `typing_symbol` (only for tests)
* `typing_extensions_symbol`
* `builtins_module_scope`
* `core_module_scope`
* Add `symbol_from_bindings_impl` and `symbol_from_declarations_impl` to
keep `RequiresExplicitReExport` an implementation detail
* Make `declaration_type` a `pub(crate)` as it's required in
`symbol_from_declarations` (`binding_type` is already `pub(crate)`
The main motivation is to keep the implementation details private and
only expose an ergonomic API which uses sane defaults for various
scenario to avoid any mistakes from the caller. Refer to
https://github.com/astral-sh/ruff/pull/16133#discussion_r1955262772,
https://github.com/astral-sh/ruff/pull/16133#issue-2850146612 for
details.
## Summary
This PR makes the following changes:
- It adjusts various callsites to use the new
`ast::StringLiteral::contents_range()` method that was introduced in
https://github.com/astral-sh/ruff/pull/16183. This is less verbose and
more type-safe than using the `ast::str::raw_contents()` helper
function.
- It adds a new `ast::ExprStringLiteral::as_unconcatenated_literal()`
helper method, and adjusts various callsites to use it. This addresses
@MichaReiser's review comment at
https://github.com/astral-sh/ruff/pull/16183#discussion_r1957334365.
There is no functional change here, but it helps readability to make it
clearer that we're differentiating between implicitly concatenated
strings and unconcatenated strings at various points.
- It renames the `StringLiteralValue::flags()` method to
`StringLiteralFlags::first_literal_flags()`. If you're dealing with an
implicitly concatenated string `string_node`,
`string_node.value.flags().closer_len()` could give an incorrect result;
this renaming makes it clearer that the `StringLiteralFlags` instance
returned by the method is only guaranteed to give accurate information
for the first `StringLiteral` contained in the `ExprStringLiteral` node.
- It deletes the unused `BytesLiteralValue::flags()` method. This seems
prone to misuse in the same way as `StringLiteralValue::flags()`: if
it's an implicitly concatenated bytestring, the `BytesLiteralFlags`
instance returned by the method would only give accurate information for
the first `BytesLiteral` in the bytestring.
## Test Plan
`cargo test`
Micha Reiser
David Peter
Felix Scherz
Vasanth
David Sherret
Wei Lee
Jo
Dhruv Manilawala
Charlie Marsh
chiri
Alex Waygood
lipefree
InSync
Carl Meyer
Dylan
Brandt Bucher
Sumana Harihareswara
Denys Kyslytsyn
Max Mynter
justin
Ramil Aleskerov
हिमांशु
Brent Westbrook
Vasco Schiavo
Adam Aaronson
Emily B. Zhang
Douglas Creager
renovate[bot]
Dragon
Chandra Kiran G
Matthew Mckee
TomerBin
Victor Hugo Gomes
Andrew Gallant
github-actions[bot]
Dan Parizher
Usul-Dev
Dimitri Papadopoulos Orfanos
Abhijeet Prasad Bodas
ZGY
Zanie Blue
Yunchi Pang
Marcus Näslund
Ibraheem Ahmed
Shunsuke Shibayama
Rogdham
omahs
Shaygan Hooshyari
Aria Desires
Eric Botti
Hans
Brendan Cooley
jie211
camper42
Navdeep K
David Salvisberg
w0nder1ng
Eric Mark Martin
Nuri Jung
cake-monotone
Mike Perlov
trag1c
Mohammad Amin Ghasemi
Trevor Manz
Aarni Koskela
Aleksei Latyshev
John Stilley
Daniel Wilton
Ash Berlin-Taylor
Junhson Jean-Baptiste
Josh Cannon
Peter Hill
Kaxil Naik
Mauro Fontana
Peter Hill
Joey Bar
Joey Bar
Jelle Zijlstra
Adam Johnson
Darius Carrier
Muspi Merol
Ari Pollak
Victorien
sobolevn
purajit