We mostly keep things the same here, but the message has been moved from
the annotation to the diagnostic's top-line message. I think this is
perhaps a little worse, but some bigger improvements could be made here.
Indeed, we could perhaps even add a "fix" here.
This moves all INVALID_ASSIGNMENT lints related to unpacking over to the new
diagnostic model.
While we're here, we improve the diagnostic a bit by adding a secondary
annotation covering where the value is. We also split apart the original
singular message into one message for the diagnostic and the "expected
versus got" into annotation messages.
This tests the diagnostic rendering of a case that wasn't previously
covered by snapshots: when unpacking fails because there are too few
values, but where the left hand side can tolerate "N or more." In the
code, this is a distinct diagnostic, so we capture it here.
(Sorry about the diff here, but it made sense to rename the other
sections and that changes the name of the snapshot file.)
I believe this was an artifact of an older iteration of the diagnostic
reporting API. But this is strictly not necessary now, and indeed, might
even be annoying. It is okay, but perhaps looks a little odd, to do
`builder.into_diagnostic("...")` if you don't want to add anything else
to the diagnostic.
I suspect this will be used pretty frequently (I wanted it
immediately). And more practically, this avoids needing to
import `Annotation` to create it.
## Summary
I ran red-knot on every project in mypy-primer. I moved every project
where red-knot ran to completion (fast enough, and mypy-primer could
handle its output) into `good.txt`, so it will run in our CI.
The remaining projects I left listed in `bad.txt`, with a comment
summarizing the failure mode (a few don't fail, they are just slow -- on
a debug build, at least -- or output too many diagnostics for
mypy-primer to handle.)
We will now run CI on 109 projects; 34 are left in `bad.txt`.
## Test Plan
CI on this PR!
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Takes the `good.txt` changes from #17474, and removes the following
projects:
- arrow (not part of mypy_primer upstream)
- freqtrade, hydpy, ibis, pandera, xarray (saw panics locally, all
related to try_metaclass cycles)
Increases the mypy_primer CI run time to ~4 min.
## Test Plan
Three successful CI runs.
## Summary
* Add initial support for `typing.dataclass_transform`
* Support decorating a function decorator with `@dataclass_transform(…)`
(used by `attrs`, `strawberry`)
* Support decorating a metaclass with `@dataclass_transform(…)` (used by
`pydantic`, but doesn't work yet, because we don't seem to model
`__new__` calls correctly?)
* *No* support yet for decorating base classes with
`@dataclass_transform(…)`. I haven't figured out how this even supposed
to work. And haven't seen it being used.
* Add `strawberry` as an ecosystem project, as it makes heavy use of
`@dataclass_transform`
## Test Plan
New Markdown tests
## Summary
This PR is a follow-up to #16852.
Instance variables bound in comprehensions are recorded, allowing type
inference to work correctly.
This required adding support for unpacking in comprehension which
resolves https://github.com/astral-sh/ruff/issues/15369.
## Test Plan
One TODO in `mdtest/attributes.md` is now resolved, and some new test
cases are added.
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
## Summary
If two types are gradually-equivalent, that means they share the same
set of possible materializations. There's no need to keep two such types
in the same union or intersection; we should simplify them.
Fixes https://github.com/astral-sh/ruff/issues/17465
The one downside here is that now we will simplify e.g. `Unknown |
Todo(...)` to just `Unknown`, if `Unknown` was added to the union first.
This is correct from a type perspective (they are equivalent types), but
it can mean we lose visibility into part of the cause for the type
inferring as unknown. I think this is OK, but if we think it's important
to avoid this, I can add a special case to try to preserve `Todo` over
`Unknown`, if we see them both in the same union or intersection.
## Test Plan
Added and updated mdtests.
## Summary
The long line of projects in `mypy_primer.yaml` is hard to work with
when adding projects or checking whether they are currently run. Use a
one-per-line text file instead.
## Test Plan
Ecosystem check on this PR.
## Summary
Member lookup can be cyclic, with type inference of implicit members. A
sample case is shown in the added mdtest.
There's no clear way to handle such cases other than to fixpoint-iterate
the cycle.
Fixes#17457.
## Test Plan
Added test.
You can now use subscript expressions in a type expression to explicitly
specialize generic classes, just like you could already do in value
expressions.
This still does not implement bidirectional checking, so a type
annotation on an assignment does not influence how we infer a
specialization for a (not explicitly specialized) constructor call. You
might get an `invalid-assignment` error if (a) we cannot infer a class
specialization from the constructor call (in which case you end up e.g.
trying to assign `C[Unknown]` to `C[int]`) or if (b) we can infer a
specialization, but it doesn't match the annotation.
Closes https://github.com/astral-sh/ruff/issues/17432
## Summary
There was some narrowing constraints not covered from the previous PR
```py
def _(x: object):
if (type(y := x)) is bool:
reveal_type(y) # revealed: bool
```
Also, refactored a bit
## Test Plan
Update type_api.md
In #17403 I added a comment asserting that all same-kind literal types
share all the same super-types. This is true, with two notable
exceptions: the types `AlwaysTruthy` and `AlwaysFalsy`. These two types
are super-types of some literal types within a given kind and not
others: `Literal[0]`, `Literal[""]`, and `Literal[b""]` inhabit
`AlwaysFalsy`, while other literals inhabit `AlwaysTruthy`.
This PR updates the literal-unions optimization to handle these types
correctly.
Fixes https://github.com/astral-sh/ruff/issues/17447
Verified locally that `QUICKCHECK_TESTS=100000 cargo test -p
red_knot_python_semantic -- --ignored types::property_tests::stable` now
passes again.
## Summary
Fixes#17147.
This was landed in #17149 and then reverted in #17335 because it caused
cycle panics in checking pybind11. #17456 fixed the cause of that panic.
## Test Plan
Add new narrow/assert.md test file
Co-authored-by: Matthew Mckee <matthewmckee04@yahoo.co.uk>
## Summary
We were over-conflating the conditions for deferred name resolution.
`from __future__ import annotations` defers annotations, but not class
bases. In stub files, class bases are also deferred. Modeling this
correctly also reduces likelihood of cycles in Python files using `from
__future__ import annotations` (since deferred resolution is inherently
cycle-prone). The same cycles are still possible in `.pyi` files, but
much less likely, since typically there isn't anything in a `pyi` file
that would cause an early return from a scope, or otherwise cause
visibility constraints to persist to end of scope. Usually there is only
code at module global scope and class scope, which can't have `return`
statements, and `raise` or `assert` statements in a stub file would be
very strange. (Technically according to the spec we'd be within our
rights to just forbid a whole bunch of syntax outright in a stub file,
but I kinda like minimizing unnecessary differences between the handling
of Python files and stub files.)
## Test Plan
Added mdtests.
## Summary
Part of #15383, this PR adds support for overloaded callables.
Typing spec: https://typing.python.org/en/latest/spec/overload.html
Specifically, it does the following:
1. Update the `FunctionType::signature` method to return signatures from
a possibly overloaded callable using a new `FunctionSignature` enum
2. Update `CallableType` to accommodate overloaded callable by updating
the inner type to `Box<[Signature]>`
3. Update the relation methods on `CallableType` with logic specific to
overloads
4. Update the display of callable type to display a list of signatures
enclosed by parenthesis
5. Update `CallableTypeOf` special form to recognize overloaded callable
6. Update subtyping, assignability and fully static check to account for
callables (equivalence is planned to be done as a follow-up)
For (2), it is required to be done in this PR because otherwise I'd need
to add some workaround for `into_callable_type` and I though it would be
best to include it in here.
For (2), another possible design would be convert `CallableType` in an
enum with two variants `CallableType::Single` and
`CallableType::Overload` but I decided to go with `Box<[Signature]>` for
now to (a) mirror it to be equivalent to `overload` field on
`CallableSignature` and (b) to avoid any refactor in this PR. This could
be done in a follow-up to better split the two kind of callables.
### Design
There were two main candidates on how to represent the overloaded
definition:
1. To include it in the existing infrastructure which is what this PR is
doing by recognizing all the signatures within the
`FunctionType::signature` method
2. To create a new `Overload` type variant
<details><summary>For context, this is what I had in mind with the new
type variant:</summary>
<p>
```rs
pub enum Type {
FunctionLiteral(FunctionType),
Overload(OverloadType),
BoundMethod(BoundMethodType),
...
}
pub struct OverloadType {
// FunctionLiteral or BoundMethod
overloads: Box<[Type]>,
// FunctionLiteral or BoundMethod
implementation: Option<Type>
}
pub struct BoundMethodType {
kind: BoundMethodKind,
self_instance: Type,
}
pub enum BoundMethodKind {
Function(FunctionType),
Overload(OverloadType),
}
```
</p>
</details>
The main reasons to choose (1) are the simplicity in the implementation,
reusing the existing infrastructure, avoiding any complications that the
new type variant has specifically around the different variants between
function and methods which would require the overload type to use `Type`
instead.
### Implementation
The core logic is how to collect all the overloaded functions. The way
this is done in this PR is by recording a **use** on the `Identifier`
node that represents the function name in the use-def map. This is then
used to fetch the previous symbol using the same name. This way the
signatures are going to be propagated from top to bottom (from first
overload to the final overload or the implementation) with each function
/ method. For example:
```py
from typing import overload
@overload
def foo(x: int) -> int: ...
@overload
def foo(x: str) -> str: ...
def foo(x: int | str) -> int | str:
return x
```
Here, each definition of `foo` knows about all the signatures that comes
before itself. So, the first overload would only see itself, the second
would see the first and itself and so on until the implementation or the
final overload.
This approach required some updates specifically recognizing
`Identifier` node to record the function use because it doesn't use
`ExprName`.
## Test Plan
Update existing test cases which were limited by the overload support
and add test cases for the following cases:
* Valid overloads as functions, methods, generics, version specific
* Invalid overloads as stated in
https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions
(implementation will be done in a follow-up)
* Various relation: fully static, subtyping, and assignability (others
in a follow-up)
## Ecosystem changes
_WIP_
After going through the ecosystem changes (there are a lot!), here's
what I've found:
We need assignability check between a callable type and a class literal
because a lot of builtins are defined as classes in typeshed whose
constructor method is overloaded e.g., `map`, `sorted`, `list.sort`,
`max`, `min` with the `key` parameter, `collections.abc.defaultdict`,
etc. (https://github.com/astral-sh/ruff/issues/17343). This makes up
most of the ecosystem diff **roughly 70 diagnostics**. For example:
```py
from collections import defaultdict
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(int)
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(list)
class Foo:
def __init__(self, x: int):
self.x = x
# red-knot: No overload of function `__new__` matches arguments [lint:no-matching-overload]
map(Foo, ["a", "b", "c"])
```
Duplicate diagnostics in unpacking
(https://github.com/astral-sh/ruff/issues/16514) has **~16
diagnostics**.
Support for the `callable` builtin which requires `TypeIs` support. This
is **5 diagnostics**. For example:
```py
from typing import Any
def _(x: Any | None) -> None:
if callable(x):
# red-knot: `Any | None`
# Pyright: `(...) -> object`
# mypy: `Any`
# pyrefly: `(...) -> object`
reveal_type(x)
```
Narrowing on `assert` which has **11 diagnostics**. This is being worked
on in https://github.com/astral-sh/ruff/pull/17345. For example:
```py
import re
match = re.search("", "")
assert match
match.group() # error: [possibly-unbound-attribute]
```
Others:
* `Self`: 2
* Type aliases: 6
* Generics: 3
* Protocols: 13
* Unpacking in comprehension: 1
(https://github.com/astral-sh/ruff/pull/17396)
## Performance
Refer to
https://github.com/astral-sh/ruff/pull/17366#issuecomment-2814053046.
## Summary
Add more narrowing analysis for match statements:
* add narrowing constraints from guard expressions
* add negated constraints from previous predicates and guards to
subsequent cases
This PR doesn't address that guards can mutate your subject, and so
theoretically invalidate some of these narrowing constraints that you've
previously accumulated. Some prior art on this issue [here][mutable
guards].
[mutable guards]:
https://www.irif.fr/~scherer/research/mutable-patterns/mutable-patterns-mlworkshop2024-abstract.pdf
## Test Plan
Add some new tests, and update some existing ones
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fixes#14866Fixes#17437
## Test Plan
Update mdtests in `narrow/`
## Summary
This PR extends version-related syntax error detection to red-knot. The
main changes here are:
1. Passing `ParseOptions` specifying a `PythonVersion` to parser calls
2. Adding a `python_version` method to the `Db` trait to make this
possible
3. Converting `UnsupportedSyntaxError`s to `Diagnostic`s
4. Updating existing mdtests to avoid unrelated syntax errors
My initial draft of (1) and (2) in #16090 instead tried passing a
`PythonVersion` down to every parser call, but @MichaReiser suggested
the `Db` approach instead
[here](https://github.com/astral-sh/ruff/pull/16090#discussion_r1969198407),
and I think it turned out much nicer.
All of the new `python_version` methods look like this:
```rust
fn python_version(&self) -> ruff_python_ast::PythonVersion {
Program::get(self).python_version(self)
}
```
with the exception of the `TestDb` in `ruff_db`, which hard-codes
`PythonVersion::latest()`.
## Test Plan
Existing mdtests, plus a new mdtest to see at least one of the new
diagnostics.
## Summary
This changeset allows us to generate the signature of synthesized
`__init__` functions in dataclasses by analyzing the fields on the class
(and its superclasses). There are certain things that I have not yet
attempted to model in this PR, like `kw_only`,
[`dataclasses.KW_ONLY`](https://docs.python.org/3/library/dataclasses.html#dataclasses.KW_ONLY)
or functionality around
[`dataclasses.field`](https://docs.python.org/3/library/dataclasses.html#dataclasses.field).
ticket: https://github.com/astral-sh/ruff/issues/16651
## Ecosystem analysis
These two seem to depend on missing features in generics (see [relevant
code
here](9898ccbb78/tests/core/test_generics.py (L54))):
> ```diff
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:24:
Argument `x` does not match any known parameter
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:38:
Argument `y` does not match any known parameter
> ```
These two are true positives. See [relevant code
here](9898ccbb78/tests/core/test_config.py (L154-L161)).
> ```diff
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:161:24:
Argument to this function is incorrect: Expected `int`, found
`Literal["test"]`
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:172:24:
Argument to this function is incorrect: Expected `int | float`, found
`Literal["test"]`
> ```
This one depends on `**` unpacking of dictionaries, which we don't
support yet:
> ```diff
> + error[lint:missing-argument]
/tmp/mypy_primer/projects/mypy_primer/mypy_primer/globals.py:218:11: No
arguments provided for required parameters `new`, `old`, `repo`,
`type_checker`, `mypyc_compile_level`, `custom_typeshed_repo`,
`new_typeshed`, `old_typeshed`, `new_prepend_path`, `old_prepend_path`,
`additional_flags`, `project_selector`, `known_dependency_selector`,
`local_project`, `expected_success`, `project_date`, `shard_index`,
`num_shards`, `output`, `old_success`, `coverage`, `bisect`,
`bisect_output`, `validate_expected_success`,
`measure_project_runtimes`, `concurrency`, `base_dir`, `debug`, `clear`
> ```
## Test Plan
New Markdown tests.
## Summary
Support dataclasses with `order=True`:
```py
@dataclass(order=True)
class WithOrder:
x: int
WithOrder(1) < WithOrder(2) # no error
```
Also adds some additional tests to `dataclasses.md`.
ticket: #16651
## Test Plan
New Markdown tests
This PR adds **_very_** basic inference of generic typevars at call
sites. It does not bring in a full unification algorithm, and there are
a few TODOs in the test suite that are not discharged by this. But it
handles a good number of useful cases! And the PR does not add anything
that would go away with a more sophisticated constraint solver.
In short, we just look for typevars in the formal parameters, and assume
that the inferred type of the corresponding argument is what that
typevar should map to. If a typevar appears more than once, we union
together the corresponding argument types.
Cases we are not yet handling:
- We are not widening literals.
- We are not recursing into parameters that are themselves generic
aliases.
- We are not being very clever with parameters that are union types.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This is similar to https://github.com/astral-sh/ruff/pull/17095, it adds
assignability check for bound methods to callables.
## Test Plan
Add test cases to for assignability; specifically it uses gradual types
because otherwise it would just delegate to `is_subtype_of`.
## Summary
closes#16615
This PR includes:
- Introduces a new type: `Type::BoundSuper`
- Implements member lookup for `Type::BoundSuper`, resolving attributes
by traversing the MRO starting from the specified class
- Adds support for inferring appropriate arguments (`pivot_class` and
`owner`) for `super()` when it is used without arguments
When `super(..)` appears in code, it can be inferred into one of the
following:
- `Type::Unknown`: when a runtime error would occur (e.g. calling
`super()` out of method scope, or when parameter validation inside
`super` fails)
- `KnownClass::Super::to_instance()`: when the result is an *unbound
super object* or when a dynamic type is used as parameters (MRO
traversing is meaningless)
- `Type::BoundSuper`: the common case, representing a properly
constructed `super` instance that is ready for MRO traversal and
attribute resolution
### Terminology
Python defines the terms *bound super object* and *unbound super
object*.
An **unbound super object** is created when `super` is called with only
one argument (e.g.
`super(A)`). This object may later be bound via the `super.__get__`
method. However, this form is rarely used in practice.
A **bound super object** is created either by calling
`super(pivot_class, owner)` or by using the implicit form `super()`,
where both arguments are inferred from the context. This is the most
common usage.
### Follow-ups
- Add diagnostics for `super()` calls that would result in runtime
errors (marked as TODO)
- Add property tests for `Type::BoundSuper`
## Test Plan
- Added `mdtest/class/super.md`
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Until we optimize our full union/intersection representation to
efficiently handle large numbers of same-kind literal types "as a
block", set a fairly low limit on the size of unions of literals.
We will want to increase this limit once we've made the broader
efficiency improvement (tracked in
https://github.com/astral-sh/ruff/issues/17420).
## Test Plan
`cargo bench --bench red_knot`
## Summary
Special-case literal types in `UnionBuilder` to speed up building large
unions of literals.
This optimization is extremely effective at speeding up building even a
very large union (it improves the large-unions benchmark by 41x!). The
problem we can run into is that it is easy to then run into another
operation on the very large union (for instance, narrowing may add it to
an intersection, which then distributes it over the intersection) which
is still slow.
I think it is possible to avoid this by extending this optimized
"grouped" representation throughout not just `UnionBuilder`, but all of
our union and intersection representations. I have some work in this
direction, but rather than spending more time on it right now, I'd
rather just land this much, along with a limit on the size of these
unions (to avoid building really big unions quickly and then hitting
issues where they are used.)
## Test Plan
Existing tests and benchmarks.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Fixes incorrect negated type eq and ne assertions in
infer_binary_intersection_type_comparison
fixes#17360
## Test Plan
Remove and update some now incorrect tests
This reworks the assignability/subtyping relations a bit to handle
typevars better:
1. For the most part, types are not assignable to typevars, since
there's no guarantee what type the typevar will be specialized to.
2. An intersection is an exception, if it contains the typevar itself as
one of the positive elements. This should fall out from the other
clauses automatically, since a typevar is assignable to itself, and an
intersection is assignable to something if any positive element is
assignable to that something.
3. Constrained typevars are an exception, since they must be specialized
to _exactly_ one of the constraints, not to a _subtype_ of a constraint.
If a type is assignable to every constraint, then the type is also
assignable to the constrained typevar.
We already had a special case for (3), but the ordering of it relative
to the intersection clauses meant we weren't catching (2) correctly. To
fix this, we keep the special case for (3), but fall through to the
other match arms for non-constrained typevars and if the special case
isn't true for a constrained typevar.
Closes https://github.com/astral-sh/ruff/issues/17364
## Summary
Add very early support for dataclasses. This is mostly to make sure that
we do not emit false positives on dataclass construction, but it also
lies some foundations for future extensions.
This seems like a good initial step to merge to me, as it basically
removes all false positives on dataclass constructor calls. This allows
us to use the ecosystem checks for making sure we don't introduce new
false positives as we continue to work on dataclasses.
## Ecosystem analysis
I re-ran the mypy_primer evaluation of [the `__init__`
PR](https://github.com/astral-sh/ruff/pull/16512) locally with our
current mypy_primer version and project selection. It introduced 1597
new diagnostics. Filtering those by searching for `__init__` and
rejecting those that contain `invalid-argument-type` (those could not
possibly be solved by this PR) leaves 1281 diagnostics. The current
version of this PR removes 1171 diagnostics, which leaves 110
unaccounted for. I extracted the lint + file path for all of these
diagnostics and generated a diff (of diffs), to see which
`__init__`-diagnostics remain. I looked at a subset of these: There are
a lot of `SomeClass(*args)` calls where we don't understand the
unpacking yet (this is not even related to `__init__`). Some others are
related to `NamedTuple`, which we also don't support yet. And then there
are some errors related to `@attrs.define`-decorated classes, which
would probably require support for `dataclass_transform`, which I made
no attempt to include in this PR.
## Test Plan
New Markdown tests.
## Summary
* Partial #17238
* Flyby from discord discussion - `todo_type!` now statically checks for
no parens in the message to avoid issues between debug & release build
tests
## Test Plan
many mdtests are changing
## Summary
This PR moves all the relation methods from `CallableType` to
`Signature`.
The main reason for this is that `Signature` is going to be the common
denominator between normal and overloaded callables and the core logic
to check a certain relationship is going to just require the information
that would exists on `Signature`. For example, to check whether an
overloaded callable is a subtype of a normal callable, we need to check
whether _every_ overloaded signature is a subtype of the normal
callable's signature. This "every" logic would become part of the
`CallableType` and the core logic of checking the subtyping would exists
on `Signature`.
## Summary
Allows us to establish that two literals do not have a subtype
relationship with each other, without having to fallback to a typeshed
Instance type, which is comparatively slow.
Improves the performance of the many-string-literals union benchmark by
5x.
## Test Plan
`cargo test -p red_knot_python_semantic` and `cargo bench --bench
red_knot`.
This commit shuffles the reporting API around a little bit such that a
range is required, up front, when reporting a lint diagnostic. This in
turn enables us to make suppression checking eager.
In order to avoid callers needing to provide the range twice, we create
a primary annotation *without* a message inside the `Diagnostic`
encapsulated by the guard. We do this instead of requiring the message
up front because we're concerned about API complexity and the effort
involved in creating the message.
In order to provide a means of attaching a message to the primary
annotation, we expose a convenience API on `LintDiagnosticGuard` for
setting the message. This isn't generally possible for a `Diagnostic`,
but since a `LintDiagnosticGuard` knows how the `Diagnostic` was
constructed, we can offer this API correctly.
It strikes me that it might be easy to forget to attach a primary
annotation message, btu I think this the "least" bad failure mode. And
in particular, it should be somewhat obvious that it's missing once one
adds a snapshot test for how the diagnostic renders.
Otherwise, this API gives us the ability to eagerly check whether a
diagnostic should be reported with nearly minimal information. It also
shouldn't have any footguns since it guarantees that the primary
annotation is tied to the file in the typing context. And it keeps
things pretty simple: callers only need to provide what is actually
strictly necessary to make a diagnostic.
This is the payoff from removing a bit of indirection. The types still
exist, but now callers don't need to do builder -> reporter ->
diagnostic. They can just conceptually think of it as builder ->
diagnostic.
We're going to make the guards deref to `Diagnostic` in order to remove
a layer of indirection in the reporter API. (Well, technically the layer
is not removed since the types still exist, but in actual _usage_ the
layer will be removed. We'll see how it shakes out in the next commit.)
## Summary
Infer precise Boolean literal types for `str.startswith` calls where the
instance and the prefix are both string literals. This allows us to
understand `sys.platform.startswith(…)` branches.
## Test Plan
New Markdown tests
## Summary
For silencing `invalid-type-form` diagnostics in unreachable code, we
use the same approach that we use before and check the reachability that
we already record.
For silencing `invalid-bases`, we simply check if the type of the base
is `Never`. If so, we silence the diagnostic with the argument that the
class construction would never happen.
## Test Plan
Updated Markdown tests.
## Summary
Similar to what we did for `unresolved-reference` and
`unresolved-attribute`, we now also silence `unresolved-import`
diagnostics if the corresponding `import` statement is unreachable.
This addresses the (already closed) issue #17049.
## Test Plan
Adapted Markdown tests.
This finally completes the deletion of all old diagnostic types.
We do this by migrating the second (and last) use of secondary
diagnostic messages: to highlight the return type of a function
definition when its return value is inconsistent with the type.
Like the last diagnostic, we do actually change the message here a bit.
We don't need a sub-diagnostic here, and we can instead just add a
secondary annotation to highlight the return type.
This is the first use of the new `lint()` reporter.
I somewhat skipped a step here and also modified the actual diagnostic
message itself. The snapshots should tell the story.
We couldn't do this before because we had no way of differentiating
between "message for the diagnostic as a whole" and "message for a
specific code annotation." Now we can, so we can write more precise
messages based on the assumption that users are also seeing the code
snippet.
The downside here is that the actual message text can become quite vague
in the absence of the code snippet. This occurs, for example, with
concise diagnostic formatting. It's unclear if we should do anything
about it. I don't really see a way to make it better that doesn't
involve creating diagnostics with messages for each mode, which I think
would be a major PITA.
The upside is that this code gets a bit simpler, and we very
specifically avoid doing extra work if this specific lint is disabled.
This required a bit of surgery in the diagnostic matching and more
faffing about using a "concise" message from a diagnostic instead of
only printing the "primary" message.
In the new diagnostic data model, we really should have a main
diagnostic message *and* a primary span (with an optional message
attached to it) for every diagnostic.
In this commit, I try to make this true for the "revealed type"
diagnostic. Instead of the annotation saying both "revealed type is"
and also the revealed type itself, the annotation is now just the
revealed type and the main diagnostic message is "Revealed type."
I expect this may be controversial. I'm open to doing something
different. I tried to avoid redundancy, but maybe this is a special case
where we want the redundancy. I'm honestly not sure. I do *like* how it
looks with this commit, but I'm not working with Red Knot's type
checking daily, so my opinion doesn't count for much.
This did also require some tweaking to concise diagnostic formatting in
order to preserve the essential information.
This commit doesn't update every relevant snapshot. Just a few. I split
the rest out into the next commit.
... and replace it with use of `report()`.
Interestingly, this is the only instance of `report_diagnostic` used
directly, and thus anticipated to be the only instance of using
`report()`. If this ends up being a true single use method, we could
make it less generic and tailored specifically to "reveal type."
Two other things to note:
I left the "primary message" as empty. This avoids changing snapshots.
I address this in a subsequent commit.
The creation of a diagnostic here is a bit verbose/annoying. Certainly
more so than it was. This is somewhat expected since our diagnostic
model is more expressive and because we don't have a proc macro. I
avoided creating helpers for this case since there's only one use of
`report()`. But I expect to create helpers for the `lint()` case.
This is a surgical change that adds new `report()` and `lint()`
APIs to `InferContext`. These are intended to replace the existing
`report_*` APIs.
The comments should explain what these reporters are meant to do. For
the most part, this is "just" shuffling some code around. The actual
logic for determining whether a lint *should* be reported or not remains
unchanged and we don't make any changes to how a `Diagnostic` is
actually constructed (yet).
I initially tried to just use `LintReporter` and `DiagnosticReporter`
without the builder types, since I perceive the builder types to be an
annoying additional layer. But I found it also exceedingly annoying to
have to construct and provide the diagnostic message before you even
know if you are going to build the diagnostic. I also felt like this
could result in potentially unnecessary and costly querying in some
cases, although this is somewhat hand wavy. So I overall felt like the
builder route was the way to go. If the builders end up being super
annoying, we can probably add convenience APIs for common patterns to
paper over them.
## Summary
Basically just repeat the same thing that we did for
`unresolved-reference`, but now for attribute expressions.
We now also handle the case where the unresolved attribute (or the
unresolved reference) diagnostic originates from a stringified type
annotation.
And I made the evaluation of reachability constraints lazy (will only be
evaluated right before we are about to emit a diagnostic).
## Test Plan
New Markdown tests for stringified annotations.
I merged #17149 without checking the ecosystem results, and it still
caused a cycle panic in pybind11. Reverting for now until I fix that, so
we don't lose the ecosystem signal on other PRs.
This causes spurious query cycles.
This PR also includes an update to Salsa, which gives us db events on
cycle iteration, so we can write tests asserting the absence of a cycle.
## Summary
Track the reachability of nested scopes within their parent scopes. We
use this as an additional requirement for emitting
`unresolved-reference` diagnostics (and in the future,
`unresolved-attribute` and `unresolved-import`). This means that we only
emit `unresolved-reference` for a given use of a symbol if the use
itself is reachable (within its own scope), *and if the scope itself is
reachable*. For example, no diagnostic should be emitted for the use of
`x` here:
```py
if False:
x = 1
def f():
print(x) # this use of `x` is reachable inside the `f` scope,
# but the whole `f` scope is not reachable.
```
There are probably more fine-grained ways of solving this problem, but
they require a more sophisticated understanding of nested scopes (see
#15777, in particular
https://github.com/astral-sh/ruff/issues/15777#issuecomment-2788950267).
But it doesn't seem completely unreasonable to silence *this specific
kind of error* in unreachable scopes.
## Test Plan
Observed changes in reachability tests and ecosystem.
## Summary
Update Salsa to pull in https://github.com/salsa-rs/salsa/pull/788 which
fixes the, by now, famous *access to field whilst the value is being
initialized*.
This PR also re-enables all tests that previously triggered the panic.
## Test Plan
`cargo test`
## Summary
There is a new official URL for the typing documentation:
https://typing.python.org/
Change all https://typing.readthedocs.io/ links to use the new sub
domain, which is slightly shorter and looks more official.
## Test Plan
Tested to see if each and every new URL is accessible. I noticed that
some links go to https://typing.python.org/en/latest/source/stubs.html
which seems to be outdated, but that is a separate issue. The same page
shows up for the old URL.
This PR lets you explicitly specialize a generic class using a subscript
expression. It introduces three new Rust types for representing classes:
- `NonGenericClass`
- `GenericClass` (not specialized)
- `GenericAlias` (specialized)
and two enum wrappers:
- `ClassType` (a non-generic class or generic alias, represents a class
_type_ at runtime)
- `ClassLiteralType` (a non-generic class or generic class, represents a
class body in the AST)
We also add internal support for specializing callables, in particular
function literals. (That is, the internal `Type` representation now
attaches an optional specialization to a function literal.) This is used
in this PR for the methods of a generic class, but should also give us
most of what we need for specializing generic _functions_ (which this PR
does not yet tackle).
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
As discussed in https://github.com/astral-sh/ruff/issues/16983 and
"mitigate" said issue for the alpha.
This PR changes the default for `PythonPlatform` to be the current
platform rather than `all`.
I'm not sure if we should be as sophisticated as supporting `ios` and
`android` as defaults but it was easy...
## Test Plan
Updated Markdown tests.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
This is a new test case that I don't know how to handle yet. It leads to
many false positives in `rich/tests/test_win32_console.py`, which does
something like:
```py
if sys.platform == "win32":
from windows_only_module import some_symbol
some_other_symbol = 1
def some_test_case():
use(some_symbol) # Red Knot: unresolved-reference
use(some_other_symbol) # Red Knot: unresolved-reference
```
Also adds a test for using unreachable symbols in type annotations or as
class bases.
## Summary
* Addresses #16511 for simple cases where only `__init__` method is
bound on class or doesn't exist at all.
* fixes a bug with argument counting in bound method diagnostics
Caveats:
* No handling of `__new__` or modified `__call__` on metaclass.
* This leads to a couple of false positive errors in tests
## Test Plan
- A couple new cases in mdtests
- cargo nextest run -p red_knot_python_semantic --no-fail-fast
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
## Summary
We already have partial "support" for `assert_never`, because it is
annotated as
```pyi
def assert_never(arg: Never, /) -> Never: ...
```
in typeshed. So we already emit a `invalid-argument-type` diagnostic if
the argument type to `assert_never` is not assignable to `Never`.
That is not enough, however. Gradual types like `Any`, `Unknown`,
`@Todo(…)` or `Any & int` can be assignable to `Never`. Which means that
we didn't issue any diagnostic in those cases.
Also, it seems like `assert_never` deserves a dedicated diagnostic
message, not just a generic "invalid argument type" error.
## Test Plan
New Markdown tests.
## Summary
This implements a new approach to silencing `unresolved-reference`
diagnostics by keeping track of the reachability of each use of a
symbol. The changes merged in
https://github.com/astral-sh/ruff/pull/17169 are still needed for the
"Use of variable in nested function" test case, but that could also be
solved in another way eventually (see
https://github.com/astral-sh/ruff/issues/15777). We can use the same
technique to silence `unresolved-import` and `unresolved-attribute`
false-positives, but I think this could be merged in isolation.
## Test Plan
New Markdown tests, ecosystem tests
## Summary
This PR adds support for stub packages, except for partial stub packages
(a stub package is always considered non-partial).
I read the specification at
[typing.python.org/en/latest/spec/distributing.html#stub-only-packages](https://typing.python.org/en/latest/spec/distributing.html#stub-only-packages)
but I found it lacking some details, especially on how to handle
namespace packages or when the regular and stub packages disagree on
whether they're namespace packages. I tried to document my decisions in
the mdtests where the specification isn't clear and compared the
behavior to Pyright.
Mypy seems to only support stub packages in the venv folder. At least,
it never picked up my stub packages otherwise. I decided not to spend
too much time fighting mypyp, which is why I focused the comparison
around Pyright
Closes https://github.com/astral-sh/ruff/issues/16612
## Test plan
Added mdtests
## Summary
Some more edge cases that I thought of while working on integrating
knowledge of statically known branches into the `*`-import machinery
## Test Plan
`cargo test -p red_knot_python_semantic`
## Summary
This PR does the following things:
- Fixes the `python` configuration setting for mdtest (added in
https://github.com/astral-sh/ruff/pull/17221) so that it expects a path
pointing to a venv's `sys.prefix` variable rather than the a path
pointing to the venv's `site-packages` subdirectory. This brings the
`python` setting in mdtest in sync with our CLI `--python` flag.
- Tweaks mdtest so that it automatically creates a valid `pyvenv.cfg`
file for you if you don't specify one. This makes it much more ergonomic
to write an mdtest with a custom `python` setting: red-knot will reject
a `python` setting that points to a directory that doesn't have a
`pyvenv.cfg` file in it
- Tweaks mdtest so that it doesn't check a custom `pyvenv.cfg` as Python
source code if you _do_ add a custom `pyvenv.cfg` file for your mock
virtual environment in an mdtest. (You get a lot of diagnostics about
Python syntax errors in the `pyvenv.cfg` file, otherwise!)
- Rewrites the test added in
https://github.com/astral-sh/ruff/pull/17178 as an mdtest, and deletes
the original test that was added in that PR
## Test Plan
I verified that the new mdtest fails if I revert the changes to
`resolver.rs` that were added in
https://github.com/astral-sh/ruff/pull/17178
## Summary
This PR fixes the cycle issue that was causing problems in the `support
super` PR.
### Affected queries
- `all_narrowing_constraints_for_expression`
- `all_negative_narrowing_constraints_for_expression`
--
Additionally, `bidict` and `werkzeug` have been added to the
project-selection list in `mypy_primer`.
This PR also addresses the panics that occurred while analyzing those
packages:
- `bidict`: panic triggered by
`all_narrowing_constraints_for_expression`
- `werkzeug`: panic triggered by
`all_negative_narrowing_constraints_for_expression`
I think the mypy-primer results for this PR can serve as sufficient test
:)
## Summary
This is a follow up to the goto type definition PR. Specifically, that
we want to avoid exposing too many semantic model internals publicly.
I want to get some feedback on the approach taken. I think it goes into
the right direction but I'm not super happy with it.
The basic idea is that we add a `Type::definition` method which does the
"goto type definition". The parts that I think make it awkward:
* We can't directly return `Definition` because we don't create a
`Definition` for modules (but we could?). Although I think it makes
sense to possibly have a more public wrapper type anyway?
* It doesn't handle unions and intersections. Mainly because not all
elements in an intersection may have a definition and we only want to
show a navigation target for intersections if there's only a single
positive element (besides maybe `Unknown`).
An alternative design or an addition to this design is to introduce a
`SemanticAnalysis(Db)` struct that has methods like
`type_definition(&self, type)` which explicitly exposes the methods we
want. I don't feel comfortable design this API yet because it's unclear
how fine granular it has to be (and if it is very fine granular,
directly using `Type` might be better after all)
## Test Plan
`cargo test`
For two non-disjoint types `P` and `Q`, the simplification of `(P | Q) &
~Q` is not `P`, but `P & ~Q`. In other words, the non-empty set `P & Q`
is also excluded from the type.
The same applies for a constrained typevar `[T: (P, Q)]`: `T & ~Q`
should simplify to `P & ~Q`, not just `P`.
Implementing this is actually purely a matter of removing code from the
constrained typevar simplification logic; we just need to not bother
removing the negations. If the negations are actually redundant (because
the constraint types are disjoint), normal intersection simplification
will already eliminate them (as shown in the added test.)
## Summary
It turns out that `a.` isn't a list format supported by rustdoc. I
changed the documentation to use `1.`, `2.` instead.
## Test Plan
`cargo clippy`
This adds a new `Type` variant for holding an instance of a typevar
inside of a generic function or class. We don't handle specializing the
typevars yet, but this should implement most of the typing rules for
inside the generic function/class, where we don't know yet which
specific type the typevar will be specialized to.
This PR does _not_ yet handle the constraint that multiple occurrences
of the typevar must be specialized to the _same_ time. (There is an
existing test case for this in `generics/functions.md` which is still
marked as TODO.)
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
<!--
Thank you for contributing to Ruff! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
I decided to disable the new
[`needless_continue`](https://rust-lang.github.io/rust-clippy/master/index.html#needless_continue)
rule because I often found the explicit `continue` more readable over an
empty block or having to invert the condition of an other branch.
## Test Plan
`cargo test`
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Andrew Gallant
Alex Waygood
Carl Meyer
David Peter
Matthew Mckee
Shunsuke Shibayama
Douglas Creager
Dhruv Manilawala
Eric Mark Martin
Brent Westbrook
cake-monotone
Mike Perlov
Micha Reiser