## Summary
This PR extends version-related syntax error detection to red-knot. The
main changes here are:
1. Passing `ParseOptions` specifying a `PythonVersion` to parser calls
2. Adding a `python_version` method to the `Db` trait to make this
possible
3. Converting `UnsupportedSyntaxError`s to `Diagnostic`s
4. Updating existing mdtests to avoid unrelated syntax errors
My initial draft of (1) and (2) in #16090 instead tried passing a
`PythonVersion` down to every parser call, but @MichaReiser suggested
the `Db` approach instead
[here](https://github.com/astral-sh/ruff/pull/16090#discussion_r1969198407),
and I think it turned out much nicer.
All of the new `python_version` methods look like this:
```rust
fn python_version(&self) -> ruff_python_ast::PythonVersion {
Program::get(self).python_version(self)
}
```
with the exception of the `TestDb` in `ruff_db`, which hard-codes
`PythonVersion::latest()`.
## Test Plan
Existing mdtests, plus a new mdtest to see at least one of the new
diagnostics.
## Summary
This changeset allows us to generate the signature of synthesized
`__init__` functions in dataclasses by analyzing the fields on the class
(and its superclasses). There are certain things that I have not yet
attempted to model in this PR, like `kw_only`,
[`dataclasses.KW_ONLY`](https://docs.python.org/3/library/dataclasses.html#dataclasses.KW_ONLY)
or functionality around
[`dataclasses.field`](https://docs.python.org/3/library/dataclasses.html#dataclasses.field).
ticket: https://github.com/astral-sh/ruff/issues/16651
## Ecosystem analysis
These two seem to depend on missing features in generics (see [relevant
code
here](9898ccbb78/tests/core/test_generics.py (L54))):
> ```diff
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:24:
Argument `x` does not match any known parameter
> + error[lint:unknown-argument]
/tmp/mypy_primer/projects/dacite/tests/core/test_generics.py:54:38:
Argument `y` does not match any known parameter
> ```
These two are true positives. See [relevant code
here](9898ccbb78/tests/core/test_config.py (L154-L161)).
> ```diff
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:161:24:
Argument to this function is incorrect: Expected `int`, found
`Literal["test"]`
> + error[lint:invalid-argument-type]
/tmp/mypy_primer/projects/dacite/tests/core/test_config.py:172:24:
Argument to this function is incorrect: Expected `int | float`, found
`Literal["test"]`
> ```
This one depends on `**` unpacking of dictionaries, which we don't
support yet:
> ```diff
> + error[lint:missing-argument]
/tmp/mypy_primer/projects/mypy_primer/mypy_primer/globals.py:218:11: No
arguments provided for required parameters `new`, `old`, `repo`,
`type_checker`, `mypyc_compile_level`, `custom_typeshed_repo`,
`new_typeshed`, `old_typeshed`, `new_prepend_path`, `old_prepend_path`,
`additional_flags`, `project_selector`, `known_dependency_selector`,
`local_project`, `expected_success`, `project_date`, `shard_index`,
`num_shards`, `output`, `old_success`, `coverage`, `bisect`,
`bisect_output`, `validate_expected_success`,
`measure_project_runtimes`, `concurrency`, `base_dir`, `debug`, `clear`
> ```
## Test Plan
New Markdown tests.
## Summary
Support dataclasses with `order=True`:
```py
@dataclass(order=True)
class WithOrder:
x: int
WithOrder(1) < WithOrder(2) # no error
```
Also adds some additional tests to `dataclasses.md`.
ticket: #16651
## Test Plan
New Markdown tests
This PR adds **_very_** basic inference of generic typevars at call
sites. It does not bring in a full unification algorithm, and there are
a few TODOs in the test suite that are not discharged by this. But it
handles a good number of useful cases! And the PR does not add anything
that would go away with a more sophisticated constraint solver.
In short, we just look for typevars in the formal parameters, and assume
that the inferred type of the corresponding argument is what that
typevar should map to. If a typevar appears more than once, we union
together the corresponding argument types.
Cases we are not yet handling:
- We are not widening literals.
- We are not recursing into parameters that are themselves generic
aliases.
- We are not being very clever with parameters that are union types.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This is similar to https://github.com/astral-sh/ruff/pull/17095, it adds
assignability check for bound methods to callables.
## Test Plan
Add test cases to for assignability; specifically it uses gradual types
because otherwise it would just delegate to `is_subtype_of`.
## Summary
closes#16615
This PR includes:
- Introduces a new type: `Type::BoundSuper`
- Implements member lookup for `Type::BoundSuper`, resolving attributes
by traversing the MRO starting from the specified class
- Adds support for inferring appropriate arguments (`pivot_class` and
`owner`) for `super()` when it is used without arguments
When `super(..)` appears in code, it can be inferred into one of the
following:
- `Type::Unknown`: when a runtime error would occur (e.g. calling
`super()` out of method scope, or when parameter validation inside
`super` fails)
- `KnownClass::Super::to_instance()`: when the result is an *unbound
super object* or when a dynamic type is used as parameters (MRO
traversing is meaningless)
- `Type::BoundSuper`: the common case, representing a properly
constructed `super` instance that is ready for MRO traversal and
attribute resolution
### Terminology
Python defines the terms *bound super object* and *unbound super
object*.
An **unbound super object** is created when `super` is called with only
one argument (e.g.
`super(A)`). This object may later be bound via the `super.__get__`
method. However, this form is rarely used in practice.
A **bound super object** is created either by calling
`super(pivot_class, owner)` or by using the implicit form `super()`,
where both arguments are inferred from the context. This is the most
common usage.
### Follow-ups
- Add diagnostics for `super()` calls that would result in runtime
errors (marked as TODO)
- Add property tests for `Type::BoundSuper`
## Test Plan
- Added `mdtest/class/super.md`
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Until we optimize our full union/intersection representation to
efficiently handle large numbers of same-kind literal types "as a
block", set a fairly low limit on the size of unions of literals.
We will want to increase this limit once we've made the broader
efficiency improvement (tracked in
https://github.com/astral-sh/ruff/issues/17420).
## Test Plan
`cargo bench --bench red_knot`
## Summary
Special-case literal types in `UnionBuilder` to speed up building large
unions of literals.
This optimization is extremely effective at speeding up building even a
very large union (it improves the large-unions benchmark by 41x!). The
problem we can run into is that it is easy to then run into another
operation on the very large union (for instance, narrowing may add it to
an intersection, which then distributes it over the intersection) which
is still slow.
I think it is possible to avoid this by extending this optimized
"grouped" representation throughout not just `UnionBuilder`, but all of
our union and intersection representations. I have some work in this
direction, but rather than spending more time on it right now, I'd
rather just land this much, along with a limit on the size of these
unions (to avoid building really big unions quickly and then hitting
issues where they are used.)
## Test Plan
Existing tests and benchmarks.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Fixes incorrect negated type eq and ne assertions in
infer_binary_intersection_type_comparison
fixes#17360
## Test Plan
Remove and update some now incorrect tests
This reworks the assignability/subtyping relations a bit to handle
typevars better:
1. For the most part, types are not assignable to typevars, since
there's no guarantee what type the typevar will be specialized to.
2. An intersection is an exception, if it contains the typevar itself as
one of the positive elements. This should fall out from the other
clauses automatically, since a typevar is assignable to itself, and an
intersection is assignable to something if any positive element is
assignable to that something.
3. Constrained typevars are an exception, since they must be specialized
to _exactly_ one of the constraints, not to a _subtype_ of a constraint.
If a type is assignable to every constraint, then the type is also
assignable to the constrained typevar.
We already had a special case for (3), but the ordering of it relative
to the intersection clauses meant we weren't catching (2) correctly. To
fix this, we keep the special case for (3), but fall through to the
other match arms for non-constrained typevars and if the special case
isn't true for a constrained typevar.
Closes https://github.com/astral-sh/ruff/issues/17364
## Summary
Add very early support for dataclasses. This is mostly to make sure that
we do not emit false positives on dataclass construction, but it also
lies some foundations for future extensions.
This seems like a good initial step to merge to me, as it basically
removes all false positives on dataclass constructor calls. This allows
us to use the ecosystem checks for making sure we don't introduce new
false positives as we continue to work on dataclasses.
## Ecosystem analysis
I re-ran the mypy_primer evaluation of [the `__init__`
PR](https://github.com/astral-sh/ruff/pull/16512) locally with our
current mypy_primer version and project selection. It introduced 1597
new diagnostics. Filtering those by searching for `__init__` and
rejecting those that contain `invalid-argument-type` (those could not
possibly be solved by this PR) leaves 1281 diagnostics. The current
version of this PR removes 1171 diagnostics, which leaves 110
unaccounted for. I extracted the lint + file path for all of these
diagnostics and generated a diff (of diffs), to see which
`__init__`-diagnostics remain. I looked at a subset of these: There are
a lot of `SomeClass(*args)` calls where we don't understand the
unpacking yet (this is not even related to `__init__`). Some others are
related to `NamedTuple`, which we also don't support yet. And then there
are some errors related to `@attrs.define`-decorated classes, which
would probably require support for `dataclass_transform`, which I made
no attempt to include in this PR.
## Test Plan
New Markdown tests.
## Summary
* Partial #17238
* Flyby from discord discussion - `todo_type!` now statically checks for
no parens in the message to avoid issues between debug & release build
tests
## Test Plan
many mdtests are changing
## Summary
This PR moves all the relation methods from `CallableType` to
`Signature`.
The main reason for this is that `Signature` is going to be the common
denominator between normal and overloaded callables and the core logic
to check a certain relationship is going to just require the information
that would exists on `Signature`. For example, to check whether an
overloaded callable is a subtype of a normal callable, we need to check
whether _every_ overloaded signature is a subtype of the normal
callable's signature. This "every" logic would become part of the
`CallableType` and the core logic of checking the subtyping would exists
on `Signature`.
## Summary
Allows us to establish that two literals do not have a subtype
relationship with each other, without having to fallback to a typeshed
Instance type, which is comparatively slow.
Improves the performance of the many-string-literals union benchmark by
5x.
## Test Plan
`cargo test -p red_knot_python_semantic` and `cargo bench --bench
red_knot`.
This commit shuffles the reporting API around a little bit such that a
range is required, up front, when reporting a lint diagnostic. This in
turn enables us to make suppression checking eager.
In order to avoid callers needing to provide the range twice, we create
a primary annotation *without* a message inside the `Diagnostic`
encapsulated by the guard. We do this instead of requiring the message
up front because we're concerned about API complexity and the effort
involved in creating the message.
In order to provide a means of attaching a message to the primary
annotation, we expose a convenience API on `LintDiagnosticGuard` for
setting the message. This isn't generally possible for a `Diagnostic`,
but since a `LintDiagnosticGuard` knows how the `Diagnostic` was
constructed, we can offer this API correctly.
It strikes me that it might be easy to forget to attach a primary
annotation message, btu I think this the "least" bad failure mode. And
in particular, it should be somewhat obvious that it's missing once one
adds a snapshot test for how the diagnostic renders.
Otherwise, this API gives us the ability to eagerly check whether a
diagnostic should be reported with nearly minimal information. It also
shouldn't have any footguns since it guarantees that the primary
annotation is tied to the file in the typing context. And it keeps
things pretty simple: callers only need to provide what is actually
strictly necessary to make a diagnostic.
This is the payoff from removing a bit of indirection. The types still
exist, but now callers don't need to do builder -> reporter ->
diagnostic. They can just conceptually think of it as builder ->
diagnostic.
We're going to make the guards deref to `Diagnostic` in order to remove
a layer of indirection in the reporter API. (Well, technically the layer
is not removed since the types still exist, but in actual _usage_ the
layer will be removed. We'll see how it shakes out in the next commit.)
## Summary
Infer precise Boolean literal types for `str.startswith` calls where the
instance and the prefix are both string literals. This allows us to
understand `sys.platform.startswith(…)` branches.
## Test Plan
New Markdown tests
## Summary
For silencing `invalid-type-form` diagnostics in unreachable code, we
use the same approach that we use before and check the reachability that
we already record.
For silencing `invalid-bases`, we simply check if the type of the base
is `Never`. If so, we silence the diagnostic with the argument that the
class construction would never happen.
## Test Plan
Updated Markdown tests.
## Summary
Similar to what we did for `unresolved-reference` and
`unresolved-attribute`, we now also silence `unresolved-import`
diagnostics if the corresponding `import` statement is unreachable.
This addresses the (already closed) issue #17049.
## Test Plan
Adapted Markdown tests.
This finally completes the deletion of all old diagnostic types.
We do this by migrating the second (and last) use of secondary
diagnostic messages: to highlight the return type of a function
definition when its return value is inconsistent with the type.
Like the last diagnostic, we do actually change the message here a bit.
We don't need a sub-diagnostic here, and we can instead just add a
secondary annotation to highlight the return type.
This is the first use of the new `lint()` reporter.
I somewhat skipped a step here and also modified the actual diagnostic
message itself. The snapshots should tell the story.
We couldn't do this before because we had no way of differentiating
between "message for the diagnostic as a whole" and "message for a
specific code annotation." Now we can, so we can write more precise
messages based on the assumption that users are also seeing the code
snippet.
The downside here is that the actual message text can become quite vague
in the absence of the code snippet. This occurs, for example, with
concise diagnostic formatting. It's unclear if we should do anything
about it. I don't really see a way to make it better that doesn't
involve creating diagnostics with messages for each mode, which I think
would be a major PITA.
The upside is that this code gets a bit simpler, and we very
specifically avoid doing extra work if this specific lint is disabled.
This required a bit of surgery in the diagnostic matching and more
faffing about using a "concise" message from a diagnostic instead of
only printing the "primary" message.
In the new diagnostic data model, we really should have a main
diagnostic message *and* a primary span (with an optional message
attached to it) for every diagnostic.
In this commit, I try to make this true for the "revealed type"
diagnostic. Instead of the annotation saying both "revealed type is"
and also the revealed type itself, the annotation is now just the
revealed type and the main diagnostic message is "Revealed type."
I expect this may be controversial. I'm open to doing something
different. I tried to avoid redundancy, but maybe this is a special case
where we want the redundancy. I'm honestly not sure. I do *like* how it
looks with this commit, but I'm not working with Red Knot's type
checking daily, so my opinion doesn't count for much.
This did also require some tweaking to concise diagnostic formatting in
order to preserve the essential information.
This commit doesn't update every relevant snapshot. Just a few. I split
the rest out into the next commit.
... and replace it with use of `report()`.
Interestingly, this is the only instance of `report_diagnostic` used
directly, and thus anticipated to be the only instance of using
`report()`. If this ends up being a true single use method, we could
make it less generic and tailored specifically to "reveal type."
Two other things to note:
I left the "primary message" as empty. This avoids changing snapshots.
I address this in a subsequent commit.
The creation of a diagnostic here is a bit verbose/annoying. Certainly
more so than it was. This is somewhat expected since our diagnostic
model is more expressive and because we don't have a proc macro. I
avoided creating helpers for this case since there's only one use of
`report()`. But I expect to create helpers for the `lint()` case.
This is a surgical change that adds new `report()` and `lint()`
APIs to `InferContext`. These are intended to replace the existing
`report_*` APIs.
The comments should explain what these reporters are meant to do. For
the most part, this is "just" shuffling some code around. The actual
logic for determining whether a lint *should* be reported or not remains
unchanged and we don't make any changes to how a `Diagnostic` is
actually constructed (yet).
I initially tried to just use `LintReporter` and `DiagnosticReporter`
without the builder types, since I perceive the builder types to be an
annoying additional layer. But I found it also exceedingly annoying to
have to construct and provide the diagnostic message before you even
know if you are going to build the diagnostic. I also felt like this
could result in potentially unnecessary and costly querying in some
cases, although this is somewhat hand wavy. So I overall felt like the
builder route was the way to go. If the builders end up being super
annoying, we can probably add convenience APIs for common patterns to
paper over them.
## Summary
Basically just repeat the same thing that we did for
`unresolved-reference`, but now for attribute expressions.
We now also handle the case where the unresolved attribute (or the
unresolved reference) diagnostic originates from a stringified type
annotation.
And I made the evaluation of reachability constraints lazy (will only be
evaluated right before we are about to emit a diagnostic).
## Test Plan
New Markdown tests for stringified annotations.
I merged #17149 without checking the ecosystem results, and it still
caused a cycle panic in pybind11. Reverting for now until I fix that, so
we don't lose the ecosystem signal on other PRs.
This causes spurious query cycles.
This PR also includes an update to Salsa, which gives us db events on
cycle iteration, so we can write tests asserting the absence of a cycle.
## Summary
Track the reachability of nested scopes within their parent scopes. We
use this as an additional requirement for emitting
`unresolved-reference` diagnostics (and in the future,
`unresolved-attribute` and `unresolved-import`). This means that we only
emit `unresolved-reference` for a given use of a symbol if the use
itself is reachable (within its own scope), *and if the scope itself is
reachable*. For example, no diagnostic should be emitted for the use of
`x` here:
```py
if False:
x = 1
def f():
print(x) # this use of `x` is reachable inside the `f` scope,
# but the whole `f` scope is not reachable.
```
There are probably more fine-grained ways of solving this problem, but
they require a more sophisticated understanding of nested scopes (see
#15777, in particular
https://github.com/astral-sh/ruff/issues/15777#issuecomment-2788950267).
But it doesn't seem completely unreasonable to silence *this specific
kind of error* in unreachable scopes.
## Test Plan
Observed changes in reachability tests and ecosystem.
## Summary
Update Salsa to pull in https://github.com/salsa-rs/salsa/pull/788 which
fixes the, by now, famous *access to field whilst the value is being
initialized*.
This PR also re-enables all tests that previously triggered the panic.
## Test Plan
`cargo test`
## Summary
There is a new official URL for the typing documentation:
https://typing.python.org/
Change all https://typing.readthedocs.io/ links to use the new sub
domain, which is slightly shorter and looks more official.
## Test Plan
Tested to see if each and every new URL is accessible. I noticed that
some links go to https://typing.python.org/en/latest/source/stubs.html
which seems to be outdated, but that is a separate issue. The same page
shows up for the old URL.
This PR lets you explicitly specialize a generic class using a subscript
expression. It introduces three new Rust types for representing classes:
- `NonGenericClass`
- `GenericClass` (not specialized)
- `GenericAlias` (specialized)
and two enum wrappers:
- `ClassType` (a non-generic class or generic alias, represents a class
_type_ at runtime)
- `ClassLiteralType` (a non-generic class or generic class, represents a
class body in the AST)
We also add internal support for specializing callables, in particular
function literals. (That is, the internal `Type` representation now
attaches an optional specialization to a function literal.) This is used
in this PR for the methods of a generic class, but should also give us
most of what we need for specializing generic _functions_ (which this PR
does not yet tackle).
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
Brent Westbrook
Alex Waygood
David Peter
Douglas Creager
Dhruv Manilawala
cake-monotone
Carl Meyer
Matthew Mckee
Mike Perlov
Shunsuke Shibayama
Andrew Gallant
Micha Reiser