mirror of https://github.com/astral-sh/ruff
5 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Dhruv Manilawala
|
44ad201262
|
[red-knot] Add support for overloaded functions (#17366)
## Summary Part of #15383, this PR adds support for overloaded callables. Typing spec: https://typing.python.org/en/latest/spec/overload.html Specifically, it does the following: 1. Update the `FunctionType::signature` method to return signatures from a possibly overloaded callable using a new `FunctionSignature` enum 2. Update `CallableType` to accommodate overloaded callable by updating the inner type to `Box<[Signature]>` 3. Update the relation methods on `CallableType` with logic specific to overloads 4. Update the display of callable type to display a list of signatures enclosed by parenthesis 5. Update `CallableTypeOf` special form to recognize overloaded callable 6. Update subtyping, assignability and fully static check to account for callables (equivalence is planned to be done as a follow-up) For (2), it is required to be done in this PR because otherwise I'd need to add some workaround for `into_callable_type` and I though it would be best to include it in here. For (2), another possible design would be convert `CallableType` in an enum with two variants `CallableType::Single` and `CallableType::Overload` but I decided to go with `Box<[Signature]>` for now to (a) mirror it to be equivalent to `overload` field on `CallableSignature` and (b) to avoid any refactor in this PR. This could be done in a follow-up to better split the two kind of callables. ### Design There were two main candidates on how to represent the overloaded definition: 1. To include it in the existing infrastructure which is what this PR is doing by recognizing all the signatures within the `FunctionType::signature` method 2. To create a new `Overload` type variant <details><summary>For context, this is what I had in mind with the new type variant:</summary> <p> ```rs pub enum Type { FunctionLiteral(FunctionType), Overload(OverloadType), BoundMethod(BoundMethodType), ... } pub struct OverloadType { // FunctionLiteral or BoundMethod overloads: Box<[Type]>, // FunctionLiteral or BoundMethod implementation: Option<Type> } pub struct BoundMethodType { kind: BoundMethodKind, self_instance: Type, } pub enum BoundMethodKind { Function(FunctionType), Overload(OverloadType), } ``` </p> </details> The main reasons to choose (1) are the simplicity in the implementation, reusing the existing infrastructure, avoiding any complications that the new type variant has specifically around the different variants between function and methods which would require the overload type to use `Type` instead. ### Implementation The core logic is how to collect all the overloaded functions. The way this is done in this PR is by recording a **use** on the `Identifier` node that represents the function name in the use-def map. This is then used to fetch the previous symbol using the same name. This way the signatures are going to be propagated from top to bottom (from first overload to the final overload or the implementation) with each function / method. For example: ```py from typing import overload @overload def foo(x: int) -> int: ... @overload def foo(x: str) -> str: ... def foo(x: int | str) -> int | str: return x ``` Here, each definition of `foo` knows about all the signatures that comes before itself. So, the first overload would only see itself, the second would see the first and itself and so on until the implementation or the final overload. This approach required some updates specifically recognizing `Identifier` node to record the function use because it doesn't use `ExprName`. ## Test Plan Update existing test cases which were limited by the overload support and add test cases for the following cases: * Valid overloads as functions, methods, generics, version specific * Invalid overloads as stated in https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions (implementation will be done in a follow-up) * Various relation: fully static, subtyping, and assignability (others in a follow-up) ## Ecosystem changes _WIP_ After going through the ecosystem changes (there are a lot!), here's what I've found: We need assignability check between a callable type and a class literal because a lot of builtins are defined as classes in typeshed whose constructor method is overloaded e.g., `map`, `sorted`, `list.sort`, `max`, `min` with the `key` parameter, `collections.abc.defaultdict`, etc. (https://github.com/astral-sh/ruff/issues/17343). This makes up most of the ecosystem diff **roughly 70 diagnostics**. For example: ```py from collections import defaultdict # red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload] defaultdict(int) # red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload] defaultdict(list) class Foo: def __init__(self, x: int): self.x = x # red-knot: No overload of function `__new__` matches arguments [lint:no-matching-overload] map(Foo, ["a", "b", "c"]) ``` Duplicate diagnostics in unpacking (https://github.com/astral-sh/ruff/issues/16514) has **~16 diagnostics**. Support for the `callable` builtin which requires `TypeIs` support. This is **5 diagnostics**. For example: ```py from typing import Any def _(x: Any | None) -> None: if callable(x): # red-knot: `Any | None` # Pyright: `(...) -> object` # mypy: `Any` # pyrefly: `(...) -> object` reveal_type(x) ``` Narrowing on `assert` which has **11 diagnostics**. This is being worked on in https://github.com/astral-sh/ruff/pull/17345. For example: ```py import re match = re.search("", "") assert match match.group() # error: [possibly-unbound-attribute] ``` Others: * `Self`: 2 * Type aliases: 6 * Generics: 3 * Protocols: 13 * Unpacking in comprehension: 1 (https://github.com/astral-sh/ruff/pull/17396) ## Performance Refer to https://github.com/astral-sh/ruff/pull/17366#issuecomment-2814053046. |
|
Brent Westbrook
|
9c47b6dbb0
|
[red-knot] Detect version-related syntax errors (#16379)
## Summary This PR extends version-related syntax error detection to red-knot. The main changes here are: 1. Passing `ParseOptions` specifying a `PythonVersion` to parser calls 2. Adding a `python_version` method to the `Db` trait to make this possible 3. Converting `UnsupportedSyntaxError`s to `Diagnostic`s 4. Updating existing mdtests to avoid unrelated syntax errors My initial draft of (1) and (2) in #16090 instead tried passing a `PythonVersion` down to every parser call, but @MichaReiser suggested the `Db` approach instead [here](https://github.com/astral-sh/ruff/pull/16090#discussion_r1969198407), and I think it turned out much nicer. All of the new `python_version` methods look like this: ```rust fn python_version(&self) -> ruff_python_ast::PythonVersion { Program::get(self).python_version(self) } ``` with the exception of the `TestDb` in `ruff_db`, which hard-codes `PythonVersion::latest()`. ## Test Plan Existing mdtests, plus a new mdtest to see at least one of the new diagnostics. |
|
David Peter
|
b32407b6f3
|
[red-knot] Dataclasses: synthesize `__init__` with proper signature (#17428)
## Summary This changeset allows us to generate the signature of synthesized `__init__` functions in dataclasses by analyzing the fields on the class (and its superclasses). There are certain things that I have not yet attempted to model in this PR, like `kw_only`, [`dataclasses.KW_ONLY`](https://docs.python.org/3/library/dataclasses.html#dataclasses.KW_ONLY) or functionality around [`dataclasses.field`](https://docs.python.org/3/library/dataclasses.html#dataclasses.field). ticket: https://github.com/astral-sh/ruff/issues/16651 ## Ecosystem analysis These two seem to depend on missing features in generics (see [relevant code here]( |
|
|
David Peter
|
b4de245a5a
|
[red-knot] Dataclasses: support `order=True` (#17406)
## Summary
Support dataclasses with `order=True`:
```py
@dataclass(order=True)
class WithOrder:
x: int
WithOrder(1) < WithOrder(2) # no error
```
Also adds some additional tests to `dataclasses.md`.
ticket: #16651
## Test Plan
New Markdown tests
|
|
|
David Peter
|
03adae80dc
|
[red-knot] Initial support for `dataclass`es (#17353)
## Summary Add very early support for dataclasses. This is mostly to make sure that we do not emit false positives on dataclass construction, but it also lies some foundations for future extensions. This seems like a good initial step to merge to me, as it basically removes all false positives on dataclass constructor calls. This allows us to use the ecosystem checks for making sure we don't introduce new false positives as we continue to work on dataclasses. ## Ecosystem analysis I re-ran the mypy_primer evaluation of [the `__init__` PR](https://github.com/astral-sh/ruff/pull/16512) locally with our current mypy_primer version and project selection. It introduced 1597 new diagnostics. Filtering those by searching for `__init__` and rejecting those that contain `invalid-argument-type` (those could not possibly be solved by this PR) leaves 1281 diagnostics. The current version of this PR removes 1171 diagnostics, which leaves 110 unaccounted for. I extracted the lint + file path for all of these diagnostics and generated a diff (of diffs), to see which `__init__`-diagnostics remain. I looked at a subset of these: There are a lot of `SomeClass(*args)` calls where we don't understand the unpacking yet (this is not even related to `__init__`). Some others are related to `NamedTuple`, which we also don't support yet. And then there are some errors related to `@attrs.define`-decorated classes, which would probably require support for `dataclass_transform`, which I made no attempt to include in this PR. ## Test Plan New Markdown tests. |