* origin/main:
Fluent formatting of method chains (#21369)
[ty] Avoid stack overflow when calculating inferable typevars (#21971)
[ty] Add "qualify ..." code fix for undefined references (#21968)
[ty] Use jemalloc on linux (#21975)
Update MSRV to 1.90 (#21987)
[ty] Improve check enforcing that an overloaded function must have an implementation (#21978)
Update actions/checkout digest to 8e8c483 (#21982)
[ty] Use `ParamSpec` without the attr for inferable check (#21934)
[ty] Emit diagnostic when a type variable with a default is followed by one without a default (#21787)
* dcreager/source-order-constraints: (30 commits)
clippy
fix test expectations (again)
include source_order in display_graph output
place bounds/constraints first
don't always bump
only fold once
document display source_order
more comments
remove now-unused items
fix test expectation
use source order in specialize_constrained too
document overall approach
more comment
reuse self source_order
sort specialize_constrained by source_order
lots of renaming
remove source_order_for
simpler source_order_for
doc
restore TODOs
...
When we calculate which typevars are inferable in a generic context, the
result might include more than the typevars bound by the generic
context. The canonical example is a generic method of a generic class:
```py
class C[A]:
def method[T](self, t: T): ...
```
Here, the inferable typevar set of `method` contains `Self` and `T`, as
you'd expect. (Those are the typevars bound by the method.) But it also
contains `A@C`, since the implicit `Self` typevar is defined as `Self:
C[A]`. That means when we call `method`, we need to mark `A@C` as
inferable, so that we can determine the correct mapping for `A@C` at the
call site.
Fixes https://github.com/astral-sh/ty/issues/1874
* origin/main: (22 commits)
[ty] Allow gradual lower/upper bounds in a constraint set (#21957)
[ty] disallow explicit specialization of type variables themselves (#21938)
[ty] Improve diagnostics for unsupported binary operations and unsupported augmented assignments (#21947)
[ty] update implicit root docs (#21955)
[ty] Enable even more goto-definition on inlay hints (#21950)
Document known lambda formatting deviations from Black (#21954)
[ty] fix hover type on named expression target (#21952)
Bump benchmark dependencies (#21951)
Keep lambda parameters on one line and parenthesize the body if it expands (#21385)
[ty] Improve resolution of absolute imports in tests (#21817)
[ty] Support `__all__ += submodule.__all__`
[ty] Change frequency of invalid `__all__` debug message
[ty] Add `KnownUnion::to_type()` (#21948)
[ty] Classify `cls` as class parameter (#21944)
[ty] Stabilize rename (#21940)
[ty] Ignore `__all__` for document and workspace symbol requests
[ty] Attach db to background request handler task (#21941)
[ty] Fix outdated version in publish diagnostics after `didChange` (#21943)
[ty] avoid fixpoint unioning of types containing current-cycle Divergent (#21910)
[ty] improve bad specialization results & error messages (#21840)
...
We now allow the lower and upper bounds of a constraint to be gradual.
Before, we would take the top/bottom materializations of the bounds.
This required us to pass in whether the constraint was intended for a
subtyping check or an assignability check, since that would control
whether we took the "restrictive" or "permissive" materializations,
respectively.
Unfortunately, doing so means that we lost information about whether the
original query involves a non-fully-static type. This would cause us to
create specializations like `T = object` for the constraint `T ≤ Any`,
when it would be nicer to carry through the gradual type and produce `T
= Any`.
We're not currently using constraint sets for subtyping checks, nor are
we going to in the very near future. So for now, we're going to assume
that constraint sets are always used for assignability checks, and allow
the lower/upper bounds to not be fully static. Once we get to the point
where we need to use constraint sets for subtyping checks, we will
consider how best to record this information in constraints.
## Summary
This PR makes explicit specialization of a type variable itself an
error, and the result of the specialization is `Unknown`.
The change also fixes https://github.com/astral-sh/ty/issues/1794.
## Test Plan
mdtests updated
new corpus test
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR takes the improvements we made to unsupported-comparison
diagnostics in https://github.com/astral-sh/ruff/pull/21737, and extends
them to other `unsupported-operator` diagnostics.
## Test Plan
Mdtests and snapshots
## Summary
Working on py-fuzzer recently (AKA, a Python project!) reminded me how
cool our "inlay hint goto-definition feature" is. So this PR adds a
bunch more of that!
I also made a couple of other minor changes to type display. For
example, in the playground, this snippet:
```py
def f(): ...
reveal_type(f.__get__)
```
currently leads to this diagnostic:
```
Revealed type: `<method-wrapper `__get__` of `f`>` (revealed-type) [Ln 2, Col 13]
```
But the fact that we have backticks both around the type display and
inside the type display isn't _great_ there. This PR changes it to
```
Revealed type: `<method-wrapper '__get__' of function 'f'>` (revealed-type) [Ln 2, Col 13]
```
which avoids the nested-backticks issue in diagnostics, and is more
similar to our display for various other `Type` variants such as
class-literal types (`<class 'Foo'>`, etc., not ``<class `Foo`>``).
## Test Plan
inlay snapshots added; mdtests updated
## Summary
This PR includes the following changes:
* When attempting to specialize a non-generic type (or a type that is
already specialized), the result is `Unknown`. Also, the error message
is improved.
* When an implicit type alias is incorrectly specialized, the result is
`Unknown`. Also, the error message is improved.
* When only some of the type alias bounds and constraints are not
satisfied, not all substitutions are `Unknown`.
* Double specialization is prohibited. e.g. `G[int][int]`
Furthermore, after applying this PR, the fuzzing tests for seeds 1052
and 4419, which panic in main, now pass.
This is because the false recursions on type variables have been
removed.
```python
# name_2[0] => Unknown
class name_1[name_2: name_2[0]]:
def name_4(name_3: name_2, /):
if name_3:
pass
# (name_5 if unique_name_0 else name_1)[0] => Unknown
def name_4[name_5: (name_5 if unique_name_0 else name_1)[0], **name_1](): ...
```
## Test Plan
New corpus test
mdtest files updated
* origin/main: (36 commits)
[ty] Defer all parameter and return type annotations (#21906)
[ty] Fix workspace symbols to return members too (#21926)
Document range suppressions, reorganize suppression docs (#21884)
Ignore ruff:isort like ruff:noqa in new suppressions (#21922)
[ty] Handle `Definition`s in `SemanticModel::scope` (#21919)
[ty] Attach salsa db when running ide tests for easier debugging (#21917)
[ty] Don't show hover for expressions with no inferred type (#21924)
[ty] avoid unions of generic aliases of the same class in fixpoint (#21909)
[ty] Squash false positive logs for failing to find `builtins` as a real module
[ty] Uniformly use "not supported" in diagnostics (#21916)
[ty] Reduce size of ty-ide snapshots (#21915)
[ty] Adjust scope completions to use all reachable symbols
[ty] Rename `all_members_of_scope` to `all_end_of_scope_members`
[ty] Remove `all_` prefix from some routines on UseDefMap
Enable `--document-private-items` for `ruff_python_formatter` (#21903)
Remove `BackwardsTokenizer` based `parenthesized_range` references in `ruff_linter` (#21836)
[ty] Revert "Do not infer types for invalid binary expressions in annotations" (#21914)
Skip over trivia tokens after re-lexing (#21895)
[ty] Avoid inferring types for invalid binary expressions in string annotations (#21911)
[ty] Improve overload call resolution tracing (#21913)
...
As described in astral-sh/ty#1729, we previously had a salsa cycle when
inferring the signature of many function definitions.
The most obvious case happened when (a) the function was decorated, (b)
it had no PEP-695 type params, and (c) annotations were not always
deferred (e.g. in a stub file). We currently evaluate and apply function
decorators eagerly, as part of `infer_function_definition`. Applying a
decorator requires knowing the signature of the function being
decorated. There were two places where signature construction called
`infer_definition_types` cyclically.
The simpler case was that we were looking up the generic context and
decorator list of the function to determine whether it has an implicit
`self` parameter. Before, we used `infer_definition_types` to determine
that information. But since we're in the middle of signature
construction for the function, we can just thread the information
through directly.
The harder case is that signature construction requires knowing the
inferred parameter and return type annotations. When (b) and (c) hold,
those type annotations are inferred in `infer_function_definition`! (In
theory, we've already finished that by the time we start applying
decorators, but signature construction doesn't know that.)
If annotations are deferred, the params/return annotations are inferred
in `infer_deferred_types`; if there are PEP-695 type params, they're
inferred in `infer_function_type_params`. Both of those are different
salsa queries, and don't induce this cycle.
So the quick fix here is to always defer inference of the function
params/return, so that they are always inferred under a different salsa
query.
A more principled fix would be to apply decorators lazily, just like we
construct signatures lazily. But that is a more invasive fix.
Fixesastral-sh/ty#1729
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
Partially addresses https://github.com/astral-sh/ty/issues/1732
Fixes https://github.com/astral-sh/ty/issues/1800
## Summary
At each fixpoint iteration, we union the "previous" and "current"
iteration types, to ensure that the type can only widen at each
iteration. This prevents oscillation and ensures convergence.
But some unions triggered by this behavior (in particular, unions of
differently-specialized generic-aliases of the same class) never
simplify, and cause spurious errors. Since we haven't seen examples of
oscillating types involving class-literal or generic-alias types, just
don't union those.
There may be more thorough/principled ways to avoid undesirable unions
in fixpoint iteration, but this narrow change seems like it results in
strict improvement.
## Test Plan
Removes two false positive `unsupported-class-base` in mdtests, and
several in the ecosystem, without causing other regression.
## Summary
Respect typevar bounds and constraints when matching against a union.
For example:
```py
def accepts_t_or_int[T_str: str](x: T_str | int) -> T_str:
raise NotImplementedError
reveal_type(accepts_t_or_int("a")) # ok, reveals `Literal["a"]`
reveal_type(accepts_t_or_int(1)) # ok, reveals `Unknown`
class Unrelated: ...
# error: [invalid-argument-type] "Argument type `Unrelated` does not
# satisfy upper bound `str` of type variable `T_str`"
accepts_t_or_int(Unrelated())
```
Previously, the last call succeed without any errors. Worse than that,
we also incorrectly solved `T_str = Unrelated`, which often lead to
downstream errors.
closes https://github.com/astral-sh/ty/issues/1837
## Ecosystem impact
Looks good!
* Lots of removed false positives, often because we previously selected
a wrong overload for a generic function (because we didn't respect the
typevar bound in an earlier overload).
* We now understand calls to functions accepting an argument of type
`GenericPath: TypeAlias = AnyStr | PathLike[AnyStr]`. Previously, we
would incorrectly match a `Path` argument against the `AnyStr` typevar
(violating its constraints), but now we match against `PathLike`.
## Performance
Another regression on `colour`. This package uses `numpy` heavily. And
`numpy` is the codebase that originally lead me to this bug. The fix
here allows us to infer more precise `np.array` types in some cases, so
it's reasonable that we just need to perform more work.
The fix here also requires us to look at more union elements when we
would previously short-circuit incorrectly, so some more work needs to
be done in the solver.
## Test Plan
New Markdown tests
* origin/main: (33 commits)
[ty] Simplify union lower bounds and intersection upper bounds in constraint sets (#21871)
[ty] Collapse `never` paths in constraint set BDDs (#21880)
Fix leading comment formatting for lambdas with multiple parameters (#21879)
[ty] Type inference for `@asynccontextmanager` (#21876)
Fix comment placement in lambda parameters (#21868)
[`pylint`] Detect subclasses of builtin exceptions (`PLW0133`) (#21382)
Fix stack overflow with recursive generic protocols (depth limit) (#21858)
New diagnostics for unused range suppressions (#21783)
[ty] Use default settings in completion tests
[ty] Infer type variables within generic unions (#21862)
[ty] Fix overload filtering to prefer more "precise" match (#21859)
[ty] Stabilize auto-import
[ty] Fix reveal-type E2E test (#21865)
[ty] Use concise message for LSP clients not supporting related diagnostic information (#21850)
Include more details in Tokens 'offset is inside token' panic message (#21860)
apply range suppressions to filter diagnostics (#21623)
[ty] followup: add-import action for `reveal_type` too (#21668)
[ty] Enrich function argument auto-complete suggestions with annotated types
[ty] Add autocomplete suggestions for function arguments
[`flake8-bugbear`] Accept immutable slice default arguments (`B008`) (#21823)
...
## Summary
This PR allows our generics solver to find a solution for `T` in cases
like the following:
```py
def extract_t[T](x: P[T] | Q[T]) -> T:
raise NotImplementedError
reveal_type(extract_t(P[int]())) # revealed: int
reveal_type(extract_t(Q[str]())) # revealed: str
```
closes https://github.com/astral-sh/ty/issues/1772
closes https://github.com/astral-sh/ty/issues/1314
## Ecosystem
The impact here looks very good!
It took me a long time to figure this out, but the new diagnostics on
bokeh are actually true positives. I should have tested with another
type-checker immediately, I guess. All other type checkers also emit
errors on these `__init__` calls. MRE
[here](https://play.ty.dev/5c19d260-65e2-4f70-a75e-1a25780843a2) (no
error on main, diagnostic on this branch)
A lot of false positives on home-assistant go away for calls to
functions like
[`async_listen`](180053fe98/homeassistant/core.py (L1581-L1587))
which take a `event_type: EventType[_DataT] | str` parameter. We can now
solve for `_DataT` here, which was previously falling back to its
default value, and then caused problems because it was used as an
argument to an invariant generic class.
## Test Plan
New Markdown tests
* origin/main:
[ty] Allow `tuple[Any, ...]` to assign to `tuple[int, *tuple[int, ...]]` (#21803)
[ty] Support renaming import aliases (#21792)
[ty] Add redeclaration LSP tests (#21812)
[ty] more detailed description of "Size limit on unions of literals" in mdtest (#21804)
[ty] Complete support for `ParamSpec` (#21445)
[ty] Update benchmark dependencies (#21815)
## Summary
Closes: https://github.com/astral-sh/ty/issues/157
This PR adds support for the following capabilities involving a
`ParamSpec` type variable:
- Representing `P.args` and `P.kwargs` in the type system
- Matching against a callable containing `P` to create a type mapping
- Specializing `P` against the stored parameters
The value of a `ParamSpec` type variable is being represented using
`CallableType` with a `CallableTypeKind::ParamSpecValue` variant. This
`CallableTypeKind` is expanded from the existing `is_function_like`
boolean flag. An `enum` is used as these variants are mutually
exclusive.
For context, an initial iteration made an attempt to expand the
`Specialization` to use `TypeOrParameters` enum that represents that a
type variable can specialize into either a `Type` or `Parameters` but
that increased the complexity of the code as all downstream usages would
need to handle both the variants appropriately. Additionally, we'd have
also need to establish an invariant that a regular type variable always
maps to a `Type` while a paramspec type variable always maps to a
`Parameters`.
I've intentionally left out checking and raising diagnostics when the
`ParamSpec` type variable and it's components are not being used
correctly to avoid scope increase and it can easily be done as a
follow-up. This would also include the scoping rules which I don't think
a regular type variable implements either.
## Test Plan
Add new mdtest cases and update existing test cases.
Ran this branch on pyx, no new diagnostics.
### Ecosystem analysis
There's a case where in an annotated assignment like:
```py
type CustomType[P] = Callable[...]
def value[**P](...): ...
def another[**P](...):
target: CustomType[P] = value
```
The type of `value` is a callable and it has a paramspec that's bound to
`value`, `CustomType` is a type alias that's a callable and `P` that's
used in it's specialization is bound to `another`. Now, ty infers the
type of `target` same as `value` and does not use the declared type
`CustomType[P]`. [This is the
assignment](0980b9d9ab/src/async_utils/gen_transform.py (L108))
that I'm referring to which then leads to error in downstream usage.
Pyright and mypy does seem to use the declared type.
There are multiple diagnostics in `dd-trace-py` that requires support
for `cls`.
I'm seeing `Divergent` type for an example like which ~~I'm not sure
why, I'll look into it tomorrow~~ is because of a cycle as mentioned in
https://github.com/astral-sh/ty/issues/1729#issuecomment-3612279974:
```py
from typing import Callable
def decorator[**P](c: Callable[P, int]) -> Callable[P, str]: ...
@decorator
def func(a: int) -> int: ...
# ((a: int) -> str) | ((a: Divergent) -> str)
reveal_type(func)
```
I ~~need to look into why are the parameters not being specialized
through multiple decorators in the following code~~ think this is also
because of the cycle mentioned in
https://github.com/astral-sh/ty/issues/1729#issuecomment-3612279974 and
the fact that we don't support `staticmethod` properly:
```py
from contextlib import contextmanager
class Foo:
@staticmethod
@contextmanager
def method(x: int):
yield
foo = Foo()
# ty: Revealed type: `() -> _GeneratorContextManager[Unknown, None, None]` [revealed-type]
reveal_type(foo.method)
```
There's some issue related to `Protocol` that are generic over a
`ParamSpec` in `starlette` which might be related to
https://github.com/astral-sh/ty/issues/1635 but I'm not sure. Here's a
minimal example to reproduce:
<details><summary>Code snippet:</summary>
<p>
```py
from collections.abc import Awaitable, Callable, MutableMapping
from typing import Any, Callable, ParamSpec, Protocol
P = ParamSpec("P")
Scope = MutableMapping[str, Any]
Message = MutableMapping[str, Any]
Receive = Callable[[], Awaitable[Message]]
Send = Callable[[Message], Awaitable[None]]
ASGIApp = Callable[[Scope, Receive, Send], Awaitable[None]]
_Scope = Any
_Receive = Callable[[], Awaitable[Any]]
_Send = Callable[[Any], Awaitable[None]]
# Since `starlette.types.ASGIApp` type differs from `ASGIApplication` from `asgiref`
# we need to define a more permissive version of ASGIApp that doesn't cause type errors.
_ASGIApp = Callable[[_Scope, _Receive, _Send], Awaitable[None]]
class _MiddlewareFactory(Protocol[P]):
def __call__(
self, app: _ASGIApp, *args: P.args, **kwargs: P.kwargs
) -> _ASGIApp: ...
class Middleware:
def __init__(
self, factory: _MiddlewareFactory[P], *args: P.args, **kwargs: P.kwargs
) -> None:
self.factory = factory
self.args = args
self.kwargs = kwargs
class ServerErrorMiddleware:
def __init__(
self,
app: ASGIApp,
value: int | None = None,
flag: bool = False,
) -> None:
self.app = app
self.value = value
self.flag = flag
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: ...
# ty: Argument to bound method `__init__` is incorrect: Expected `_MiddlewareFactory[(...)]`, found `<class 'ServerErrorMiddleware'>` [invalid-argument-type]
Middleware(ServerErrorMiddleware, value=500, flag=True)
```
</p>
</details>
### Conformance analysis
> ```diff
> -constructors_callable.py:36:13: info[revealed-type] Revealed type:
`(...) -> Unknown`
> +constructors_callable.py:36:13: info[revealed-type] Revealed type:
`(x: int) -> Unknown`
> ```
Requires return type inference i.e.,
https://github.com/astral-sh/ruff/pull/21551
> ```diff
> +constructors_callable.py:194:16: error[invalid-argument-type]
Argument is incorrect: Expected `list[T@__init__]`, found `list[Unknown
| str]`
> +constructors_callable.py:194:22: error[invalid-argument-type]
Argument is incorrect: Expected `list[T@__init__]`, found `list[Unknown
| str]`
> +constructors_callable.py:195:4: error[invalid-argument-type] Argument
is incorrect: Expected `list[T@__init__]`, found `list[Unknown | int]`
> +constructors_callable.py:195:9: error[invalid-argument-type] Argument
is incorrect: Expected `list[T@__init__]`, found `list[Unknown | str]`
> ```
I might need to look into why this is happening...
> ```diff
> +generics_defaults.py:79:1: error[type-assertion-failure] Type
`type[Class_ParamSpec[(str, int, /)]]` does not match asserted type
`<class 'Class_ParamSpec'>`
> ```
which is on the following code
```py
DefaultP = ParamSpec("DefaultP", default=[str, int])
class Class_ParamSpec(Generic[DefaultP]): ...
assert_type(Class_ParamSpec, type[Class_ParamSpec[str, int]])
```
It's occurring because there's no equivalence relationship defined
between `ClassLiteral` and `KnownInstanceType::TypeGenericAlias` which
is what these types are.
Everything else looks good to me!
* origin/main: (41 commits)
[ty] Carry generic context through when converting class into `Callable` (#21798)
[ty] Add more tests for renamings (#21810)
[ty] Minor improvements to `assert_type` diagnostics (#21811)
[ty] Add some attribute/method renaming test cases (#21809)
Update mkdocs-material to 9.7.0 (Insiders now free) (#21797)
Remove unused whitespaces in test cases (#21806)
[ty] fix panic when instantiating a type variable with invalid constraints (#21663)
[ty] fix build failure caused by conflicts between #21683 and #21800 (#21802)
[ty] do nothing with `store_expression_type` if `inner_expression_inference_state` is `Get` (#21718)
[ty] increase the limit on the number of elements in a non-recursively defined literal union (#21683)
[ty] normalize typevar bounds/constraints in cycles (#21800)
[ty] Update completion eval to include modules
[ty] Add modules to auto-import
[ty] Add support for module-only import requests
[ty] Refactor auto-import symbol info
[ty] Clarify the use of `SymbolKind` in auto-import
[ty] Redact ranking of completions from e2e LSP tests
[ty] Tweaks tests to use clearer language
[ty] Update evaluation results
[ty] Make auto-import ignore symbols in modules starting with a `_`
...
When converting a class (whether specialized or not) into a `Callable`
type, we should carry through any generic context that the constructor
has. This includes both the generic context of the class itself (if it's
generic) and of the constructor methods (if they are separately
generic).
To help test this, this also updates the `generic_context` extension
function to work on `Callable` types and unions; and adds a new
`into_callable` extension function that works just like
`CallableTypeOf`, but on value forms instead of type forms.
Pulled this out of #21551 for separate review.
* origin/main:
[ty] Reachability constraints: minor documentation fixes (#21774)
[ty] Fix non-determinism in `ConstraintSet.specialize_constrained` (#21744)
[ty] Improve `@override`, `@final` and Liskov checks in cases where there are multiple reachable definitions (#21767)
[ty] Extend `invalid-explicit-override` to also cover properties decorated with `@override` that do not override anything (#21756)
[ty] Enable LRU collection for parsed module (#21749)
[ty] Support typevar-specialized dynamic types in generic type aliases (#21730)
Add token based `parenthesized_ranges` implementation (#21738)
[ty] Default-specialization of generic type aliases (#21765)
[ty] Suppress false positives when `dataclasses.dataclass(...)(cls)` is called imperatively (#21729)
[syntax-error] Default type parameter followed by non-default type parameter (#21657)
Douglas Creager
Dhruv Manilawala
Bhuminjay Soni
Alex Waygood
Shunsuke Shibayama
Carl Meyer
Luca Chiodini
David Peter