We're seeing a lot of nondeterminism in the ecosystem tests at the
moment, which started (or at least got worse) once `Callable` inference
landed.
This PR attempts to remove this nondeterminism. We recently
(https://github.com/astral-sh/ruff/pull/21983) added a `source_order`
field to BDD nodes, which tracks when their constraint was added to the
BDD. Since we build up constraints based on the order that they appear
in the underlying source, that gives us a stable ordering even though we
use an arbitrary salsa-derived ordering for the BDD variables.
The issue (at least for some of the flakiness) is that we add "derived"
constraints when walking a BDD tree, and those derived constraints
inherit or borrow the `source_order` of the "real" constraint that
implied them. That means we can get multiple constraints in our
specialization that all have the same `source_order`. If we're not
careful, those "tied" constraints can be ordered arbitrarily.
The fix requires ~three~ ~four~ several steps:
- When starting to construct a sequent map (the data structure that
stores the derived constraints), we first sort all of the "real"
constraints by their `source_order`. That ensures that we insert things
into the sequent map in a stable order.
- During sequent map construction, derived facts are discovered by a
deterministic process applied to constraints in a (now) stable order. So
derived facts are now also inserted in a stable order.
- We update the fields of `SequentMap` to use `FxOrderSet` instead of
`FxHashSet`, so that we retain that stable insertion order.
- When walking BDD paths when constructing a specialization, we were
already sorting the constraints by their `source_order`. However, we
were not considering that we might get derived constraints, and
therefore constraints with "ties". Because of that, we need to make sure
to use a _stable_ sort, that retains the insertion order for those ties.
All together, this...should...fix the nondeterminism. (Unfortunately, I
haven't been able to effectively test this, since I haven't been able to
coerce local tests to flop into the other order that we sometimes see in
CI.)
@carljm put forth a reasonably compelling argument that just disabling
this lint might be advisable. If we agree, here's the implementation.
* Fixes https://github.com/astral-sh/ty/issues/309
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
This refactor is intended to give more structure to how we generate
completions. There's now a `Context` for "how do we figure out what kind
of completions to offer" and also a `CollectionContext` for "how do we
figure out which completions are appropriate or not." We double down on
`Completions` as a collector and a single point of truth for this. It
now handles adding information to `Completion` (based on the context)
and also skipping completions that are inappropriate (instead of
filtering them after-the-fact).
We also bundle a bunch of state into a new `ContextCursor` type, and
then define a bunch of predicates/accessors on that type that were
previously free functions with loads of parameters.
Finally, we introduce more structure to ranking. Instead of an anonymous
tuple, we define an explicit type with some helper types to hopefully
make the influence on ranking from each constituent piece a bit clearer.
This does seem to fix one bug around detecting the target for non-import
completions, but otherwise should not have any changes in behavior.
This is meant to be a precursor to improving completion ranking.
## Summary
I should have factored this better but this includes a drive-by move of
find_node to ruff_python_ast so ty_python_semantic can use it too.
* Fixes https://github.com/astral-sh/ty/issues/2017
## Test Plan
Snapshots galore
When inferring a specialization of a `Callable` type, we use the new
constraint set implementation. In the example in
https://github.com/astral-sh/ty/issues/1968, we end up with a constraint
set that includes all of the following clauses:
```
U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M1 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M2 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M3 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M4 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M5 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M6 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M7 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
```
In general, we take the upper bounds of those constraints to get the
specialization. However, the upper bounds of those constraints are not
all guaranteed to be the same, and so first we need to intersect them
all together. In this case, the upper bounds are all identical, so their
intersection is trivial:
```
U_co = M1 | M2 | M3 | M4 | M5 | M6 | M7
```
But we were still doing the work of calculating that trivial
intersection 7 times. And each time we have to do 7^2 comparisons of the
`M*` classes, ending up with O(n^3) overall work.
This pattern is common enough that we can put in a quick heuristic to
prune identical copies of the same type before performing the
intersection.
Fixes https://github.com/astral-sh/ty/issues/1968
## Summary
This contains two bug fixes:
- [Handle field specifier functions that accept
`**kwargs`](ad6918d505)
- [Recognize metaclass-based transformers as instances of
`DataclassInstance`](1a8e29b23c)
closes https://github.com/astral-sh/ty/issues/1987
## Test Plan
* New Markdown tests
* Made sure that the example in 1987 checks without errors
## Summary
We're actually quite good at computing this but the main issue is just
that we compute it at the type-level and so wrap it in `Literal[...]`.
So just special-case the rendering of these to omit `Literal[...]` and
fallback to `...` in cases where the thing we'll show is probably
useless (i.e. `x: str = str`).
Fixes https://github.com/astral-sh/ty/issues/1882
This fixes a bug @zsol found running ty against pyx. His original repro
is:
```py
class Base:
def __init__(self) -> None: pass
class A(Base):
pass
def foo[T](callable: Callable[..., T]) -> T:
return callable()
a: A = foo(A)
```
The call at the bottom would fail, since we would infer `() -> Base` as
the callable type of `A`, when it should be `() -> A`.
The issue was how we add implicit annotations to `self` parameters.
Typically, we turn it into `self: Self`. But in cases where we don't
need to introduce a full typevar, we turn it into `self: [the class
itself]` — in this case, `self: Base`. Then, when turning the class
constructor into a callable, we would see this non-`Self` annotation and
think that it was important and load-bearing.
The fix is that we skip all implicit annotations when determining
whether the `self` annotation should take precedence in the callable's
return type.
This is a first stab at solving
https://github.com/astral-sh/ty/issues/500, at least in part, with the
old solver. We add a new `TypeRelation` that lets us opt into using
constraint sets to describe when a typevar is assignability to some
type, and then use that to calculate a constraint set that describes
when two callable types are assignable. If the callable types contain
typevars, that constraint set will describe their valid specializations.
We can then walk through all of the ways the constraint set can be
satisfied, and record a type mapping in the old solver for each one.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
Fixes https://github.com/astral-sh/ty/issues/1787
## Summary
Allow method decorators returning Callables to presumptively propagate
"classmethod-ness" in the same way that they already presumptively
propagate "function-like-ness". We can't actually be sure that this is
the case, based on the decorator's annotations, but (along with other
type checkers) we heuristically assume it to be the case for decorators
applied via decorator syntax.
## Test Plan
Added mdtest.
## Summary
Infer `Literal[True]` for `isinstance(x, C)` calls when `x: T` and `T`
has a bound `B` that satisfies the `isinstance` check against `C`.
Similar for constrained typevars.
closes https://github.com/astral-sh/ty/issues/1895
## Test Plan
* New Markdown tests
* Verified the the example in the linked ticket checks without errors
In https://github.com/astral-sh/ruff/pull/21957, we tried to use
`union_or_intersection_elements_ordering` to provide a stable ordering
of the union and intersection elements that are created when determining
which type a typevar should specialize to. @AlexWaygood [pointed
out](https://github.com/astral-sh/ruff/pull/21551#discussion_r2616543762)
that this won't work, since that provides a consistent ordering within a
single process run, but does not provide a stable ordering across runs.
This is an attempt to produce a proper stable ordering for constraint
sets, so that we end up with consistent diagnostic and test output.
We do this by maintaining a new `source_order` field on each interior
BDD node, which records when that node's constraint was added to the
set. Several of the BDD operators (`and`, `or`, etc) now have
`_with_offset` variants, which update each `source_order` in the rhs to
be larger than any of the `source_order`s in the lhs. This is what
causes that field to be in line with (a) when you add each constraint to
the set, and (b) the order of the parameters you provide to `and`, `or`,
etc. Then we sort by that new field before constructing the
union/intersection types when creating a specialization.
When we calculate which typevars are inferable in a generic context, the
result might include more than the typevars bound by the generic
context. The canonical example is a generic method of a generic class:
```py
class C[A]:
def method[T](self, t: T): ...
```
Here, the inferable typevar set of `method` contains `Self` and `T`, as
you'd expect. (Those are the typevars bound by the method.) But it also
contains `A@C`, since the implicit `Self` typevar is defined as `Self:
C[A]`. That means when we call `method`, we need to mark `A@C` as
inferable, so that we can determine the correct mapping for `A@C` at the
call site.
Fixes https://github.com/astral-sh/ty/issues/1874
## Summary
- Treat `if TYPE_CHECKING` blocks the same as stub files (the feature
requested in https://github.com/astral-sh/ty/issues/1216)
- We currently only allow `@abstractmethod`-decorated methods to omit
the implementation if they're methods in classes that have _exactly_
`ABCMeta` as their metaclass. That seems wrong -- `@abstractmethod` has
the same semantics if a class has a subclass of `ABCMeta` as its
metaclass. This PR fixes that too. (I'm actually not _totally_ sure we
should care what the class's metaclass is at all -- see discussion in
https://github.com/astral-sh/ty/issues/1877#issue-3725937441... but the
change this PR is making seems less wrong than what we have currently,
anyway.)
Fixes https://github.com/astral-sh/ty/issues/1216
## Test Plan
Mdtests and snapshots
We now allow the lower and upper bounds of a constraint to be gradual.
Before, we would take the top/bottom materializations of the bounds.
This required us to pass in whether the constraint was intended for a
subtyping check or an assignability check, since that would control
whether we took the "restrictive" or "permissive" materializations,
respectively.
Unfortunately, doing so means that we lost information about whether the
original query involves a non-fully-static type. This would cause us to
create specializations like `T = object` for the constraint `T ≤ Any`,
when it would be nicer to carry through the gradual type and produce `T
= Any`.
We're not currently using constraint sets for subtyping checks, nor are
we going to in the very near future. So for now, we're going to assume
that constraint sets are always used for assignability checks, and allow
the lower/upper bounds to not be fully static. Once we get to the point
where we need to use constraint sets for subtyping checks, we will
consider how best to record this information in constraints.
## Summary
This PR makes explicit specialization of a type variable itself an
error, and the result of the specialization is `Unknown`.
The change also fixes https://github.com/astral-sh/ty/issues/1794.
## Test Plan
mdtests updated
new corpus test
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR takes the improvements we made to unsupported-comparison
diagnostics in https://github.com/astral-sh/ruff/pull/21737, and extends
them to other `unsupported-operator` diagnostics.
## Test Plan
Mdtests and snapshots
## Summary
Working on py-fuzzer recently (AKA, a Python project!) reminded me how
cool our "inlay hint goto-definition feature" is. So this PR adds a
bunch more of that!
I also made a couple of other minor changes to type display. For
example, in the playground, this snippet:
```py
def f(): ...
reveal_type(f.__get__)
```
currently leads to this diagnostic:
```
Revealed type: `<method-wrapper `__get__` of `f`>` (revealed-type) [Ln 2, Col 13]
```
But the fact that we have backticks both around the type display and
inside the type display isn't _great_ there. This PR changes it to
```
Revealed type: `<method-wrapper '__get__' of function 'f'>` (revealed-type) [Ln 2, Col 13]
```
which avoids the nested-backticks issue in diagnostics, and is more
similar to our display for various other `Type` variants such as
class-literal types (`<class 'Foo'>`, etc., not ``<class `Foo`>``).
## Test Plan
inlay snapshots added; mdtests updated
By teaching desperate resolution to try every possible ancestor that
doesn't have an `__init__.py(i)` when resolving absolute imports.
* Fixes https://github.com/astral-sh/ty/issues/1782
Partially addresses https://github.com/astral-sh/ty/issues/1732
## Summary
Don't union the previous type in fixpoint iteration if the previous type
contains a `Divergent` from the current cycle and the latest type does
not. The theory here, as outlined by @mtshiba at
https://github.com/astral-sh/ty/issues/1732#issuecomment-3609937420, is
that oscillation can't occur by removing and then reintroducing a
`Divergent` type repeatedly, since `Divergent` types are only introduced
at the start of fixpoint iteration.
## Test Plan
Removes a `Divergent` type from the added mdtest, doesn't otherwise
regress any tests.
## Summary
This PR includes the following changes:
* When attempting to specialize a non-generic type (or a type that is
already specialized), the result is `Unknown`. Also, the error message
is improved.
* When an implicit type alias is incorrectly specialized, the result is
`Unknown`. Also, the error message is improved.
* When only some of the type alias bounds and constraints are not
satisfied, not all substitutions are `Unknown`.
* Double specialization is prohibited. e.g. `G[int][int]`
Furthermore, after applying this PR, the fuzzing tests for seeds 1052
and 4419, which panic in main, now pass.
This is because the false recursions on type variables have been
removed.
```python
# name_2[0] => Unknown
class name_1[name_2: name_2[0]]:
def name_4(name_3: name_2, /):
if name_3:
pass
# (name_5 if unique_name_0 else name_1)[0] => Unknown
def name_4[name_5: (name_5 if unique_name_0 else name_1)[0], **name_1](): ...
```
## Test Plan
New corpus test
mdtest files updated
As described in astral-sh/ty#1729, we previously had a salsa cycle when
inferring the signature of many function definitions.
The most obvious case happened when (a) the function was decorated, (b)
it had no PEP-695 type params, and (c) annotations were not always
deferred (e.g. in a stub file). We currently evaluate and apply function
decorators eagerly, as part of `infer_function_definition`. Applying a
decorator requires knowing the signature of the function being
decorated. There were two places where signature construction called
`infer_definition_types` cyclically.
The simpler case was that we were looking up the generic context and
decorator list of the function to determine whether it has an implicit
`self` parameter. Before, we used `infer_definition_types` to determine
that information. But since we're in the middle of signature
construction for the function, we can just thread the information
through directly.
The harder case is that signature construction requires knowing the
inferred parameter and return type annotations. When (b) and (c) hold,
those type annotations are inferred in `infer_function_definition`! (In
theory, we've already finished that by the time we start applying
decorators, but signature construction doesn't know that.)
If annotations are deferred, the params/return annotations are inferred
in `infer_deferred_types`; if there are PEP-695 type params, they're
inferred in `infer_function_type_params`. Both of those are different
salsa queries, and don't induce this cycle.
So the quick fix here is to always defer inference of the function
params/return, so that they are always inferred under a different salsa
query.
A more principled fix would be to apply decorators lazily, just like we
construct signatures lazily. But that is a more invasive fix.
Fixesastral-sh/ty#1729
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
Charlie Marsh
Douglas Creager
Jack O'Connor
Aria Desires
Micha Reiser
Andrew Gallant
Alex Waygood
David Peter
Carl Meyer
Dhruv Manilawala
Bhuminjay Soni
Simon Lamon
Shunsuke Shibayama