## Summary
This PR implements the strategy described in
https://github.com/astral-sh/ty/issues/1871: we iterate over the
positive types, resolve them, then intersect the results.
## Summary
This should make revealed types a little nicer, as well as avoid
confusing the constraint solver in some cases (which were showing up in
https://github.com/astral-sh/ruff/pull/21930).
## Summary
Add lockfiles for all mdtests which make use of external dependencies.
When running tests normally, we use this lockfile when creating the
temporary venv using `uv sync --locked`. A new
`MDTEST_UPGRADE_LOCKFILES` environment variable is used to switch to a
mode in which those lockfiles can be updated or regenerated. When using
the Python mdtest runner, this environment variable is automatically set
(because we use this command while developing, not to simulate exactly
what happens in CI). A command-line flag is provided to opt out of this.
## Test Plan
### Using the mdtest runner
#### Adding a new test (no lockfile yet)
* Removed `attrs.lock` to simulate this
* Ran `uv run crates/ty_python_semantic/mdtest.py -e external/`. The
lockfile is generated and the test succeeds.
#### Upgrading/downgrading a dependency
* Changed pydantic requirement from `pydantic==2.12.2` to
`pydantic==2.12.5` (also tested with `2.12.0`)
* Ran `uv run crates/ty_python_semantic/mdtest.py -e external/`. The
lockfile is updated and the test succeeds.
### Using cargo
#### Adding a new test (no lockfile yet)
* Removed `attrs.lock` to simulate this
* Ran `MDTEST_EXTERNAL=1 cargo test -p ty_python_semantic --test mdtest
mdtest__external` "naively", which outputs:
> Failed to setup in-memory virtual environment with dependencies:
Lockfile not found at
'/home/shark/ruff/crates/ty_python_semantic/resources/mdtest/external/attrs.lock'.
Run with `MDTEST_UPGRADE_LOCKFILES=1` to generate it.
* Ran `MDTEST_UPGRADE_LOCKFILES=1 MDTEST_EXTERNAL=1 cargo test -p
ty_python_semantic --test mdtest mdtest__external`. The lockfile is
updated and the test succeeds.
#### Upgrading/downgrading a dependency
* Changed pydantic requirement from `pydantic==2.12.2` to
`pydantic==2.12.5` (also tested with `2.12.0`)
* Ran `MDTEST_EXTERNAL=1 cargo test -p ty_python_semantic --test mdtest
mdtest__external` "naively", which outputs a similar error message as
above.
* Ran the command suggested in the error message (`MDTEST_EXTERNAL=1
MDTEST_UPGRADE_LOCKFILES=1 cargo test -p ty_python_semantic --test
mdtest mdtest__external`). The lockfile is updated and the test
succeeds.
We're seeing a lot of nondeterminism in the ecosystem tests at the
moment, which started (or at least got worse) once `Callable` inference
landed.
This PR attempts to remove this nondeterminism. We recently
(https://github.com/astral-sh/ruff/pull/21983) added a `source_order`
field to BDD nodes, which tracks when their constraint was added to the
BDD. Since we build up constraints based on the order that they appear
in the underlying source, that gives us a stable ordering even though we
use an arbitrary salsa-derived ordering for the BDD variables.
The issue (at least for some of the flakiness) is that we add "derived"
constraints when walking a BDD tree, and those derived constraints
inherit or borrow the `source_order` of the "real" constraint that
implied them. That means we can get multiple constraints in our
specialization that all have the same `source_order`. If we're not
careful, those "tied" constraints can be ordered arbitrarily.
The fix requires ~three~ ~four~ several steps:
- When starting to construct a sequent map (the data structure that
stores the derived constraints), we first sort all of the "real"
constraints by their `source_order`. That ensures that we insert things
into the sequent map in a stable order.
- During sequent map construction, derived facts are discovered by a
deterministic process applied to constraints in a (now) stable order. So
derived facts are now also inserted in a stable order.
- We update the fields of `SequentMap` to use `FxOrderSet` instead of
`FxHashSet`, so that we retain that stable insertion order.
- When walking BDD paths when constructing a specialization, we were
already sorting the constraints by their `source_order`. However, we
were not considering that we might get derived constraints, and
therefore constraints with "ties". Because of that, we need to make sure
to use a _stable_ sort, that retains the insertion order for those ties.
All together, this...should...fix the nondeterminism. (Unfortunately, I
haven't been able to effectively test this, since I haven't been able to
coerce local tests to flop into the other order that we sometimes see in
CI.)
@carljm put forth a reasonably compelling argument that just disabling
this lint might be advisable. If we agree, here's the implementation.
* Fixes https://github.com/astral-sh/ty/issues/309
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
This refactor is intended to give more structure to how we generate
completions. There's now a `Context` for "how do we figure out what kind
of completions to offer" and also a `CollectionContext` for "how do we
figure out which completions are appropriate or not." We double down on
`Completions` as a collector and a single point of truth for this. It
now handles adding information to `Completion` (based on the context)
and also skipping completions that are inappropriate (instead of
filtering them after-the-fact).
We also bundle a bunch of state into a new `ContextCursor` type, and
then define a bunch of predicates/accessors on that type that were
previously free functions with loads of parameters.
Finally, we introduce more structure to ranking. Instead of an anonymous
tuple, we define an explicit type with some helper types to hopefully
make the influence on ranking from each constituent piece a bit clearer.
This does seem to fix one bug around detecting the target for non-import
completions, but otherwise should not have any changes in behavior.
This is meant to be a precursor to improving completion ranking.
## Summary
I should have factored this better but this includes a drive-by move of
find_node to ruff_python_ast so ty_python_semantic can use it too.
* Fixes https://github.com/astral-sh/ty/issues/2017
## Test Plan
Snapshots galore
When inferring a specialization of a `Callable` type, we use the new
constraint set implementation. In the example in
https://github.com/astral-sh/ty/issues/1968, we end up with a constraint
set that includes all of the following clauses:
```
U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M1 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M2 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M3 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M4 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M5 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M6 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
M7 ≤ U_co ≤ M1 | M2 | M3 | M4 | M5 | M6 | M7
```
In general, we take the upper bounds of those constraints to get the
specialization. However, the upper bounds of those constraints are not
all guaranteed to be the same, and so first we need to intersect them
all together. In this case, the upper bounds are all identical, so their
intersection is trivial:
```
U_co = M1 | M2 | M3 | M4 | M5 | M6 | M7
```
But we were still doing the work of calculating that trivial
intersection 7 times. And each time we have to do 7^2 comparisons of the
`M*` classes, ending up with O(n^3) overall work.
This pattern is common enough that we can put in a quick heuristic to
prune identical copies of the same type before performing the
intersection.
Fixes https://github.com/astral-sh/ty/issues/1968
## Summary
This contains two bug fixes:
- [Handle field specifier functions that accept
`**kwargs`](ad6918d505)
- [Recognize metaclass-based transformers as instances of
`DataclassInstance`](1a8e29b23c)
closes https://github.com/astral-sh/ty/issues/1987
## Test Plan
* New Markdown tests
* Made sure that the example in 1987 checks without errors
## Summary
We're actually quite good at computing this but the main issue is just
that we compute it at the type-level and so wrap it in `Literal[...]`.
So just special-case the rendering of these to omit `Literal[...]` and
fallback to `...` in cases where the thing we'll show is probably
useless (i.e. `x: str = str`).
Fixes https://github.com/astral-sh/ty/issues/1882
This fixes a bug @zsol found running ty against pyx. His original repro
is:
```py
class Base:
def __init__(self) -> None: pass
class A(Base):
pass
def foo[T](callable: Callable[..., T]) -> T:
return callable()
a: A = foo(A)
```
The call at the bottom would fail, since we would infer `() -> Base` as
the callable type of `A`, when it should be `() -> A`.
The issue was how we add implicit annotations to `self` parameters.
Typically, we turn it into `self: Self`. But in cases where we don't
need to introduce a full typevar, we turn it into `self: [the class
itself]` — in this case, `self: Base`. Then, when turning the class
constructor into a callable, we would see this non-`Self` annotation and
think that it was important and load-bearing.
The fix is that we skip all implicit annotations when determining
whether the `self` annotation should take precedence in the callable's
return type.
This is a first stab at solving
https://github.com/astral-sh/ty/issues/500, at least in part, with the
old solver. We add a new `TypeRelation` that lets us opt into using
constraint sets to describe when a typevar is assignability to some
type, and then use that to calculate a constraint set that describes
when two callable types are assignable. If the callable types contain
typevars, that constraint set will describe their valid specializations.
We can then walk through all of the ways the constraint set can be
satisfied, and record a type mapping in the old solver for each one.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
Fixes https://github.com/astral-sh/ty/issues/1787
## Summary
Allow method decorators returning Callables to presumptively propagate
"classmethod-ness" in the same way that they already presumptively
propagate "function-like-ness". We can't actually be sure that this is
the case, based on the decorator's annotations, but (along with other
type checkers) we heuristically assume it to be the case for decorators
applied via decorator syntax.
## Test Plan
Added mdtest.
## Summary
Infer `Literal[True]` for `isinstance(x, C)` calls when `x: T` and `T`
has a bound `B` that satisfies the `isinstance` check against `C`.
Similar for constrained typevars.
closes https://github.com/astral-sh/ty/issues/1895
## Test Plan
* New Markdown tests
* Verified the the example in the linked ticket checks without errors
In https://github.com/astral-sh/ruff/pull/21957, we tried to use
`union_or_intersection_elements_ordering` to provide a stable ordering
of the union and intersection elements that are created when determining
which type a typevar should specialize to. @AlexWaygood [pointed
out](https://github.com/astral-sh/ruff/pull/21551#discussion_r2616543762)
that this won't work, since that provides a consistent ordering within a
single process run, but does not provide a stable ordering across runs.
This is an attempt to produce a proper stable ordering for constraint
sets, so that we end up with consistent diagnostic and test output.
We do this by maintaining a new `source_order` field on each interior
BDD node, which records when that node's constraint was added to the
set. Several of the BDD operators (`and`, `or`, etc) now have
`_with_offset` variants, which update each `source_order` in the rhs to
be larger than any of the `source_order`s in the lhs. This is what
causes that field to be in line with (a) when you add each constraint to
the set, and (b) the order of the parameters you provide to `and`, `or`,
etc. Then we sort by that new field before constructing the
union/intersection types when creating a specialization.
When we calculate which typevars are inferable in a generic context, the
result might include more than the typevars bound by the generic
context. The canonical example is a generic method of a generic class:
```py
class C[A]:
def method[T](self, t: T): ...
```
Here, the inferable typevar set of `method` contains `Self` and `T`, as
you'd expect. (Those are the typevars bound by the method.) But it also
contains `A@C`, since the implicit `Self` typevar is defined as `Self:
C[A]`. That means when we call `method`, we need to mark `A@C` as
inferable, so that we can determine the correct mapping for `A@C` at the
call site.
Fixes https://github.com/astral-sh/ty/issues/1874
## Summary
- Treat `if TYPE_CHECKING` blocks the same as stub files (the feature
requested in https://github.com/astral-sh/ty/issues/1216)
- We currently only allow `@abstractmethod`-decorated methods to omit
the implementation if they're methods in classes that have _exactly_
`ABCMeta` as their metaclass. That seems wrong -- `@abstractmethod` has
the same semantics if a class has a subclass of `ABCMeta` as its
metaclass. This PR fixes that too. (I'm actually not _totally_ sure we
should care what the class's metaclass is at all -- see discussion in
https://github.com/astral-sh/ty/issues/1877#issue-3725937441... but the
change this PR is making seems less wrong than what we have currently,
anyway.)
Fixes https://github.com/astral-sh/ty/issues/1216
## Test Plan
Mdtests and snapshots
We now allow the lower and upper bounds of a constraint to be gradual.
Before, we would take the top/bottom materializations of the bounds.
This required us to pass in whether the constraint was intended for a
subtyping check or an assignability check, since that would control
whether we took the "restrictive" or "permissive" materializations,
respectively.
Unfortunately, doing so means that we lost information about whether the
original query involves a non-fully-static type. This would cause us to
create specializations like `T = object` for the constraint `T ≤ Any`,
when it would be nicer to carry through the gradual type and produce `T
= Any`.
We're not currently using constraint sets for subtyping checks, nor are
we going to in the very near future. So for now, we're going to assume
that constraint sets are always used for assignability checks, and allow
the lower/upper bounds to not be fully static. Once we get to the point
where we need to use constraint sets for subtyping checks, we will
consider how best to record this information in constraints.
## Summary
This PR makes explicit specialization of a type variable itself an
error, and the result of the specialization is `Unknown`.
The change also fixes https://github.com/astral-sh/ty/issues/1794.
## Test Plan
mdtests updated
new corpus test
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR takes the improvements we made to unsupported-comparison
diagnostics in https://github.com/astral-sh/ruff/pull/21737, and extends
them to other `unsupported-operator` diagnostics.
## Test Plan
Mdtests and snapshots
## Summary
Working on py-fuzzer recently (AKA, a Python project!) reminded me how
cool our "inlay hint goto-definition feature" is. So this PR adds a
bunch more of that!
I also made a couple of other minor changes to type display. For
example, in the playground, this snippet:
```py
def f(): ...
reveal_type(f.__get__)
```
currently leads to this diagnostic:
```
Revealed type: `<method-wrapper `__get__` of `f`>` (revealed-type) [Ln 2, Col 13]
```
But the fact that we have backticks both around the type display and
inside the type display isn't _great_ there. This PR changes it to
```
Revealed type: `<method-wrapper '__get__' of function 'f'>` (revealed-type) [Ln 2, Col 13]
```
which avoids the nested-backticks issue in diagnostics, and is more
similar to our display for various other `Type` variants such as
class-literal types (`<class 'Foo'>`, etc., not ``<class `Foo`>``).
## Test Plan
inlay snapshots added; mdtests updated
By teaching desperate resolution to try every possible ancestor that
doesn't have an `__init__.py(i)` when resolving absolute imports.
* Fixes https://github.com/astral-sh/ty/issues/1782
Charlie Marsh
Ibraheem Ahmed
Alex Waygood
David Peter
Micha Reiser
Douglas Creager
Jack O'Connor
Aria Desires
Andrew Gallant
Carl Meyer
Dhruv Manilawala
Bhuminjay Soni
Simon Lamon
Shunsuke Shibayama