Python
/
ruff
mirror of https://github.com/astral-sh/ruff
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[ty] tighten up handling of subscripts in type expressions (#21503) 

## Summary

Get rid of the catch-all todo type from subscripting a base type we
haven't implemented handling for yet in a type expression, and turn it
into a diagnostic instead.

Handle a few more cases explicitly, to avoid false positives from the
above change:
1. Subscripting any dynamic type (not just a todo type) in a type
expression should just result in that same dynamic type. This is
important for gradual guarantee, and matches other type checkers.
2. Subscripting a generic alias may be an error or not, depending
whether the specialization itself contains typevars. Don't try to handle
this yet (it should be handled in a later PR for specializing generic
non-PEP695 type aliases), just use a dedicated todo type for it.
3. Add a temporary todo branch to avoid false positives from string PEP
613 type aliases. This can be removed in the next PR, with PEP 613 type
alias support.

## Test Plan

Adjusted mdtests, ecosystem.

All new diagnostics in conformance suite are supposed to be diagnostics,
so this PR is a strict improvement there.

New diagnostics in the ecosystem are surfacing cases where we already
don't understand an annotation, but now we emit a diagnostic about it.
They are mostly intentional choices. Analysis of particular cases:

* `attrs`, `bokeh`, `django-stubs`, `dulwich`, `ibis`, `kornia`,
`mitmproxy`, `mongo-python-driver`, `mypy`, `pandas`, `poetry`,
`prefect`, `pydantic`, `pytest`, `scrapy`, `trio`, `werkzeug`, and
`xarray` are all cases where under `from __future__ import annotations`
or Python 3.14 deferred-annotations semantics, we follow normal
name-scoping rules, whereas some other type checkers prefer global names
over local names. This means we don't like it if e.g. you have a class
with a method or attribute named `type` or `tuple`, and you also try to
use `type` or `tuple` in method/attribute annotations of that class.
This PR isn't changing those semantics, just revealing them in more
cases where previously we just silently fell back to `Unknown`. I think
failing with a diagnostic (so authors can alias names as needed to avoid
relying on scoping rules that differ between type checkers) is better
than failing silently here.
* `beartype` assumes we support `TypeForm` (because it only supports
mypy and pyright, it uses `if MYPY:` to hide the `TypeForm` from mypy,
and pyright supports `TypeForm`), and we don't yet.
* `graphql-core` likes to use a `try: ... except ImportError: ...`
pattern for importing special forms from `typing` with fallback to
`typing_extensions`, instead of using `sys.version_info` checks. We
don't handle this well when type checking under an older Python version
(where the import from `typing` is not found); we see the imported name
as of type e.g. `Unknown | SpecialFormType(...)`, and because of the
union with `Unknown` we fail to handle it as the special form type. Mypy
and pyright also don't seem to support this pattern. They don't complain
about subscripting such special forms, but they do silently fail to
treat them as the desired special form. Again here, if we are going to
fail I'd rather fail with a diagnostic rather than silently.
* `ibis` is [trying to
use](https://github.com/ibis-project/ibis/blob/main/ibis/common/collections.py#L372)
`frozendict: type[FrozenDict]` as a way to create a "type alias" to
`FrozenDict`, but this is wrong: that means `frozendict:
type[FrozenDict[Any, Any]]`.
* `mypy` has some errors due to the fact that type-checking `typing.pyi`
itself (without knowing that it's the real `typing.pyi`) doesn't work
very well.
* `mypy-protobuf` imports some types from the protobufs library that end
up unioned with `Unknown` for some reason, and so we don't allow
explicit-specialization of them. Depending on the reason they end up
unioned with `Unknown`, we might want to better support this? But it's
orthogonal to this PR -- we aren't failing any worse here, just alerting
the author that we didn't understand their annotation.
* `pwndbg` has unresolved references due to star-importing from a
dependency that isn't installed, and uses un-imported names like `Dict`
in annotation expressions. Some of the unresolved references were hidden
by
https://github.com/astral-sh/ruff/blob/main/crates/ty_python_semantic/src/types/infer/builder.rs#L7223-L7228
when some annotations previously resolved to a Todo type that no longer
do.
This commit is contained in:
Carl Meyer 2025-11-18 10:43:07 -08:00 committed by GitHub
parent 0645418f00
commit 192c37d540
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
6 changed files with 59 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
crates/ruff_benchmark/benches/ty.rs
View File

@ -667,7 +667,7 @@ fn attrs(criterion: &mut Criterion) {
max_dep_date: "2025-06-17",
python_version: PythonVersion::PY313,
},
110,
120,
);
bench_project(&benchmark, criterion);

6
crates/ty_python_semantic/resources/mdtest/annotations/invalid.md
View File

@ -98,7 +98,9 @@ async def outer_async(): # avoid unrelated syntax errors on `yield` and `await`
n: 1 < 2, # error: [invalid-type-form] "Comparison expressions are not allowed in type expressions"
o: bar(), # error: [invalid-type-form] "Function calls are not allowed in type expressions"
p: int | f"foo", # error: [invalid-type-form] "F-strings are not allowed in type expressions"
q: [1, 2, 3][1:2], # error: [invalid-type-form] "Slices are not allowed in type expressions"
# error: [invalid-type-form] "Slices are not allowed in type expressions"
# error: [invalid-type-form] "Invalid subscript"
q: [1, 2, 3][1:2],
):
reveal_type(a) # revealed: Unknown
reveal_type(b) # revealed: Unknown
@ -116,7 +118,7 @@ async def outer_async(): # avoid unrelated syntax errors on `yield` and `await`
reveal_type(n) # revealed: Unknown
reveal_type(o) # revealed: Unknown
reveal_type(p) # revealed: int | Unknown
reveal_type(q) # revealed: @Todo(unknown type subscript)
reveal_type(q) # revealed: Unknown
class Mat:
def __init__(self, value: int):

3
crates/ty_python_semantic/resources/mdtest/annotations/literal.md
View File

@ -330,10 +330,11 @@ from other import Literal
# ?
#
# error: [invalid-type-form] "Int literals are not allowed in this context in a type expression"
# error: [invalid-type-form] "Invalid subscript of object of type `_SpecialForm` in type expression"
a1: Literal[26]
def f():
reveal_type(a1) # revealed: @Todo(unknown type subscript)
reveal_type(a1) # revealed: Unknown
```
## Detecting typing_extensions.Literal

26
crates/ty_python_semantic/resources/mdtest/implicit_type_aliases.md
View File

@ -33,9 +33,11 @@ g(None)
We also support unions in type aliases:
```py
from typing_extensions import Any, Never, Literal, LiteralString, Tuple, Annotated, Optional, Union, Callable
from typing_extensions import Any, Never, Literal, LiteralString, Tuple, Annotated, Optional, Union, Callable, TypeVar
from ty_extensions import Unknown
T = TypeVar("T")
IntOrStr = int | str
IntOrStrOrBytes1 = int | str | bytes
IntOrStrOrBytes2 = (int | str) | bytes
@ -70,6 +72,10 @@ IntOrTypeOfStr = int | type[str]
TypeOfStrOrInt = type[str] | int
IntOrCallable = int | Callable[[str], bytes]
CallableOrInt = Callable[[str], bytes] | int
TypeVarOrInt = T | int
IntOrTypeVar = int | T
TypeVarOrNone = T | None
NoneOrTypeVar = None | T
reveal_type(IntOrStr) # revealed: types.UnionType
reveal_type(IntOrStrOrBytes1) # revealed: types.UnionType
@ -105,6 +111,10 @@ reveal_type(IntOrTypeOfStr) # revealed: types.UnionType
reveal_type(TypeOfStrOrInt) # revealed: types.UnionType
reveal_type(IntOrCallable) # revealed: types.UnionType
reveal_type(CallableOrInt) # revealed: types.UnionType
reveal_type(TypeVarOrInt) # revealed: types.UnionType
reveal_type(IntOrTypeVar) # revealed: types.UnionType
reveal_type(TypeVarOrNone) # revealed: types.UnionType
reveal_type(NoneOrTypeVar) # revealed: types.UnionType
def _(
int_or_str: IntOrStr,
@ -141,6 +151,10 @@ def _(
type_of_str_or_int: TypeOfStrOrInt,
int_or_callable: IntOrCallable,
callable_or_int: CallableOrInt,
type_var_or_int: TypeVarOrInt,
int_or_type_var: IntOrTypeVar,
type_var_or_none: TypeVarOrNone,
none_or_type_var: NoneOrTypeVar,
):
reveal_type(int_or_str) # revealed: int | str
reveal_type(int_or_str_or_bytes1) # revealed: int | str | bytes
@ -176,6 +190,14 @@ def _(
reveal_type(type_of_str_or_int) # revealed: type[str] | int
reveal_type(int_or_callable) # revealed: int | ((str, /) -> bytes)
reveal_type(callable_or_int) # revealed: ((str, /) -> bytes) | int
# TODO should be Unknown | int
reveal_type(type_var_or_int) # revealed: T@_ | int
# TODO should be int | Unknown
reveal_type(int_or_type_var) # revealed: int | T@_
# TODO should be Unknown | None
reveal_type(type_var_or_none) # revealed: T@_ | None
# TODO should be None | Unknown
reveal_type(none_or_type_var) # revealed: None | T@_
```
If a type is unioned with itself in a value expression, the result is just that type. No
@ -357,7 +379,7 @@ MyList = list[T]
def _(my_list: MyList[int]):
# TODO: This should be `list[int]`
reveal_type(my_list) # revealed: @Todo(unknown type subscript)
reveal_type(my_list) # revealed: @Todo(specialized generic alias in type expression)
ListOrTuple = list[T] | tuple[T, ...]

9
crates/ty_python_semantic/src/types/infer/builder.rs
View File

@ -9507,7 +9507,8 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
| KnownInstanceType::Literal(_)
| KnownInstanceType::Annotated(_)
| KnownInstanceType::TypeGenericAlias(_)
| KnownInstanceType::Callable(_),
| KnownInstanceType::Callable(_)
| KnownInstanceType::TypeVar(_),
),
Type::ClassLiteral(..)
| Type::SubclassOf(..)
@ -9518,7 +9519,8 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
| KnownInstanceType::Literal(_)
| KnownInstanceType::Annotated(_)
| KnownInstanceType::TypeGenericAlias(_)
| KnownInstanceType::Callable(_),
| KnownInstanceType::Callable(_)
| KnownInstanceType::TypeVar(_),
),
ast::Operator::BitOr,
) if pep_604_unions_allowed() => {
@ -10926,6 +10928,9 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
.map(Type::from)
.unwrap_or_else(Type::unknown);
}
Type::KnownInstance(KnownInstanceType::UnionType(_)) => {
return todo_type!("Specialization of union type alias");
}
_ => {}
}

26
crates/ty_python_semantic/src/types/infer/builder/type_expression.rs
View File

@ -858,7 +858,7 @@ impl<'db> TypeInferenceBuilder<'db, '_> {
Type::unknown()
}
},
Type::Dynamic(DynamicType::Todo(_)) => {
Type::Dynamic(_) => {
self.infer_type_expression(slice);
value_ty
}
@ -887,11 +887,27 @@ impl<'db> TypeInferenceBuilder<'db, '_> {
}
}
}
_ => {
// TODO: Emit a diagnostic once we've implemented all valid subscript type
// expressions.
Type::GenericAlias(_) => {
self.infer_type_expression(slice);
todo_type!("unknown type subscript")
// If the generic alias is already fully specialized, this is an error. But it
// could have been specialized with another typevar (e.g. a type alias like `MyList
// = list[T]`), in which case it's later valid to do `MyList[int]`.
todo_type!("specialized generic alias in type expression")
}
Type::StringLiteral(_) => {
self.infer_type_expression(slice);
// For stringified TypeAlias; remove once properly supported
todo_type!("string literal subscripted in type expression")
}
_ => {
self.infer_type_expression(slice);
if let Some(builder) = self.context.report_lint(&INVALID_TYPE_FORM, subscript) {
builder.into_diagnostic(format_args!(
"Invalid subscript of object of type `{}` in type expression",
value_ty.display(self.db())
));
}
Type::unknown()
}
}
}