mirror of https://github.com/astral-sh/ruff synced 2026-01-06 06:04:16 -05:00

Files

Carl Meyer 192c37d540 [ty] tighten up handling of subscripts in type expressions (#21503 )

## Summary

Get rid of the catch-all todo type from subscripting a base type we
haven't implemented handling for yet in a type expression, and turn it
into a diagnostic instead.

Handle a few more cases explicitly, to avoid false positives from the
above change:
1. Subscripting any dynamic type (not just a todo type) in a type
expression should just result in that same dynamic type. This is
important for gradual guarantee, and matches other type checkers.
2. Subscripting a generic alias may be an error or not, depending
whether the specialization itself contains typevars. Don't try to handle
this yet (it should be handled in a later PR for specializing generic
non-PEP695 type aliases), just use a dedicated todo type for it.
3. Add a temporary todo branch to avoid false positives from string PEP
613 type aliases. This can be removed in the next PR, with PEP 613 type
alias support.

## Test Plan

Adjusted mdtests, ecosystem.

All new diagnostics in conformance suite are supposed to be diagnostics,
so this PR is a strict improvement there.

New diagnostics in the ecosystem are surfacing cases where we already
don't understand an annotation, but now we emit a diagnostic about it.
They are mostly intentional choices. Analysis of particular cases:

* `attrs`, `bokeh`, `django-stubs`, `dulwich`, `ibis`, `kornia`,
`mitmproxy`, `mongo-python-driver`, `mypy`, `pandas`, `poetry`,
`prefect`, `pydantic`, `pytest`, `scrapy`, `trio`, `werkzeug`, and
`xarray` are all cases where under `from __future__ import annotations`
or Python 3.14 deferred-annotations semantics, we follow normal
name-scoping rules, whereas some other type checkers prefer global names
over local names. This means we don't like it if e.g. you have a class
with a method or attribute named `type` or `tuple`, and you also try to
use `type` or `tuple` in method/attribute annotations of that class.
This PR isn't changing those semantics, just revealing them in more
cases where previously we just silently fell back to `Unknown`. I think
failing with a diagnostic (so authors can alias names as needed to avoid
relying on scoping rules that differ between type checkers) is better
than failing silently here.
* `beartype` assumes we support `TypeForm` (because it only supports
mypy and pyright, it uses `if MYPY:` to hide the `TypeForm` from mypy,
and pyright supports `TypeForm`), and we don't yet.
* `graphql-core` likes to use a `try: ... except ImportError: ...`
pattern for importing special forms from `typing` with fallback to
`typing_extensions`, instead of using `sys.version_info` checks. We
don't handle this well when type checking under an older Python version
(where the import from `typing` is not found); we see the imported name
as of type e.g. `Unknown | SpecialFormType(...)`, and because of the
union with `Unknown` we fail to handle it as the special form type. Mypy
and pyright also don't seem to support this pattern. They don't complain
about subscripting such special forms, but they do silently fail to
treat them as the desired special form. Again here, if we are going to
fail I'd rather fail with a diagnostic rather than silently.
* `ibis` is [trying to
use](https://github.com/ibis-project/ibis/blob/main/ibis/common/collections.py#L372)
`frozendict: type[FrozenDict]` as a way to create a "type alias" to
`FrozenDict`, but this is wrong: that means `frozendict:
type[FrozenDict[Any, Any]]`.
* `mypy` has some errors due to the fact that type-checking `typing.pyi`
itself (without knowing that it's the real `typing.pyi`) doesn't work
very well.
* `mypy-protobuf` imports some types from the protobufs library that end
up unioned with `Unknown` for some reason, and so we don't allow
explicit-specialization of them. Depending on the reason they end up
unioned with `Unknown`, we might want to better support this? But it's
orthogonal to this PR -- we aren't failing any worse here, just alerting
the author that we didn't understand their annotation.
* `pwndbg` has unresolved references due to star-importing from a
dependency that isn't installed, and uses un-imported names like `Dict`
in annotation expressions. Some of the unresolved references were hidden
by
https://github.com/astral-sh/ruff/blob/main/crates/ty_python_semantic/src/types/infer/builder.rs#L7223-L7228
when some annotations previously resolved to a Todo type that no longer
do.

2025-11-18 10:43:07 -08:00

10 KiB

Raw Blame History

Tests for invalid types in type expressions

Invalid types are rejected

Many types are illegal in the context of a type expression:

import typing
from ty_extensions import AlwaysTruthy, AlwaysFalsy
from typing_extensions import Literal, Never

class A: ...

def _(
    a: type[int],
    b: AlwaysTruthy,
    c: AlwaysFalsy,
    d: Literal[True],
    e: Literal["bar"],
    f: Literal[b"foo"],
    g: tuple[int, str],
    h: Never,
    i: int,
    j: A,
):
    def foo(): ...
    def invalid(
        a_: a,  # error: [invalid-type-form] "Variable of type `type[int]` is not allowed in a type expression"
        b_: b,  # error: [invalid-type-form]
        c_: c,  # error: [invalid-type-form]
        d_: d,  # error: [invalid-type-form]
        e_: e,  # error: [invalid-type-form]
        f_: f,  # error: [invalid-type-form]
        g_: g,  # error: [invalid-type-form]
        h_: h,  # error: [invalid-type-form]
        i_: typing,  # error: [invalid-type-form]
        j_: foo,  # error: [invalid-type-form]
        k_: i,  # error: [invalid-type-form] "Variable of type `int` is not allowed in a type expression"
        l_: j,  # error: [invalid-type-form] "Variable of type `A` is not allowed in a type expression"
    ):
        reveal_type(a_)  # revealed: Unknown
        reveal_type(b_)  # revealed: Unknown
        reveal_type(c_)  # revealed: Unknown
        reveal_type(d_)  # revealed: Unknown
        reveal_type(e_)  # revealed: Unknown
        reveal_type(f_)  # revealed: Unknown
        reveal_type(g_)  # revealed: Unknown
        reveal_type(h_)  # revealed: Unknown
        reveal_type(i_)  # revealed: Unknown
        reveal_type(j_)  # revealed: Unknown

# Inspired by the conformance test suite at
# https://github.com/python/typing/blob/d4f39b27a4a47aac8b6d4019e1b0b5b3156fabdc/conformance/tests/aliases_implicit.py#L88-L122
B = [x for x in range(42)]
C = {x for x in range(42)}
D = {x: y for x, y in enumerate(range(42))}
E = (x for x in range(42))

def _(
    b: B,  # error: [invalid-type-form]
    c: C,  # error: [invalid-type-form]
    d: D,  # error: [invalid-type-form]
    e: E,  # error: [invalid-type-form]
):
    reveal_type(b)  # revealed: Unknown
    reveal_type(c)  # revealed: Unknown
    reveal_type(d)  # revealed: Unknown
    reveal_type(e)  # revealed: Unknown

Invalid AST nodes

def bar() -> None:
    return None

def outer_sync():  # `yield` from is only valid syntax inside a synchronous function
    def _(
        a: (yield from [1]),  # error: [invalid-type-form] "`yield from` expressions are not allowed in type expressions"
    ): ...

async def baz(): ...
async def outer_async():  # avoid unrelated syntax errors on `yield` and `await`
    def _(
        a: 1,  # error: [invalid-type-form] "Int literals are not allowed in this context in a type expression"
        b: 2.3,  # error: [invalid-type-form] "Float literals are not allowed in type expressions"
        c: 4j,  # error: [invalid-type-form] "Complex literals are not allowed in type expressions"
        d: True,  # error: [invalid-type-form] "Boolean literals are not allowed in this context in a type expression"
        e: int | b"foo",  # error: [invalid-type-form] "Bytes literals are not allowed in this context in a type expression"
        f: 1 and 2,  # error: [invalid-type-form] "Boolean operations are not allowed in type expressions"
        g: 1 or 2,  # error: [invalid-type-form] "Boolean operations are not allowed in type expressions"
        h: (foo := 1),  # error: [invalid-type-form] "Named expressions are not allowed in type expressions"
        i: not 1,  # error: [invalid-type-form] "Unary operations are not allowed in type expressions"
        j: lambda: 1,  # error: [invalid-type-form] "`lambda` expressions are not allowed in type expressions"
        k: 1 if True else 2,  # error: [invalid-type-form] "`if` expressions are not allowed in type expressions"
        l: await baz(),  # error: [invalid-type-form] "`await` expressions are not allowed in type expressions"
        m: (yield 1),  # error: [invalid-type-form] "`yield` expressions are not allowed in type expressions"
        n: 1 < 2,  # error: [invalid-type-form] "Comparison expressions are not allowed in type expressions"
        o: bar(),  # error: [invalid-type-form] "Function calls are not allowed in type expressions"
        p: int | f"foo",  # error: [invalid-type-form] "F-strings are not allowed in type expressions"
        # error: [invalid-type-form] "Slices are not allowed in type expressions"
        # error: [invalid-type-form] "Invalid subscript"
        q: [1, 2, 3][1:2],
    ):
        reveal_type(a)  # revealed: Unknown
        reveal_type(b)  # revealed: Unknown
        reveal_type(c)  # revealed: Unknown
        reveal_type(d)  # revealed: Unknown
        reveal_type(e)  # revealed: int | Unknown
        reveal_type(f)  # revealed: Unknown
        reveal_type(g)  # revealed: Unknown
        reveal_type(h)  # revealed: Unknown
        reveal_type(i)  # revealed: Unknown
        reveal_type(j)  # revealed: Unknown
        reveal_type(k)  # revealed: Unknown
        reveal_type(l)  # revealed: Unknown
        reveal_type(m)  # revealed: Unknown
        reveal_type(n)  # revealed: Unknown
        reveal_type(o)  # revealed: Unknown
        reveal_type(p)  # revealed: int | Unknown
        reveal_type(q)  # revealed: Unknown

class Mat:
    def __init__(self, value: int):
        self.value = value

    def __matmul__(self, other) -> int:
        return 42

def invalid_binary_operators(
    a: "1" + "2",  # error: [invalid-type-form] "Invalid binary operator `+` in type annotation"
    b: 3 - 5.0,  # error: [invalid-type-form] "Invalid binary operator `-` in type annotation"
    c: 4 * -2,  # error: [invalid-type-form] "Invalid binary operator `*` in type annotation"
    d: Mat(4) @ Mat(2),  # error: [invalid-type-form] "Invalid binary operator `@` in type annotation"
    e: 10 / 2,  # error: [invalid-type-form] "Invalid binary operator `/` in type annotation"
    f: 10 % 3,  # error: [invalid-type-form] "Invalid binary operator `%` in type annotation"
    g: 2**-0.5,  # error: [invalid-type-form] "Invalid binary operator `**` in type annotation"
    h: 10 // 3,  # error: [invalid-type-form] "Invalid binary operator `//` in type annotation"
    i: 1 << 2,  # error: [invalid-type-form] "Invalid binary operator `<<` in type annotation"
    j: 4 >> 42,  # error: [invalid-type-form] "Invalid binary operator `>>` in type annotation"
    k: 5 ^ 3,  # error: [invalid-type-form] "Invalid binary operator `^` in type annotation"
    l: 5 & 3,  # error: [invalid-type-form] "Invalid binary operator `&` in type annotation"
):
    reveal_type(a)  # revealed: Unknown
    reveal_type(b)  # revealed: Unknown
    reveal_type(c)  # revealed: Unknown
    reveal_type(d)  # revealed: Unknown
    reveal_type(e)  # revealed: Unknown
    reveal_type(f)  # revealed: Unknown
    reveal_type(g)  # revealed: Unknown
    reveal_type(h)  # revealed: Unknown
    reveal_type(i)  # revealed: Unknown
    reveal_type(j)  # revealed: Unknown
    reveal_type(k)  # revealed: Unknown
    reveal_type(l)  # revealed: Unknown

Invalid Collection based AST nodes

[environment]
python-version = "3.12"

def _(
    a: {1: 2},  # error: [invalid-type-form] "Dict literals are not allowed in type expressions"
    b: {1, 2},  # error: [invalid-type-form] "Set literals are not allowed in type expressions"
    c: {k: v for k, v in [(1, 2)]},  # error: [invalid-type-form] "Dict comprehensions are not allowed in type expressions"
    d: [k for k in [1, 2]],  # error: [invalid-type-form] "List comprehensions are not allowed in type expressions"
    e: {k for k in [1, 2]},  # error: [invalid-type-form] "Set comprehensions are not allowed in type expressions"
    f: (k for k in [1, 2]),  # error: [invalid-type-form] "Generator expressions are not allowed in type expressions"
    # error: [invalid-type-form] "List literals are not allowed in this context in a type expression: Did you mean `tuple[int, str]`?"
    g: [int, str],
    # error: [invalid-type-form] "Tuple literals are not allowed in this context in a type expression: Did you mean `tuple[int, str]`?"
    h: (int, str),
    i: (),  # error: [invalid-type-form] "Tuple literals are not allowed in this context in a type expression: Did you mean `tuple[()]`?"
):
    reveal_type(a)  # revealed: Unknown
    reveal_type(b)  # revealed: Unknown
    reveal_type(c)  # revealed: Unknown
    reveal_type(d)  # revealed: Unknown
    reveal_type(e)  # revealed: Unknown
    reveal_type(f)  # revealed: Unknown
    reveal_type(g)  # revealed: Unknown
    reveal_type(h)  # revealed: Unknown
    reveal_type(i)  # revealed: Unknown

# error: [invalid-type-form] "List literals are not allowed in this context in a type expression: Did you mean `list[int]`?"
class name_0[name_2: [int]]:
    pass

# error: [invalid-type-form] "List literals are not allowed in this context in a type expression"
# error: [invalid-type-form] "Dict literals are not allowed in type expressions"
class name_4[name_1: [{}]]:
    pass

Diagnostics for common errors

Module-literal used when you meant to use a class from that module

It's pretty common in Python to accidentally use a module-literal type in a type expression when you meant to use a class by the same name that comes from that module. We emit a nice subdiagnostic for this case:

foo.py:

import datetime

def f(x: datetime): ...  # error: [invalid-type-form]

PIL/Image.py:

class Image: ...

bar.py:

from PIL import Image

def g(x: Image): ...  # error: [invalid-type-form]

List-literal used when you meant to use a list or tuple

def _(
    x: [int],  # error: [invalid-type-form]
) -> [int]:  # error: [invalid-type-form]
    return x

def _(
    x: [int, str],  # error: [invalid-type-form]
) -> [int, str]:  # error: [invalid-type-form]
    return x

Tuple-literal used when you meant to use a tuple

def _(
    x: (),  # error: [invalid-type-form]
) -> ():  # error: [invalid-type-form]
    return x

def _(
    x: (int,),  # error: [invalid-type-form]
) -> (int,):  # error: [invalid-type-form]
    return x

def _(
    x: (int, str),  # error: [invalid-type-form]
) -> (int, str):  # error: [invalid-type-form]
    return x

10 KiB Raw Blame History