## Summary
This PR adds the possibility to write mdtests that specify external
dependencies in a `project` section of TOML blocks. For example, here is
a test that makes sure that we understand Pydantic's dataclass-transform
setup:
````markdown
```toml
[environment]
python-version = "3.12"
python-platform = "linux"
[project]
dependencies = ["pydantic==2.12.2"]
```
```py
from pydantic import BaseModel
class User(BaseModel):
id: int
name: str
user = User(id=1, name="Alice")
reveal_type(user.id) # revealed: int
reveal_type(user.name) # revealed: str
# error: [missing-argument] "No argument provided for required parameter
`name`"
invalid_user = User(id=2)
```
````
## How?
Using the `python-version` and the `dependencies` fields from the
Markdown section, we generate a `pyproject.toml` file, write it to a
temporary directory, and use `uv sync` to install the dependencies into
a virtual environment. We then copy the Python source files from that
venv's `site-packages` folder to a corresponding directory structure in
the in-memory filesystem. Finally, we configure the search paths
accordingly, and run the mdtest as usual.
I fully understand that there are valid concerns here:
* Doesn't this require network access? (yes, it does)
* Is this fast enough? (`uv` caching makes this almost unnoticeable,
actually)
* Is this deterministic? ~~(probably not, package resolution can depend
on the platform you're on)~~ (yes, hopefully)
For this reason, this first version is opt-in, locally. ~~We don't even
run these tests in CI (even though they worked fine in a previous
iteration of this PR).~~ You need to set `MDTEST_EXTERNAL=1`, or use the
new `-e/--enable-external` command line option of the `mdtest.py`
runner. For example:
```bash
# Skip mdtests with external dependencies (default):
uv run crates/ty_python_semantic/mdtest.py
# Run all mdtests, including those with external dependencies:
uv run crates/ty_python_semantic/mdtest.py -e
# Only run the `pydantic` tests. Use `-e` to make sure it is not skipped:
uv run crates/ty_python_semantic/mdtest.py -e pydantic
```
## Why?
I believe that this can be a useful addition to our testing strategy,
which lies somewhere between ecosystem tests and normal mdtests.
Ecosystem tests cover much more code, but they have the disadvantage
that we only see second- or third-order effects via diagnostic diffs. If
we unexpectedly gain or lose type coverage somewhere, we might not even
notice (assuming the gradual guarantee holds, and ecosystem code is
mostly correct). Another disadvantage of ecosystem checks is that they
only test checked-in code that is usually correct. However, we also want
to test what happens on wrong code, like the code that is momentarily
written in an editor, before fixing it. On the other end of the spectrum
we have normal mdtests, which have the disadvantage that they do not
reflect the reality of complex real-world code. We experience this
whenever we're surprised by an ecosystem report on a PR.
That said, these tests should not be seen as a replacement for either of
these things. For example, we should still strive to write detailed
self-contained mdtests for user-reported issues. But we might use this
new layer for regression tests, or simply as a debugging tool. It can
also serve as a tool to document our support for popular third-party
libraries.
## Test Plan
* I've been locally using this for a couple of weeks now.
* `uv run crates/ty_python_semantic/mdtest.py -e`
## Summary
Pulls in an ecosystem-analyzer change with a few updates to the diff
report:
* Breakdown of added/removed/changed diagnostics by project
* Option to filter diagnostics by project
* Small button to copy a file path to the clipboard
* `(-R +A ~C)` indicators in the filter dropdowns (removed, added,
changed)
* More concise layout, less scrolling
## Test Plan
Tested on https://github.com/astral-sh/ruff/pull/21553 =>
https://david-generic-implicit-alias.ecosystem-663.pages.dev/diff
## Summary
This PR sets up CI jobs to run ty from the `main` branch on the files
and subdirectories in our `scripts` directory
## Test Plan
Both these commands pass for me locally:
- `uv run --project=./scripts cargo run -p ty check --project=./scripts`
- `uv run --project=./scripts/ty_benchmark cargo run -p ty check
--project=./scripts/ty_benchmark`
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| actions/checkout | action | digest | `ff7abcd` -> `c2d88d3` |
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Configuration
📅 **Schedule**: Branch creation - "before 4am on Monday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/astral-sh/ruff).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0Mi4xNi4xIiwidXBkYXRlZEluVmVyIjoiNDIuMTYuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiaW50ZXJuYWwiXX0=-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
## Summary
Further improve subscript assignment diagnostics, especially for
`dict`s:
```py
config: dict[str, int] = {}
config["retries"] = "three"
```
<img width="1276" height="274" alt="image"
src="https://github.com/user-attachments/assets/9762c733-8d1c-4a57-8c8a-99825071dc7d"
/>
I have many more ideas, but this looks like a reasonable first step.
Thank you @AlexWaygood for some of the suggestions here.
## Test Plan
Update tests
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[cargo-bins/cargo-binstall](https://redirect.github.com/cargo-bins/cargo-binstall)
| action | patch | `v1.15.10` -> `v1.15.11` |
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>cargo-bins/cargo-binstall (cargo-bins/cargo-binstall)</summary>
###
[`v1.15.11`](https://redirect.github.com/cargo-bins/cargo-binstall/releases/tag/v1.15.11)
[Compare
Source](https://redirect.github.com/cargo-bins/cargo-binstall/compare/v1.15.10...v1.15.11)
*Binstall is a tool to fetch and install Rust-based executables as
binaries. It aims to be a drop-in replacement for `cargo install` in
most cases. Install it today with `cargo install cargo-binstall`, from
the binaries below, or if you already have it, upgrade with `cargo
binstall cargo-binstall`.*
##### In this release:
- Fix binstalk-downloader cannot decode some zip files on macos (x64 and
arm64) platforms
([#​2049](https://redirect.github.com/cargo-bins/cargo-binstall/issues/2049)
[#​2362](https://redirect.github.com/cargo-bins/cargo-binstall/issues/2362))
- Fix grammer in `HELP.md` and `--help` output
([#​2357](https://redirect.github.com/cargo-bins/cargo-binstall/issues/2357)
[#​2359](https://redirect.github.com/cargo-bins/cargo-binstall/issues/2359))
- Update documentation link in Cargo.toml
([#​2355](https://redirect.github.com/cargo-bins/cargo-binstall/issues/2355))
##### Other changes:
- Upgrade dependencies
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "before 4am on Monday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/astral-sh/ruff).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS4xNTkuNCIsInVwZGF0ZWRJblZlciI6IjQxLjE1OS40IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJpbnRlcm5hbCJdfQ==-->
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
## Summary
This reduces the walltime benchmarks from 15m to 10m, and we should see an even bigger improvement once build caching kicks in, so I think it's worth the downsides.
## Summary
There have been some larger-scale updates to the conformance suite since
we introduced our CI job, so it seems sensible to bump the version of
the conformance suite to the latest state.
## Test plan
This is a bit awkward to test. Here is the diff of running ty on the
conformance suite before and after this bump. I filtered out line/column
information (`sed -re 's/\.py:[0-9]+:[0-9]+:/.py/'`) to avoid spurious
changes from content that has simply been moved around.
```diff
1,2c1
< fatal[panic] Panicked at /home/shark/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/cdd0b85/src/function/execute.rs:419:17 when checking `/home/shark/typing/conformance/tests/aliases_typealiastype.py`: `infer_definition_types(Id(1a99c)): execute: too many cycle iterations`
< src/type_checker.py error[unresolved-import] Cannot resolve imported module `tqdm`
---
> fatal[panic] Panicked at /home/shark/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/cdd0b85/src/function/execute.rs:419:17 when checking `/home/shark/typing/conformance/tests/aliases_typealiastype.py`: `infer_definition_types(Id(6e4c)): execute: too many cycle iterations`
205,206d203
< tests/constructors_call_metaclass.py error[type-assertion-failure] Argument does not have asserted type `Never`
< tests/constructors_call_metaclass.py error[missing-argument] No argument provided for required parameter `x` of function `__new__`
268a266,273
> tests/dataclasses_match_args.py error[type-assertion-failure] Argument does not have asserted type `tuple[Literal["x"]]`
> tests/dataclasses_match_args.py error[unresolved-attribute] Class `DC1` has no attribute `__match_args__`
> tests/dataclasses_match_args.py error[type-assertion-failure] Argument does not have asserted type `tuple[Literal["x"]]`
> tests/dataclasses_match_args.py error[unresolved-attribute] Class `DC2` has no attribute `__match_args__`
> tests/dataclasses_match_args.py error[type-assertion-failure] Argument does not have asserted type `tuple[Literal["x"]]`
> tests/dataclasses_match_args.py error[unresolved-attribute] Class `DC3` has no attribute `__match_args__`
> tests/dataclasses_match_args.py error[unresolved-attribute] Class `DC4` has no attribute `__match_args__`
> tests/dataclasses_match_args.py error[type-assertion-failure] Argument does not have asserted type `tuple[()]`
339a345
> tests/directives_assert_type.py error[type-assertion-failure] Argument does not have asserted type `Any`
424a431
> tests/generics_defaults.py error[type-assertion-failure] Argument does not have asserted type `Any`
520a528,529
> tests/generics_syntax_infer_variance.py error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `T@ShouldBeCovariant2 | Sequence[T@ShouldBeCovariant2]`
> tests/generics_syntax_infer_variance.py error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `int`
711a721
> tests/namedtuples_define_class.py error[too-many-positional-arguments] Too many positional arguments: expected 3, got 4
795d804
< tests/protocols_explicit.py error[invalid-attribute-access] Cannot assign to ClassVar `cm1` from an instance of type `Self@__init__`
822,823d830
< tests/qualifiers_annotated.py error[invalid-syntax] named expression cannot be used within a type annotation
< tests/qualifiers_annotated.py error[invalid-syntax] await expression cannot be used within a type annotation
922a930,953
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `Movie`: Unknown key "novel_adaptation"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `Movie`: Unknown key "year"
> tests/typeddicts_extra_items.py error[type-assertion-failure] Argument does not have asserted type `bool`
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `Movie`: Unknown key "novel_adaptation"
> tests/typeddicts_extra_items.py error[invalid-argument-type] Invalid argument to key "year" with declared type `int` on TypedDict `InheritedMovie`: value of type `None`
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `InheritedMovie`: Unknown key "other_extra_key"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `MovieEI`: Unknown key "year"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `MovieExtraInt`: Unknown key "year"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `MovieExtraStr`: Unknown key "description"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `MovieExtraInt`: Unknown key "year"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `NonClosedMovie`: Unknown key "year"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `ExtraMovie`: Unknown key "year"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `ExtraMovie`: Unknown key "language"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `ClosedMovie`: Unknown key "year"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `MovieExtraStr`: Unknown key "summary"
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `MovieExtraInt`: Unknown key "year"
> tests/typeddicts_extra_items.py error[invalid-assignment] Object of type `dict[Unknown | str, Unknown | str | int]` is not assignable to `Mapping[str, int]`
> tests/typeddicts_extra_items.py error[type-assertion-failure] Argument does not have asserted type `list[tuple[str, int | str]]`
> tests/typeddicts_extra_items.py error[type-assertion-failure] Argument does not have asserted type `list[int | str]`
> tests/typeddicts_extra_items.py error[unresolved-attribute] Object of type `IntDict` has no attribute `clear`
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `IntDictWithNum`: Unknown key "bar" - did you mean "num"?
> tests/typeddicts_extra_items.py error[type-assertion-failure] Argument does not have asserted type `tuple[str, int]`
> tests/typeddicts_extra_items.py error[invalid-key] Cannot access `IntDictWithNum` with a key of type `str`. Only string literals are allowed as keys on TypedDicts.
> tests/typeddicts_extra_items.py error[invalid-key] Invalid key for TypedDict `IntDictWithNum` of type `str`
950c981
< Found 949 diagnostics
---
> Found 980 diagnostics
```
## Summary
Also bumps `cargo dist` to 0.30, and moves us
back to the upstream copy of `dist` now that
the latest version has integrated our fork's
patches.
## Test Plan
See what happens in CI 🙂
---------
Signed-off-by: William Woodruff <william@astral.sh>
David Peter
mahiro
William Woodruff
Micha Reiser
renovate[bot]
Alex Waygood
github-actions[bot]
William Woodruff
Ibraheem Ahmed
Dan Parizher