## Summary
This PR implements support for providing LSP client settings.
The complementary PR in the ty VS Code extension:
astral-sh/ty-vscode#106.
Notes for the previous iteration of this PR is in
https://github.com/astral-sh/ruff/pull/19614#issuecomment-3136477864
(click on "Details").
Specifically, this PR splits the client settings into 3 distinct groups.
Keep in mind that these groups are not visible to the user, they're
merely an implementation detail. The groups are:
1. `GlobalOptions` - these are the options that are global to the
language server and will be the same for all the workspaces that are
handled by the server
2. `WorkspaceOptions` - these are the options that are specific to a
workspace and will be applied only when running any logic for that
workspace
3. `InitializationOptions` - these are the options that can be specified
during initialization
The initialization options are a superset that contains both the global
and workspace options flattened into a 1-dimensional structure. This
means that the user can specify any and all fields present in
`GlobalOptions` and `WorkspaceOptions` in the initialization options in
addition to the fields that are _specific_ to initialization options.
From the current set of available settings, following are only available
during initialization because they are required at that time, are static
during the runtime of the server and changing their values require a
restart to take effect:
- `logLevel`
- `logFile`
And, following are available under `GlobalOptions`:
- `diagnosticMode`
And, following under `WorkspaceOptions`:
- `disableLanguageServices`
- `pythonExtension` (Python environment information that is populated by
the ty VS Code extension)
### `workspace/configuration`
This request allows server to ask the client for configuration to a
specific workspace. But, this is only supported by the client that has
the `workspace.configuration` client capability set to `true`. What to
do for clients that don't support pulling configurations?
In that case, the settings needs to be provided in the initialization
options and updating the values of those settings can only be done by
restarting the server. With the way this is implemented, this means that
if the client does not support pulling workspace configuration then
there's no way to specify settings specific to a workspace. Earlier,
this would've been possible by providing an array of client options with
an additional field which specifies which workspace the options belong
to but that adds complexity and clients that actually do not support
`workspace/configuration` would usually not support multiple workspaces
either.
Now, for the clients that do support this, the server will initiate the
request to get the configuration for all the workspaces at the start of
the server. Once the server receives these options, it will resolve them
for each workspace as follows:
1. Combine the client options sent during initialization with the
options specific to the workspace creating the final client options
that's specific to this workspace
2. Create a global options by combining the global options from (1) for
all workspaces which in turn will also combine the global options sent
during initialization
The global options are resolved into the global settings and are
available on the `Session` which is initialized with the default global
settings. The workspace options are resolved into the workspace settings
and are available on the respective `Workspace`.
The `SessionSnapshot` contains the global settings while the document
snapshot contains the workspace settings. We could add the global
settings to the document snapshot but that's currently not needed.
### Document diagnostic dynamic registration
Currently, the document diagnostic server capability is created based on
the `diagnosticMode` sent during initialization. But, that wouldn't
provide us with the complete picture. This means the server needs to
defer registering the document diagnostic capability at a later point
once the settings have been resolved.
This is done using dynamic registration for clients that support it. For
clients that do not support dynamic registration for document diagnostic
capability, the server advertises itself as always supporting workspace
diagnostics and work done progress token.
This dynamic registration now allows us to change the server capability
for workspace diagnostics based on the resolved `diagnosticMode` value.
In the future, once `workspace/didChangeConfiguration` is supported, we
can avoid the server restart when users have changed any client
settings.
## Test Plan
Add integration tests and recorded videos on the user experience in
various editors:
### VS Code
For VS Code users, the settings experience is unchanged because the
extension defines it's own interface on how the user can specify the
server setting. This means everything is under the `ty.*` namespace as
usual.
https://github.com/user-attachments/assets/c2e5ba5c-7617-406e-a09d-e397ce9c3b93
### Zed
For Zed, the settings experience has changed. Users can specify settings
during initialization:
```json
{
"lsp": {
"ty": {
"initialization_options": {
"logLevel": "debug",
"logFile": "~/.cache/ty.log",
"diagnosticMode": "workspace",
"disableLanguageServices": true
}
},
}
}
```
Or, can specify the options under the `settings` key:
```json
{
"lsp": {
"ty": {
"settings": {
"ty": {
"diagnosticMode": "openFilesOnly",
"disableLanguageServices": true
}
},
"initialization_options": {
"logLevel": "debug",
"logFile": "~/.cache/ty.log"
}
},
}
}
```
The `logLevel` and `logFile` setting still needs to go under the
initialization options because they're required by the server during
initialization.
We can remove the nesting of the settings under the "ty" namespace by
updating the return type of
db9ea0cdfd/src/tychecker.rs (L45-L49)
to be wrapped inside `ty` directly so that users can avoid doing the
double nesting.
There's one issue here which is that if the `diagnosticMode` is
specified in both the initialization option and settings key, then the
resolution is a bit different - if either of them is set to be
`workspace`, then it wins which means that in the following
configuration, the diagnostic mode is `workspace`:
```json
{
"lsp": {
"ty": {
"settings": {
"ty": {
"diagnosticMode": "openFilesOnly"
}
},
"initialization_options": {
"diagnosticMode": "workspace"
}
},
}
}
```
This behavior is mainly a result of combining global options from
various workspace configuration results. Users should not be able to
provide global options in multiple workspaces but that restriction
cannot be done on the server side. The ty VS Code extension restricts
these global settings to only be set in the user settings and not in
workspace settings but we do not control extensions in other editors.
https://github.com/user-attachments/assets/8e2d6c09-18e6-49e5-ab78-6cf942fe1255
### Neovim
Same as in Zed.
### Other
Other editors that do not support `workspace/configuration`, the users
would need to provide the server settings during initialization.
## Summary
This was originally stacked on #19129, but some of the changes I made
for JSON also impacted the Azure format, so I went ahead and combined
them. The main changes here are:
- Implementing `FileResolver` for Ruff's `EmitterContext`
- Adding `FileResolver::notebook_index` and `FileResolver::is_notebook`
methods
- Adding a `DisplayDiagnostics` (with an "s") type for rendering a group
of diagnostics at once
- Adding `Azure`, `Json`, and `JsonLines` as new `DiagnosticFormat`s
I tried a couple of alternatives to the `FileResolver::notebook` methods
like passing down the `NotebookIndex` separately and trying to reparse a
`Notebook` from Ruff's `SourceFile`. The latter seemed promising, but
the `SourceFile` only stores the concatenated plain text of the
notebook, not the re-parsable JSON. I guess the current version is just
a variation on passing the `NotebookIndex`, but at least we can reuse
the existing `resolver` argument. I think a lot of this can be cleaned
up once Ruff has its own actual file resolver.
As suggested, I also tried deleting the corresponding `Emitter` files in
`ruff_linter`, but it doesn't look like git was able to follow this as a
rename. It did, however, track that the tests were moved, so the
snapshots should be easy to review.
## Test Plan
Existing Ruff tests ported to tests in `ruff_db`. I think some other
existing ruff tests also cover parts of this refactor.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Print the [new salsa memory usage
dumps](https://github.com/astral-sh/ruff/pull/18928) in mypy primer CI
runs to help us catch memory regressions. The numbers are rounded to the
nearest power of 1.1 (about a 5% threshold between buckets) to avoid overly sensitive diffs.
## Summary
Setting `TY_MEMORY_REPORT=full` will generate and print a memory usage
report to the CLI after a `ty check` run:
```
=======SALSA STRUCTS=======
`Definition` metadata=7.24MB fields=17.38MB count=181062
`Expression` metadata=4.45MB fields=5.94MB count=92804
`member_lookup_with_policy_::interned_arguments` metadata=1.97MB fields=2.25MB count=35176
...
=======SALSA QUERIES=======
`File -> ty_python_semantic::semantic_index::SemanticIndex`
metadata=11.46MB fields=88.86MB count=1638
`Definition -> ty_python_semantic::types::infer::TypeInference`
metadata=24.52MB fields=86.68MB count=146018
`File -> ruff_db::parsed::ParsedModule`
metadata=0.12MB fields=69.06MB count=1642
...
=======SALSA SUMMARY=======
TOTAL MEMORY USAGE: 577.61MB
struct metadata = 29.00MB
struct fields = 35.68MB
memo metadata = 103.87MB
memo fields = 409.06MB
```
Eventually, we should integrate these numbers into CI in some form. The
one limitation currently is that heap allocations in salsa structs (e.g.
interned values) are not tracked, but memoized values should have full
coverage. We may also want a peak memory usage counter (that accounts
for non-salsa memory), but that is relatively simple to profile manually
(e.g. `time -v ty check`) and would require a compile-time option to
avoid runtime overhead.
## Summary
https://github.com/astral-sh/ty/issues/214 will require a couple
invasive changes that I would like to get merged even before garbage
collection is fully implemented (to avoid rebasing):
- `ParsedModule` can no longer be dereferenced directly. Instead you
need to load a `ParsedModuleRef` to access the AST, which requires a
reference to the salsa database (as it may require re-parsing the AST if
it was collected).
- `AstNodeRef` can only be dereferenced with the `node` method, which
takes a reference to the `ParsedModuleRef`. This allows us to encode the
fact that ASTs do not live as long as the database and may be collected
as soon a given instance of a `ParsedModuleRef` is dropped. There are a
number of places where we currently merge the `'db` and `'ast`
lifetimes, so this requires giving some types/functions two separate
lifetime parameters.
## Summary
Updates the `--python` flag to accept Python executables in virtual
environments. Notably, we do not query the executable and it _must_ be
in a canonical location in a virtual environment. This is pretty naive,
but solves for the trivial case of `ty check --python .venv/bin/python3`
which will be a common mistake (and `ty check --python $(which python)`)
I explored this while trying to understand Python discovery in ty in
service of https://github.com/astral-sh/ty/issues/272, I'm not attached
to it, but figure it's worth sharing.
As an alternative, we can add more variants to the
`SearchPathValidationError` and just improve the _error_ message, i.e.,
by hinting that this looks like a virtual environment and suggesting the
concrete alternative path they should provide. We'll probably want to do
that for some other cases anyway (e.g., `3.13` as described in the
linked issue)
This functionality is also briefly mentioned in
https://github.com/astral-sh/ty/issues/193
Closes https://github.com/astral-sh/ty/issues/318
## Test Plan
e.g.,
```
uv run ty check --python .venv/bin/python3
```
needs test coverage still
Summary
--
This was suggested on Discord, I hope this is roughly what we had in
mind. I took the message from the ty README, but I'm more than happy to
update it. Otherwise I just tried to mimic the appearance of the `ruff
analyze graph` warning (although I'm realizing now the whole text is
bold for ruff).
Test Plan
--
New warnings in the CLI tests. I thought this might be undesirable but
it looks like uv did the same thing
(https://github.com/astral-sh/uv/pull/6166).
## Summary
Adds a simple progress bar for the `ty check` CLI command. The style is
taken from uv, and like uv the bar is always shown - for smaller
projects it is fast enough that it isn't noticeable. We could
alternatively hide it completely based on some heuristic for the number
of files, or only show it after some amount of time.
I also disabled it when `--watch` is passed, cancelling inflight checks
was leading to zombie progress bars. I think we can fix this by using
[`MultiProgress`](https://docs.rs/indicatif/latest/indicatif/struct.MultiProgress.html)
and managing all the bars globally, but I left that out for now.
Resolves https://github.com/astral-sh/ty/issues/98.
## Summary
This PR is a first step toward integration of the new `Diagnostic` type
into ruff. There are two main changes:
- A new `UnifiedFile` enum wrapping `File` for red-knot and a
`SourceFile` for ruff
- ruff's `Message::SyntaxError` variant is now a `Diagnostic` instead of
a `SyntaxErrorMessage`
The second of these changes was mostly just a proof of concept for the
first, and it went pretty smoothly. Converting `DiagnosticMessage`s will
be most of the work in replacing `Message` entirely.
## Test Plan
Existing tests, which show no changes.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Resolves#15502.
`ty generate-shell-completion` now works in a similar manner to `ruff
generate-shell-completion`.
## Test Plan
Manually:
<details>
```shell
$ cargo run --package ty generate-shell-completion nushell
module completions {
# An extremely fast Python type checker.
export extern ty [
--help(-h) # Print help
--version(-V) # Print version
]
# ...
}
export use completions *
```
</details>
e.g.,
```
❯ uv run -q -- ty -V
ty 0.0.0-alpha.4 (08881edba 2025-05-05)
❯ uv run -q -- ty --version
ty 0.0.0-alpha.4 (08881edba 2025-05-05)
```
Previously, this just displayed `ty 0.0.0` because it didn't use our
custom version implementation. We no longer have a short version —
matching the interface in uv. We could add a variant for it, if it seems
important to people. However, I think we found it more confusing than
not over there and didn't get any complaints about the change.
Closes https://github.com/astral-sh/ty/issues/54
Extends https://github.com/astral-sh/ruff/pull/17866, using
`dist-workspace.toml` as a source of truth for versions to enable
version retrieval in distributions that are not Git repositories (i.e.,
Python source distributions and source tarballs consumed by Linux
distros).
I retain the Git tag lookup from
https://github.com/astral-sh/ruff/pull/17866 as a fallback — it seems
harmless, but we could drop it to simplify things here.
I confirmed this works from the repository as well as Python source and
binary distributions:
```
❯ uv run --refresh-package ty --reinstall-package ty -q -- ty version
ty 0.0.1-alpha.1+5 (2eadc9e61 2025-05-05)
❯ uv build
...
❯ uvx --from ty@dist/ty-0.0.0a1.tar.gz --no-cache -q -- ty version
ty 0.0.1-alpha.1
❯ uvx --from ty@dist/ty-0.0.0a1-py3-none-macosx_11_0_arm64.whl -q -- ty version
ty 0.0.1-alpha.1
```
Requires https://github.com/astral-sh/ty/pull/36
cc @Gankra and @MichaReiser for review.
Currently, `ty version` pulls its information from the Ruff repository —
but we want this to pull from the repository in the directory _above_
when Ruff is a submodule.
I tested this in the `ty` repository after tagging an arbitrary commit:
```
❯ uv run --refresh-package ty --reinstall-package ty ty version
Built ty @ file:///Users/zb/workspace/ty
Uninstalled 1 package in 2ms
Installed 1 package in 1ms
ty 0.0.0+3 (34253b1d4 2025-05-05)
```
We also use the last Git tag as the source of truth for the version,
instead of the crate version. However, we'll need a way to set the
version for releases still, as the tag is published _after_ the build.
We can either tag early (without pushing the tag to the remote), or add
another environment variable. (**Note, this approach is changed in a
follow-up. See https://github.com/astral-sh/ruff/pull/17868**)
From this repository, the version will be `unknown`:
```
❯ cargo run -q --bin ty -- version
ty unknown
```
We could add special handling like... `ty unknown (ruff@...)` but I see
that as a secondary goal.
Closes https://github.com/astral-sh/ty/issues/5
The reviewer situation in this repository is unhinged, cc @Gankra and
@MichaReiser for review.
## Summary
closes#17472
This is obviously just a band-aid solution to this problem (in that you
can always make your [pathological
inputs](28994edd82/sympy/polys/numberfields/resolvent_lookup.py)
bigger and it will still crash), but I think this is not an unreasonable
change — even if we add more sophisticated solutions later. I tried
using `stacker` as suggested by @MichaReiser, and it works. But it's
unclear where exactly would be the right place to put it, and even for
the `sympy` problem, we would need to add it both in the semantic index
builder AST traversal and in type inference. Increasing the default
stack size for worker threads, as proposed here, doesn't solve the
underlying problem (that there is a hard limit), but it is more
universal in the sense that it is not specific to large binary-operator
expression chains.
To determine a reasonable stack size, I created files that look like
*right associative*:
```py
from typing import reveal_type
total = (1 + (1 + (1 + (1 + (… + 1)))))
reveal_type(total)
```
*left associative*
```py
from typing import reveal_type
total = 1 + 1 + 1 + 1 + … + 1
reveal_type(total)
```
with a variable amount of operands (`N`). I then chose the stack size
large enough to still be able to handle cases that existing type
checkers can not:
```
right
N = 20: mypy takes ~ 1min
N = 350: pyright crashes with a stack overflow (mypy fails with "too many nested parentheses")
N = 800: ty(main) infers Literal[800] instantly
N = 1000: ty(main) crashes with "thread '<unknown>' has overflowed its stack"
N = 7000: ty(this branch) infers Literal[7000] instantly
N = 8000+: ty(this branch) crashes
left
N = 300: pyright emits "Maximum parse depth exceeded; break expression into smaller sub-expressions"
total is inferred as Unknown
N = 5500: mypy crashes with "INTERNAL ERROR"
N = 2500: ty(main) infers Literal[2500] instantly
N = 3000: ty(main) crashes with "thread '<unknown>' has overflowed its stack"
N = 22000: ty(this branch) infers Literal[22000] instantly
N = 23000+: ty(this branch) crashes
```
## Test Plan
New regression test.
Micha Reiser
Dhruv Manilawala
Leandro Braga
Zanie Blue
Brent Westbrook
Charlie Marsh
Ibraheem Ahmed
Alex Waygood
justin
Jo
Sumana Harihareswara
InSync
David Peter