048f02fb6b
The implementation here is (to me) surprisingly complicated. The main complications are: 1. Trying to limit the redundant detection to as few of the symbols we extract as possible. In particular, while I haven't done benchmarking on this, I perceive the redundancy detection to be somewhat expensive and auto-import can return lots of symbols. So we're careful to only do this extra checking on (typically) small groups of symbols that could possibly be merged. 2. Even by restricting our work, this merging process could still be called quite a bit. (Thousands of times in my "standard data scienc-y test environment.") So I went out of my way to amortize allocs. 3. Re-exports can form a chain and we want to find all of them. 4. We (probably) don't want to remove redundant re-exports unless they share the same top-level module. Otherwise, e.g., a library that re-exports another library's symbols could have all of its re-exports dropped. 5. We want to only keep the top-most re-exports, and there may be multiple such re-exports. So we keep all of them. 6. We can't assume anything about the relationship of re-exports and the original definition. A re-export could be deeper in the module hierarchy than the original definition or above it.
This directory contains a framework for evaluating completion suggestions returned by the ty LSP.
Running an evaluation
To run a full evaluation, run the ty_completion_eval crate with the
all command from the root of this repository:
cargo run --profile profiling --package ty_completion_eval -- all
The output should look like this:
Finished `release` profile [optimized] target(s) in 0.09s
Running `target/release/ty_completion_eval all`
mean reciprocal rank: 0.20409790112917506
MRR exceeds threshold of 0.001
If you want to look at the results of each individual evaluation task, you can ask the evaluation to write CSV data that contains the rank of the expected answer in each completion request:
cargo r --profile profiling -p ty_completion_eval -- all --tasks ./crates/ty_completion_eval/completion-evaluation-tasks.csv
To debug a specific task and look at the actual results, use the show-one
command:
cargo r -q -p ty_completion_eval show-one higher-level-symbols-preferred --index 1
(The --index flag is only needed if there are multiple <CURSOR> directives in the same file.)
Has output that should look like this:
ZQZQZQ_SOMETHING_IMPORTANT (*, 1/31)
__annotations__
__class__
__delattr__
__dict__
__dir__
__doc__
__eq__
__file__
__format__
__getattr__
__getattribute__
__getstate__
__hash__
__init__
__init_subclass__
__loader__
__module__
__name__
__ne__
__new__
__package__
__path__
__reduce__
__reduce_ex__
__repr__
__setattr__
__sizeof__
__spec__
__str__
__subclasshook__
-----
found 31 completions
The expected answer is marked with a *. The higher the rank, the better. In this example, the
rank is perfect. Note that the expected answer may not always appear in the completion results!
(Which is considered the worst possible outcome by this evaluation framework.)
Evaluation model
This evaluation is based on mean reciprocal rank (MRR). That is, it assumes
that for every evaluation task (i.e., a single completion request) there is
precisely one correct answer. The higher the correct answer appears in each
completion request, the better. The mean reciprocal rank is computed as the
average of 1/rank across all evaluation tasks. The higher the mean reciprocal
rank, the better.
The evaluation starts by preparing its truth data, which is contained in the ./truth directory.
Within ./truth is a list of Python projects. Every project contains one or more <CURSOR>
directives. Each <CURSOR> directive corresponds to an instruction to initiate a completion
request at that position. For example:
class Foo:
def frobnicate(self): pass
foo = Foo()
foo.frob<CURSOR: frobnicate>
The above example says that completions should be requested immediately after foo.frob
and that the expected answer is frobnicate.
When testing auto-import, one should also include the module in the expected answer. For example:
RegexFl<CURSOR: re.RegexFlag>
Settings for completion requests can be configured via a completion.toml file within
each Python project directory.
When an evaluation is run, the truth data is copied to a temporary directory.
uv sync is then run within each directory to prepare it.
Continuous Integration
At time of writing (2025-10-07), an evaluation is run in CI. CI will fail if the MRR is
below a set threshold. When this occurs, it means that the evaluation's results have likely
gotten worse in some measurable way. Ideally, the way to fix this would be to fix whatever
regression occurred in ranking. One can follow the steps above to run an evaluation and
emit the individual task results in CSV format. This difference between this CSV data and
whatever is committed at ./crates/ty_completion_eval/completion-evaluation-tasks.csv should
point to where the regression occurs.
If the change is not a regression or is otherwise expected, then the MRR threshold can be
lowered. This requires changing how ty_completion_eval is executed within CI.
CI will also fail if the individual task results have changed. To make CI pass, you can just re-run the evaluation locally and commit the results:
cargo r --profile profiling -p ty_completion_eval -- all --tasks ./crates/ty_completion_eval/completion-evaluation-tasks.csv
CI fails in this case because it would be best to scrutinize the differences here. It's possible that the ranking has improved in some measurable way, for example. (Think of this as if it were a snapshot test.)