mirror of
https://github.com/astral-sh/ruff
synced 2026-01-11 00:24:13 -05:00
fc94857a20
Closes #7239 - Refactors `scripts/check_ecosystem.py` into a new Python project at `python/ruff-ecosystem` - Includes [documentation](https://github.com/astral-sh/ruff/blob/zanie/ecosystem-format/python/ruff-ecosystem/README.md) now - Provides a `ruff-ecosystem` CLI - Fixes bug where `ruff check` report included "fixable" summary line - Adds truncation to `ruff check` reports - Otherwise we often won't see the `ruff format` reports - The truncation uses some very simple heuristics and could be improved in the future - Identifies diagnostic changes that occur just because a violation's fix available changes - We still show the diff for the line because it's could matter _where_ this changes, but we could improve this - Similarly, we could improve detection of diagnostic changes where just the message changes - Adds support for JSON ecosystem check output - I added this primarily for development purposes - If there are no changes, only errors while processing projects, we display a different summary message - When caching repositories, we now checkout the requested ref - Adds `ruff format` reports, which format with the baseline then the use `format --diff` to generate a report - Runs all CI jobs when the CI workflow is changed ## Known problems - Since we must format the project to get a baseline, the permalink line numbers do not exactly correspond to the correct range - This looks... hard. I tried using `git diff` and some wonky hunk matching to recover the original line numbers but it doesn't seem worth it. I think we should probably commit the formatted changes to a fork or something if we want great results here. Consequently, I've just used the start line instead of a range for now. - I don't love the comment structure — it'd be nice, perhaps, to have separate headings for the linter and formatter. - However, the `pr-comment` workflow is an absolute pain to change because it runs _separately_ from this pull request so I if I want to make edits to it I can only test it via manual workflow dispatch. - Lines are not printed "as we go" which means they're all held in memory, presumably this would be a problem for large-scale ecosystem checks - We are encountering a hard limit with the maximum comment length supported by GitHub. We will need to move the bulk of the report elsewhere. ## Future work - Update `ruff-ecosystem` to support non-default projects and `check_ecosystem_all.py` behavior - Remove existing ecosystem check scripts - Add preview mode toggle (#8076) - Add a toggle for truncation - Add hints for quick reproduction of runs locally - Consider parsing JSON output of Ruff instead of using regex to parse the text output - Links to project repositories should use the commit hash we checked against - When caching repositories, we should pull the latest changes for the ref - Sort check diffs by path and rule code only (changes in messages should not change order) - Update check diffs to distinguish between new violations and changes in messages - Add "fix" diffs - Remove existing formatter similarity reports - On release pull request, compare to the previous tag instead --------- Co-authored-by: konsti <konstin@mailbox.org>
1.1 KiB
1.1 KiB
ruff-ecosystem
Compare lint and format results for two different ruff versions (e.g. main and a PR) on real world projects.
Installation
From the Ruff project root, install with pip:
pip install -e ./python/ruff-ecosystem
Usage
ruff-ecosystem <check | format> <baseline executable> <comparison executable>
Note executable paths may be absolute, relative to the current working directory, or will be looked up in the current Python environment and PATH.
Run ruff check ecosystem checks comparing your debug build to your system Ruff:
ruff-ecosystem check ruff "./target/debug/ruff"
Run ruff format ecosystem checks comparing your debug build to your system Ruff:
ruff-ecosystem format ruff "./target/debug/ruff"
Development
When developing, it can be useful to set the --pdb flag to drop into a debugger on failure:
ruff-ecosystem check ruff "./target/debug/ruff" --pdb
You can also provide a path to cache checkouts to speed up repeated runs:
ruff-ecosystem check ruff "./target/debug/ruff" --cache ./repos