4df9e07a79
## Summary This PR adds a benchmarking script for the formatter, which benchmarks the Ruff formatter against Black, yapf, and autopep8. Three benchmarks are included: 1. Format everything. 2. Format everything, but use a single thread. 3. Format everything, but `--check` (don't write to disk). There's some nuance in figuring out the right combination of arguments to each command, but the _main_ nuance is to ensure that we always run the given formatter (and modify the target repo in-place) prior to benchmarking it, so that the formatters aren't disadvantaged by the existing formatting of the target repo. (E.g.: prior to benchmarking Black's preview style, we need to make sure we format the target repo with Black's preview style; otherwise, preview style appears much slower.) Part of https://github.com/astral-sh/ruff/issues/7309.
benchmarks
Utilities for benchmarking Ruff.
Getting Started
Run ./scripts/benchmarks/run.sh to clone the benchmarking target (CPython).
If you're looking to benchmark Ruff against other tools, you'll also need to run poetry install
to create a virtual environment with the required dependencies.
Running Benchmarks
Run ./scripts/benchmarks/run.sh to run Ruff over the target repo (CPython). The
./scripts/benchmarks folder contains a few other benchmarks (e.g., scripts/benchmarks/run_comparisons.sh
compares Ruff to a variety of other tools).
Generating Plots
The Vega specification for the benchmark plot depicted in the root README can be found at
scripts/benchmarks/graph-spec.json. You can render this JSON spec in the Vega Editor.
The images seen in the README are generated by exporting the rendered Vega spec as SVG (at around
688px wide) and manually bolding the Ruff title and benchmark time. The dark mode variant is
generated by changing the fill from fill="#333333" to fill="#C9D1D9".