# Ruff Micro-benchmarks

Benchmarks for the different Ruff-tools.

## Run Benchmark

You can run the benchmarks with

```shell
cargo benchmark
```

## Benchmark driven Development

You can use `--save-baseline=<name>` to store an initial baseline benchmark (e.g. on `main`) and
then use `--benchmark=<name>` to compare against that benchmark. Criterion will print a message
telling you if the benchmark improved/regressed compared to that baseline.

```shell
# Run once on your "baseline" code
cargo benchmark --save-baseline=main

# Then iterate with
cargo benchmark --baseline=main
```

## PR Summary

You can use `--save-baseline` and `critcmp` to get a pretty comparison between two recordings.
This is useful to illustrate the improvements of a PR.

```shell
# On main
cargo benchmark --save-baseline=main

# After applying your changes
cargo benchmark --save-baseline=pr

critcmp main pr
```

You must install [`critcmp`](https://github.com/BurntSushi/critcmp) for the comparison.

```bash
cargo install critcmp
```

## Tips

- Use `cargo benchmark <filter>` to only run specific benchmarks. For example: `cargo benchmark linter/pydantic`
  to only run the pydantic tests.
- Use `cargo benchmark --quiet` for a more cleaned up output (without statistical relevance)
- Use `cargo benchmark --quick` to get faster results (more prone to noise)

## Profiling

### Linux

Install `perf` and build `ruff_benchmark` with the `release-debug` profile and then run it with perf

```shell
cargo bench -p ruff_benchmark --no-run --profile=release-debug && perf record -g -F 9999 cargo bench -p ruff_benchmark --profile=release-debug -- --profile-time=1
```

Then convert the recorded profile

```shell
perf script -F +pid > /tmp/test.perf
```

You can now view the converted file with [firefox profiler](https://profiler.firefox.com/)

You can find a more in-depth guide [here](https://profiler.firefox.com/docs/#/./guide-perf-profiling)

### Mac

Install [`cargo-instruments`](https://crates.io/crates/cargo-instruments):

```shell
cargo install cargo-instruments
```

Then run the profiler with

```shell
cargo instruments -t time --bench linter --profile release-debug -p ruff_benchmark -- --profile-time=1
```

- `-t`: Specifies what to profile. Useful options are `time` to profile the wall time and `alloc`
  for profiling the allocations.
- You may want to pass an additional filter to run a single test file