From 58bc0694205fcfad0e5fdb5834ce2c032c0e897f Mon Sep 17 00:00:00 2001 From: Micha Reiser Date: Fri, 16 Feb 2024 15:27:32 +0100 Subject: [PATCH] Add troubleshooting section to benchmarks guide (#1485) ## Summary It took me a while to figure out why the benchmarks are extremelly flaky (and slow) on my machine. I documented my finding in the benchmarks.md file in case someone else runs into the same problems as I --- BENCHMARKS.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index cb3cf7182..5d5c6b053 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -104,3 +104,10 @@ You need to install the [Roboto Font](https://fonts.google.com/specimen/Roboto) The inclusion of this `BENCHMARKS.md` file was inspired by the excellent benchmarking documentation in [Orogene](https://github.com/orogene/orogene/blob/472e481b4fc6e97c2b57e69240bf8fe995dfab83/BENCHMARKS.md). + + +## Troubleshooting + +### Flaky benchmarks + +If you're seeing high variance when running the cold benchmarks, then it's likely that you're running into throttling or DDoS prevention from your ISP. In that case, ISPs forcefully terminate TCP connections with a TCP reset. We believe this is due to the benchmarks making the exact same requests in a very short time (especially true for `uv`). A possible workaround is to connect to VPN to bypass your ISPs filtering mechanism.