Alexander J. Semenuk
5d5e35fb9b
## Problem Windows builds break with a current local toolchain (Scoop LLVM 22.1.8, CMake 4.4.0, VS 2026), in two independent ways: 1. The build stops at curl's deliberate guard: `#error "no non-blocking method was found/used/set"` in `third-party/curl/lib/nonblock.c`. 2. From the second configure onward, `cmake --build` re-runs CMake in an endless loop (observed 42 consecutive reconfigure cycles in a single build). Likely the same mechanism behind the "endlessly building" VS 2026 note in `docs/setup/dev/vs.md`. ## Root cause 1. `third-party/curl/CMake/CurlTests.c` passes `int *` to `ioctlsocket()`, whose third parameter is `u_long *`. Clang 22 promotes `-Wincompatible-pointer-types` to a hard error in C, so the `HAVE_IOCTLSOCKET_FIONBIO` try_compile silently fails and `curl_config.h` never defines it. Upstream CI does not see this because the windows-2022 runner image ships an older LLVM. GCC 14 promotes the same warning to a hard error, which is very likely the `CurlTests.c.obj` failure reported from MSYS2 in open-goal/jak-project#3551. Upstream curl hit the identical problem with GCC 14 and fixed the probe in curl 8.8.0 (curl/curl#13578). 2. The root CMakeLists copies the build tree's `compile_commands.json` into `<src>/build/` for clangd using `configure_file()`, which registers its input as a configure dependency. CMake rewrites `compile_commands.json` late in every generation, after `CTestTestfile.cmake` and `cmake_install.cmake` (outputs of the same Ninja regen rule), so once the dependency is registered the rule is deterministically dirty and every `ninja` invocation re-runs CMake. A pristine first configure is safe (the file does not exist yet, so the `if(EXISTS ...)` guard skips the copy), which is why the loop looks machine- or IDE-specific. ## Fix 1. Per review, re-vendor `third-party/curl` at the `curl-8_21_0` tag (previously `curl-8_3_0`), which carries the upstream probe fix plus two years of upstream development; `vendor.yaml` updated to match. Adjustments the version jump forced: - curl 8.15 removed the native macOS Secure Transport backend (`CURL_USE_SECTRANSP`), so macOS now builds curl against OpenSSL like Linux. The two macOS workflows install Homebrew `openssl@3` and export `OPENSSL_ROOT_DIR` (keg-only), and the macOS setup docs gained the same two lines. - `CURL_BROTLI` / `CURL_ZSTD` switched to AUTO-detection in curl 8.10; pinned OFF to keep the previous no-compression behavior and avoid silently linking whatever the CI images happen to have. - curl's new top-level `BUILD_EXAMPLES` cache option (default ON) leaked into discord-rpc's identically named option and broke configure at a nonexistent `examples/send-presence` directory; pinned OFF ahead of the third-party subdirectories. The diff is dominated by the mechanical tag-tree swap under `third-party/curl` (linguist-vendored, collapsed in review). The hand-written changes are `CMakeLists.txt`, the two macOS workflows, `docs/setup/system/macos.md`, and `vendor.yaml`. 2. Swap `configure_file()` for `file(COPY ...)`: the same clangd copy with no configure dependency registered. (`file(COPY_FILE ... ONLY_IF_DIFFERENT)` would be cleaner still but requires CMake 3.21, above the declared `cmake_minimum_required(VERSION 3.10)`.) ## Test plan - [x] Fresh `cmake --preset Release-windows-clang` (LLVM 22, no cache seeding) completes and logs `Enabled SSL backends: Schannel`; the FIONBIO probe passes without the previous `#error` - [x] Full Windows Release build from scratch in the branch worktree (all 1422 targets) - [x] goalc-test suite: 1509 passed, 0 failed - [x] Second consecutive configure with `compile_commands.json` present: the regen rule in `build.ninja` has no `compile_commands.json` input; `<src>/build/compile_commands.json` is still refreshed for clangd - [x] Repeated `ninja` invocations after a full build no longer re-run CMake - [x] macOS Intel and ARM CI green (first exercise of the OpenSSL backend switch) --- I work off a self-hosted forge, so this GitHub account is quiet; the configure logs and ninja dirty-node traces from the investigation are available if anyone wants the raw data. (AI-assisted)
3.9 KiB
Vendored
Generated
Rate Limiting Transfers
Rate limiting a transfer means that no more than "n bytes per second"
shall be sent or received. It can be set individually for both directions
via CURLOPT_MAX_RECV_SPEED_LARGE and CURLOPT_MAX_SEND_SPEED_LARGE. These
options may be adjusted for an ongoing transfer.
Implementation Base
ratelimit.[ch] implements struct Curl_rlimit and functions to manage
such limits. It has the following properties:
rate_per_sec: how many "tokens" can be used per second, 0 for infinite.tokens: the currently available tokens to consumeburst_per_sec: an upper limit on tokens availablets: the microsecond timestamp of the last tokens updatespare_us: elapsed microseconds that have not counted yet for a token updateblocked: if the limit is blocked
Tokens can be drained from an rlimit. This reduces tokens, even to
negative values. To enforce the limits, tokens should not be drained
further when they reach 0, but such things may happen.
An rlimitcan be asked how long to wait until tokens are positive again.
This is given in milliseconds. When token are available, this wait
time is 0.
Ideally a user of rlimit would consume the available tokens to 0, then
get a wait times of 1000ms, after which the set rate of tokens has
regenerated. Rinse and repeat.
Should a user drain twice the amount of the rate, tokens are negative
and the wait time is 2 seconds. The spare_us account for the
time that has passed for the consumption. When a user takes 250ms to
consume the rate, the wait time is then 750ms.
When a user drains nothing for two seconds, the available tokens would grow to twice the rate, unless a burst rate is set.
Finally, an rlimit may be set to blocked and later unblocked again.
A blocked rlimit has no tokens available. This works also when the rate
is unlimited (rate_per_sec set to 0).
Downloads
rlimit is in data->progress.dl.rlimit. setopt.c initializes it whenever
the application sets CURLOPT_MAX_RECV_SPEED_LARGE. This may be done
in the middle of a transfer.
rlimit tokens are drained in the "protocol" client writer. Checks for
capacity depend on the protocol:
- HTTP and other plain protocols:
transfer.c:sendrecv_dl()reads only up to capacity. - HTTP/2: capacity is used to adjust a stream's window size. Since all
streams start with
64kb,rlimittakes a few seconds to take effect. - HTTP/3: ngtcp2 acknowledges stream data according to capacity. It keeps track of bytes not acknowledged yet. This has the same effect as HTTP/2 window sizes.
(The quiche API does not offer control of ACKs and rlimits for download
do not work in that backend.)
Uploads
rlimit is in data->progress.ul.rlimit. setopt.c initializes it whenever
the application sets CURLOPT_MAX_SEND_SPEED_LARGE. This may be done
in the middle of a transfer.
The upload capacity is checked in Curl_client_read() and readers are
only asked to read bytes up to the rlimit capacity. This limits upload
of data for all protocols in the same way.
Pause/Unpause
Pausing of up-/downloads sets the corresponding rlimit to blocked. Unpausing
removes that block.
Suspending transfers
While obeying the rlimit for up-/download leads to the desired transfer
rates, the other issue that needs care is CPU consumption.
rlimits are inspected when computing the "pollset" of a transfer. When
a transfer wants to send, but not send tokens are available, the POLLOUT
is removed from the pollset. Same for receiving.
For a transfer that is, due to rlimit, not able to progress, the pollset
is then empty. No socket events are monitored, no CPU activity
happens. For paused transfers, this is sufficient.
Draining rlimit happens when a transfer is in PERFORM state and
exhausted limits cause the timer TOOFAST to be set. When the fires,
the transfer runs again and rlimits are re-evaluated.