Alexander J. Semenuk
5d5e35fb9b
## Problem Windows builds break with a current local toolchain (Scoop LLVM 22.1.8, CMake 4.4.0, VS 2026), in two independent ways: 1. The build stops at curl's deliberate guard: `#error "no non-blocking method was found/used/set"` in `third-party/curl/lib/nonblock.c`. 2. From the second configure onward, `cmake --build` re-runs CMake in an endless loop (observed 42 consecutive reconfigure cycles in a single build). Likely the same mechanism behind the "endlessly building" VS 2026 note in `docs/setup/dev/vs.md`. ## Root cause 1. `third-party/curl/CMake/CurlTests.c` passes `int *` to `ioctlsocket()`, whose third parameter is `u_long *`. Clang 22 promotes `-Wincompatible-pointer-types` to a hard error in C, so the `HAVE_IOCTLSOCKET_FIONBIO` try_compile silently fails and `curl_config.h` never defines it. Upstream CI does not see this because the windows-2022 runner image ships an older LLVM. GCC 14 promotes the same warning to a hard error, which is very likely the `CurlTests.c.obj` failure reported from MSYS2 in open-goal/jak-project#3551. Upstream curl hit the identical problem with GCC 14 and fixed the probe in curl 8.8.0 (curl/curl#13578). 2. The root CMakeLists copies the build tree's `compile_commands.json` into `<src>/build/` for clangd using `configure_file()`, which registers its input as a configure dependency. CMake rewrites `compile_commands.json` late in every generation, after `CTestTestfile.cmake` and `cmake_install.cmake` (outputs of the same Ninja regen rule), so once the dependency is registered the rule is deterministically dirty and every `ninja` invocation re-runs CMake. A pristine first configure is safe (the file does not exist yet, so the `if(EXISTS ...)` guard skips the copy), which is why the loop looks machine- or IDE-specific. ## Fix 1. Per review, re-vendor `third-party/curl` at the `curl-8_21_0` tag (previously `curl-8_3_0`), which carries the upstream probe fix plus two years of upstream development; `vendor.yaml` updated to match. Adjustments the version jump forced: - curl 8.15 removed the native macOS Secure Transport backend (`CURL_USE_SECTRANSP`), so macOS now builds curl against OpenSSL like Linux. The two macOS workflows install Homebrew `openssl@3` and export `OPENSSL_ROOT_DIR` (keg-only), and the macOS setup docs gained the same two lines. - `CURL_BROTLI` / `CURL_ZSTD` switched to AUTO-detection in curl 8.10; pinned OFF to keep the previous no-compression behavior and avoid silently linking whatever the CI images happen to have. - curl's new top-level `BUILD_EXAMPLES` cache option (default ON) leaked into discord-rpc's identically named option and broke configure at a nonexistent `examples/send-presence` directory; pinned OFF ahead of the third-party subdirectories. The diff is dominated by the mechanical tag-tree swap under `third-party/curl` (linguist-vendored, collapsed in review). The hand-written changes are `CMakeLists.txt`, the two macOS workflows, `docs/setup/system/macos.md`, and `vendor.yaml`. 2. Swap `configure_file()` for `file(COPY ...)`: the same clangd copy with no configure dependency registered. (`file(COPY_FILE ... ONLY_IF_DIFFERENT)` would be cleaner still but requires CMake 3.21, above the declared `cmake_minimum_required(VERSION 3.10)`.) ## Test plan - [x] Fresh `cmake --preset Release-windows-clang` (LLVM 22, no cache seeding) completes and logs `Enabled SSL backends: Schannel`; the FIONBIO probe passes without the previous `#error` - [x] Full Windows Release build from scratch in the branch worktree (all 1422 targets) - [x] goalc-test suite: 1509 passed, 0 failed - [x] Second consecutive configure with `compile_commands.json` present: the regen rule in `build.ninja` has no `compile_commands.json` input; `<src>/build/compile_commands.json` is still refreshed for clangd - [x] Repeated `ninja` invocations after a full build no longer re-run CMake - [x] macOS Intel and ARM CI green (first exercise of the OpenSSL backend switch) --- I work off a self-hosted forge, so this GitHub account is quiet; the configure logs and ninja dirty-node traces from the investigation are available if anyone wants the raw data. (AI-assisted)
4.5 KiB
Vendored
Generated
Adding a new protocol?
Every once in a while, someone comes up with the idea of adding support for yet another protocol to curl. After all, curl already supports 25 something protocols and it is the Internet transfer machine for the world.
In the curl project we love protocols and we love supporting many protocols and doing it well.
How do you proceed to add a new protocol and what are the requirements?
No fixed set of requirements
This document is an attempt to describe things to consider. There is no checklist of the twenty-seven things you need to cross off. We view the entire effort as a whole and then judge if it seems to be the right thing - for now. The more things that look right, fit our patterns and are done in ways that align with our thinking, the better are the chances that we agree that supporting this protocol is a grand idea.
Mutual benefit is preferred
curl is not here for your protocol. Your protocol is not here for curl. The best cooperation and end result occur when all involved parties mutually see and agree that supporting this protocol in curl would be good for everyone. Heck, for the world.
Consider "selling us" the idea that we need an implementation merged in curl, to be fairly important. Why do we want curl to support this new protocol?
Protocol requirements
Client-side
The protocol implementation is for a client's side of a "communication session".
Transfer oriented
The protocol itself should be focused on transfers. Be it uploads or downloads or both. It should at least be possible to view the transfers as such, like we can view reading emails over POP3 as a download and sending emails over SMTP as an upload.
If you cannot even shoehorn the protocol into a transfer focused view, then you are up for a tough argument.
URL
There should be a documented URL format. If there is an RFC for it there is no question about it but the syntax does not have to be a published RFC. It could be enough if it is already in use by other implementations.
If you make up the syntax in order to be able to propose it to curl, then you are in a bad place. URLs are designed and defined for interoperability. There should at least be a good chance that other clients and servers can be implemented supporting the same URL syntax and work the same or similar way.
URLs work on registered 'schemes'. There is a register of all officially recognized schemes. If your protocol is not in there, is it really a protocol we want?
Wide and public use
The protocol shall already be used or have an expectation of getting used widely. Experimental protocols are better off worked on in experiments first, to prove themselves before they are adopted by curl.
Code
Of course the code needs to be written, provided, licensed agreeably and it should follow our code guidelines and review comments have to be dealt with. If the implementation needs third party code, that third party code should not have noticeably lesser standards than the curl project itself.
Tests
As much of the protocol implementation as possible needs to be verified by curl test cases. We must have the implementation get tested by CI jobs, torture tests and more.
We have experienced many times in the past how new implementations were brought to curl and immediately once the code had been merged, the originator vanished from the face of the earth. That is fine, but we need to take the necessary precautions so when it happens we are still fine.
Our test infrastructure is powerful enough to test almost every protocol - but it might require a bit of an effort to make it happen.
Documentation
We cannot assume that users are particularly familiar with details and peculiarities of the protocol. It needs documentation.
Maybe it even needs some internal documentation so that the developers who try to debug something five years from now can figure out functionality a little easier.
The protocol specification itself should be freely available without requiring a non-disclosure agreement or similar.
Do not compare
We are constantly raising the bar and we are constantly improving the project. A lot of things we did in the past would not be acceptable if done today. Therefore, you might be tempted to use shortcuts or "hacks" you can spot other - existing - protocol implementations have used, but there is nothing to gain from that. The bar has been raised. Former "cheats" may not tolerated anymore.