mirror of
https://github.com/open-goal/jak-project
synced 2026-09-02 02:02:51 -04:00
Alexander J. Semenuk
5d5e35fb9b
## Problem Windows builds break with a current local toolchain (Scoop LLVM 22.1.8, CMake 4.4.0, VS 2026), in two independent ways: 1. The build stops at curl's deliberate guard: `#error "no non-blocking method was found/used/set"` in `third-party/curl/lib/nonblock.c`. 2. From the second configure onward, `cmake --build` re-runs CMake in an endless loop (observed 42 consecutive reconfigure cycles in a single build). Likely the same mechanism behind the "endlessly building" VS 2026 note in `docs/setup/dev/vs.md`. ## Root cause 1. `third-party/curl/CMake/CurlTests.c` passes `int *` to `ioctlsocket()`, whose third parameter is `u_long *`. Clang 22 promotes `-Wincompatible-pointer-types` to a hard error in C, so the `HAVE_IOCTLSOCKET_FIONBIO` try_compile silently fails and `curl_config.h` never defines it. Upstream CI does not see this because the windows-2022 runner image ships an older LLVM. GCC 14 promotes the same warning to a hard error, which is very likely the `CurlTests.c.obj` failure reported from MSYS2 in open-goal/jak-project#3551. Upstream curl hit the identical problem with GCC 14 and fixed the probe in curl 8.8.0 (curl/curl#13578). 2. The root CMakeLists copies the build tree's `compile_commands.json` into `<src>/build/` for clangd using `configure_file()`, which registers its input as a configure dependency. CMake rewrites `compile_commands.json` late in every generation, after `CTestTestfile.cmake` and `cmake_install.cmake` (outputs of the same Ninja regen rule), so once the dependency is registered the rule is deterministically dirty and every `ninja` invocation re-runs CMake. A pristine first configure is safe (the file does not exist yet, so the `if(EXISTS ...)` guard skips the copy), which is why the loop looks machine- or IDE-specific. ## Fix 1. Per review, re-vendor `third-party/curl` at the `curl-8_21_0` tag (previously `curl-8_3_0`), which carries the upstream probe fix plus two years of upstream development; `vendor.yaml` updated to match. Adjustments the version jump forced: - curl 8.15 removed the native macOS Secure Transport backend (`CURL_USE_SECTRANSP`), so macOS now builds curl against OpenSSL like Linux. The two macOS workflows install Homebrew `openssl@3` and export `OPENSSL_ROOT_DIR` (keg-only), and the macOS setup docs gained the same two lines. - `CURL_BROTLI` / `CURL_ZSTD` switched to AUTO-detection in curl 8.10; pinned OFF to keep the previous no-compression behavior and avoid silently linking whatever the CI images happen to have. - curl's new top-level `BUILD_EXAMPLES` cache option (default ON) leaked into discord-rpc's identically named option and broke configure at a nonexistent `examples/send-presence` directory; pinned OFF ahead of the third-party subdirectories. The diff is dominated by the mechanical tag-tree swap under `third-party/curl` (linguist-vendored, collapsed in review). The hand-written changes are `CMakeLists.txt`, the two macOS workflows, `docs/setup/system/macos.md`, and `vendor.yaml`. 2. Swap `configure_file()` for `file(COPY ...)`: the same clangd copy with no configure dependency registered. (`file(COPY_FILE ... ONLY_IF_DIFFERENT)` would be cleaner still but requires CMake 3.21, above the declared `cmake_minimum_required(VERSION 3.10)`.) ## Test plan - [x] Fresh `cmake --preset Release-windows-clang` (LLVM 22, no cache seeding) completes and logs `Enabled SSL backends: Schannel`; the FIONBIO probe passes without the previous `#error` - [x] Full Windows Release build from scratch in the branch worktree (all 1422 targets) - [x] goalc-test suite: 1509 passed, 0 failed - [x] Second consecutive configure with `compile_commands.json` present: the regen rule in `build.ninja` has no `compile_commands.json` input; `<src>/build/compile_commands.json` is still refreshed for clangd - [x] Repeated `ninja` invocations after a full build no longer re-run CMake - [x] macOS Intel and ARM CI green (first exercise of the OpenSSL backend switch) --- I work off a self-hosted forge, so this GitHub account is quiet; the configure logs and ninja dirty-node traces from the investigation are available if anyone wants the raw data. (AI-assisted)
162 lines
6.8 KiB
Markdown
Vendored
Generated
162 lines
6.8 KiB
Markdown
Vendored
Generated
<!--
|
|
Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
|
|
|
|
SPDX-License-Identifier: curl
|
|
-->
|
|
|
|
# TLS Sessions and Tickets
|
|
|
|
The TLS protocol offers methods of "resuming" a previous "session". A
|
|
TLS "session" is a negotiated security context across a connection
|
|
(which may be via TCP or UDP or other transports.)
|
|
|
|
By "resuming", the TLS protocol means that the security context from
|
|
before can be fully or partially resurrected when the TLS client presents
|
|
the proper crypto stuff to the server. This saves on the amount of
|
|
TLS packets that need to be sent back and forth, reducing amount
|
|
of data and even latency. In the case of QUIC, resumption may send
|
|
application data without having seen any reply from the server, hence
|
|
this is named 0-RTT data.
|
|
|
|
The exact mechanism of session tickets in TLSv1.2 (and earlier) and
|
|
TLSv1.3 differs. TLSv1.2 tickets have several weaknesses (that can
|
|
be exploited by attackers) which TLSv1.3 then fixed. See
|
|
[Session Tickets in the real world](https://words.filippo.io/we-need-to-talk-about-session-tickets/)
|
|
for an insight into this topic.
|
|
|
|
These difference between TLS protocol versions are reflected in curl's
|
|
handling of session tickets. More below.
|
|
|
|
## curl's `ssl_peer_key`
|
|
|
|
In order to find a ticket from a previous TLS session, curl
|
|
needs a name for TLS sessions that uniquely identifies the peer
|
|
it talks to.
|
|
|
|
This name has to reflect also the various TLS parameters that can
|
|
be configured in curl for a connection. We do not want to use
|
|
a ticket from an different configuration. Example: when setting
|
|
the maximum TLS version to 1.2, we do not want to reuse a ticket
|
|
we got from a TLSv1.3 session, although we are talking to the
|
|
same host.
|
|
|
|
Internally, we call this name a `ssl_peer_key`. It is a printable
|
|
string that carries hostname and port and any non-default TLS
|
|
parameters involved in the connection.
|
|
|
|
Examples:
|
|
- `curl.se:443:CA-/etc/ssl/cert.pem:IMPL-GnuTLS/3.8.7` is a peer key for
|
|
a connection to `curl.se:443` using `/etc/ssl/cert.pem` as CA
|
|
trust anchors and GnuTLS/3.8.7 as TLS backend.
|
|
- `curl.se:443:TLSVER-6-6:CA-/etc/ssl/cert.pem:IMPL-GnuTLS/3.8.7` is the
|
|
same as the previous, except it is configured to use TLSv1.2 as
|
|
min and max versions.
|
|
|
|
Different configurations produce different keys which is what curl needs when
|
|
handling SSL session tickets.
|
|
|
|
One important thing: peer keys do not contain confidential information. If you
|
|
configure a client certificate or SRP authentication with username/password,
|
|
these are not part of the peer key.
|
|
|
|
Peer keys carry the hostnames you use curl for. They *do* leak the privacy of
|
|
your communication. We recommend to *not* persist peer keys for this reason.
|
|
|
|
**Caveat**: The key may contain filenames or paths. It does not reflect the
|
|
*contents* in the file system. If you change `/etc/ssl/cert.pem` and reuse
|
|
a previous ticket, curl might trust a server which no longer has a root
|
|
certificate in the file.
|
|
|
|
## Session Cache Access
|
|
|
|
#### Lookups
|
|
|
|
When a new connection is being established, each SSL connection filter creates
|
|
its own peer_key and calls into the cache. The cache then looks for a ticket
|
|
with exactly this peer_key. Peer keys between proxy SSL filters and SSL
|
|
filters talking through a tunnel differ, as they talk to different peers.
|
|
|
|
If the connection filter wants to use a client certificate or SRP
|
|
authentication, the cache checks those as well. If the cache peer carries
|
|
client cert or SRP auth, the connection filter must have those with the same
|
|
values (and vice versa).
|
|
|
|
On a match, the connection filter gets the session ticket and feeds that to
|
|
the TLS implementation which, on accepting it, tries to resume it for a
|
|
shorter handshake. In addition, the filter gets the ALPN used before and the
|
|
amount of 0-RTT data that the server announced to be willing to accept. The
|
|
filter can then decide if it wants to attempt 0-RTT or not. (The ALPN is
|
|
needed to know if the server speaks the protocol you want to send in 0-RTT. It
|
|
makes no sense to send HTTP/2 requests to a server that only knows HTTP/1.1.)
|
|
|
|
#### Updates
|
|
|
|
When a new TLS session ticket is received by a filter, it adds it to the
|
|
cache using its peer_key and SSL configuration. The cache looks for
|
|
a matching entry and, should it find one, adds the ticket for this
|
|
peer.
|
|
|
|
### Put, Take and Return
|
|
|
|
when a filter accesses the session cache, it *takes*
|
|
a ticket from the cache, meaning a returned ticket is removed. The filter
|
|
then configures its TLS backend and *returns* the ticket to the cache.
|
|
|
|
The cache needs to treat tickets from TLSv1.2 and 1.3 differently. 1.2 tickets
|
|
should be reused, but 1.3 tickets SHOULD NOT (RFC 8446). The session cache
|
|
drops 1.3 tickets when they are returned after use, but keeps a 1.2
|
|
ticket.
|
|
|
|
When a ticket is *put* into the cache, there is also a difference. There
|
|
can be several 1.3 tickets at the same time, but only a single 1.2 ticket.
|
|
TLSv1.2 tickets replace any other. 1.3 tickets accumulate up to a max
|
|
amount.
|
|
|
|
By having a "put/take/return" we reflect the 1.3 use case nicely. Two
|
|
concurrent connections do not reuse the same ticket.
|
|
|
|
## Session Ticket Persistence
|
|
|
|
#### Privacy and Security
|
|
|
|
As mentioned above, SSL peer keys are not intended for storage in a file
|
|
system. They clearly show which hosts the user talked to. This is not only
|
|
privacy relevant, but also has security implications as an attacker might find
|
|
worthy targets among your peer keys.
|
|
|
|
Also, we do not recommend to persist TLSv1.2 tickets.
|
|
|
|
### Salted Hashes
|
|
|
|
The TLS session cache offers an alternative to storing peer keys:
|
|
it provides a salted SHA256 hash of the peer key for import and export.
|
|
|
|
#### Export
|
|
|
|
The salt is generated randomly for each peer key on export. The SHA256 makes
|
|
sure that the peer key cannot be reversed and that a slightly different key
|
|
still produces a different result.
|
|
|
|
This means an attacker cannot "grep" a session file for a particular entry,
|
|
e.g. if they want to know if you accessed a specific host. They *can* however
|
|
compute the SHA256 hashes for all salts in the file and find a specific entry.
|
|
They *cannot* find a hostname they do not know. They would have to brute force
|
|
by guessing.
|
|
|
|
#### Import
|
|
|
|
When session tickets are imported from a file, curl only gets the salted
|
|
hashes. The imported tickets belong to an *unknown* peer key.
|
|
|
|
When a connection filter tries to *take* a session ticket, it passes its peer
|
|
key. This peer key initially does not match any tickets in the cache. The
|
|
cache then checks all entries with unknown peer keys if the passed key matches
|
|
their salted hash. If it does, the peer key is recovered and remembered at the
|
|
cache entry.
|
|
|
|
This is a performance penalty in the order of "unknown" peer keys which
|
|
diminishes over time when keys are rediscovered. Note that this also works for
|
|
putting a new ticket into the cache: when no present entry matches, a new one
|
|
with peer key is created. This peer key then no longer bears the cost of hash
|
|
computes.
|