Files
jak-project/third-party/curl/lib/bufq.h
T
Alexander J. Semenuk 5d5e35fb9b fix: Windows toolchain compatibility (curl 8.21 re-vendor, endless reconfigure loop) (#4355)
## Problem

Windows builds break with a current local toolchain (Scoop LLVM 22.1.8,
CMake 4.4.0, VS 2026), in two independent ways:

1. The build stops at curl's deliberate guard: `#error "no non-blocking
method was found/used/set"` in `third-party/curl/lib/nonblock.c`.
2. From the second configure onward, `cmake --build` re-runs CMake in an
endless loop (observed 42 consecutive reconfigure cycles in a single
build). Likely the same mechanism behind the "endlessly building" VS
2026 note in `docs/setup/dev/vs.md`.

## Root cause

1. `third-party/curl/CMake/CurlTests.c` passes `int *` to
`ioctlsocket()`, whose third parameter is `u_long *`. Clang 22 promotes
`-Wincompatible-pointer-types` to a hard error in C, so the
`HAVE_IOCTLSOCKET_FIONBIO` try_compile silently fails and
`curl_config.h` never defines it. Upstream CI does not see this because
the windows-2022 runner image ships an older LLVM. GCC 14 promotes the
same warning to a hard error, which is very likely the `CurlTests.c.obj`
failure reported from MSYS2 in open-goal/jak-project#3551. Upstream curl
hit the identical problem with GCC 14 and fixed the probe in curl 8.8.0
(curl/curl#13578).
2. The root CMakeLists copies the build tree's `compile_commands.json`
into `<src>/build/` for clangd using `configure_file()`, which registers
its input as a configure dependency. CMake rewrites
`compile_commands.json` late in every generation, after
`CTestTestfile.cmake` and `cmake_install.cmake` (outputs of the same
Ninja regen rule), so once the dependency is registered the rule is
deterministically dirty and every `ninja` invocation re-runs CMake. A
pristine first configure is safe (the file does not exist yet, so the
`if(EXISTS ...)` guard skips the copy), which is why the loop looks
machine- or IDE-specific.

## Fix

1. Per review, re-vendor `third-party/curl` at the `curl-8_21_0` tag
(previously `curl-8_3_0`), which carries the upstream probe fix plus two
years of upstream development; `vendor.yaml` updated to match.
Adjustments the version jump forced:
- curl 8.15 removed the native macOS Secure Transport backend
(`CURL_USE_SECTRANSP`), so macOS now builds curl against OpenSSL like
Linux. The two macOS workflows install Homebrew `openssl@3` and export
`OPENSSL_ROOT_DIR` (keg-only), and the macOS setup docs gained the same
two lines.
- `CURL_BROTLI` / `CURL_ZSTD` switched to AUTO-detection in curl 8.10;
pinned OFF to keep the previous no-compression behavior and avoid
silently linking whatever the CI images happen to have.
- curl's new top-level `BUILD_EXAMPLES` cache option (default ON) leaked
into discord-rpc's identically named option and broke configure at a
nonexistent `examples/send-presence` directory; pinned OFF ahead of the
third-party subdirectories.

The diff is dominated by the mechanical tag-tree swap under
`third-party/curl` (linguist-vendored, collapsed in review). The
hand-written changes are `CMakeLists.txt`, the two macOS workflows,
`docs/setup/system/macos.md`, and `vendor.yaml`.
2. Swap `configure_file()` for `file(COPY ...)`: the same clangd copy
with no configure dependency registered. (`file(COPY_FILE ...
ONLY_IF_DIFFERENT)` would be cleaner still but requires CMake 3.21,
above the declared `cmake_minimum_required(VERSION 3.10)`.)

## Test plan

- [x] Fresh `cmake --preset Release-windows-clang` (LLVM 22, no cache
seeding) completes and logs `Enabled SSL backends: Schannel`; the
FIONBIO probe passes without the previous `#error`
- [x] Full Windows Release build from scratch in the branch worktree
(all 1422 targets)
- [x] goalc-test suite: 1509 passed, 0 failed
- [x] Second consecutive configure with `compile_commands.json` present:
the regen rule in `build.ninja` has no `compile_commands.json` input;
`<src>/build/compile_commands.json` is still refreshed for clangd
- [x] Repeated `ninja` invocations after a full build no longer re-run
CMake
- [x] macOS Intel and ARM CI green (first exercise of the OpenSSL
backend switch)

---

I work off a self-hosted forge, so this GitHub account is quiet; the
configure logs and ninja dirty-node traces from the investigation are
available if anyone wants the raw data.

(AI-assisted)
2026-07-27 19:19:18 -04:00

258 lines
9.6 KiB
C
Vendored
Generated

#ifndef HEADER_CURL_BUFQ_H
#define HEADER_CURL_BUFQ_H
/***************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
* SPDX-License-Identifier: curl
*
***************************************************************************/
#include "curl_setup.h"
/**
* A chunk of bytes for reading and writing.
* The size is fixed a creation with read and write offset
* for where unread content is.
*/
struct buf_chunk {
struct buf_chunk *next; /* to keep it in a list */
size_t dlen; /* the amount of allocated x.data[] */
size_t r_offset; /* first unread bytes */
size_t w_offset; /* one after last written byte */
union {
uint8_t data[1]; /* the buffer for `dlen` bytes */
void *dummy; /* alignment */
} x;
};
/**
* A pool for providing/keeping a number of chunks of the same size
*
* The same pool can be shared by many `bufq` instances. A pool is not thread
* safe. All bufqs using it are supposed to operate in the same thread.
*/
struct bufc_pool {
struct buf_chunk *spare; /* list of available spare chunks */
size_t chunk_size; /* the size of chunks in this pool */
size_t spare_count; /* current number of spare chunks in list */
size_t spare_max; /* max number of spares to keep */
};
void Curl_bufcp_init(struct bufc_pool *pool,
size_t chunk_size, size_t spare_max);
void Curl_bufcp_free(struct bufc_pool *pool);
/**
* A queue of byte chunks for reading and writing.
* Reading is done from `head`, writing is done to `tail`.
*
* `bufq`s can be empty or full or neither. Its `len` is the number
* of bytes that can be read. For an empty bufq, `len` will be 0.
*
* By default, a bufq can hold up to `max_chunks * chunk_size` number
* of bytes. When `max_chunks` are used (in the `head` list) and the
* `tail` chunk is full, the bufq will report that it is full.
*
* On a full bufq, `len` may be less than the maximum number of bytes,
* e.g. when the head chunk is partially read. `len` may also become
* larger than the max when option `BUFQ_OPT_SOFT_LIMIT` is used.
*
* By default, writing to a full bufq will return (-1, CURLE_AGAIN). Same
* as reading from an empty bufq.
* With `BUFQ_OPT_SOFT_LIMIT` set, a bufq will allow writing beyond this limit
* and use more than `max_chunks`. It will report that it is full
* nevertheless. This is provided for situation where writes preferably never
* fail (except for memory exhaustion).
*
* By default and without a pool, a bufq will keep chunks that read
* empty in its `spare` list. Option `BUFQ_OPT_NO_SPARES` will
* disable that and free chunks once they become empty.
*
* When providing a pool to a bufq, all chunk creation and spare handling
* will be delegated to that pool.
*/
struct bufq {
struct buf_chunk *head; /* chunk with bytes to read from */
struct buf_chunk *tail; /* chunk to write to */
struct buf_chunk *spare; /* list of free chunks, unless `pool` */
struct bufc_pool *pool; /* optional pool for free chunks */
size_t chunk_count; /* current number of chunks in `head+spare` */
size_t max_chunks; /* max `head` chunks to use */
size_t chunk_size; /* size of chunks to manage */
int opts; /* options for handling queue, see below */
};
/**
* Default behaviour: chunk limit is "hard", meaning attempts to write
* more bytes than can be hold in `max_chunks` is refused and will return
* -1, CURLE_AGAIN. */
#define BUFQ_OPT_NONE 0
/**
* Make `max_chunks` a "soft" limit. A bufq will report that it is "full"
* when `max_chunks` are used, but allows writing beyond this limit.
*/
#define BUFQ_OPT_SOFT_LIMIT (1 << 0)
/**
* Do not keep spare chunks.
*/
#define BUFQ_OPT_NO_SPARES (1 << 1)
/**
* Initialize a buffer queue that can hold up to `max_chunks` buffers
* each of size `chunk_size`. The bufq will not allow writing of
* more bytes than can be held in `max_chunks`.
*/
void Curl_bufq_init(struct bufq *q, size_t chunk_size, size_t max_chunks);
/**
* Initialize a buffer queue that can hold up to `max_chunks` buffers
* each of size `chunk_size` with the given options. See `BUFQ_OPT_*`.
*/
void Curl_bufq_init2(struct bufq *q, size_t chunk_size,
size_t max_chunks, int opts);
void Curl_bufq_initp(struct bufq *q, struct bufc_pool *pool,
size_t max_chunks, int opts);
/**
* Reset the buffer queue to be empty. Keep any allocated buffer chunks
* around.
*/
void Curl_bufq_reset(struct bufq *q);
/**
* Free all resources held by the buffer queue.
*/
void Curl_bufq_free(struct bufq *q);
/**
* Return the total amount of data in the queue.
*/
size_t Curl_bufq_len(const struct bufq *q);
/**
* Returns TRUE iff there is no data in the buffer queue.
*/
bool Curl_bufq_is_empty(const struct bufq *q);
/**
* Returns TRUE iff there is no space left in the buffer queue.
*/
bool Curl_bufq_is_full(const struct bufq *q);
/**
* Write buf to the end of the buffer queue. The buf is copied
* and the amount of copied bytes is returned.
* CURLE_AGAIN is returned if the buffer queue is full.
*/
CURLcode Curl_bufq_write(struct bufq *q,
const uint8_t *buf, size_t len,
size_t *pnwritten);
CURLcode Curl_bufq_cwrite(struct bufq *q,
const char *buf, size_t len,
size_t *pnwritten);
/**
* Read buf from the start of the buffer queue. The buf is copied
* and the amount of copied bytes is returned.
*/
CURLcode Curl_bufq_read(struct bufq *q, uint8_t *buf, size_t len,
size_t *pnread);
CURLcode Curl_bufq_cread(struct bufq *q, char *buf, size_t len,
size_t *pnread);
/**
* Peek at the head chunk in the buffer queue. Returns a pointer to
* the chunk buf (at the current offset) and its length. Does not
* modify the buffer queue.
* Returns TRUE iff bytes are available. Sets `pbuf` to NULL and `plen`
* to 0 when no bytes are available.
* Repeated calls return the same information until the buffer queue
* is modified, see `Curl_bufq_skip()``
*/
bool Curl_bufq_peek(struct bufq *q,
const uint8_t **pbuf, size_t *plen);
bool Curl_bufq_peek_at(struct bufq *q, size_t offset,
const uint8_t **pbuf, size_t *plen);
/**
* Tell the buffer queue to discard `amount` buf bytes at the head of the
* queue. Skipping more buf than is currently buffered will empty the queue.
*/
void Curl_bufq_skip(struct bufq *q, size_t amount);
typedef CURLcode Curl_bufq_writer(void *writer_ctx,
const uint8_t *buf, size_t len,
size_t *pwritten);
/**
* Passes the chunks in the buffer queue to the writer and returns
* the amount of buf written. A writer may return -1 and CURLE_AGAIN
* to indicate blocking at which point the queue will stop and return
* the amount of buf passed so far.
* -1 is returned on any other errors reported by the writer.
* Note that in case of a -1 chunks may have been written and
* the buffer queue will have different length than before.
*/
CURLcode Curl_bufq_pass(struct bufq *q, Curl_bufq_writer *writer,
void *writer_ctx, size_t *pwritten);
typedef CURLcode Curl_bufq_reader(void *reader_ctx,
uint8_t *buf, size_t len,
size_t *pnread);
/**
* Read bytes and append them to the end of the buffer queue until the
* reader returns blocking or the queue is full. A reader returns
* CURLE_AGAIN to indicate blocking.
* Returns the total amount of buf read (may be 0) in `pnread` on success.
* Note that in case of an error chunks may have been read and
* the buffer queue will have different length than before.
*/
CURLcode Curl_bufq_slurp(struct bufq *q, Curl_bufq_reader *reader,
void *reader_ctx, size_t *pnread);
/**
* Read *once* up to `max_len` bytes and append it to the buffer.
* if `max_len` is 0, no limit is imposed besides the chunk space.
* Returns the total amount of buf read (may be 0) or -1 on other
* reader errors.
*/
CURLcode Curl_bufq_sipn(struct bufq *q, size_t max_len,
Curl_bufq_reader *reader, void *reader_ctx,
size_t *pnread);
/**
* Write buf to the end of the buffer queue.
* Write bufq content or passed `buf` directly using the `writer`
* callback when it sees fit. 'buf' might get passed directly
* on or is placed into the buffer, depending on `len` and current
* amount buffered, chunk size, etc.
*/
CURLcode Curl_bufq_write_pass(struct bufq *q,
const uint8_t *buf, size_t len,
Curl_bufq_writer *writer, void *writer_ctx,
size_t *pwritten);
#endif /* HEADER_CURL_BUFQ_H */