Files
jak-project/third-party/curl/tests/testutil.pm
T
Alexander J. Semenuk 5d5e35fb9b fix: Windows toolchain compatibility (curl 8.21 re-vendor, endless reconfigure loop) (#4355)
## Problem

Windows builds break with a current local toolchain (Scoop LLVM 22.1.8,
CMake 4.4.0, VS 2026), in two independent ways:

1. The build stops at curl's deliberate guard: `#error "no non-blocking
method was found/used/set"` in `third-party/curl/lib/nonblock.c`.
2. From the second configure onward, `cmake --build` re-runs CMake in an
endless loop (observed 42 consecutive reconfigure cycles in a single
build). Likely the same mechanism behind the "endlessly building" VS
2026 note in `docs/setup/dev/vs.md`.

## Root cause

1. `third-party/curl/CMake/CurlTests.c` passes `int *` to
`ioctlsocket()`, whose third parameter is `u_long *`. Clang 22 promotes
`-Wincompatible-pointer-types` to a hard error in C, so the
`HAVE_IOCTLSOCKET_FIONBIO` try_compile silently fails and
`curl_config.h` never defines it. Upstream CI does not see this because
the windows-2022 runner image ships an older LLVM. GCC 14 promotes the
same warning to a hard error, which is very likely the `CurlTests.c.obj`
failure reported from MSYS2 in open-goal/jak-project#3551. Upstream curl
hit the identical problem with GCC 14 and fixed the probe in curl 8.8.0
(curl/curl#13578).
2. The root CMakeLists copies the build tree's `compile_commands.json`
into `<src>/build/` for clangd using `configure_file()`, which registers
its input as a configure dependency. CMake rewrites
`compile_commands.json` late in every generation, after
`CTestTestfile.cmake` and `cmake_install.cmake` (outputs of the same
Ninja regen rule), so once the dependency is registered the rule is
deterministically dirty and every `ninja` invocation re-runs CMake. A
pristine first configure is safe (the file does not exist yet, so the
`if(EXISTS ...)` guard skips the copy), which is why the loop looks
machine- or IDE-specific.

## Fix

1. Per review, re-vendor `third-party/curl` at the `curl-8_21_0` tag
(previously `curl-8_3_0`), which carries the upstream probe fix plus two
years of upstream development; `vendor.yaml` updated to match.
Adjustments the version jump forced:
- curl 8.15 removed the native macOS Secure Transport backend
(`CURL_USE_SECTRANSP`), so macOS now builds curl against OpenSSL like
Linux. The two macOS workflows install Homebrew `openssl@3` and export
`OPENSSL_ROOT_DIR` (keg-only), and the macOS setup docs gained the same
two lines.
- `CURL_BROTLI` / `CURL_ZSTD` switched to AUTO-detection in curl 8.10;
pinned OFF to keep the previous no-compression behavior and avoid
silently linking whatever the CI images happen to have.
- curl's new top-level `BUILD_EXAMPLES` cache option (default ON) leaked
into discord-rpc's identically named option and broke configure at a
nonexistent `examples/send-presence` directory; pinned OFF ahead of the
third-party subdirectories.

The diff is dominated by the mechanical tag-tree swap under
`third-party/curl` (linguist-vendored, collapsed in review). The
hand-written changes are `CMakeLists.txt`, the two macOS workflows,
`docs/setup/system/macos.md`, and `vendor.yaml`.
2. Swap `configure_file()` for `file(COPY ...)`: the same clangd copy
with no configure dependency registered. (`file(COPY_FILE ...
ONLY_IF_DIFFERENT)` would be cleaner still but requires CMake 3.21,
above the declared `cmake_minimum_required(VERSION 3.10)`.)

## Test plan

- [x] Fresh `cmake --preset Release-windows-clang` (LLVM 22, no cache
seeding) completes and logs `Enabled SSL backends: Schannel`; the
FIONBIO probe passes without the previous `#error`
- [x] Full Windows Release build from scratch in the branch worktree
(all 1422 targets)
- [x] goalc-test suite: 1509 passed, 0 failed
- [x] Second consecutive configure with `compile_commands.json` present:
the regen rule in `build.ninja` has no `compile_commands.json` input;
`<src>/build/compile_commands.json` is still refreshed for clangd
- [x] Repeated `ninja` invocations after a full build no longer re-run
CMake
- [x] macOS Intel and ARM CI green (first exercise of the OpenSSL
backend switch)

---

I work off a self-hosted forge, so this GitHub account is quiet; the
configure logs and ninja dirty-node traces from the investigation are
available if anyone wants the raw data.

(AI-assisted)
2026-07-27 19:19:18 -04:00

282 lines
7.8 KiB
Perl
Vendored
Generated

#***************************************************************************
# _ _ ____ _
# Project ___| | | | _ \| |
# / __| | | | |_) | |
# | (__| |_| | _ <| |___
# \___|\___/|_| \_\_____|
#
# Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
#
# This software is licensed as described in the file COPYING, which
# you should have received as part of this distribution. The terms
# are also available at https://curl.se/docs/copyright.html.
#
# You may opt to use, copy, modify, merge, publish, distribute and/or sell
# copies of the Software, and permit persons to whom the Software is
# furnished to do so, under the terms of the COPYING file.
#
# This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
# KIND, either express or implied.
#
# SPDX-License-Identifier: curl
#
###########################################################################
# This module contains miscellaneous functions needed in several parts of
# the test suite.
package testutil;
use strict;
use warnings;
BEGIN {
use base qw(Exporter);
our @EXPORT = qw(
runclient
runclientoutput
setlogfunc
exerunner
subtextfile
subchars
subbase64
subnewlines
subsha256base64file
substrippemfile
);
our @EXPORT_OK = qw(
clearlogs
logmsg
);
}
use Digest::SHA qw(sha256);
use MIME::Base64;
use globalconfig qw(
$torture
$verbose
$dev_null
);
my $logfunc; # optional reference to function for logging
my @logmessages; # array holding logged messages
#######################################################################
# Log an informational message
# If a log callback function was set in setlogfunc, it is called. If not,
# then the log message is buffered until retrieved by clearlogs.
#
# logmsg must only be called by one of the runner_* entry points and functions
# called by them, or else logs risk being lost, since those are the only
# functions that know about and return buffered logs.
sub logmsg {
if(!scalar(@_)) {
return;
}
if(defined $logfunc) {
&$logfunc(@_);
return;
}
push @logmessages, @_;
}
#######################################################################
# Set the function to use for logging
sub setlogfunc {
($logfunc) = @_;
}
#######################################################################
# Clear the buffered log messages after returning them
sub clearlogs {
my $loglines = join('', @logmessages);
undef @logmessages;
return $loglines;
}
#######################################################################
sub includefile {
my ($f, $text) = @_;
open(F, "<$f");
if($text) {
binmode F, ':crlf';
}
my @a = <F>;
close(F);
return join("", @a);
}
sub subtextfile {
my ($thing) = @_;
my $count = ($$thing =~ s/%includetext ([^%]*)%[\n\r]+/includefile($1, 1)/ge);
return $count > 0;
}
sub subchars {
my ($thing) = @_;
$$thing =~ s/%SP/ /g; # space
$$thing =~ s/%TAB/\t/g; # horizontal tab
$$thing =~ s/%CR/\r/g; # carriage return aka \r aka 0x0d
$$thing =~ s/%LT/</g;
$$thing =~ s/%GT/>/g;
$$thing =~ s/%AMP/&/g;
}
sub subbase64 {
my ($thing) = @_;
# cut out the base64 piece
while($$thing =~ s/%b64\[(.*?)\]b64%/%%B64%%/i) {
my $d = $1;
# encode %NN characters
$d =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;
my $enc = encode_base64($d, "");
# put the result into there
$$thing =~ s/%%B64%%/$enc/;
}
# hex decode
while($$thing =~ s/%hex\[(.*?)\]hex%/%%HEX%%/i) {
# decode %NN characters
my $d = $1;
$d =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;
$$thing =~ s/%%HEX%%/$d/;
}
# repeat
while($$thing =~ s/%repeat\[(\d+) x (.*?)\]%/%%REPEAT%%/i) {
# decode %NN characters
my ($d, $n) = ($2, $1);
$d =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;
$n =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;
my $all = $d x $n;
$$thing =~ s/%%REPEAT%%/$all/;
}
# days
while($$thing =~ s/%days\[(.*?)\]/%%DAYS%%/i) {
# convert to now + given days in epoch seconds, align to a 60 second
# boundary. Then provide two alternatives.
my $now = time();
my $d = ($1 * 24 * 3600) + $now + 30;
$d = int($d / 60) * 60;
my $d2 = $d + 60;
$$thing =~ s/%%DAYS%%/%alternatives[$d,$d2]/;
}
# include a file
$$thing =~ s/%include ([^%]*)%[\n\r]+/includefile($1, 0)/ge;
}
my $prevupdate; # module scope so it remembers the last value
sub subnewlines {
my ($force, $thing) = @_;
if($force) {
# enforce CRLF newline
$$thing =~ s/\x0d*\x0a/\x0d\x0a/;
return;
}
if(($$thing =~ /^HTTP\/(1.1|1.0|2|3) ([1-5]|9)[^\x0d]*\z/) ||
($$thing =~ /^(GET|HEAD|POST|PUT|DELETE|CONNECT) \S+ HTTP\/\d+(\.\d+)?/) ||
($$thing =~ /^(SETUP|GET_PARAMETER|OPTIONS|ANNOUNCE|DESCRIBE) \S+ RTSP\/\d+(\.\d+)?/) ||
(($$thing =~ /^[a-z0-9_-]+: [^\x0d]*\z/i) &&
# skip curl error messages
($$thing !~ /^curl: \(\d+\) /))) {
# enforce CRLF newline
$$thing =~ s/\x0d*\x0a/\x0d\x0a/;
$prevupdate = 1;
}
else {
if(($$thing =~ /^\n\z/) && $prevupdate) {
# if there is a blank line after a line we update, we hope it is
# the empty line following headers
$$thing =~ s/\x0a/\x0d\x0a/;
}
$prevupdate = 0;
}
}
#######################################################################
# Run the application under test and return its return code
#
sub runclient {
my ($cmd) = @_;
my $ret = system($cmd);
print "CMD ($ret): $cmd\n" if($verbose && !$torture);
return $ret;
# This is one way to test curl on a remote machine
# my $out = system("ssh $CLIENTIP cd \'$pwd\' \\; \'$cmd\'");
# sleep 2; # time to allow the NFS server to be updated
# return $out;
}
#######################################################################
# Run the application under test and return its stdout
#
sub runclientoutput {
my ($cmd) = @_;
return qx($cmd 2>$dev_null);
# This is one way to test curl on a remote machine
# my @out = qx(ssh $CLIENTIP cd \'$pwd\' \\; \'$cmd\');
# sleep 2; # time to allow the NFS server to be updated
# return @out;
}
#######################################################################
# Return custom tool (e.g. wine or qemu) to run curl binaries.
#
sub exerunner {
if($ENV{'CURL_TEST_EXE_RUNNER'}) {
return $ENV{'CURL_TEST_EXE_RUNNER'} . ' ';
}
return '';
}
sub get_sha256_base64 {
my ($file_path) = @_;
return encode_base64(sha256(do { local $/; open my $fh, '<:raw', $file_path or die $!; <$fh> }), "");
}
sub subsha256base64file {
my ($thing) = @_;
# SHA-256 base64
while($$thing =~ s/%sha256b64file\[(.*?)\]sha256b64file%/%%SHA256B64FILE%%/i) {
my $file_path = $1;
$file_path =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;
my $hash_b64 = get_sha256_base64($file_path);
$$thing =~ s/%%SHA256B64FILE%%/$hash_b64/;
}
}
sub get_file_content {
my ($file_path) = @_;
my $content = do { local $/; open my $fh, '<', $file_path or die $!; <$fh> };
$content =~ s/(^|-----END .*?-----[\r\n]?)(.*?)(-----BEGIN .*?-----|$)/$1$3/gs;
$content =~ s/\r\n/\n/g;
chomp($content);
return $content;
}
sub substrippemfile {
my ($thing) = @_;
# File content substitution
while($$thing =~ s/%strippemfile\[(.*?)\]strippemfile%/%%FILE%%/i) {
my $file_path = $1;
$file_path =~ s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg;
my $file_content = get_file_content($file_path);
$$thing =~ s/%%FILE%%/$file_content/;
}
}
1;