mirror of
https://github.com/open-goal/jak-project
synced 2026-09-10 12:25:21 -04:00
Alexander J. Semenuk
5d5e35fb9b
## Problem Windows builds break with a current local toolchain (Scoop LLVM 22.1.8, CMake 4.4.0, VS 2026), in two independent ways: 1. The build stops at curl's deliberate guard: `#error "no non-blocking method was found/used/set"` in `third-party/curl/lib/nonblock.c`. 2. From the second configure onward, `cmake --build` re-runs CMake in an endless loop (observed 42 consecutive reconfigure cycles in a single build). Likely the same mechanism behind the "endlessly building" VS 2026 note in `docs/setup/dev/vs.md`. ## Root cause 1. `third-party/curl/CMake/CurlTests.c` passes `int *` to `ioctlsocket()`, whose third parameter is `u_long *`. Clang 22 promotes `-Wincompatible-pointer-types` to a hard error in C, so the `HAVE_IOCTLSOCKET_FIONBIO` try_compile silently fails and `curl_config.h` never defines it. Upstream CI does not see this because the windows-2022 runner image ships an older LLVM. GCC 14 promotes the same warning to a hard error, which is very likely the `CurlTests.c.obj` failure reported from MSYS2 in open-goal/jak-project#3551. Upstream curl hit the identical problem with GCC 14 and fixed the probe in curl 8.8.0 (curl/curl#13578). 2. The root CMakeLists copies the build tree's `compile_commands.json` into `<src>/build/` for clangd using `configure_file()`, which registers its input as a configure dependency. CMake rewrites `compile_commands.json` late in every generation, after `CTestTestfile.cmake` and `cmake_install.cmake` (outputs of the same Ninja regen rule), so once the dependency is registered the rule is deterministically dirty and every `ninja` invocation re-runs CMake. A pristine first configure is safe (the file does not exist yet, so the `if(EXISTS ...)` guard skips the copy), which is why the loop looks machine- or IDE-specific. ## Fix 1. Per review, re-vendor `third-party/curl` at the `curl-8_21_0` tag (previously `curl-8_3_0`), which carries the upstream probe fix plus two years of upstream development; `vendor.yaml` updated to match. Adjustments the version jump forced: - curl 8.15 removed the native macOS Secure Transport backend (`CURL_USE_SECTRANSP`), so macOS now builds curl against OpenSSL like Linux. The two macOS workflows install Homebrew `openssl@3` and export `OPENSSL_ROOT_DIR` (keg-only), and the macOS setup docs gained the same two lines. - `CURL_BROTLI` / `CURL_ZSTD` switched to AUTO-detection in curl 8.10; pinned OFF to keep the previous no-compression behavior and avoid silently linking whatever the CI images happen to have. - curl's new top-level `BUILD_EXAMPLES` cache option (default ON) leaked into discord-rpc's identically named option and broke configure at a nonexistent `examples/send-presence` directory; pinned OFF ahead of the third-party subdirectories. The diff is dominated by the mechanical tag-tree swap under `third-party/curl` (linguist-vendored, collapsed in review). The hand-written changes are `CMakeLists.txt`, the two macOS workflows, `docs/setup/system/macos.md`, and `vendor.yaml`. 2. Swap `configure_file()` for `file(COPY ...)`: the same clangd copy with no configure dependency registered. (`file(COPY_FILE ... ONLY_IF_DIFFERENT)` would be cleaner still but requires CMake 3.21, above the declared `cmake_minimum_required(VERSION 3.10)`.) ## Test plan - [x] Fresh `cmake --preset Release-windows-clang` (LLVM 22, no cache seeding) completes and logs `Enabled SSL backends: Schannel`; the FIONBIO probe passes without the previous `#error` - [x] Full Windows Release build from scratch in the branch worktree (all 1422 targets) - [x] goalc-test suite: 1509 passed, 0 failed - [x] Second consecutive configure with `compile_commands.json` present: the regen rule in `build.ninja` has no `compile_commands.json` input; `<src>/build/compile_commands.json` is still refreshed for clangd - [x] Repeated `ninja` invocations after a full build no longer re-run CMake - [x] macOS Intel and ARM CI green (first exercise of the OpenSSL backend switch) --- I work off a self-hosted forge, so this GitHub account is quiet; the configure logs and ninja dirty-node traces from the investigation are available if anyone wants the raw data. (AI-assisted)
169 lines
6.8 KiB
Markdown
Vendored
Generated
169 lines
6.8 KiB
Markdown
Vendored
Generated
<!--
|
|
Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
|
|
|
|
SPDX-License-Identifier: curl
|
|
-->
|
|
|
|
# Verify
|
|
|
|
Do not trust, verify!
|
|
|
|
## Signed releases
|
|
|
|
Every curl release is shipped as a set of tarballs. They all have the exact
|
|
same content but use different archivers, visible by the different file
|
|
extensions used.
|
|
|
|
Each tarball is signed by the curl release manager Daniel. The digital
|
|
signatures for each tarball are always provided. The digital signatures can be
|
|
used to verify that the tarballs were produced by Daniel.
|
|
|
|
If the curl website were breached and fake curl releases were
|
|
provided, they could be detected using these signatures.
|
|
|
|
Daniel's public GPG key: [27ED EAF2 2F3A BCEB 50DB 9A12 5CC9 08FD B71E 12C2](https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x27edeaf22f3abceb50db9a125cc908fdb71e12c2)
|
|
|
|
## Reproducible releases
|
|
|
|
The curl project ships *reproducible releases*. This means that everyone is
|
|
able - and encouraged - to independently verify the contents of every curl
|
|
release. Verify that it contains exactly the bits that are supposed to be in
|
|
the release and nothing extra.
|
|
|
|
The curl releases are generated using a Docker image to make it easy to get an
|
|
identical setup. To verify an existing curl release, we provide a convenient
|
|
script that generates a new curl release from source code and then compares
|
|
this newly generated release tarball with the tarball file you downloaded from
|
|
curl.se.
|
|
|
|
For full verification, invoke the script inside an up-to-date curl source code
|
|
git repository. Without a git repository present, it does a lighter check by
|
|
rebuilding the release using the files in the tarball.
|
|
|
|
Note: full verification mode checks out the release tag in your repository.
|
|
Run it in a clean working tree (no local changes) or a dedicated clone.
|
|
|
|
Invoke it like this:
|
|
|
|
git clone https://github.com/curl/curl
|
|
cd curl
|
|
mv [download-dir]/curl-8.19.0.tar.xz .
|
|
./scripts/verify-release curl-8.19.0.tar.xz
|
|
|
|
A successful check ends up with a final output similar to:
|
|
|
|
curl-8.19.0.tar.xz: OK
|
|
|
|
By verifying the release tarballs, you verify that Daniel does not infect the
|
|
release on purpose or involuntarily because of anything malicious running in
|
|
his setup.
|
|
|
|
### Verify the verify
|
|
|
|
Of course you should not blindly trust the verification script. It is short
|
|
and simple and should be quick to verify. Or you write your own script that
|
|
you trust, to do the same job.
|
|
|
|
## Source code
|
|
|
|
How do you then verify that what is in git is fine to build a product from?
|
|
|
|
In the curl project we verify the source code in multiple ways, and one way to
|
|
gain trust is to verify and review our testing procedures.
|
|
|
|
- we have a consistent code style (invalid style causes errors)
|
|
|
|
- we ban and avoid a number of "sensitive" and "hard-to-use" C functions (use
|
|
of such functions causes errors)
|
|
|
|
- we have a ceiling for complexity in functions to keep them easy to follow,
|
|
read and understand (failing to do so causes errors)
|
|
|
|
- we review all pull requests before merging, both with humans and with bots.
|
|
We link back commits to their origin pull requests in commit messages.
|
|
|
|
- we ban use of "binary blobs" in git to not provide means for malicious
|
|
actors to bundle encrypted payloads (trying to include a blob causes errors)
|
|
|
|
- every single file in the git repository has a clear copyright and license
|
|
statement. Complete knowledge and tracking of provenience.
|
|
|
|
- we actively avoid base64 encoded chunks as they too could function as ways
|
|
to obfuscate malicious contents
|
|
|
|
- we forbid and prevent git force push on the master branch. History cannot be
|
|
rewritten.
|
|
|
|
- we ban most uses of UTF-8 in code and documentation to avoid easily mixed
|
|
up Unicode characters that look like other characters. (adding Unicode
|
|
characters causes errors)
|
|
|
|
- we document everything to make it clear how things are supposed to work. No
|
|
surprises. Lots of documentation is tested and verified in addition to
|
|
spellchecks and consistent wording.
|
|
|
|
- we have thousands of tests and we add test cases for (ideally) every
|
|
functionality. Finding "white spots" and adding coverage is a top priority.
|
|
curl runs on countless operating systems, CPU architectures and you can
|
|
build curl in billions of different configuration setups: not every
|
|
combination is practically possible to test
|
|
|
|
- we build curl and run tests in over two hundred CI jobs that are run for
|
|
every commit and every PR. We do not merge commits that have unexplained
|
|
test failures.
|
|
|
|
- we run all tests as "torture tests", where each test case is rerun to have
|
|
every invoked fallible function call fail once each, to make sure curl
|
|
never leaks memory or crashes due to this.
|
|
|
|
- we build curl in CI with the most picky compiler options enabled and we
|
|
never allow compiler warnings to linger. We always use `-Werror` that
|
|
converts warnings to errors and fail the builds.
|
|
|
|
- we run all tests using valgrind and several combinations of sanitizers to
|
|
find and reduce the risk for memory problems, undefined behavior and
|
|
similar
|
|
|
|
- we keep running static code analyzers on the code, both traditional ones
|
|
(clang-tidy, CodeSonar, Coverity) but also new generation AI powered ones
|
|
like Zeropath and Codex Security.
|
|
|
|
- we run fuzzing on curl: non-stop as part of Google's OSS-Fuzz project, but
|
|
also briefly as part of the CI setup for every commit and PR
|
|
|
|
- we make sure that the CI jobs we have for curl never "write back" to curl.
|
|
They access the source repository read-only and even if they would be
|
|
breached, they cannot infect or taint source code.
|
|
|
|
- we run `zizmor` and other code analyzer tools on the CI job config scripts
|
|
to reduce the risk of us running or using insecure CI jobs.
|
|
|
|
- we do reproducible releases to allow anyone to verify that the contents is
|
|
untainted
|
|
|
|
- we digitally sign releases, git tags and git commits
|
|
|
|
- there is a git backup on [codeberg](https://codeberg.org/curl/) for enhanced
|
|
resilience to infrastructure disturbance
|
|
|
|
- we are committed to always fix reported vulnerabilities in the following
|
|
release. Security problems never linger around once they have been
|
|
reported.
|
|
|
|
- we document everything and every detail about all curl vulnerabilities ever
|
|
reported
|
|
|
|
- our code has been audited several times by external security experts, and
|
|
the few issues that have been detected in those were immediately addressed
|
|
|
|
- Strong two-factor authentication on GitHub is mandatory for all committers
|
|
|
|
- our commitment to never breaking ABI or API allows all users to easily
|
|
upgrade to new releases. This enables users to run recent security-fixed
|
|
versions instead of legacy insecure versions.
|
|
|
|
- we have a vulnerability disclosure program that allows researchers to submit
|
|
suspected vulnerabilities in a private and secure fashion, so that we can
|
|
work on fixing curl and announcing the flaw in a responsible manner to
|
|
minimize risks for users.
|