uv itself is a large package with many dependencies and lots of
features. To build a package using the uv build backend, you shouldn't
have to download and install the entirety of uv. For platform where we
don't provide wheels, it should be possible and fast to compile the uv
build backend. To that end, we're introducing a python package that
contains a trimmed down version of uv that only contains the build
backend, with a minimal dependency tree in rust.
The `uv_build` package is publish from CI just like uv itself. It is
part of the workspace, but has much less dependencies for its own
binary. We're using cargo deny to enforce that the network stack is not
part of the dependencies. A new build profile ensure we're getting the
minimum possible binary size for a rust binary.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
I believe `zlib-rs` is now a better choice on ARM and x86, so I'm just
going to assume it's a better choice everywhere. It's much easier to
build (removes our CMake dependency), and in my benchmarking, it's
substantially faster on ARM and faster or ~exactly even on my x86
Windows machine.
We migrated to `zlib-rs` once before (#9184); however, I later reverted
it as I learned that they were only doing compile-time feature
detection, and so `zlib-rs` was meaningfully slower on x86. They now
perform runtime feature detection:
https://trifectatech.org/blog/zlib-rs-is-faster-than-c/.
To benchmark, I wrote a script to create a local Simple API-compliant
registry (see the commit history) for a single package. Then I ran the
`install-cold` benchmark against that registry to install NumPy.
On ARM:
```
❯ uv run resolver --uv-pip-path ../../zlib-ng --uv-pip-path ../../zlib-rs \
--benchmark install-cold \
req.txt --warmup 10 --min-runs 30
Benchmark 1: ../../zlib-ng (install-cold)
Time (mean ± σ): 165.7 ms ± 34.7 ms [User: 64.4 ms, System: 93.2 ms]
Range (min … max): 141.8 ms … 293.2 ms 30 runs
Benchmark 2: ../../zlib-rs (install-cold)
Time (mean ± σ): 150.9 ms ± 16.2 ms [User: 57.4 ms, System: 86.4 ms]
Range (min … max): 135.3 ms … 202.4 ms 30 runs
Summary
../../zlib-rs (install-cold) ran
1.10 ± 0.26 times faster than ../../zlib-ng (install-cold)
```
I benchmarked this about 100 times on my Windows machine and found it
difficult to conclude anything beyond "They're nearly the same". Here's
an example:
```
PS C:\Users\crmar\workspace\puffin> hyperfine --prepare "uv venv" "zlib-rs.exe pip sync ./scripts/benchmark/req.txt" "zlib-ng.exe pip sync ./scripts/benchmark/req.txt" "zlib-rs.exe pip sync ./scripts/benchmark/req.txt" "zlib-ng.exe pip sync ./scripts/benchmark/req.txt" --runs 10 --warmup 5
Benchmark 1: zlib-rs.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 240.6 ms ± 10.8 ms [User: 6.1 ms, System: 92.2 ms]
Range (min … max): 229.4 ms … 267.9 ms 10 runs
Benchmark 2: zlib-ng.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 241.3 ms ± 6.2 ms [User: 7.7 ms, System: 90.6 ms]
Range (min … max): 233.9 ms … 252.1 ms 10 runs
Benchmark 3: zlib-rs.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 242.8 ms ± 7.7 ms [User: 6.2 ms, System: 23.4 ms]
Range (min … max): 236.1 ms … 262.8 ms 10 runs
Benchmark 4: zlib-ng.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 245.9 ms ± 5.7 ms [User: 1.5 ms, System: 59.4 ms]
Range (min … max): 240.9 ms … 257.3 ms 10 runs
Summary
zlib-rs.exe pip sync ./scripts/benchmark/req.txt ran
1.00 ± 0.05 times faster than zlib-ng.exe pip sync ./scripts/benchmark/req.txt
1.01 ± 0.06 times faster than zlib-rs.exe pip sync ./scripts/benchmark/req.txt
1.02 ± 0.05 times faster than zlib-ng.exe pip sync ./scripts/benchmark/req.txt
```
Closes#11885.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Follow on to #11706. In the original PR, I tried to solve the issue by
getting rid of the `ctrlc::set_handler` call. Unfortunately, this didn't
work on windows due to an issue with the console crate. console 0.15.11
includes https://github.com/console-rs/console/pull/235, which resolves
the issue, so now we can get rid of the call.
<!-- What's the purpose of the change? What does it do, and why? -->
This change is not super important but I still think it's worthwhile.
For one, spinning up a background thread to handle `SIGINT`s when we're
going to be raising the `SIGINT` from within the function is more
technical complexity than needed, now that there's an easy way to
explicitly catch the Ctrl-C from the terminal input. Secondly,
`ctrlc::set_handler`'s
[docs](https://docs.rs/ctrlc/3.4.5/ctrlc/fn.set_handler.html) advise
that you set the handler just once, at the beginning of the program, so
this use seems somewhat error prone. In fact, uv already has a second
[callsite](461f4d9007/crates/uv/src/commands/project/add.rs (L596-L611))
for this function (though I'm not sure if the two callsites could
currently ever both occur on the same run of uv)
## Test Plan
I've tested this manually on linux (WSL ubuntu) and windows, though not
on aarch64-apple-darwin as I don't have a machine running that. I would
appreciate if someone would double-check that it works on such machines.
As discussed in the original PR, this change is pretty hard to test due
to the fact that the behavior only occurs if stderr is connected to a
tty. I experimented with using pseudoterminals to test this but it's
still quite tricky due to the lack of x-platform non-blocking reads on
the pty.
<!-- How was it tested? -->
Fixes#11217
By default, a 64-bit uv does not see a 32-bit global (HKLM) installation
of Python in the registry
(https://github.com/astral-sh/uv/issues/11217). To work around this, we
manually request both 32-bit and 64-bit access using registry access
flags (https://peps.python.org/pep-0514/#sample-code). The flags have no
effect on 32-bit (https://stackoverflow.com/a/12796797/3549270).
This effect is that there is an asymmetry between discovery modes: For
the registry-based discovery using PEP 514, we discover both 32-bit and
64-bit Pythons, while for managed installations, we are stricter and
only discover those matching in bit-ness.
I tested this manually with an additional 32-bit installation of CPython
on a 64-bit machine and windows with 32-bit and 64-bit (x86_64 and i686)
builds of uv.
Currently, we're using both the official `windows-*` with
`windows-registry` crates as well as `winreg`, an older,
community-maintained crate.
To unify the codebase, we follow the lead of rustup that already
performed this migration
(bce3ed67d2).
This is also a prerequisite to unblock the unification of the
windows-sys crate versions.
I've manually tested that `uv tool update-shell` works for adding to
PATH and correctly detects when PATH was already added.
We want to build `uv-build` without depending on the network crates. In
preparation for that, we split uv-git into uv-git and uv-git-types,
where only uv-git depends on reqwest, so that uv-build can use
uv-git-types.
In https://github.com/astral-sh/tokio-tar/pull/2, we accidentally
changed the `target_base` from the target base to the parent of the
file. This would cause hardlink unpacking to fail.
Example: A hardlink at `hardlinked-0.1.0/pyproject.toml` pointing to
`hardlinked-0.1.0/pyproject.toml.real` would try pointing to
`hardlinked-0.1.0/hardlinked-0.1.0/pyproject.toml.real` instead and fail
the unpacking.
The actual fix is in astral-tokio-tar, on the uv side there are only tests.
Fixes#11213
Looks like the set based prioritize tracking from
https://github.com/pubgrub-rs/pubgrub/pull/313 is a slight speedup.
I assume the changed derivation tree in the error snapshot is due to
out-of-sync virtual package priorities, while the main package priority
defining the solution remains stable.
```
$ hyperfine --warmup 2 "./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal" "./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal"
Benchmark 1: ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 115.0 ms ± 4.8 ms [User: 131.0 ms, System: 113.6 ms]
Range (min … max): 108.1 ms … 125.8 ms 25 runs
Benchmark 2: ./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 105.4 ms ± 2.6 ms [User: 118.5 ms, System: 113.5 ms]
Range (min … max): 101.1 ms … 111.9 ms 28 runs
Summary
./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal ran
1.09 ± 0.05 times faster than ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
```
## Summary
This lets us drop a dependency entirely. `percent-encoding` is used by
`url` and so is already in the graph, whereas `urlencoding` isn't used
by anything else.
cosign uses the GitHub action ID token to retrieve an ephemeral code
signing certificate from Fulcio, and store the signature in the Rekor
transparency log.
Once an image has been successfully signed, you should be able to verify
the signature with:
```sh
cosign verify ghcr.io/astral-sh/uv:latest --certificate-identity-regexp='.*' --certificate-oidc-issuer-regexp='.*'
```
Closes#8670
## Summary
This appears to be a consistent 1% performance improvement and should
also reduce memory quite a bit. We've also decided to use these for
markers, so it's nice to use the same optimization here.
```
❯ hyperfine "./uv pip compile --universal scripts/requirements/airflow.in" "./arcstr pip compile --universal scripts/requirements/airflow.in" --min-runs 50 --warmup 20
Benchmark 1: ./uv pip compile --universal scripts/requirements/airflow.in
Time (mean ± σ): 136.3 ms ± 4.0 ms [User: 139.1 ms, System: 241.9 ms]
Range (min … max): 131.5 ms … 149.5 ms 50 runs
Benchmark 2: ./arcstr pip compile --universal scripts/requirements/airflow.in
Time (mean ± σ): 134.9 ms ± 3.2 ms [User: 137.6 ms, System: 239.0 ms]
Range (min … max): 130.1 ms … 151.8 ms 50 runs
Summary
./arcstr pip compile --universal scripts/requirements/airflow.in ran
1.01 ± 0.04 times faster than ./uv pip compile --universal scripts/requirements/airflow.in
```
This should be essentially the exact same behaviour, but backon is a
total API redesign, so things had to be expressed slightly differently.
Overall I think the code is more readable, which is nice.
Fixes#10001
## Summary
Allows uv to recognize the ARMv5TE platform. This platform is currently
supported on Debian distributions. It is an older 32 bit platform mostly
used in embedded devices, currently in rust tier 2.5 so it requires
cross compilation.
Fixes#10157 .
## Test Plan
Tested directly on device by applying a slightly different patch to tag
0.5.4 which is used by the current Home Assistant version (2024.12.5).
After the patch Home Assistant is able to recognize the Python venv and
setup its dependencies.
Patched uv was built with
```
$ CARGO_TARGET_ARMV5TE_UNKNOWN_LINUX_GNUEABI_LINKER="/usr/bin/arm-linux-gnueabi-gcc" maturin build --release --target armv5te-unknown-linux-gnueabi --manylinux off
```
The target wheel was then moved on the device and installed via pip
install.
uv gives priorities to packages by package name, not by virtual package
(`PubGrubPackage`). pubgrub otoh when prioritizing order the virtual
packages. When the order of virtual packages changes, uv changes its
resolutions and error messages. This means uv was depending on
implementation details of pubgrub's prioritization caching.
This broke with https://github.com/pubgrub-rs/pubgrub/pull/299, which
added a tiebreaker term that made pubgrub's sorting deterministic given
a deterministic ordering of allocating the packages (which happens the
first time pubgrub sees a package).
The new custom tiebreaker decreases the difference to upstream pubgrub.
Background reading: https://github.com/astral-sh/uv/issues/8157
Companion PR: https://github.com/astral-sh/pubgrub/pull/36
Requires for test coverage: https://github.com/astral-sh/packse/pull/230
When two packages A and B conflict, we have the option to choose a lower
version of A, or a lower version of B. Currently, we determine this by
the order we saw a package (assuming equal specificity of the
requirement): If we saw A before B, we pin A until all versions of B are
exhausted. This can lead to undesirable outcomes, from cases where it's
just slow (sentry) to others cases without lower bounds where be
backtrack to a very old version of B. This old version may fail to build
(terminating the resolution), or it's a version so old that it doesn't
depend on A (or the shared conflicting package) anymore - but also is
too old for the user's application (fastapi). #8157 collects such cases,
and the `wrong-backtracking` packse scenario contains a minimized
example.
We try to solve this by tracking which packages are "A"s, culprits, and
"B"s, affected, and manually interfering with project selection and
backtracking. Whenever a version we just chose is rejected, we give the
current package a counter for being affected, and the package it
conflicted with a counter for being a culprit. If a package accumulates
more counts than a threshold, we reprioritize: Undecided after the
culprits, after the affected, after packages that only have a single
version (URLs, `==<version>`). We then ask pubgrub to backtrack just
before the culprit. Due to the changed priorities, we now select package
B, the affected, instead of package A, the culprit.
To do this efficiently, we ask pubgrub for the incompatibility that
caused backtracking, or just the last version to be discarded (due to
its dependencies). For backtracking, we use the last incompatibility
from unit propagation as a heuristic. When a version is discarded
because one of its dependencies conflicts with the partial solution, the
incompatibility tells us the package in the partial solution that
conflicted.
We only backtrack once per package, on the first time it passes the
threshold. This prevents backtracking loops in which we make the same
decisions over and over again. But we also changed the priority, so that
we shouldn't take the same path even after the one time we backtrack (it
would defeat the purpose of this change).
There are some parameters that can be tweaked: Currently, the threshold
is set to 5, which feels not too eager with so me of the conflicts that
we want to tolerate but also changes strategies quickly. The relative
order of the new priorities can also be changed, as for each (A, B) pair
the priority of B is afterwards lower than that for A. Currently,
culprits capture conflict for the whole package, but we could limit that
to a specific version. We could discard conflict counters after
backtracking instead of keeping them eternally as we do now. Note that
we're always taking about pairs (A, B), but in practice we track
individual packages, not pairs.
A case that we wouldn't capture is when B is only introduced to the
dependency graph after A, but I think that would require cyclical
dependency for A and B to conflict? There may also be cases where
looking at the last incompatibility is insufficient.
Another example that we can't repair with prioritization is
urllib3/boto3/botocore: We actually have to check all the newer versions
of boto3 and botocore to identify the version that allows with the older
urllib3, no shortcuts allowed.
```
urllib3<1.25.4
boto3
```
All examples I tested were cases with two packages where we only had to
switch the order, so I've abstracted them into a single packse case.
This PR changes the resolution for certain paths, and there is the risk
for regressions.
Fixes#8157
---
All tested examples improved.
Input fastapi:
```text
starlette<=0.36.0
fastapi<=0.115.2
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/fastapi.txt
annotated-types==0.7.0
anyio==4.6.0
fastapi==0.1.17
idna==3.10
pydantic==2.9.2
pydantic-core==2.23.4
sniffio==1.3.1
starlette==0.36.0
typing-extensions==4.12.2
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/fastapi.txt
annotated-types==0.7.0
anyio==4.6.0
fastapi==0.109.1
idna==3.10
pydantic==2.9.2
pydantic-core==2.23.4
sniffio==1.3.1
starlette==0.35.1
typing-extensions==4.12.2
```
Input xarray:
```text
xarray[accel]
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/xarray-accel.txt
bottleneck==1.4.0
flox==0.9.13
llvmlite==0.36.0
numba==0.53.1
numbagg==0.8.2
numpy==2.1.1
numpy-groupies==0.11.2
opt-einsum==3.4.0
packaging==24.1
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2024.2
scipy==1.14.1
setuptools==75.1.0
six==1.16.0
toolz==0.12.1
tzdata==2024.2
xarray==2024.9.0
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/xarray-accel.txt
bottleneck==1.4.0
flox==0.9.13
llvmlite==0.43.0
numba==0.60.0
numbagg==0.8.2
numpy==2.0.2
numpy-groupies==0.11.2
opt-einsum==3.4.0
packaging==24.1
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2024.2
scipy==1.14.1
six==1.16.0
toolz==0.12.1
tzdata==2024.2
xarray==2024.9.0
```
Input sentry: The resolution is identical, but arrived at much faster:
main tries 69 versions (sentry-kafka-schemas: 63), PR tries 12 versions
(sentry-kafka-schemas: 6; 5 times conflicting, then once the right
version).
```text
python-rapidjson<=1.20,>=1.4
sentry-kafka-schemas<=0.1.113,>=0.1.50
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/sentry.txt
fastjsonschema==2.20.0
msgpack==1.1.0
python-rapidjson==1.8
pyyaml==6.0.2
sentry-kafka-schemas==0.1.111
typing-extensions==4.12.2
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/sentry.txt
fastjsonschema==2.20.0
msgpack==1.1.0
python-rapidjson==1.8
pyyaml==6.0.2
sentry-kafka-schemas==0.1.111
typing-extensions==4.12.2
```
Input apache-beam
```text
# Run on Python 3.10
dill<0.3.9,>=0.2.2
apache-beam<=2.49.0
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.10 --exclude-newer 2024-10-01 --no-annotate debug/apache-beam.txt
× Failed to download and build `apache-beam==2.0.0`
╰─▶ Build backend failed to determine requirements with `build_wheel()` (exit status: 1)
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.10 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/apache-beam.txt
apache-beam==2.49.0
certifi==2024.8.30
charset-normalizer==3.3.2
cloudpickle==2.2.1
crcmod==1.7
dill==0.3.1.1
dnspython==2.6.1
docopt==0.6.2
fastavro==1.9.7
fasteners==0.19
grpcio==1.66.2
hdfs==2.7.3
httplib2==0.22.0
idna==3.10
numpy==1.24.4
objsize==0.6.1
orjson==3.10.7
proto-plus==1.24.0
protobuf==4.23.4
pyarrow==11.0.0
pydot==1.4.2
pymongo==4.10.0
pyparsing==3.1.4
python-dateutil==2.9.0.post0
pytz==2024.2
regex==2024.9.11
requests==2.32.3
six==1.16.0
typing-extensions==4.12.2
urllib3==2.2.3
zstandard==0.23.0
```
## Summary
This is blocking the release (#9793). We seem to have hit some sort of
limit that's causing builds to fail on this target. It's a Tier 3 Rust
target with _unknown_ (???) `std` support (see the question mark
[here](https://doc.rust-lang.org/rustc/platform-support.html)).
This lint fires when an `if` negates the condition. The lint
instead suggests that the condition should be un-negated and
the `if` and `else` bodies flipped.
I find this to be a pretty annoying lint, because sometimes
I want to keep the order of the `if` and `else` bodies as-is,
and not make it subject to whether the primary condition is
negated or not. Sometimes it's for linear scanning reasons
(where the `if` block is smaller), and sometimes it's for
"code parallelism" reasons (i.e., this block of code looks like
another block of code intentionally).
Moreover, I don't think the benefits of this lint are very
big. I do agree that sometimes negating a conditional can make
it harder to read (e.g., a double negative), but it's nowhere
near a universal truth enough to ban it outright.
