Looks like the set based prioritize tracking from
https://github.com/pubgrub-rs/pubgrub/pull/313 is a slight speedup.
I assume the changed derivation tree in the error snapshot is due to
out-of-sync virtual package priorities, while the main package priority
defining the solution remains stable.
```
$ hyperfine --warmup 2 "./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal" "./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal"
Benchmark 1: ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 115.0 ms ± 4.8 ms [User: 131.0 ms, System: 113.6 ms]
Range (min … max): 108.1 ms … 125.8 ms 25 runs
Benchmark 2: ./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal
Time (mean ± σ): 105.4 ms ± 2.6 ms [User: 118.5 ms, System: 113.5 ms]
Range (min … max): 101.1 ms … 111.9 ms 28 runs
Summary
./uv-branch pip compile --no-progress scripts/requirements/airflow.in --universal ran
1.09 ± 0.05 times faster than ./uv-main pip compile --no-progress scripts/requirements/airflow.in --universal
```
## Summary
This lets us drop a dependency entirely. `percent-encoding` is used by
`url` and so is already in the graph, whereas `urlencoding` isn't used
by anything else.
cosign uses the GitHub action ID token to retrieve an ephemeral code
signing certificate from Fulcio, and store the signature in the Rekor
transparency log.
Once an image has been successfully signed, you should be able to verify
the signature with:
```sh
cosign verify ghcr.io/astral-sh/uv:latest --certificate-identity-regexp='.*' --certificate-oidc-issuer-regexp='.*'
```
Closes#8670
## Summary
This appears to be a consistent 1% performance improvement and should
also reduce memory quite a bit. We've also decided to use these for
markers, so it's nice to use the same optimization here.
```
❯ hyperfine "./uv pip compile --universal scripts/requirements/airflow.in" "./arcstr pip compile --universal scripts/requirements/airflow.in" --min-runs 50 --warmup 20
Benchmark 1: ./uv pip compile --universal scripts/requirements/airflow.in
Time (mean ± σ): 136.3 ms ± 4.0 ms [User: 139.1 ms, System: 241.9 ms]
Range (min … max): 131.5 ms … 149.5 ms 50 runs
Benchmark 2: ./arcstr pip compile --universal scripts/requirements/airflow.in
Time (mean ± σ): 134.9 ms ± 3.2 ms [User: 137.6 ms, System: 239.0 ms]
Range (min … max): 130.1 ms … 151.8 ms 50 runs
Summary
./arcstr pip compile --universal scripts/requirements/airflow.in ran
1.01 ± 0.04 times faster than ./uv pip compile --universal scripts/requirements/airflow.in
```
This should be essentially the exact same behaviour, but backon is a
total API redesign, so things had to be expressed slightly differently.
Overall I think the code is more readable, which is nice.
Fixes#10001
## Summary
Allows uv to recognize the ARMv5TE platform. This platform is currently
supported on Debian distributions. It is an older 32 bit platform mostly
used in embedded devices, currently in rust tier 2.5 so it requires
cross compilation.
Fixes#10157 .
## Test Plan
Tested directly on device by applying a slightly different patch to tag
0.5.4 which is used by the current Home Assistant version (2024.12.5).
After the patch Home Assistant is able to recognize the Python venv and
setup its dependencies.
Patched uv was built with
```
$ CARGO_TARGET_ARMV5TE_UNKNOWN_LINUX_GNUEABI_LINKER="/usr/bin/arm-linux-gnueabi-gcc" maturin build --release --target armv5te-unknown-linux-gnueabi --manylinux off
```
The target wheel was then moved on the device and installed via pip
install.
uv gives priorities to packages by package name, not by virtual package
(`PubGrubPackage`). pubgrub otoh when prioritizing order the virtual
packages. When the order of virtual packages changes, uv changes its
resolutions and error messages. This means uv was depending on
implementation details of pubgrub's prioritization caching.
This broke with https://github.com/pubgrub-rs/pubgrub/pull/299, which
added a tiebreaker term that made pubgrub's sorting deterministic given
a deterministic ordering of allocating the packages (which happens the
first time pubgrub sees a package).
The new custom tiebreaker decreases the difference to upstream pubgrub.
Background reading: https://github.com/astral-sh/uv/issues/8157
Companion PR: https://github.com/astral-sh/pubgrub/pull/36
Requires for test coverage: https://github.com/astral-sh/packse/pull/230
When two packages A and B conflict, we have the option to choose a lower
version of A, or a lower version of B. Currently, we determine this by
the order we saw a package (assuming equal specificity of the
requirement): If we saw A before B, we pin A until all versions of B are
exhausted. This can lead to undesirable outcomes, from cases where it's
just slow (sentry) to others cases without lower bounds where be
backtrack to a very old version of B. This old version may fail to build
(terminating the resolution), or it's a version so old that it doesn't
depend on A (or the shared conflicting package) anymore - but also is
too old for the user's application (fastapi). #8157 collects such cases,
and the `wrong-backtracking` packse scenario contains a minimized
example.
We try to solve this by tracking which packages are "A"s, culprits, and
"B"s, affected, and manually interfering with project selection and
backtracking. Whenever a version we just chose is rejected, we give the
current package a counter for being affected, and the package it
conflicted with a counter for being a culprit. If a package accumulates
more counts than a threshold, we reprioritize: Undecided after the
culprits, after the affected, after packages that only have a single
version (URLs, `==<version>`). We then ask pubgrub to backtrack just
before the culprit. Due to the changed priorities, we now select package
B, the affected, instead of package A, the culprit.
To do this efficiently, we ask pubgrub for the incompatibility that
caused backtracking, or just the last version to be discarded (due to
its dependencies). For backtracking, we use the last incompatibility
from unit propagation as a heuristic. When a version is discarded
because one of its dependencies conflicts with the partial solution, the
incompatibility tells us the package in the partial solution that
conflicted.
We only backtrack once per package, on the first time it passes the
threshold. This prevents backtracking loops in which we make the same
decisions over and over again. But we also changed the priority, so that
we shouldn't take the same path even after the one time we backtrack (it
would defeat the purpose of this change).
There are some parameters that can be tweaked: Currently, the threshold
is set to 5, which feels not too eager with so me of the conflicts that
we want to tolerate but also changes strategies quickly. The relative
order of the new priorities can also be changed, as for each (A, B) pair
the priority of B is afterwards lower than that for A. Currently,
culprits capture conflict for the whole package, but we could limit that
to a specific version. We could discard conflict counters after
backtracking instead of keeping them eternally as we do now. Note that
we're always taking about pairs (A, B), but in practice we track
individual packages, not pairs.
A case that we wouldn't capture is when B is only introduced to the
dependency graph after A, but I think that would require cyclical
dependency for A and B to conflict? There may also be cases where
looking at the last incompatibility is insufficient.
Another example that we can't repair with prioritization is
urllib3/boto3/botocore: We actually have to check all the newer versions
of boto3 and botocore to identify the version that allows with the older
urllib3, no shortcuts allowed.
```
urllib3<1.25.4
boto3
```
All examples I tested were cases with two packages where we only had to
switch the order, so I've abstracted them into a single packse case.
This PR changes the resolution for certain paths, and there is the risk
for regressions.
Fixes#8157
---
All tested examples improved.
Input fastapi:
```text
starlette<=0.36.0
fastapi<=0.115.2
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/fastapi.txt
annotated-types==0.7.0
anyio==4.6.0
fastapi==0.1.17
idna==3.10
pydantic==2.9.2
pydantic-core==2.23.4
sniffio==1.3.1
starlette==0.36.0
typing-extensions==4.12.2
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/fastapi.txt
annotated-types==0.7.0
anyio==4.6.0
fastapi==0.109.1
idna==3.10
pydantic==2.9.2
pydantic-core==2.23.4
sniffio==1.3.1
starlette==0.35.1
typing-extensions==4.12.2
```
Input xarray:
```text
xarray[accel]
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/xarray-accel.txt
bottleneck==1.4.0
flox==0.9.13
llvmlite==0.36.0
numba==0.53.1
numbagg==0.8.2
numpy==2.1.1
numpy-groupies==0.11.2
opt-einsum==3.4.0
packaging==24.1
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2024.2
scipy==1.14.1
setuptools==75.1.0
six==1.16.0
toolz==0.12.1
tzdata==2024.2
xarray==2024.9.0
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/xarray-accel.txt
bottleneck==1.4.0
flox==0.9.13
llvmlite==0.43.0
numba==0.60.0
numbagg==0.8.2
numpy==2.0.2
numpy-groupies==0.11.2
opt-einsum==3.4.0
packaging==24.1
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2024.2
scipy==1.14.1
six==1.16.0
toolz==0.12.1
tzdata==2024.2
xarray==2024.9.0
```
Input sentry: The resolution is identical, but arrived at much faster:
main tries 69 versions (sentry-kafka-schemas: 63), PR tries 12 versions
(sentry-kafka-schemas: 6; 5 times conflicting, then once the right
version).
```text
python-rapidjson<=1.20,>=1.4
sentry-kafka-schemas<=0.1.113,>=0.1.50
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/sentry.txt
fastjsonschema==2.20.0
msgpack==1.1.0
python-rapidjson==1.8
pyyaml==6.0.2
sentry-kafka-schemas==0.1.111
typing-extensions==4.12.2
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/sentry.txt
fastjsonschema==2.20.0
msgpack==1.1.0
python-rapidjson==1.8
pyyaml==6.0.2
sentry-kafka-schemas==0.1.111
typing-extensions==4.12.2
```
Input apache-beam
```text
# Run on Python 3.10
dill<0.3.9,>=0.2.2
apache-beam<=2.49.0
```
```
# BEFORE
$ uv pip --no-progress compile -p 3.10 --exclude-newer 2024-10-01 --no-annotate debug/apache-beam.txt
× Failed to download and build `apache-beam==2.0.0`
╰─▶ Build backend failed to determine requirements with `build_wheel()` (exit status: 1)
# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.10 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/apache-beam.txt
apache-beam==2.49.0
certifi==2024.8.30
charset-normalizer==3.3.2
cloudpickle==2.2.1
crcmod==1.7
dill==0.3.1.1
dnspython==2.6.1
docopt==0.6.2
fastavro==1.9.7
fasteners==0.19
grpcio==1.66.2
hdfs==2.7.3
httplib2==0.22.0
idna==3.10
numpy==1.24.4
objsize==0.6.1
orjson==3.10.7
proto-plus==1.24.0
protobuf==4.23.4
pyarrow==11.0.0
pydot==1.4.2
pymongo==4.10.0
pyparsing==3.1.4
python-dateutil==2.9.0.post0
pytz==2024.2
regex==2024.9.11
requests==2.32.3
six==1.16.0
typing-extensions==4.12.2
urllib3==2.2.3
zstandard==0.23.0
```
## Summary
This is blocking the release (#9793). We seem to have hit some sort of
limit that's causing builds to fail on this target. It's a Tier 3 Rust
target with _unknown_ (???) `std` support (see the question mark
[here](https://doc.rust-lang.org/rustc/platform-support.html)).
This lint fires when an `if` negates the condition. The lint
instead suggests that the condition should be un-negated and
the `if` and `else` bodies flipped.
I find this to be a pretty annoying lint, because sometimes
I want to keep the order of the `if` and `else` bodies as-is,
and not make it subject to whether the primary condition is
negated or not. Sometimes it's for linear scanning reasons
(where the `if` block is smaller), and sometimes it's for
"code parallelism" reasons (i.e., this block of code looks like
another block of code intentionally).
Moreover, I don't think the benefits of this lint are very
big. I do agree that sometimes negating a conditional can make
it harder to read (e.g., a double negative), but it's nowhere
near a universal truth enough to ban it outright.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
While working on potential bug fixes with temporary files on Windows (I
think I am currently ecountering the same issue as #2810)
I noticed that sub-workspaces were not all having the same `tempfile`
version. And they were not relying on the cargo root project dependency.
I don't know at all if it was done on purpose or not.
(I also wanted to override the root dependency with a local source but
it was not possible due to sub-workspaces not relying on the same).
The root lockfile already pinned to the `3.14.0`. Some sub-workspaces
were depending on the `3.12.0`, some others on the `3.14.0`. So I
updated the root `Cargo.toml` to the `3.14.0`.
Feel free to decline if it was done on purpose! No worries at all
🙂
Thanks!
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
All units tests are still passing on my side. Let's see with the
pull-request CI 😄
## Summary
A lot of good new lints, and most importantly, error stabilizations. I
tried to find a few usages of the new stabilizations, but I'm sure there
are more.
IIUC, this _does_ require bumping our MSRV.
When building only a single crate in the workspace to run its tests, we
often recompile a lot of other, unrelated crates. Whenever cargo has a
different set of crate features, it needs to recompile. By moving some
features (non-exhaustive for now) to the workspace level, we always
activate them an avoid recompiling.
The cargo docs mismatch the behavior of cargo around default-deps, so I
filed that upstream and left most `default-features` mismatches:
https://github.com/rust-lang/cargo/issues/14841.
Reference script:
```python
import tomllib
from collections import defaultdict
from pathlib import Path
uv = Path("/home/konsti/projects/uv")
skip_list = ["uv-trampoline", "uv-dev", "uv-performance-flate2-backend", "uv-performance-memory-allocator"]
root_feature_map = defaultdict(set)
root_default_features = defaultdict(bool)
cargo_toml = tomllib.loads(uv.joinpath("Cargo.toml").read_text())
for dep, declaration in cargo_toml["workspace"]["dependencies"].items():
root_default_features[dep] = root_default_features[dep] or declaration.get("default-features", True)
root_feature_map[dep].update(declaration.get("features", []))
feature_map = defaultdict(set)
default_features = defaultdict(bool)
for crate in uv.joinpath("crates").iterdir():
if crate.name in skip_list:
continue
if not crate.joinpath("Cargo.toml").is_file():
continue
cargo_toml = tomllib.loads(crate.joinpath("Cargo.toml").read_text())
for dep, declaration in cargo_toml.get("dependencies", {}).items():
# If any item uses default features, they are used everywhere
default_features[dep] = default_features[dep] or declaration.get("default-features", True)
feature_map[dep].update(declaration.get("features", []))
for dep, features in sorted(feature_map.items()):
features = features - root_feature_map.get(dep, set())
if not features and default_features[dep] == root_default_features[dep]:
continue
print(dep, default_features[dep], sorted(features))
```
## Summary
Update the URL to the "tl" crate since the repository has been
transferred to astral-sh/. This is of no real consequence, except it
keeps triggering Gentoo linter that detects permanently redirected URL.
## Test Plan
`cargo test`
When doing a directory traversal for source dist inclusion, we want to
offer the user include and exclude options, and we want to avoid
traversing irrelevant directories. The latter is important for
performance, especially on network file systems, but also with large
data directories, or (not-included) directories with other permissions.
To support this, we introduce `GlobDirFilter`, which uses a DFA from
regex_automata to determine whether any children of a directory can be
included and skips the directory if not.
The globs are based on PEP 639. The syntax is more restricted than glob
or globset, but it's standardized. I chose it over glob or globset
because we're already using this syntax for `project.license-files` a
required by PEP 639, so it makes sense to use the same globs for all
includes (see e.g.
4f52a3bb62/pyproject.toml (L36-L48)
for example with same semantics for include and exclude)
### Semantics
Glob semantics are complex due to mixing directories and files,
expectations around simplicity and our need to exclude most of the tree
in the project from traversal. The current draft uses a syntax that
optimizes for simple default use cases for the start.
#### includes
Glob expressions which files and directories to include in the source
distribution.
Includes are anchored, which means that `pyproject.toml` includes only
`<project root>/pyproject.toml`. Use for example `assets/**/sample.csv`
to include for all
`sample.csv` files in `<project root>/assets` or any child directory. To
recursively include
all files under a directory, use a `/**` suffix, e.g. `src/**`. For
performance and
reproducibility, avoid unanchored matches such as `**/sample.csv`.
The glob syntax is the reduced portable glob from
[PEP 639](https://peps.python.org/pep-0639/#add-license-FILES-key).
#### excludes
Glob expressions which files and directories to exclude from the
previous source
distribution includes.
Excludes are not, which means that `__pycache__` excludes all
directories named
`__pycache__` and it's children anywhere. To anchor a directory, use a
`/` prefix, e.g.,
`/dist` will exclude only `<project root>/dist`.
The glob syntax is the reduced portable glob from
[PEP 639](https://peps.python.org/pep-0639/#add-license-FILES-key).
This adds support for providing conflicting group names in addition to
extra names to `Conflicts`.
This merely makes "room" for it in the types while keeping everything
working. We'll add proper support for it in the next commit.
Note that one interesting trick we do here is depend directly on
`hashbrown` so that we can make use of its `Equivalent` trait. This in
turn lets us use things like `ConflictItemRef` as a lookup key for a
hashset that contains `ConflictItem`. This mirrors using a `&str` as a
lookup key for a hashset that contains `String`, but works for arbitrary
types. `std` doesn't support this, but `hashbrown` does. This trick in
turn lets us simplify some of our data structures.
This also rejiggers some of the serde-interaction with the conflicting
types. We now use a wire type to represent our conflicting items for
more flexibility. i.e., Support `extra` XOR `group` fields.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
This PR builds off of https://github.com/astral-sh/uv/pull/6738 to fix
#6724 (sorry for the new PR @charliermarsh I didn't want to push to your
branch, not even sure if I could). The reason the original PR doesn't
fix the issue described in #6724 is because the fastapi is ran in the
project context (as I assume a lot of use cases are). This PR adds an
extra commit to handle the signals in the project/run.rs file
~It also addresses the comment
[here](https://github.com/astral-sh/uv/pull/6738/files#r1734757548) to
not use the tokio ctrl-c method since we are now handling SIGINT
ourselves~ update, tokio handles SIGINT in a platform agnostic way,
intercepting this ouselves makes the logic more complicated with
windows, decided to leave the tokio ctrl-c handler
~[This
comment](https://github.com/astral-sh/uv/pull/6738/files#r1743510140)
remains unaddressed, however, the Child process does not have any other
methods besides kill() so I don't see how we can "preserve" the
interrupt call :/ I tried looking around but no luck.~ updated, this PR
is reduced to only handling SIGTERM propagation on unix machines, and
the sigterm call to the child is preserved by making use of the nix
package, instead of relying on tokio which only allowed for `kill()` on
a child process
## Test Plan
I tested this by building the docker container locally with these
changes and tagging it "myuv", and then using that as the base image in
uv-docker-example, (and ofc following the rest of the repro issues in
#6724. In my tests I see that ctrl-c in the docker-compose up command
exits the process almost immediately 👍
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
Very basic source distribution support. What's included:
- Include and exclude patterns (hard-coded): Currently, we have
globset+walkdir in one part and glob in the other. I'll migrate
everything to globset+walkset and some custom perf optimizations to
avoid traversing irrelevant directories on top. I'll also pick a glob
syntax (or subset), PEP 639 seems like a good candidate since it's
consistent with what we already have to support.
- Add the `PKG-INFO` file with metadata: Thanks to Code Metadata 2.2,
this metadata is reliable and can be read statically by external tools.
Example output:
```
$ tar -ztvf dist/dummy-0.1.0.tar.gz
-rw-r--r-- 0/0 154 1970-01-01 01:00 dummy-0.1.0/PKG-INFO
-rw-rw-r-- 0/0 509 1970-01-01 01:00 dummy-0.1.0/pyproject.toml
drwxrwxr-x 0/0 0 1970-01-01 01:00 dummy-0.1.0/src/dummy
drwxrwxr-x 0/0 0 1970-01-01 01:00 dummy-0.1.0/src/dummy/submodule
-rw-rw-r-- 0/0 30 1970-01-01 01:00 dummy-0.1.0/src/dummy/submodule/impl.py
-rw-rw-r-- 0/0 14 1970-01-01 01:00 dummy-0.1.0/src/dummy/submodule/__init__.py
-rw-rw-r-- 0/0 12 1970-01-01 01:00 dummy-0.1.0/src/dummy/__init__.py
```
No tests since the source distributions don't build valid wheels yet.
## Summary
This PR pulls in https://github.com/astral-sh/uv/pull/8263 and
https://github.com/astral-sh/uv/pull/8463, which were originally merged
into the v0.5 tracking branch but can now be committed separately, as
we've made `.env` loading opt-in.
In summary:
- `.env` loading is now opt-in (`--env-file .env`).
- `.env` remains supported on `uv run`, so it's meant for providing
environment variables to the run command, rather than to uv itself.
---------
Co-authored-by: Eduardo González Vaquero <47718648+edugzlez@users.noreply.github.com>
