Go to file
konsti 431ddc1d74
Change backtracking when packages conflict too much (#9843)
Background reading: https://github.com/astral-sh/uv/issues/8157
Companion PR: https://github.com/astral-sh/pubgrub/pull/36
Requires for test coverage: https://github.com/astral-sh/packse/pull/230

When two packages A and B conflict, we have the option to choose a lower
version of A, or a lower version of B. Currently, we determine this by
the order we saw a package (assuming equal specificity of the
requirement): If we saw A before B, we pin A until all versions of B are
exhausted. This can lead to undesirable outcomes, from cases where it's
just slow (sentry) to others cases without lower bounds where be
backtrack to a very old version of B. This old version may fail to build
(terminating the resolution), or it's a version so old that it doesn't
depend on A (or the shared conflicting package) anymore - but also is
too old for the user's application (fastapi). #8157 collects such cases,
and the `wrong-backtracking` packse scenario contains a minimized
example.

We try to solve this by tracking which packages are "A"s, culprits, and
"B"s, affected, and manually interfering with project selection and
backtracking. Whenever a version we just chose is rejected, we give the
current package a counter for being affected, and the package it
conflicted with a counter for being a culprit. If a package accumulates
more counts than a threshold, we reprioritize: Undecided after the
culprits, after the affected, after packages that only have a single
version (URLs, `==<version>`). We then ask pubgrub to backtrack just
before the culprit. Due to the changed priorities, we now select package
B, the affected, instead of package A, the culprit.

To do this efficiently, we ask pubgrub for the incompatibility that
caused backtracking, or just the last version to be discarded (due to
its dependencies). For backtracking, we use the last incompatibility
from unit propagation as a heuristic. When a version is discarded
because one of its dependencies conflicts with the partial solution, the
incompatibility tells us the package in the partial solution that
conflicted.

We only backtrack once per package, on the first time it passes the
threshold. This prevents backtracking loops in which we make the same
decisions over and over again. But we also changed the priority, so that
we shouldn't take the same path even after the one time we backtrack (it
would defeat the purpose of this change).

There are some parameters that can be tweaked: Currently, the threshold
is set to 5, which feels not too eager with so me of the conflicts that
we want to tolerate but also changes strategies quickly. The relative
order of the new priorities can also be changed, as for each (A, B) pair
the priority of B is afterwards lower than that for A. Currently,
culprits capture conflict for the whole package, but we could limit that
to a specific version. We could discard conflict counters after
backtracking instead of keeping them eternally as we do now. Note that
we're always taking about pairs (A, B), but in practice we track
individual packages, not pairs.

A case that we wouldn't capture is when B is only introduced to the
dependency graph after A, but I think that would require cyclical
dependency for A and B to conflict? There may also be cases where
looking at the last incompatibility is insufficient.

Another example that we can't repair with prioritization is
urllib3/boto3/botocore: We actually have to check all the newer versions
of boto3 and botocore to identify the version that allows with the older
urllib3, no shortcuts allowed.

```
urllib3<1.25.4
boto3
```

All examples I tested were cases with two packages where we only had to
switch the order, so I've abstracted them into a single packse case.

This PR changes the resolution for certain paths, and there is the risk
for regressions.

Fixes #8157

---

All tested examples improved.

Input fastapi:
```text
starlette<=0.36.0
fastapi<=0.115.2
```

```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/fastapi.txt
annotated-types==0.7.0
anyio==4.6.0
fastapi==0.1.17
idna==3.10
pydantic==2.9.2
pydantic-core==2.23.4
sniffio==1.3.1
starlette==0.36.0
typing-extensions==4.12.2

# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/fastapi.txt 
annotated-types==0.7.0
anyio==4.6.0
fastapi==0.109.1
idna==3.10
pydantic==2.9.2
pydantic-core==2.23.4
sniffio==1.3.1
starlette==0.35.1
typing-extensions==4.12.2
```


Input xarray:
```text
xarray[accel]
```

```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/xarray-accel.txt
bottleneck==1.4.0
flox==0.9.13
llvmlite==0.36.0
numba==0.53.1
numbagg==0.8.2
numpy==2.1.1
numpy-groupies==0.11.2
opt-einsum==3.4.0
packaging==24.1
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2024.2
scipy==1.14.1
setuptools==75.1.0
six==1.16.0
toolz==0.12.1
tzdata==2024.2
xarray==2024.9.0

# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/xarray-accel.txt
bottleneck==1.4.0
flox==0.9.13
llvmlite==0.43.0
numba==0.60.0
numbagg==0.8.2
numpy==2.0.2
numpy-groupies==0.11.2
opt-einsum==3.4.0
packaging==24.1
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2024.2
scipy==1.14.1
six==1.16.0
toolz==0.12.1
tzdata==2024.2
xarray==2024.9.0
```


Input sentry: The resolution is identical, but arrived at much faster:
main tries 69 versions (sentry-kafka-schemas: 63), PR tries 12 versions
(sentry-kafka-schemas: 6; 5 times conflicting, then once the right
version).

```text
python-rapidjson<=1.20,>=1.4
sentry-kafka-schemas<=0.1.113,>=0.1.50
```

```
# BEFORE
$ uv pip --no-progress compile -p 3.11 --exclude-newer 2024-10-01 --no-annotate debug/sentry.txt
fastjsonschema==2.20.0
msgpack==1.1.0
python-rapidjson==1.8
pyyaml==6.0.2
sentry-kafka-schemas==0.1.111
typing-extensions==4.12.2

# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.11 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/sentry.txt
fastjsonschema==2.20.0
msgpack==1.1.0
python-rapidjson==1.8
pyyaml==6.0.2
sentry-kafka-schemas==0.1.111
typing-extensions==4.12.2
```


Input apache-beam
```text
# Run on Python 3.10
dill<0.3.9,>=0.2.2
apache-beam<=2.49.0
```

```
# BEFORE
$ uv pip --no-progress compile -p 3.10 --exclude-newer 2024-10-01 --no-annotate debug/apache-beam.txt
  × Failed to download and build `apache-beam==2.0.0`
  ╰─▶ Build backend failed to determine requirements with `build_wheel()` (exit status: 1)

# AFTER
$ cargo run --profile fast-build --no-default-features pip compile -p 3.10 --no-progress --exclude-newer 2024-10-01 --no-annotate debug/apache-beam.txt
apache-beam==2.49.0
certifi==2024.8.30
charset-normalizer==3.3.2
cloudpickle==2.2.1
crcmod==1.7
dill==0.3.1.1
dnspython==2.6.1
docopt==0.6.2
fastavro==1.9.7
fasteners==0.19
grpcio==1.66.2
hdfs==2.7.3
httplib2==0.22.0
idna==3.10
numpy==1.24.4
objsize==0.6.1
orjson==3.10.7
proto-plus==1.24.0
protobuf==4.23.4
pyarrow==11.0.0
pydot==1.4.2
pymongo==4.10.0
pyparsing==3.1.4
python-dateutil==2.9.0.post0
pytz==2024.2
regex==2024.9.11
requests==2.32.3
six==1.16.0
typing-extensions==4.12.2
urllib3==2.2.3
zstandard==0.23.0
```
2024-12-16 11:39:50 +01:00
.cargo Don't build uv-dev by default (#6827) 2024-08-29 15:44:13 -04:00
.config Abort tests running for more than 60s to catch deadlocks (#3694) 2024-05-21 13:19:57 +00:00
.github Add lzma to benchmark install (#9888) 2024-12-13 16:54:20 -05:00
assets Add top-level benchmark to the README (#2622) 2024-03-22 16:07:31 -04:00
crates Change backtracking when packages conflict too much (#9843) 2024-12-16 11:39:50 +01:00
docs Change backtracking when packages conflict too much (#9843) 2024-12-16 11:39:50 +01:00
ecosystem Remove test dependency on `pg_config` (#9853) 2024-12-13 12:45:08 +01:00
python/uv Add hint on build backend import with preview disabled (#9691) 2024-12-06 14:08:00 -06:00
scripts Fix local packse workflow (#9808) 2024-12-11 09:32:46 -06:00
.editorconfig Ignore `UV_CACHE_DIR` in `help` tests (#7895) 2024-10-04 09:41:25 -05:00
.gitattributes Mark `.inc` files as Rust for GitHub Linguist (#9890) 2024-12-13 22:17:23 +00:00
.gitignore ignore macOS .DS_Store files (#7200) 2024-09-08 21:29:18 +00:00
.pre-commit-config.yaml Update pre-commit dependencies (#9927) 2024-12-15 20:12:00 -05:00
.prettierignore Generate environment variables doc from code (#8493) 2024-11-03 08:31:38 -06:00
.python-versions Use 3.13 in CI (#8014) 2024-11-07 14:29:54 -06:00
BENCHMARKS.md Update `BENCHMARKS.md` (#6258) 2024-08-20 16:42:57 +00:00
CHANGELOG.md Bump version to v0.5.9 (#9889) 2024-12-13 17:28:19 -05:00
CONTRIBUTING.md Increase UV_STACK_SIZE to 3MB (#9311) 2024-11-21 11:46:33 +00:00
Cargo.lock Change backtracking when packages conflict too much (#9843) 2024-12-16 11:39:50 +01:00
Cargo.toml Change backtracking when packages conflict too much (#9843) 2024-12-16 11:39:50 +01:00
Dockerfile Docker: put uv and uvx in a single layer (#7697) 2024-09-26 09:19:14 -05:00
LICENSE-APACHE Add README and LICENSE files 2023-10-05 12:45:38 -04:00
LICENSE-MIT Add README and LICENSE files 2023-10-05 12:45:38 -04:00
PIP_COMPATIBILITY.md Move pip compatibility guide to the documentation (#6213) 2024-08-20 11:31:46 -05:00
README.md Remove `pypy` from top-level pin example (#9896) 2024-12-14 12:00:13 -06:00
STYLE.md Improve and fix some documents (#8749) 2024-11-01 09:44:04 -04:00
_typos.toml uv/tests: add new 'ecosystem' integration tests (#5970) 2024-08-13 09:48:00 -04:00
clippy.toml Improve the `uv python` CLI documentation (#5961) 2024-08-09 14:46:21 -05:00
mkdocs.insiders.yml Watch for changes to the template file during documentation serve (#9244) 2024-11-19 22:39:38 -06:00
mkdocs.public.yml Watch for changes to the template file during documentation serve (#9244) 2024-11-19 22:39:38 -06:00
mkdocs.template.yml Add policies reference section and license document (#9367) 2024-12-03 10:56:32 -06:00
pyproject.toml Bump version to v0.5.9 (#9889) 2024-12-13 17:28:19 -05:00
ruff.toml uv/tests: add new 'ecosystem' integration tests (#5970) 2024-08-13 09:48:00 -04:00
rust-toolchain.toml Upgrade minimum Rust version to 1.83 (#9815) 2024-12-11 10:06:19 -06:00
uv.schema.json Introduce a `--fork-strategy` preference mode (#9868) 2024-12-13 16:05:07 -05:00

README.md

uv

uv image image image Actions status Discord

An extremely fast Python package and project manager, written in Rust.

Shows a bar chart with benchmark results.

Installing Trio's dependencies with a warm cache.

Highlights

uv is backed by Astral, the creators of Ruff.

Installation

Install uv with our standalone installers:

# On macOS and Linux.
curl -LsSf https://astral.sh/uv/install.sh | sh
# On Windows.
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Or, from PyPI:

# With pip.
pip install uv
# Or pipx.
pipx install uv

If installed via the standalone installer, uv can update itself to the latest version:

uv self update

See the installation documentation for details and alternative installation methods.

Documentation

uv's documentation is available at docs.astral.sh/uv.

Additionally, the command line reference documentation can be viewed with uv help.

Features

Project management

uv manages project dependencies and environments, with support for lockfiles, workspaces, and more, similar to rye or poetry:

$ uv init example
Initialized project `example` at `/home/user/example`

$ cd example

$ uv add ruff
Creating virtual environment at: .venv
Resolved 2 packages in 170ms
   Built example @ file:///home/user/example
Prepared 2 packages in 627ms
Installed 2 packages in 1ms
 + example==0.1.0 (from file:///home/user/example)
 + ruff==0.5.9

$ uv run ruff check
All checks passed!

See the project documentation to get started.

uv also supports building and publishing projects, even if they're not managed with uv. See the publish guide to learn more.

Tool management

uv executes and installs command-line tools provided by Python packages, similar to pipx.

Run a tool in an ephemeral environment using uvx (an alias for uv tool run):

$ uvx pycowsay 'hello world!'
Resolved 1 package in 167ms
Installed 1 package in 9ms
 + pycowsay==0.0.0.2
  """

  ------------
< hello world! >
  ------------
   \   ^__^
    \  (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

Install a tool with uv tool install:

$ uv tool install ruff
Resolved 1 package in 6ms
Installed 1 package in 2ms
 + ruff==0.5.9
Installed 1 executable: ruff

$ ruff --version
ruff 0.5.9

See the tools documentation to get started.

Python management

uv installs Python and allows quickly switching between versions.

Install multiple Python versions:

$ uv python install 3.10 3.11 3.12
Searching for Python versions matching: Python 3.10
Searching for Python versions matching: Python 3.11
Searching for Python versions matching: Python 3.12
Installed 3 versions in 3.42s
 + cpython-3.10.14-macos-aarch64-none
 + cpython-3.11.9-macos-aarch64-none
 + cpython-3.12.4-macos-aarch64-none

Download Python versions as needed:

$ uv venv --python 3.12.0
Using Python 3.12.0
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate

$ uv run --python pypy@3.8 -- python --version
Python 3.8.16 (a9dbdca6fc3286b0addd2240f11d97d8e8de187a, Dec 29 2022, 11:45:30)
[PyPy 7.3.11 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>>

Use a specific Python version in the current directory:

$ uv python pin 3.11
Pinned `.python-version` to `3.11`

See the Python installation documentation to get started.

Script support

uv manages dependencies and environments for single-file scripts.

Create a new script and add inline metadata declaring its dependencies:

$ echo 'import requests; print(requests.get("https://astral.sh"))' > example.py

$ uv add --script example.py requests
Updated `example.py`

Then, run the script in an isolated virtual environment:

$ uv run example.py
Reading inline script metadata from: example.py
Installed 5 packages in 12ms
<Response [200]>

See the scripts documentation to get started.

A pip-compatible interface

uv provides a drop-in replacement for common pip, pip-tools, and virtualenv commands.

uv extends their interfaces with advanced features, such as dependency version overrides, platform-independent resolutions, reproducible resolutions, alternative resolution strategies, and more.

Migrate to uv without changing your existing workflows — and experience a 10-100x speedup — with the uv pip interface.

Compile requirements into a platform-independent requirements file:

$ uv pip compile docs/requirements.in \
   --universal \
   --output-file docs/requirements.txt
Resolved 43 packages in 12ms

Create a virtual environment:

$ uv venv
Using Python 3.12.3
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate

Install the locked requirements:

$ uv pip sync docs/requirements.txt
Resolved 43 packages in 11ms
Installed 43 packages in 208ms
 + babel==2.15.0
 + black==24.4.2
 + certifi==2024.7.4
 ...

See the pip interface documentation to get started.

Platform support

See uv's platform support document.

Versioning policy

See uv's versioning policy document.

Contributing

We are passionate about supporting contributors of all levels of experience and would love to see you get involved in the project. See the contributing guide to get started.

Acknowledgements

uv's dependency resolver uses PubGrub under the hood. We're grateful to the PubGrub maintainers, especially Jacob Finkelman, for their support.

uv's Git implementation is based on Cargo.

Some of uv's optimizations are inspired by the great work we've seen in pnpm, Orogene, and Bun. We've also learned a lot from Nathaniel J. Smith's Posy and adapted its trampoline for Windows support.

License

uv is licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in uv by you, as defined in the Apache-2.0 license, shall be dually licensed as above, without any additional terms or conditions.