diff --git a/BENCHMARKS.md b/BENCHMARKS.md index bb64fae90..c83a53d5f 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -9,11 +9,11 @@ important caveats: - Benchmark performance may vary dramatically depending on the set of packages being installed. For example, a resolution that requires building a single intensive source distribution may appear very similar across tools, since the bottleneck is tool-agnostic. -- Unlike Poetry, both uv and pip-tools do _not_ generate multi-platform lockfiles. As such, Poetry - is (by design) doing significantly more work than other tools in the resolution benchmarks. Poetry - is included for completeness, as many projects may not _need_ a multi-platform lockfile. However, - it's critical to understand that benchmarking uv's resolution time against Poetry is an unfair - comparison. (Benchmarking installation, however, _is_ a fair comparison.) +- Unlike Poetry, both uv and pip-tools do _not_ generate platform-independent lockfiles. As such, + Poetry is (by design) doing significantly more work than other tools in the resolution benchmarks. + Poetry is included for completeness, as many projects may not _need_ a platform-independent + lockfile. However, it's critical to understand that benchmarking uv's resolution time against + Poetry is an unfair comparison. (Benchmarking installation, however, _is_ a fair comparison.) This document benchmarks against Trio's `docs-requirements.in`, as a representative example of a real-world project. diff --git a/README.md b/README.md index 15601fc1f..4e7302ea6 100644 --- a/README.md +++ b/README.md @@ -459,7 +459,7 @@ While constraints are purely _additive_, and thus cannot _expand_ the set of acc a package, overrides _can_ expand the set of acceptable versions for a package, providing an escape hatch for erroneous upper version bounds. -### Multi-platform resolution +### Platform-independent resolution By default, uv's `pip-compile` command produces a resolution that's known to be compatible with the current platform and Python version. Unlike Poetry and PDM, uv does not yet produce a diff --git a/docs/concepts/resolution.md b/docs/concepts/resolution.md index fa6ca7b6d..08c0e4b70 100644 --- a/docs/concepts/resolution.md +++ b/docs/concepts/resolution.md @@ -1,20 +1,88 @@ # Resolution +Dependency resolution is the process of taking your requirements and converting them to a list of +package versions that fulfil your requirements and the requirements of all included packages. + +## Overview + +Imagine you have the following dependency tree: + +- Your project depends on `foo>=1,<3` and `bar>=1,<3`. +- `foo` has two versions, 1.0.0 and 2.0.0. `foo` 2.0.0 depends on `lib==2.0.0`, `foo` 1.0.0 has no + dependencies. +- `bar` has two versions, 1.0.0 and 2.0.0. `bar` 2.0.0 depends on `lib==1.0.0`, `bar` 1.0.0 has no + dependencies. +- `lib` has two versions, 1.0.0 and 2.0.0. Both versions have no dependencies. + +We can't install both `foo` 2.0.0 and `bar` 2.0.0 because they conflict on the version of `lib`, so +the resolver will pick either `foo` 1.0.0 or `bar` 1.0.0. Both are valid solutions, at the resolvers +choice. + +## Platform-specific and universal resolution + +uv supports two modes of resolution: Platform-specific and universal (platform-independent). + +Like `pip` and `pip-tools`, `uv pip compile` produces a resolution that's only known to be +compatible with the current operating system, architecture, Python version and Python interpreter. +`uv pip compile --universal` and the [project](../guides/projects.md) interface on the other hand +will solve to a host-agnostic universal resolution that can be used across platforms. + +For universal resolution, you need to configure the minimum required python version. For +`uv pip compile --universal`, you can pass `--python-version`, otherwise the current Python version +will be treated as a lower bound. For example, `--universal --python-version 3.9` writes a universal +resolution for Python 3.9 and later. Project commands such as `uv sync` or `uv lock` read +`project.requires-python` from your `pyproject.toml`. + +Setting the minimum Python version is important because all package versions we select have to be +compatible with the python range. For example, a universal resolution of `numpy<2` with +`--python-version 3.8` resolves to `numpy==1.24.4`, while `--python-version 3.9` resolves to +`numpy==1.26.4`, as `numpy` releases after 1.26.4 require at least Python 3.9. Note that we only +consider the lower bound of any Python requirement. + +In platform-specific mode, the `uv pip` interface also supports resolving for specific alternate +platforms and Python versions with `--python-platform` and `--python-version`. For example, if +you're running Python 3.12 on macOS, but want to resolve for Linux with Python 3.10, you can run +`uv pip compile --python-platform linux --python-version 3.10 requirements.in` to produce a +`manylinux2014`-compatible resolution. In this mode, `--python-version` is the exact python version +to use, not a lower bound. + +!!! note + + Python's environment markers expose far more information about the current machine + than can be expressed by a simple `--python-platform` argument. For example, the `platform_version` marker + on macOS includes the time at which the kernel was built, which can (in theory) be encoded in + package requirements. uv's resolver makes a best-effort attempt to generate a resolution that is + compatible with any machine running on the target `--python-platform`, which should be sufficient for + most use cases, but may lose fidelity for complex package and platform combinations. + +In universal mode, a package may be listed multiple times with different versions or URLs. In this +case, uv determined that we need different versions to be compatible different platforms, and the +markers decides on which platform we use which version. A universal resolution is often more +constrained than a platform-specific resolution, since we need to take the requirements for all +markers into account. + +If an output file is used with `uv pip` or `uv.lock` exist with the project commands, we try to +resolve to the versions present there, considering them preferences in the resolution. The same +applies to version already installed to the active virtual environments. You can override this with +`--upgrade`. + ## Resolution strategy -By default, uv follows the standard Python dependency resolution strategy of preferring the latest -compatible version of each package. For example, `uv pip install flask>=2.0.0` will install the -latest version of Flask (at time of writing: `3.0.0`). +By default, uv tries to use the latest version of each package. For example, +`uv pip install flask>=2.0.0` will install the latest version of Flask (at time of writing: +`3.0.0`). If you have `flask>=2.0.0` as a dependency of your library, you will only test `flask` +3.0.0 this way, but not if you are actually still compatible with `flask` 2.0.0. -However, uv's resolution strategy can be configured to support alternative workflows. With -`--resolution lowest`, uv will install the **lowest** compatible versions for all dependencies, both -**direct** and **transitive**. Alternatively, `--resolution lowest-direct` will opt for the -**lowest** compatible versions for all **direct** dependencies, while using the **latest** -compatible versions for all **transitive** dependencies. This distinction can be particularly useful -for library authors who wish to test against the lowest supported versions of direct dependencies -without restricting the versions of transitive dependencies. +With `--resolution lowest`, uv will install the lowest possible version for all dependencies, both +direct and indirect (transitive). Alternatively, `--resolution lowest-direct` will opt for the +lowest compatible versions for all direct dependencies, while using the latest compatible versions +for all other dependencies. uv will always use the latest versions for build dependencies. -For example, given the following `requirements.in` file: +For libraries, we recommend separately running tests with `--resolution lowest` or +`--resolution lowest-direct` in continuous integration to ensure compatibility with the declared +lower bounds. + +As an example, given the following `requirements.in` file: ```text title="requirements.in" flask>=2.0.0 @@ -64,7 +132,7 @@ werkzeug==2.0.0 By default, uv will accept pre-release versions during dependency resolution in two cases: -1. If the package is a direct dependency, and its version markers include a pre-release specifier +1. If the package is a direct dependency, and its version specifiers include a pre-release specifier (e.g., `flask>=2.0.0rc1`). 1. If _all_ published versions of a package are pre-releases. @@ -81,70 +149,31 @@ model, and are a frequent source of bugs in other packaging tools. uv's pre-rele _intentionally_ limited and _intentionally_ requires user opt-in for pre-releases, to ensure correctness. -For more, see ["Pre-release compatibility"](../pip/compatibility.md#pre-release-compatibility) +For more, see [Pre-release compatibility](../pip/compatibility.md#pre-release-compatibility). -## Dependency overrides +## Constraints -Historically, `pip` has supported "constraints" (`-c constraints.txt`), which allows users to narrow -the set of acceptable versions for a given package. +Like `pip`, uv supports constraints files (`--constraint constraints.txt`), which allows users to +narrow the set of acceptable versions for a given package. A constraint files is like a regular +requirements files, but it doesn't add packages, it only constrains their version range when they +are depended on by a regular requirement. -uv supports constraints, but also takes this concept further by allowing users to _override_ the -acceptable versions of a package across the dependency tree via overrides -(`--override overrides.txt`). +## Overrides -In short, overrides allow the user to lie to the resolver by overriding the declared dependencies of -a package. Overrides are a useful last resort for cases in which the user knows that a dependency is -compatible with a newer version of a package than the package declares, but the package has not yet -been updated to declare that compatibility. +Sometimes, the requirements in one of your (transitive) dependencies are too strict, and you want to +install a version of a package that you know to work, but wouldn't be allowed regularly. Overrides +allow you to lie to the resolver, replacing all other requirements for that package with the +override. They break the usual rules of version resolving and should only be used as last resort +measure. For example, if a transitive dependency declares `pydantic>=1.0,<2.0`, but the user knows that the package is compatible with `pydantic>=2.0`, the user can override the declared dependency with `pydantic>=2.0,<3` to allow the resolver to continue. -While constraints are purely _additive_, and thus cannot _expand_ the set of acceptable versions for -a package, overrides _can_ expand the set of acceptable versions for a package, providing an escape -hatch for erroneous upper version bounds. - -## Multi-platform resolution - -By default, uv's `pip-compile` command produces a resolution that's known to be compatible with the -current platform and Python version. - -uv also supports a machine agnostic resolution. uv supports writing multiplatform resolutions in -both a `requirements.txt` format and uv-specific (`uv.lock`) format. - -If using uv's `pip compile`, the `--universal` flag will generate a resolution that is compatible -with all operating systems, architectures, and Python implementations. In universal mode, the -current Python version (or provided `--python-version`) will be treated as a lower bound. For -example, `--universal --python-version 3.7` would produce a universal resolution for Python 3.7 and -later. - -If using uv's [project](../guides/projects.md) interface, the machine agnostic resolution will be -used automatically and a `uv.lock` file will be created. The lockfile can also be created with an -explicit `uv lock` invocation. - -uv also supports resolving for specific alternate platforms and Python versions via the -`--python-platform` and `--python-version` command line arguments. - -For example, if you're running uv on macOS, but want to resolve for Linux, you can run -`uv pip compile --python-platform linux requirements.in` to produce a `manylinux2014`-compatible -resolution. - -Similarly, if you're running uv on Python 3.9, but want to resolve for Python 3.8, you can run -`uv pip compile --python-version 3.8 requirements.in` to produce a Python 3.8-compatible resolution. - -The `--python-platform` and `--python-version` arguments can be combined to produce a resolution for -a specific platform and Python version, enabling users to generate multiple lockfiles for different -environments from a single machine. - -!!! note - - Python's environment markers expose far more information about the current machine - than can be expressed by a simple `--python-platform` argument. For example, the `platform_version` marker - on macOS includes the time at which the kernel was built, which can (in theory) be encoded in - package requirements. uv's resolver makes a best-effort attempt to generate a resolution that is - compatible with any machine running on the target `--python-platform`, which should be sufficient for - most use cases, but may lose fidelity for complex package and platform combinations. +Overrides are passed to `uv pip` as `--override` with an overrides file with the same syntax as +requirements or constraints files. In `pyproject.toml`, you can set `tool.uv.override-dependencies` +to a list of requirements. If you provide multiple overrides for the same package, we apply them +simultaneously, while markers are applied as usual. ## Time-restricted reproducible resolutions @@ -155,7 +184,8 @@ may be specified as an [RFC 3339](https://www.rfc-editor.org/rfc/rfc3339.html) t Note the package index must support the `upload-time` field as specified in [`PEP 700`](https://peps.python.org/pep-0700/). If the field is not present for a given -distribution, the distribution will be treated as unavailable. +distribution, the distribution will be treated as unavailable. PyPI provides `upload-time` for all +packages. To ensure reproducibility, messages for unsatisfiable resolutions will not mention that distributions were excluded due to the `--exclude-newer` flag — newer distributions will be treated diff --git a/docs/index.md b/docs/index.md index 81f7b8ba4..85b15311c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -160,9 +160,10 @@ See the [installing Python guide](./guides/install-python.md) to get started. uv provides a drop-in replacement for common `pip`, `pip-tools`, and `virtualenv` commands. uv extends their interfaces with advanced features, such as dependency version overrides, -multi-platform resolutions, reproducible resolutions, alternative resolution strategies, and more. +platform-independent resolutions, reproducible resolutions, alternative resolution strategies, and +more. -Compile requirements into a multi-platform requirements file: +Compile requirements into a platform-independent requirements file: ```console $ uv pip compile docs/requirements.in \ diff --git a/docs/reference/resolution-internals.md b/docs/reference/resolution-internals.md new file mode 100644 index 000000000..76056d496 --- /dev/null +++ b/docs/reference/resolution-internals.md @@ -0,0 +1,111 @@ +# Resolution internals + +This page explains some of the internal workings of uv, its resolver and the lockfile. For using uv, +see [Resolution](../concepts/resolution.md). + +## Dependency resolution with PubGrub + +If you look into a textbook, it will tell you that finding a set of version to install from a given +set of requirements is equivalent to the +[SAT problem](https://en.wikipedia.org/wiki/Boolean_satisfiability_problem) and thereby NP-complete, +i.e., in the worst case you have to try all possible combinations of all versions of all packages +and there are no general fast algorithms. In practice, this is fairly misleading for a number of +reasons: + +- The slowest part of uv is loading package and version metadata, even if it's cached. +- Certain solution are more preferable than others, for example we generally want to use latest + versions. +- Requirements follow lots of patterns: We use continuous versions ranges and not arbitrary boolean + inclusion/exclusions of versions, adjacent release have the same or similar requirements, etc. +- For the majority of resolutions, we wouldn't even need to backtrack, just picking versions + iteratively is sufficient. If we have preferences from a previous resolution we often barely need + to anything at all. +- We don't just need either a solution or a message that there is no solution (like for SAT), we + need an understandable error trace that tell you which packages are involved in away to allows you + to remove the conflict. + +uv uses [pubgrub-rs](https://github.com/pubgrub-rs/pubgrub), the Rust implementation of +[PubGrub](https://nex3.medium.com/pubgrub-2fb6470504f), an incremental version solver. PubGrub in uv +works in the following steps: + +- We have a partial solution that tells us for which packages we already picked versions and for + which we still need to decide. +- From the undecided packages we pick the one with the highest priority. Package with URLs + (including file, git, etc.) have the highest priority, then those with more exact specifiers (such + as `==`), then those with less strict specifiers. Inside each category, we order packages by when + we first saw them, making the resolution deterministic. +- For that package with the highest priority, pick a version that works with all specifiers from the + packages with versions in the partial solution and that is not yet marked as incompatible. We + prefer versions from a lockfile (`uv.lock` or `-o requirements.txt`) and installed versions, then + we go from highest to lowest (unless you changed the resolution mode). You can see this happening + by the `Selecting ...` messages in `uv lock -v`. +- Add all requirements of this version to pubgrub. Start prefetching their metadata in the + background. +- Now we either we repeat this process with the next package or we have a conflict. Let's say we + pick picked, among other packages, `a` 2 and then `b` 2, and those have requirements `a 2 -> c 1` + and `b 2 -> c 2`. When trying to pick a version for `c`, we see there is no version we can pick. + Using its internal incompatibilities store, PubGrub traces this back to `a 2` and `b 2` and adds + an incompatibility for `{a 2, b 2}`, meaning when either is picked we can't select the other. We + restore the state with `a` 2 before picking `b` 2 with the new learned incompatibility and pick a + new version for `b`. + +Eventually, we either have picked compatible versions for all packages and get a successful +resolution, or we get an incompatibility for the virtual root package, that is whatever versions of +the root dependencies and their transitive dependencies we'd pick, we'll always get a conflict. From +the incompatibilities in PubGrub, we can trace which packages were involved and format an error +message. For more details on the PubGrub algorithm, see +[Internals of the PubGrub algorithm](https://pubgrub-rs-guide.pages.dev/internals/intro). + +## Forking + +Python historically didn't have backtracking version resolution, and even with version resolution, +it was usually limited to single environment, which one specific architecture, operating system, +python version and python implementation. Some packages use contradictory requirements for different +environments, something like: + +```text +numpy>=2,<3 ; python_version >= "3.11" +numpy>=1.16,<2 ; python_version < "3.11" +``` + +Since Python only allows one version package, just version resolution would error here. Inspired by +[poetry](https://github.com/python-poetry/poetry), we instead use forking: Whenever there are +multiple requirements with different for one package name in the requirements of a package, we split +the resolution around these requirements. In this case, we take our partial solution and then once +solve the rest for `python_version >= "3.11"` and once for `python_version < "3.11"`. If some +markers overlap or are missing a part of the marker space, we add additional forks. There can be +more than 2 forks per package and we nest forks. You can see this in the log of `uv lock -v` by +looking for `Splitting resolution on ...`, `Solving split ... (requires-python: ...)` and +`Split ... resolution took ...`. + +One problem is that where and how we split is dependent on the order we see packages, which is in +turn dependent on the preference you get e.g. from `uv.lock`. So it can happen that we solve your +requirements with specific forks, write this to the lockfile, and when you call `uv lock` again, +we'd do a different resolution even if nothing changed because the preferences cause us to use +different fork points. To avoid this we write the `environment-markers` of each fork and each +package that diverges between forks to the lockfile. When doing a new resolution, we start with the +forks from the lockfile and use fork-dependent preference (from the `environment-markers` on each +package) to keep the resolution stable. When requirements change, we may introduce new forks from +the saved forks. We also merge forks with identical packages to keep the number of forks low. + +## Requires-python + +To ensure that a resolution with `requires-python = ">=3.9"` can actually be installed for all those +python versions, uv requires that all dependency support at least that python version. We reject +package versions that declare e.g. `requires-python = ">=3.10"` because we already know that a +resolution with that version can't be installed on Python 3.9, while the user explicitly requested +including 3.9. For simplicity and forward compatibility, we do however only consider lower bounds +for requires-python. If a dependency declares `requires-python = ">=3.8,<4"`, we don't want to +propagate that `<4` marker. + +## Wheel tags + +While our resolution is universal with respect to requirement markers, this doesn't extend to wheel +tags. Wheel tags can encode Python version, Python interpreter, operating system and architecture, +e.g. `torch-2.4.0-cp312-cp312-manylinux2014_aarch64.whl` is only compatible with CPython 3.12 on +arm64 Linux with glibc >= 2.17 (the manylinux2014 policy), while `tqdm-4.66.4-py3-none-any.whl` +works with all Python 3 versions and interpreters on any operating system and architecture. Most +projects have a (universally compatible) source distribution we can fall back to when we try to +install a package version and there is no compatible wheel, but some, such as `torch`, don't have a +source distribution. In this case an installation on e.g. Python 3.13 or an uncommon operating +system or architecture will fail with a message about a missing matching wheel. diff --git a/mkdocs.template.yml b/mkdocs.template.yml index a225032b1..a67d084e4 100644 --- a/mkdocs.template.yml +++ b/mkdocs.template.yml @@ -122,6 +122,7 @@ nav: - Reference: - Commands: reference/cli.md - Settings: reference/settings.md + - Resolution Internals: reference/resolution-internals.md - Policies: - Versioning: versioning.md - Platform support: platforms.md