Extends https://github.com/astral-sh/uv/pull/4121
Part of #2607
Adds support for managed toolchain fetching to `uv venv`, e.g.
```
❯ cargo run -q -- venv --python 3.9.18 --preview -v
DEBUG Searching for Python 3.9.18 in search path or managed toolchains
DEBUG Searching for managed toolchains at `/Users/zb/Library/Application Support/uv/toolchains`
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Found CPython 3.9.6 at `/usr/bin/python3` (search path)
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Requested Python not found, checking for available download...
DEBUG Using registry request timeout of 30s
INFO Fetching requested toolchain...
DEBUG Downloading https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst to temporary location /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp
DEBUG Extracting cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst
DEBUG Moving /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp/python to /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none
Using Python 3.9.18 interpreter at: /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none/install/bin/python3
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate
```
The preview flag is required. The fetch is performed if we can't find an
interpreter that satisfies the request. Once fetched, the toolchain will
be available for later invocations that include the `--preview` flag.
There will be follow-ups to improve toolchain management in general,
there is still outstanding work from the initial implementation.
## Summary
If we want more granular control over how these errors are handled, then
we need to move them out of the TOML deserialization.
No actual behavior changes here.
Part of https://github.com/astral-sh/uv/issues/4142.
Follow-up to #4016.
This exposes `Range` and `PubGrubSpecifier` from outside the resolver to
use pubgrub's union creating a dependency edge we don't really want.
When creating a lockfile, lock the combined dependencies for all
packages in a workspace. This make the lockfile independent of where you
are in the workspace.
Fixes#3983
## Summary
I don't think it's worth maintaining this separate test harness for ~18
tests, when they can all be tested in the `uv` package itself. Let's
drop the maintenance burden.
## Summary
Externally, development dependencies are currently structured as a flat
list of PEP 580-compatible requirements:
```toml
[tool.uv]
dev-dependencies = ["werkzeug"]
```
When locking, we lock all development dependencies; when syncing, users
can provide `--dev`.
Internally, though, we model them as dependency groups, similar to
Poetry, PDM, and [PEP 735](https://peps.python.org/pep-0735). This
enables us to change out the user-facing frontend without changing the
internal implementation, once we've decided how these should be exposed
to users.
A few important decisions encoded in the implementation (which we can
change later):
1. Groups are enabled globally, for all dependencies. This differs from
extras, which are enabled on a per-requirement basis. Note, however,
that we'll only discover groups for uv-enabled packages anyway.
2. Installing a group requires installing the base package. We rely on
this in PubGrub to ensure that we resolve to the same version (even
though we only expect groups to come from workspace dependencies anyway,
which are unique). But anyway, that's encoded in the resolver right now,
just as it is for extras.
We had previously changed the signature of
`DependencyProvider::get_dependencies` to return an iterator instead of
a hashmap to avoid the conversion cost from our dependencies `Vec` to
the pubgrub's hashmap. These changes are difficult to make in pubgrub
since they complicate the public api. But we don't actually use
`DependencyProvider::get_dependencies`, so we rolled those
customizations back in https://github.com/pubgrub-rs/pubgrub/pull/226
and instead opted to change only the internal
`add_incompatibility_from_dependencies` method that we exposed in our
fork. This aligns us closer with upstream, removes the design questions
about `DependencyProvider` from our concerns and reduces our diff (not
counting the github action) to +36 -12.
We retry several kinds of network request failures, but it's often
unclear whether a request was retried or not
(https://github.com/astral-sh/uv/issues/3514#issuecomment-2105485773).
This PR adds a small intermediary layer that logs all transient request
failures, adding the `DEBUG Transient request failure` lines:
```
DEBUG Searching for Python interpreter in virtual environments
DEBUG Found CPython 3.12.3 at `/home/konsti/projects/uv/.venv/bin/python3` (active virtual environment)
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
DEBUG Acquired lock for `.venv`
DEBUG At least one requirement is not satisfied: tqdm
DEBUG Using registry request timeout of 30s
DEBUG Solving with target Python version 3.12.3
DEBUG Adding direct dependency: tqdm*
DEBUG No cache entry for: https://pypi.org/simple/tqdm/
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
error: Could not connect, are you offline?
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
```
I decided for multi-line logging to show the complete error trace since
only `Transient request failure for https://pypi.org/simple/tqdm/,
retrying: Request error: error sending request for url
(https://pypi.org/simple/tqdm/)` doesn't tell you the actual problem (a
dns error).
Note that running with `-v` will not show messages about retry backoff
timing, but running with `RUST_LOG=debug` now shows a complete picture:
```
DEBUG starting new connection: https://pypi.org/
DEBUG resolving host="pypi.org"
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
WARN Retry attempt #2. Sleeping 528.728192ms before the next attempt
```
Fixes#3572
The workspace test directories can be used both in tests and directly
for developing/debugging. In the latter, we shouldn't copy the venv and
the lockfile when running tests. Using the ignore crate over manual
recursion we exclude those files.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
See #3834 .
This PR adds a new namespace, `override-dependencies`, to
pyproject.toml/uv.toml.
This namespace assumes that the dependencies you want to override are
written in the form of `requirements.txt`.
a example of pyproject.toml
```toml
[project]
name = "example"
version = "0.0.0"
dependencies = [
"flask==3.0.0"
]
[tool.uv]
override-dependencies = [
"werkzeug==2.3.0"
]
```
This will improve usability by allowing you to override dependencies
without having to specify the --override option when running `uv pip
compile/install`.
## Test Plan
added test to `crates/uv/tests/pip_compile.rs`.
---------
Co-authored-by: konstin <konstin@mailbox.org>
Add a `--package` option that allows switching the current project in
the workspace. Wherever you are in a workspace, you should be able to
run with any other project as root. This is the uv equivalent of `cargo
run -p`.
I don't love the `--package` name, esp. since `-p` is already taken and
in general to many things start with p already.
Part of this change is moving the workspace discovery of
`ProjectWorkspace` to `Workspace` itself.
## Usage
In albatross-virtual-workspace:
```console
$ uv venv
$ uv run --preview --package bird-feeder python -c "import albatross"
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds
Built 2 editables in 167ms
Resolved 5 packages in 4ms
Installed 5 packages in 1ms
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds)
+ sniffio==1.3.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'albatross'
$ uv venv
$ uv run --preview --package albatross python -c "import albatross"
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/albatross
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds
Built 3 editables in 173ms
Resolved 7 packages in 6ms
Installed 7 packages in 1ms
+ albatross==0.1.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/albatross)
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds)
+ sniffio==1.3.1
+ tqdm==4.66.4
```
In albatross-root-workspace:
```console
$ uv venv
$ uv run --preview --package bird-feeder python -c "import albatross"
Using Python 3.12.3 interpreter at: /home/konsti/.local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
Running `/home/konsti/projects/uv/target/debug/uv run --preview --package bird-feeder python -c 'import albatross'`
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds Built 2 editables in 161ms
Resolved 5 packages in 4ms
Installed 5 packages in 1ms
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds)
+ sniffio==1.3.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'albatross'
$ uv venv
$ cargo run run --preview --package albatross python -c "import albatross"
Using Python 3.12.3 interpreter at: /home/konsti/.local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.13s
Running `/home/konsti/projects/uv/target/debug/uv run --preview --package albatross python -c 'import albatross'`
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds
Built 3 editables in 168ms
Resolved 7 packages in 5ms
Installed 7 packages in 1ms
+ albatross==0.1.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace)
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds)
+ sniffio==1.3.1
+ tqdm==4.66.4
```
## Summary
This PR removes the static resolver map:
```rust
static RESOLVED_GIT_REFS: Lazy<Mutex<FxHashMap<RepositoryReference, GitSha>>> =
Lazy::new(Mutex::default);
```
With a `GitResolver` struct that we now pass around on the
`BuildContext`. There should be no behavior changes here; it's purely an
internal refactor with an eye towards making it cleaner for us to
"pre-populate" the list of resolved SHAs.
## Summary
This will help prevent bugs like #3934 by unifying the implementations
for editables and non-editable unnamed requirements. Specifically, both
of these now go through the same parsing paths and use the same struct
representations (with the exception that the editable flag is flipped in
the first case):
```
-e ./foo/bar
./foo/bar
```
We also now support PEP 508 in editable URLs. It turns out this is just
a limitation in pip, so it's correct to support it. For example, this
now works:
```
-e black[d] @ file://${PROJECT_ROOT}/scripts/packages/black_editable
```
Closes#3941.
Closes#3942.
With the change, we remove the special casing of workspace dependencies
and resolve `tool.uv` for all git and directory distributions. This
gives us support for non-editable workspace dependencies and path
dependencies in other workspaces. It removes a lot of special casing
around workspaces. These changes are the groundwork for supporting
`tool.uv` with dynamic metadata.
The basis for this change is moving `Requirement` from
`distribution-types` to `pypi-types` and the lowering logic from
`uv-requirements` to `uv-distribution`. This changes should be split out
in separate PRs.
I've included an example workspace `albatross-root-workspace2` where
`bird-feeder` depends on `a` from another workspace `ab`. There's a
bunch of failing tests and regressed error messages that still need
fixing. It does fix the audited package count for the workspace tests.
## Summary
In general, it's not quite right to filter preferences by `--reinstall`
-- we still want to respect existing versions, we just don't want to
respect _installed_ versions. But now that the installed versions and
preferences are decoupled, we can remove this (`--reinstall` is enforced
on the installed versions via the `Exclusions` struct that we pass to
the resolver).
While I was here, I also cleaned up the lockfile preference code to
better match the structure for `requirements.txt`.
## Summary
We currently rely on libgit2 for most git-related functionality.
However, libgit2 has long-standing performance issues, as well as lags
significantly behind git in terms of new features. For these reasons we
now use the git CLI by default for fetching repositories
(https://github.com/astral-sh/uv/pull/1781). This PR completely drops
libgit2 in favor of the git CLI for all git-related functionality, which
should allow us to use features such as partial clones and sparse
checkouts in the future for performance.
There is also a lot of technical debt in the current git code as it's
mostly taken from Cargo. Switching to the git CLI *vastly* simplifies
the `uv-git` codebase.
Eventually we might want to look into switching to
[`gitoxide`](https://github.com/Byron/gitoxide), but it's currently too
immature for our use case.
## Summary
This PR changes the lock-file format to use inline tables for wheels and
source distributions, which currently use separate tables that make the
file harder to follow.
```diff
[[distribution]]
name = "typing-extensions"
version = "4.10.0"
source = "registry+https://pypi.org/simple"
- [distribution.sdist]
- url = "0d26ce356c/typing_extensions-4.10.0.tar.gz"
- hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb"
- size = 77558
-
- [[distribution.wheel]]
- url = "dc04a3ea60/typing_extensions-4.10.0-py3-none-any.whl"
- hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475"
- size = 33926
+ sdist = { url = "0d26ce356c/typing_extensions-4.10.0.tar.gz", hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb", size = 77558 }
+ wheel = [{ url = "dc04a3ea60/typing_extensions-4.10.0-py3-none-any.whl", hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475", size = 33926 }]
```
The downside is that the inline-tables end up quite long and TOML
doesn't support line breaks in inline tables, yet.
Part of https://github.com/astral-sh/uv/issues/3611.
## Summary
There are a few behavior changes in here:
- We now enforce `--require-hashes` for editables, like pip. So if you
use `--require-hashes` with an editable requirement, we'll reject it. I
could change this if it seems off.
- We now treat source tree requirements, editable or not (e.g., both `-e
./black` and `./black`) as if `--refresh` is always enabled. This
doesn't mean that we _always_ rebuild them; but if you pass
`--reinstall`, then yes, we always rebuild them. I think this is an
improvement and is close to how editables work today.
Closes#3844.
Closes#2695.
Add workspace support when using `-r <path>/pyproject.toml` or `-e
<path>` in the pip interface. It is limited to all-editable
static-metadata workspaces, and tests only include a single main
workspace, ignoring path dependencies in another workspace. This can be
considered the MVP for workspace support: You can create a workspace,
you can install from it, but some options and conveniences are still
missing. I'll file follow-up tickets (support in lockfiles, support path
deps in other workspace, #3625)
There is also support in `uv run`, but we need
https://github.com/astral-sh/uv/issues/3700 first to properly support
using different current projects in the bluejay interface, currently the
resolution and therefore the lockfile depends on the current project.
I'd do this change first (it's big enough already), then #3700, and then
add workspace support properly to bluejay.
Fixes#3404
This bumps the versions of pep580 and pep440 to coincide with the
crates.io versions. While not strictly the same, the new types in uv us
an `Inner` struct. Practically I've found I'm still able to use the
patched versions, as can seen from the open PR here:
https://github.com/prefix-dev/pixi/pull/1436.
Would be great if this bump can be done so we can keep combining the
types :)
Extends #3726
Moves toolchain storage out of `UV_BOOTSTRAP_DIR` (`./bin`) into the
proper user data directory as defined by #3726.
Replaces `UV_BOOTSTRAP_DIR` with `UV_TOOLCHAIN_DIR` for customization.
Installed toolchains will be discovered without opt-in, but the idea is
still that these are not yet user-facing.
## Summary
This PR makes a variety of invalid states unrepresentable by changing
`Preference` to require a `PackageName` and `Version`, rather than
accepting a generic `Requirement`. There should be no meaningful
behavior changes.
## Summary
I haven't tested on Windows yet, but the idea here is that we should use
a portable representation when printing paths.
I decided to limit the scope here to paths that we write to output
files.
Closes https://github.com/astral-sh/uv/issues/3800.
