In #3514 and #2755, users had intermittent network errors, but it was
not always clear whether we had already retried these requests or not.
Building upon https://github.com/TrueLayer/reqwest-middleware/pull/159,
this PR adds the number of retries to the error message, so we can see
at first glance where we're missing retries and where we might need to
change retry settings.
Example error trace:
```
Could not connect, are you offline?
Caused by: Request failed after 3 retries
Caused by: error sending request for url (https://pypi.org/simple/uv/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
```
This code is ugly since i'm missing a better pattern for attaching
context to reqwest middleware errors in
https://github.com/TrueLayer/reqwest-middleware/pull/159.
I think `--toolchain-preference system` is sufficiently clear and
`--toolchain-preference prefer-system` is excessively verbose. This was
discussed in the original pull request at
https://github.com/astral-sh/uv/pull/4424 but because we had a case for
preferring "installed managed" toolchains I was hesitant to change it.
Now that I've dropped that in #4601, I think we can drop the prefix.
Adds a `toolchain-fetch` option alongside `toolchain-preference` with
`automatic` (default) and `manual` values allowing automatic toolchain
fetches to be disabled (replaces
https://github.com/astral-sh/uv/pull/4425). When `manual`, toolchains
must be installed with `uv toolchain install`.
Note this was previously implemented with `if-necessary`, `always`,
`never` variants but the interaction between this and
`toolchain-preference` was too confusing. By reducing to a binary
option, things should be clearer. The `if-necessary` behavior moved to
`toolchain-preference=installed`. See
https://github.com/astral-sh/uv/pull/4601#discussion_r1657839633 and
https://github.com/astral-sh/uv/pull/4601#discussion_r1658658755
The journey here can be seen in:
- #4587
- #4589
- #4594
I collapsed all the commits here because only the last one in the stack
got us to a "correct" error message.
There are a few architectural changes:
- We have a dedicated `MissingEnvironment` and `EnvironmentNotFound`
type for `PythonEnvironment::find` allowing different error messages
when searching for environments
- `ToolchainNotFound` becomes a struct with the `ToolchainRequest` which
greatly simplifies missing toolchain error formatting
- `ToolchainNotFound` tracks the `EnvironmentPreference` so it can
accurately report the locations checked
The messages look like this now, instead of the bland (and often
incorrect): "No Python interpreter found in system toolchains".
```
❯ cargo run -q -- pip sync requirements.txt
error: No virtual environment found
❯ UV_TEST_PYTHON_PATH="" cargo run -q -- pip sync requirements.txt --system
error: No system environment found
❯ UV_TEST_PYTHON_PATH="" cargo run -q -- pip sync requirements.txt --python 3.12
error: No virtual environment found for Python 3.12
❯ UV_TEST_PYTHON_PATH="" cargo run -q -- pip sync requirements.txt --python 3.12 --system
error: No system environment found for Python 3.12
❯ UV_TEST_PYTHON_PATH="" cargo run -q -- toolchain find 3.12 --preview
error: No toolchain found for Python 3.12 in system path
❯ UV_TEST_PYTHON_PATH="" cargo run -q -- pip compile requirements.in
error: No toolchain found in virtual environments or system path
```
I'd like to follow this with hints, suggesting creating an environment
or using system in some cases.
This is the minimal "working" implementation. In summary, we:
- Resolve the requested requirements
- Create an environment at `$UV_STATE_DIR/tools/$name`
- Inspect the `dist-info` for the main requirement to determine its
entry points scripts
- Link the entry points from a user-executable directory
(`$XDG_BIN_HOME`) to the environment bin
- Create an entry at `$UV_STATE_DIR/tools/tools.toml` tracking the
user's request
The idea with `tools.toml` is that it allows us to perform upgrades and
syncs, retaining the original user request (similar to declarations in a
`pyproject.toml`). I imagine using a similar schema in the
`pyproject.toml` in the future if/when we add project-levle tools. I'm
also considering exposing `tools.toml` in the standard uv configuration
directory instead of the state directory, but it seems nice to tuck it
away for now while we iterate on it. Installing a tool won't perform a
sync of other tool environments, we'll probably have an explicit `uv
tool sync` command for that?
I've split out todos into follow-up pull requests:
- #4509 (failing on Windows)
- #4501
- #4504Closes#4485
## Summary
In #4085, support was implemented for the `--prefix` option. When using
this option, however, a lock is either acquired on the virtualenv or
globally, preventing multiple installs to different `--prefix`s from the
same interpreter.
In this change, acquire the lock on just the prefix in question.
## Test Plan
Ran a `uv pip install` with `--prefix` and `RUST_LOG=trace` and observed
that the lock was acquired in the prefix.
Exposes the option added in #4416. Adds `--toolchain-preference` and
`tool.uv.toolchain-preference` to configure if system or managed
toolchains are preferred. Users can opt-out of managed toolchains or
system toolchains entirely as well.
Restores the `PythonEnvironment::find` API which was removed a while
back in favor of `Toolchain::find`. As mentioned in #4416, I'm
attempting to separate the case where you want an active environment
from the case where you want an installed toolchain in order to create
environments.
I wanted to drop `EnvironmentPreference` from `Toolchain::find` and just
have us consistently consider (or not consider) virtual environments
when discovering toolchains for creating environments. Unfortunately
this caused a few things to break so I reverted that change and will
explore it separately. Because I was exploring that change, there are
some minor changes to the `Toolchain` API here.
Adds support for the toolchain discovery preferences outlined in
https://github.com/astral-sh/uv/issues/4198 but we don't expose this to
users yet, I'll do that next to make it easier to review.
I've made some refactors in the toolchain discovery implementation to
enable this behavior and move us towards clearer abstractions. There's
still remaining work here, but I'd prefer tackle things in follow-ups
instead of expanding this pull request. I plan on opening a couple
before merging this.
I'd like to shift the public toolchain API to focus on discovering
either an **environment** or a **toolchain**. The first would be used by
commands that operate on an environment, while the latter would be used
by commands that just need an interpreter to create environments. I
haven't changed this here, but some of the refactors are in preparation
for supporting this idea.
In brief:
- We now allow different ordering of installed toolchain discovery based
on a `ToolchainPreference` type. This is the type we will expose to
users.
- `SystemPython` was changed into an `EnvironmentPreference` which is
used to determine if we should prefer virtual or system Python
environments.
- We drop the whole `ToolchainSources` selection concept, it was
confusing and the error messages from it were awkward. Most of the
functionality is now captured by the preference enums, but you can't do
things like "only find a toolchain from the parent interpreter" as
easily anymore.
Closes#4380
This is the same logic as `should_stop_discovery` but I changed the log
level and duplicated it because I don't really want that method to be
public. Maybe it should be though?
As in #4360, updates the uv project CLI to respect `.python-version`
files as default Python version requests. Additionally, updates project
interpreter discovery to fetch managed toolchains as in `uv venv
--preview`.
Allows installation of multiple toolchains in a single invocation
because I don't want to be limited to one! Most of the implementation
for concurrent downloads ported from `cargo dev fetch-python`.
Adds support for toolchain keys e.g. `cpython-3.11.2-macos` allowing you
to download toolchains for specific architectures and operating systems
using the format we use to uniquely identify a toolchain.
Closes#4240
e.g.
```
❯ cargo run -q -- pip install anyio --python "/Users/zb/Library/Application Support/uv/toolchains/cpython-3.12.0-macos-aarch64-none/install/bin/python3"
error: The interpreter at /Users/zb/Library/Application Support/uv/toolchains/cpython-3.12.0-macos-aarch64-none/install is externally managed, and indicates the following:
This toolchain is managed by uv and should not be modified.
Consider creating a virtual environment with `uv venv`.
```
Adds a command to find a toolchain on the system. Right now, it displays
the path to the first matching toolchain. We'll probably have more rich
output in the future (after implementing `toolchain show`).
The eventual plan (separate from here) is to port all of the toolchain
discovery tests to use this command. I'll add a few tests for this
command here anyway.
## Summary
This PR introduces top-level configuration for uv, such that you can do:
```toml
[tool.uv]
index-url = "https://test.pypi.org/simple"
```
And `uv pip compile`, `uv run`, `uv tool run`, etc., will all respect
that configuration.
The settings that were escalated to the top-level remain on
`tool.uv.pip` too, but they're only respected in `uv pip` commands. If
they're specified in both places, then the `pip` settings win out.
While making this change, I also wired up some of the global options,
like `connectivity` and `native_tls`, through to all the relevant
places.
Closes#4250.
Previously, we took the first executable on the `PATH` but if it was not
a usable interpreter we'd fail. Now, we'll continue searching in the
path until we find an interpreter as we do with the standard executable
names.
Includes system interpreters in `uv toolchain list`.
This includes a refactor of `find_toolchain` to support iterating over
all toolchains
that match a request rather than ending earlier.
Before 0.2.10 we would parse `--python=python` as an executable name.
After https://github.com/astral-sh/uv/pull/4214, we started treating
this as a Python version range request (with an empty version range).
This is not entirely unreasonable, but it was an unexpected regression
and I don't think `VersionRequest` should support empty ranges in its
`from_str` implementation without more consideration.
Closes https://github.com/astral-sh/uv/issues/3857
Instead of using custom `Arch`, `Os`, and `Libc` types I just use
`target-lexicon`'s which enumerate way more variants and implement
display and parsing. We use a wrapper type to represent a couple special
cases to support the "x86" alias for "i686" and "macos" for "darwin".
Alternatively we could try to use our `platform-tags` types but those
capture more information (like operating system versions) that we don't
have for downloads.
As discussed in https://github.com/astral-sh/uv/pull/4160, this is not
sufficient for proper libc detection but that work is larger and will be
handled separately.
Cherry-picked from https://github.com/astral-sh/uv/pull/4214
The first commit gets us some context on an IO error during queries:
Previously:
```
failed to canonicalize path `[VENV]/bin/python3`
Caused by: No such file or directory (os error 2)
```
Now:
```
Failed to query Python interpreter
Caused by: failed to canonicalize path `[VENV]/bin/python3`
Caused by: No such file or directory (os error 2)
```
but really we shouldn't attempt to query a missing interpreter during
discovery anyway, so we improve handling of that too.
Adds a command (following #4163) to download and install specific
toolchains. While we fetch toolchains on demand, this is useful for,
e.g., pre-downloading a toolchain in a Docker image build.
~I kind of think we should call this `install` instead of `fetch`~ I
changed the name from `fetch` to `install`.
Extends https://github.com/astral-sh/uv/pull/4121
Part of #2607
Adds support for managed toolchain fetching to `uv venv`, e.g.
```
❯ cargo run -q -- venv --python 3.9.18 --preview -v
DEBUG Searching for Python 3.9.18 in search path or managed toolchains
DEBUG Searching for managed toolchains at `/Users/zb/Library/Application Support/uv/toolchains`
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Found CPython 3.9.6 at `/usr/bin/python3` (search path)
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Requested Python not found, checking for available download...
DEBUG Using registry request timeout of 30s
INFO Fetching requested toolchain...
DEBUG Downloading https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst to temporary location /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp
DEBUG Extracting cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst
DEBUG Moving /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp/python to /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none
Using Python 3.9.18 interpreter at: /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none/install/bin/python3
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate
```
The preview flag is required. The fetch is performed if we can't find an
interpreter that satisfies the request. Once fetched, the toolchain will
be available for later invocations that include the `--preview` flag.
There will be follow-ups to improve toolchain management in general,
there is still outstanding work from the initial implementation.
Drops `find_toolchain`, `find_best_toolchain`, etc. in favor of
`Toolchain::find_...`
We can change this in the future, but there should only be one "right"
way to do it not two redundant ways in the public interface.
Extends #4120
Part of #2607
There should be no behavior changes here. Restructures the discovery API
to be focused on a toolchain first perspective in preparation for
exposing a `find_or_fetch` method for toolchains in
https://github.com/astral-sh/uv/pull/4138.
Moves all of `uv-toolchain` into `uv-interpreter`. We may split these
out in the future, but the refactoring I want to do for interpreter
discovery is easier if I don't have to deal with entanglement. Includes
some restructuring of `uv-interpreter`.
Part of #2386
Previously, this would use the "system" temporary directory.
Because we rename the resulting directory to its final destination,
and because renaming is implemented via hardlinking, and because
hardlinking doesn't work across mount points, and because /tmp is
commonly on a different mount point than where `uv` is checked out,
we "fix" this by putting the temporary directory somewhere close to
the final destination of the fetched artifact.
There are alternatives we might consider pursuing. For example,
if the `rename` fails, then we should probably do a recursive
directory copy. But this is a quick fix for now and it also
consistent with colocation of other temporary directories in `uv`.
The main downside of this change is that if a user does ^C while
`uv-dev fetch-python` is running, then there is no mechanism for
cleaning up temporary directories.
