Specifically, these are emitted when requirements fail to satisfy
`Requires-Python` or the markers associated with the current fork in the
resolver.
Closes#4373
Closes#4380
This is the same logic as `should_stop_discovery` but I changed the log
level and duplicated it because I don't really want that method to be
public. Maybe it should be though?
As in #4360, updates the uv project CLI to respect `.python-version`
files as default Python version requests. Additionally, updates project
interpreter discovery to fetch managed toolchains as in `uv venv
--preview`.
It was becoming problematic that the virtual environment test context
diverged from the other one i.e. we had to implement filtering twice.
This combines the contexts and tweaks the `TestContext` API and
filtering mechanisms for Python versions. Combined with my previous
changes to the test context at #4364 and
https://github.com/astral-sh/uv/pull/4368 this finally unblocks the
snapshots for test cases in #4360 and
https://github.com/astral-sh/uv/pull/4362.
## Summary
Make the git commit/date part of the version string matched in
cache_prune optional, so that the test also works correctly when uv is
built from an unpacked release tarball rather than a git repository.
Fixes#4374
## Test Plan
Ran `cargo test` inside the git repository and inside unpacked archive
for `0.2.12` release with the patch applied on top.
## Summary
Adds a `--no-clean` flag to `uv sync` that keeps extraneous
installations. This is the default in `uv run` and `uv add`, but not in
`uv sync` or `uv remove`. This means you need to run an explicit `uv
sync/remove` to clean the virtual environment.
Extends new filters for interpreter paths to apply to tests with
multiple Python versions. Adds patch version filtering for them as well,
which is needed for #4360 tests.
Extends #4364 automatically adding filters to the test context for
additional test virtual environments.
It turns out that the `pip sync` tests were really on the loose with
their virtual environment creation and it was difficult to use the new
helpers because they require mutability and the tests had immutable
borrows to extend the filters 😭. I refactored the tests to just reset
the environment.
A bare `uv toolchain install` invocation now reads default requests from
Python version files in the working directory. In order, a bare
invocation means:
- requests from `.python-versions`
- a single request from`.python-version`
- any installed managed toolchain
- the latest managed toolchain download
This replaces all the functionality of `cargo dev fetch-python`, which
we drop in #4337
## Summary
Closes https://github.com/astral-sh/uv/issues/4162
Changes keyring subprocess to allow display of stderr.
This aligns with pip's behavior since pip 23.1.
## Test Plan
* Tested using gnome-keyring-backend on a self-hosted private registry
as well as the keyring script described in #4162 to confirm both
existing functionality and the new stderr display.
* Existing tests using `scripts/packages/keyring_test_plugin` are now
showing its stderr output as well.
Allows installation of multiple toolchains in a single invocation
because I don't want to be limited to one! Most of the implementation
for concurrent downloads ported from `cargo dev fetch-python`.
Adds support for toolchain keys e.g. `cpython-3.11.2-macos` allowing you
to download toolchains for specific architectures and operating systems
using the format we use to uniquely identify a toolchain.
Closes#4240
e.g.
```
❯ cargo run -q -- pip install anyio --python "/Users/zb/Library/Application Support/uv/toolchains/cpython-3.12.0-macos-aarch64-none/install/bin/python3"
error: The interpreter at /Users/zb/Library/Application Support/uv/toolchains/cpython-3.12.0-macos-aarch64-none/install is externally managed, and indicates the following:
This toolchain is managed by uv and should not be modified.
Consider creating a virtual environment with `uv venv`.
```
I found this while testing the tracking of marker expressions across
resolver forks. Namely, given
sys_platform == 'darwin' and implementation_name == 'pypy'
And:
sys_platform == 'bar' or implementation_name == 'foo'
These should be disjoint, but the disjointness checker was reporting
them as overlapping. I fixed this by giving handling of disjunctions
higher precedence than conjunctions, although I am not 100% confident
that this is correct for all cases.
This commit adds marker expressions to our `Fork` type, which are in
turn passed down into `PubGrubDependencies::from_requirements` to filter
our any dependencies with markers that are disjoint from the fork's
marker expression.
This is necessary to avoid visiting packages in the dependency graph
that can never actually be installed. This is because when a fork is
created in the resolver, it always happens when there are two sibling
dependency specifications on a package with the same name, but with
non-overlapping marker expressions. Each fork corresponds to each
such conflicting dependency specification, and each fork assumes the
the corresponding marker expression as a pre-condition for any future
dependencies considered by it. That is, since the fork represents an
installation path that can only be taken when the corresponding
dependency specification (and its marker expression) is actually used,
it also therefore follows that the marker expression is true. Therefore,
any dependency visited in that fork with a marker expression that cannot
possibly be true when the markers of the fork are true can and ought to
be completely ignored.
There are some key invariants that I had to re-learn by reading the
code. This hopefully makes those invariants easier to discover by future
me (and others).
Otherwise, when testing `uv venv --preview` the default toolchain
directory will leak into the test.
I believe I've made a similar change for the standard `TestContext` in
another commit somewhere in my stack, if not I'll add it after.
Adds a command to find a toolchain on the system. Right now, it displays
the path to the first matching toolchain. We'll probably have more rich
output in the future (after implementing `toolchain show`).
The eventual plan (separate from here) is to port all of the toolchain
discovery tests to use this command. I'll add a few tests for this
command here anyway.
## Summary
This would be a lightweight solution to
https://github.com/astral-sh/uv/issues/4307 that doesn't fully engage
with all the possibilities in the design space (but would unblock
cross-platform for now).
## Summary
The fixtures here are pretty large, but it lets us test what we actually
care about (the resolved settings) rather than inferring the resolved
settings from behavior, which I think is a big improvement.
I also broke the tests down into more granular cases.
## Summary
Because the workspace member itself is part of the resolution, adding
the workspace name for the project leads to confusing errors, like:
```
❯ cargo run lock --preview
Compiling uv v0.2.11 (/Users/crmarsh/workspace/puffin/crates/uv)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.79s
Running `/Users/crmarsh/workspace/puffin/target/debug/uv lock --preview`
× No solution found when resolving dependencies:
╰─▶ Because only albatross==0.1.0 is available and albatross==0.1.0 depends on anyio<=3, we can conclude that all versions of albatross depend on anyio<=3.
And because bird-feeder==1.0.0 depends on anyio>=4.3.0,<5 and only bird-feeder==1.0.0 is available, we can conclude that all versions of albatross and all versions of bird-feeder are incompatible.
And because albatross depends on albatross and bird-feeder, we can conclude that the requirements are unsatisfiable.
```
(Notice "albatross depends on albatross".)
## Summary
In a workspace, we now read configuration from the workspace root.
Previously, we read configuration from the first `pyproject.toml` or
`uv.toml` file in path -- but in a workspace, that would often be the
_project_ rather than the workspace configuration.
We need to read configuration from the workspace root, rather than its
members, because we lock the workspace globally, so all configuration
applies to the workspace globally.
As part of this change, the `uv-workspace` crate has been renamed to
`uv-settings` and its purpose has been narrowed significantly (it no
longer discovers a workspace; instead, it just reads the settings from a
directory).
If a user has a `uv.toml` in their directory or in a parent directory
but is _not_ in a workspace, we will still respect that use-case as
before.
Closes#4249.
## Summary
This PR introduces top-level configuration for uv, such that you can do:
```toml
[tool.uv]
index-url = "https://test.pypi.org/simple"
```
And `uv pip compile`, `uv run`, `uv tool run`, etc., will all respect
that configuration.
The settings that were escalated to the top-level remain on
`tool.uv.pip` too, but they're only respected in `uv pip` commands. If
they're specified in both places, then the `pip` settings win out.
While making this change, I also wired up some of the global options,
like `connectivity` and `native_tls`, through to all the relevant
places.
Closes#4250.
Previously, we took the first executable on the `PATH` but if it was not
a usable interpreter we'd fail. Now, we'll continue searching in the
path until we find an interpreter as we do with the standard executable
names.
Previously, `b` in the test case would have been incorrectly locked to
the path of `a`. I've moved `relative_to` into uv-fs since it's now used
in two different places.
Previously failing lockfile when `a/pyproject.toml` and
`a/b/pyproject.toml` exist (not in a workspace) and `a` was depending on
`b`:
```toml
version = 1
requires-python = ">=3.11, <3.13"
[[distribution]]
name = "b"
version = "0.1.0"
source = "directory+/home/konsti/projects/uv/a"
sdist = { path = "/home/konsti/projects/uv/a" }
[[distribution]]
name = "black"
version = "0.1.0"
source = "editable+."
sdist = { path = "." }
[[distribution.dependencies]]
name = "b"
version = "0.1.0"
source = "directory+/home/konsti/projects/uv/a"
```
Includes system interpreters in `uv toolchain list`.
This includes a refactor of `find_toolchain` to support iterating over
all toolchains
that match a request rather than ending earlier.
## Summary
This i still a draft, but it gets some of the work started on issue
#4064 , which requests --no-binary functionality similar to `uv pip
install` to exist on `uv pip compile`.
So far this has moved the command line shape from install and cloned it
over to compile. The actual functionality of respecting --no-binary
<package> and generating the resulting line in requirements.txt I could
use a hand with.
My understanding is we want to create a requirements.in file such as:
```
yt
```
Then when running `cargo run -- pip compile --no-binary yt
requirements.in`
we want the file to have this line in it:
```
yt==4.3.1 --no-binary yt
```
## Test Plan
Existing unit tests continue to pass.
No new unit tests have been created yet.
The new command line options do show up when testing with `cargo run --
pip compile -h`
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
Closes https://github.com/astral-sh/uv/issues/4296.
## Test Plan
Ran `cargo run lock --verbose` from
`scripts/workspaces/albatross-virtual-workspace`:
```
DEBUG uv 0.2.11 (ef3bc1612 2024-06-12)
warning: `uv lock` is experimental and may change without warning.
DEBUG Found workspace root: `/Users/crmarsh/workspace/puffin/scripts/workspaces/albatross-virtual-workspace`
DEBUG Adding discovered workspace member: /Users/crmarsh/workspace/puffin/scripts/workspaces/albatross-virtual-workspace/packages/albatross
DEBUG Adding discovered workspace member: /Users/crmarsh/workspace/puffin/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder
DEBUG Adding discovered workspace member: /Users/crmarsh/workspace/puffin/scripts/workspaces/albatross-virtual-workspace/packages/seeds
DEBUG Searching for Python >=3.12 in search path or managed toolchains
DEBUG Searching for managed toolchains at `/Users/crmarsh/Library/Application Support/uv/toolchains`
DEBUG Found managed toolchain `cpython-3.12.3-macos-aarch64-none`
DEBUG Found CPython 3.12.3 at `/Users/crmarsh/Library/Application Support/uv/toolchains/cpython-3.12.3-macos-aarch64-none/install/bin/python3` (managed toolchains)
Using Python 3.12.3 interpreter at: /Users/crmarsh/Library/Application Support/uv/toolchains/cpython-3.12.3-macos-aarch64-none/install/bin/python3
```
Before 0.2.10 we would parse `--python=python` as an executable name.
After https://github.com/astral-sh/uv/pull/4214, we started treating
this as a Python version range request (with an empty version range).
This is not entirely unreasonable, but it was an unexpected regression
and I don't think `VersionRequest` should support empty ranges in its
`from_str` implementation without more consideration.
Updates `--no-binary <package>` to take precedence over `--only-binary
:all:` and `--only-binary <package>` to take precedence over
`--no-binary :all:`.
I'm not entirely sure about this behavior, e.g. maybe I provided
`--only-binary :all:` later on the command line and really want it to
override those earlier arguments of `--no-binary <package>` for safety.
Right now we just fail to solve though since we can't satisfy the
overlapping requests.
Closes https://github.com/astral-sh/uv/issues/4063
## Summary
This is what I consider to be the "real" fix for #8072. We now treat
directory and path URLs as separate `ParsedUrl` types and
`RequirementSource` types. This removes a lot of `.is_dir()` forking
within the `ParsedUrl::Path` arms and makes some states impossible
(e.g., you can't have a `.whl` path that is editable). It _also_ fixes
the `direct_url.json` for direct URLs that refer to files. Previously,
we wrote out to these as if they were installed as directories, which is
just wrong.
We weren't using the common interface in `uv lock` because it didn't
support finding an interpreter without touching the virtual environment.
Here I refactor the project interface to support what we need and update
`uv lock` to use the shared implementation.
In the time before universal resolving, we would always pass a
`MarkerEnvironment`, and this environment would capture any relevant
`Requires-Python` specifier (including if `-p/--python` was provided on
the CLI).
But in universal resolution, we very specifically do not use a
`MarkerEnvironment` because we want to produce a resolution that is
compatible across potentially multiple environments. This in turn meant
that we lost `Requires-Python` filtering.
This PR adds it back. We do this by converting our `PythonRequirement`
into a `MarkerTree` that encodes the version specifiers in a
`Requires-Python` specifier. We then ask whether that `MarkerTree` is
disjoint with a dependency specification's `MarkerTree`. If it is, then
we know it's impossible for that dependency specification to every be
true, and we can completely ignore it.
## Summary
Right now, we're _always_ reinstalling local wheel archives, even if the
timestamp didn't change.
I want to fix the TODO properly but I will do so in a separate PR.
## Summary
`normalize` now takes an owned value and returns `Option<MarkerTree>`,
such that if any sub-expression evaluates to `true`, we can normalize
out the entire marker.
Closes https://github.com/astral-sh/uv/issues/4267.
Closes https://github.com/astral-sh/uv/issues/3857
Instead of using custom `Arch`, `Os`, and `Libc` types I just use
`target-lexicon`'s which enumerate way more variants and implement
display and parsing. We use a wrapper type to represent a couple special
cases to support the "x86" alias for "i686" and "macos" for "darwin".
Alternatively we could try to use our `platform-tags` types but those
capture more information (like operating system versions) that we don't
have for downloads.
As discussed in https://github.com/astral-sh/uv/pull/4160, this is not
sufficient for proper libc detection but that work is larger and will be
handled separately.
## Summary
Fix the docsting where `remove` was used instead of `add` in the context
of `uv add` command.
## Test Plan
```
cargo run -- add --help
```
```
Add one or more packages to the project requirements
Usage: uv add [OPTIONS] <REQUIREMENTS>...
Arguments:
<REQUIREMENTS>...
The packages to add, as PEP 508 requirements (e.g., `flask==2.2.3`)
```
## Summary
I think this is a useful piece of connective tissue that will let us
avoid back-and-forths when folks include traces.
## Test Plan
```
❯ cargo run pip list --verbose
DEBUG uv 0.2.11 (44041bccd 2024-06-11)
DEBUG Searching for Python interpreter in virtual environments
DEBUG Found CPython 3.12.3 at `/Users/crmarsh/workspace/puffin/.venv/bin/python3` (virtual environment)
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
```
## Summary
If we're ORing an OR, we should just append rather than nesting in
another OR.
In my branch, this let us simplify:
```
python_version < '3.10' or python_version > '3.12' or (python_version < '3.8' or python_version > '3.12')
```
To:
```
python_version < '3.10' or python_version > '3.12
```
## Summary
The build tags in this case are like, e.g., `202206090410`. That's
larger than a `u32`, so we're rejecting the wheel. In theory build tags
could be even larger, but we already use `u64` for version segment so I
think it's fine to keep that constraint here.
I'm going to look into surfacing these errors separately.
Closes https://github.com/astral-sh/uv/issues/4252.
## Test Plan
`cargo run pip install monailabel`
e.g.
```
❯ uv toolchain install
Found installed toolchain 'cpython-3.9.19-macos-aarch64-none'
A toolchain is already installed. Use `uv toolchain install <request>` to install a specific toolchain
```
instead of
```
❯ uv toolchain install
Using latest Python version
Found installed toolchain 'cpython-3.9.19-macos-aarch64-none'
Already installed at /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.19-macos-aarch64-none
```
## Summary
Something that looks like it was forgotten to replace in #4164.
## Test Plan
Run `cargo run toolchain install` should display the warning: ``warning:
`uv toolchain install` is experimental and may change without warning.``
By splitting `path` into a lockable, relative (or absolute) and an
absolute installable path and by splitting between urls and paths by
dist type, we can store relative paths in the lockfile.
## Summary
We may choose to persist these eventually, but for now, it's useful to
have them colocated with the cache, and in their own dedicated bucket
(so, at the very least, we can keep track of the use-cases).
Closes https://github.com/astral-sh/uv/issues/4219.
A merge kerfuffle from #4222 and #4218
Now we fail because we genuinely can't find any interpreters since tests
contexts are isolated by default. I'll improve the error message and
maybe add another test case once `main` is fixed.
By setting the test search path to an empty path, we avoid accidentally
pulling interpreters from the system during a test case.
Cherry-picked from https://github.com/astral-sh/uv/pull/4214
Cherry-picked from https://github.com/astral-sh/uv/pull/4214
The first commit gets us some context on an IO error during queries:
Previously:
```
failed to canonicalize path `[VENV]/bin/python3`
Caused by: No such file or directory (os error 2)
```
Now:
```
Failed to query Python interpreter
Caused by: failed to canonicalize path `[VENV]/bin/python3`
Caused by: No such file or directory (os error 2)
```
but really we shouldn't attempt to query a missing interpreter during
discovery anyway, so we improve handling of that too.
## Summary
Should be no behavior changes, but one piece of technical debt I noticed
left over in the URL resolver. We already have structured paths, so we
shouldn't need to compare verbatim URLs.
## Summary
If the user requests a package as both editable and non-editable, the
editable now "wins".
Previously, `pip install -e . .` would install as editable. However,
`pip install -e . -r requirements.txt` would _not_ if `requirements.txt`
contained `.`, because we ignored `editable` when deduplicating and the
order of iteration was just dependent on internals.
Closes https://github.com/astral-sh/uv/issues/4053.
Adds a command (following #4163) to download and install specific
toolchains. While we fetch toolchains on demand, this is useful for,
e.g., pre-downloading a toolchain in a Docker image build.
~I kind of think we should call this `install` instead of `fetch`~ I
changed the name from `fetch` to `install`.
## Summary
Adds handling for a few cases to improve interoperability with Poetry:
- If the `project` schema is invalid, we now raise a hard error, rather
than treating the metadata as dynamic and then falling back to the build
backend. This could cause problems, I'm not sure. It's stricter than
before.
- If the project contains `tool.poetry` but omits
`project.dependencies`, we now treat it as dynamic. We could go even
further and treat _any_ Poetry project as dynamic, but then we'd be
ignoring user-declared dependencies, which is also confusing.
Closes https://github.com/astral-sh/uv/issues/4142.
Extends https://github.com/astral-sh/uv/pull/4121
Part of #2607
Adds support for managed toolchain fetching to `uv venv`, e.g.
```
❯ cargo run -q -- venv --python 3.9.18 --preview -v
DEBUG Searching for Python 3.9.18 in search path or managed toolchains
DEBUG Searching for managed toolchains at `/Users/zb/Library/Application Support/uv/toolchains`
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Found CPython 3.9.6 at `/usr/bin/python3` (search path)
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Requested Python not found, checking for available download...
DEBUG Using registry request timeout of 30s
INFO Fetching requested toolchain...
DEBUG Downloading https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst to temporary location /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp
DEBUG Extracting cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst
DEBUG Moving /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp/python to /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none
Using Python 3.9.18 interpreter at: /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none/install/bin/python3
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate
```
The preview flag is required. The fetch is performed if we can't find an
interpreter that satisfies the request. Once fetched, the toolchain will
be available for later invocations that include the `--preview` flag.
There will be follow-ups to improve toolchain management in general,
there is still outstanding work from the initial implementation.
## Summary
Similar to how we abstracted the dependencies into
`ResolutionDependencyNames`, I think it makes sense to abstract the base
packages into a `ResolutionPackage`. This also avoids leaking details
about the various `PubGrubPackage` enum variants to `ResolutionGraph`.
Remove the panic when there is an invalid wheel source, instead surface
the error. This error can only occur when manually editing the lock
file, but since it's an external file, we should error and not panic.
This change is helpful since the method needs to be able to error for
relative path support.
## Summary
If a package lacks a source distribution, and we can't find a compatible
wheel for the current platform, we need to just _assume_ that the
package will have a valid wheel on all platforms on which it's
requested; if not, we raise an error at install time.
It's possible that we can be smarter about this over time. For example,
if the package was requested _only_ for macOS, we could verify that
there's at least one macOS-compatible wheel. See the linked issue for
more details.
Closes https://github.com/astral-sh/uv/issues/4139.
## Summary
If we want more granular control over how these errors are handled, then
we need to move them out of the TOML deserialization.
No actual behavior changes here.
Part of https://github.com/astral-sh/uv/issues/4142.
## Summary
See the long comment inline. I think this is debatable but probably
right for now. The other options have their own problems, but there are
a few alternate ideas in the comment.
Closes https://github.com/astral-sh/uv/issues/4132.
The basic idea here is to make it so forking can only ever result in a
resolution that, for a particular marker environment, will only install
at most one version of a package. We can guarantee this by ensuring we
only fork on conflicting dependency specifications only when their
corresponding markers are completely disjoint. If they aren't, then
resolution _must_ find a single version of the package in the
intersection of the two dependency specifications.
A test for this case has been added to packse here:
https://github.com/astral-sh/packse/pull/182. Previously, that test
would result in a resolution with two different unconditional versions
of the same package. With this change, resolution fails (as it should).
A commit-by-commit review should be helpful here, since the first commit
is a refactor to make the second commit a bit more digestible.
## Summary
I think we should be able to model PubGrub such that this isn't
necessary (at least for the case described in the issue), but for now,
let's just avoid attempting to build very old distributions in
prefetching.
Closes https://github.com/astral-sh/uv/issues/4136.
Drops `find_toolchain`, `find_best_toolchain`, etc. in favor of
`Toolchain::find_...`
We can change this in the future, but there should only be one "right"
way to do it not two redundant ways in the public interface.
## Summary
Simplify and normalize marker expressions in the lockfile. Right now
this does a simple analysis by only looking at related operators at the
same level of precedence. I think anything more complex would be out of
scope.
Resolves https://github.com/astral-sh/uv/issues/4002.
## Summary
This PR adds a lowering similar to that seen in
https://github.com/astral-sh/uv/pull/3100, but this time, for markers.
Like `PubGrubPackageInner::Extra`, we now have
`PubGrubPackageInner::Marker`. The dependencies of the `Marker` are
`PubGrubPackageInner::Package` with and without the marker.
As an example of why this is useful: assume we have `urllib3>=1.22.0` as
a direct dependency. Later, we see `urllib3 ; python_version > '3.7'` as
a transitive dependency. As-is, we might (for some reason) pick a very
old version of `urllib3` to satisfy `urllib3 ; python_version > '3.7'`,
then attempt to fetch its dependencies, which could even involve
building a very old version of `urllib3 ; python_version > '3.7'`. Once
we fetch the dependencies, we would see that `urllib3` at the same
version is _also_ a dependency (because we tack it on). In the new
scheme though, as soon as we "choose" the very old version of `urllib3 ;
python_version > '3.7'`, we'd then see that `urllib3` (the base package)
is also a dependency; so we see a conflict before we even fetch the
dependencies of the old variant.
With this, I can successfully resolve the case in #4099.
Closes https://github.com/astral-sh/uv/issues/4099.
Extends #4120
Part of #2607
There should be no behavior changes here. Restructures the discovery API
to be focused on a toolchain first perspective in preparation for
exposing a `find_or_fetch` method for toolchains in
https://github.com/astral-sh/uv/pull/4138.
Partially addresses https://github.com/astral-sh/uv/issues/4056
We were incorrectly omitting the port from requests to `keyring` when
falling back to a realm/host query, e.g. `localhost` was used instead of
`localhost:1234`. We still won't include "standard" ports like `80` for
an HTTP request.
Follow-up to #4016.
This exposes `Range` and `PubGrubSpecifier` from outside the resolver to
use pubgrub's union creating a dependency edge we don't really want.
When creating a lockfile, lock the combined dependencies for all
packages in a workspace. This make the lockfile independent of where you
are in the workspace.
Fixes#3983
## Summary
This PR modifies our `Requires-Python` handling to treat
`Requires-Python` as a lower bound. There's extensive discussion around
this in https://github.com/astral-sh/uv/issues/4022 and the references
linked therein. I think it's an experiment worth trying. Even in my own
small projects, I'm running into issues whereby I'm being "forced" to
add a `<4` upper bound to my `Requires-Python` due to these caps.
Separately, we should explore adding a mechanism that's distinct from
`Requires-Python` to enable users to declare a supported range for
locking.
Closes https://github.com/astral-sh/uv/issues/4022.
## Summary
The condition enforced here isn't quite right. The same dependency can
appear multiple times, as long as the extra is different.
Closes https://github.com/astral-sh/uv/issues/4101.
## Summary
The "only include if relevant for the extra" filtering should _not_ be
applied to constraints. Otherwise, we'd only constrain when the extra
was included in the constraints file itself, which is incorrect.
Closes https://github.com/astral-sh/uv/issues/4091.
## Summary
I don't think it's worth maintaining this separate test harness for ~18
tests, when they can all be tested in the `uv` package itself. Let's
drop the maintenance burden.
## Summary
Externally, development dependencies are currently structured as a flat
list of PEP 580-compatible requirements:
```toml
[tool.uv]
dev-dependencies = ["werkzeug"]
```
When locking, we lock all development dependencies; when syncing, users
can provide `--dev`.
Internally, though, we model them as dependency groups, similar to
Poetry, PDM, and [PEP 735](https://peps.python.org/pep-0735). This
enables us to change out the user-facing frontend without changing the
internal implementation, once we've decided how these should be exposed
to users.
A few important decisions encoded in the implementation (which we can
change later):
1. Groups are enabled globally, for all dependencies. This differs from
extras, which are enabled on a per-requirement basis. Note, however,
that we'll only discover groups for uv-enabled packages anyway.
2. Installing a group requires installing the base package. We rely on
this in PubGrub to ensure that we resolve to the same version (even
though we only expect groups to come from workspace dependencies anyway,
which are unique). But anyway, that's encoded in the resolver right now,
just as it is for extras.
## Summary
As with other `.egg-info` and `.egg-link` distributions, it's easy to
support _existing_ `.egg-link` files. Like pip, we refuse to uninstall
these, since there's no way to know which files are part of the
distribution.
Closes https://github.com/astral-sh/uv/issues/4059.
## Test Plan
Verify that `vtk` is included here, which is installed as a `.egg-link`
file:
```
> conda create -c conda-forge -n uv-test python h5py vtk pyside6 cftime psutil
> cargo run pip freeze --python /opt/homebrew/Caskroom/miniforge/base/envs/uv-test/bin/python
aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1713964997382/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1704011227531/work
cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work
cftime @ file:///Users/runner/miniforge3/conda-bld/cftime_1715919201099/work
frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1702645558715/work
h5py @ file:///Users/runner/miniforge3/conda-bld/h5py_1715968397721/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1713279365350/work
loguru @ file:///Users/runner/miniforge3/conda-bld/loguru_1695547410953/work
msgpack @ file:///Users/runner/miniforge3/conda-bld/msgpack-python_1715670632250/work
multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1707040780513/work
numpy @ file:///Users/runner/miniforge3/conda-bld/numpy_1707225421156/work/dist/numpy-1.26.4-cp312-cp312-macosx_11_0_arm64.whl
pip==24.0
psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1705722460205/work
pyside6==6.7.1
setuptools==70.0.0
shiboken6==6.7.1
vtk==9.2.6
wheel==0.43.0
wslink @ file:///home/conda/feedstock_root/build_artifacts/wslink_1716591560747/work
yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1705508643525/work
```
Closes https://github.com/astral-sh/uv/issues/4072
This was an accidental change in
https://github.com/astral-sh/uv/pull/4029 in which I had updated the
pull request to support virtual environments as requested in review and
forgot to revert it.
Separately, we shouldn't fail if `VIRTUAL_ENV` points to an empty
directory and `SystemPython::Allowed` is used, will address that
separately.
## Summary
If `Requires-Python` is omitted in `uv lock` or `uv run`, we now warn
and default to `>=` the current minor version.
Closes https://github.com/astral-sh/uv/issues/4050.
## Summary
This PR adds the `Requires-Python` range to the user's lockfile. This
will enable us to validate it when installing.
For now, we repeat the `Requires-Python` back to the user;
alternatively, though, we could detect the supported Python range
automatically.
See: https://github.com/astral-sh/uv/issues/4052
We had previously changed the signature of
`DependencyProvider::get_dependencies` to return an iterator instead of
a hashmap to avoid the conversion cost from our dependencies `Vec` to
the pubgrub's hashmap. These changes are difficult to make in pubgrub
since they complicate the public api. But we don't actually use
`DependencyProvider::get_dependencies`, so we rolled those
customizations back in https://github.com/pubgrub-rs/pubgrub/pull/226
and instead opted to change only the internal
`add_incompatibility_from_dependencies` method that we exposed in our
fork. This aligns us closer with upstream, removes the design questions
about `DependencyProvider` from our concerns and reduces our diff (not
counting the github action) to +36 -12.
Retry the flaky `compile_invalid_pyc_invalidation_mode` if it fails. I
don't understand why this happening in the first place (we have code
that should catch those cases, but also those cases shouldn't be
happening at all) and this is terrible hack, but it fixes the test
flakes.
Fixes#2672
## Summary
This PR separates "gathering the requirements" from the rest of the
metadata (e.g., version), which isn't required when installing a
package's _dependencies_ (as opposed to installing the package itself).
It thus ensures that we don't need to build a package when a static
`pyproject.toml` is provided in `pip compile`.
Closes https://github.com/astral-sh/uv/issues/4040.
We know that `[project]` must exist for each workspace member, so we can
store it directly and avoid going through the `.and_then()` when we need
to access it. This requires cloning the struct due to lack of
self-referential structs. An alternative would taking the `Project` from
`PyProjectToml` instead, but this could be confusing when passing the
`PyProjectToml` around.
## Summary
Thankfully this is pretty rare since `pip sync` is usually run on `pip
compile` output, and `pip compile` never outputs markers.
Closes https://github.com/astral-sh/uv/issues/4044
## Summary
Closes#3955
Adds explicit support to `NO_COLOR` and `FORCE_COLOR` via
GlobalSettings.
The order, per specs is now `NO_COLOR` > `FORCE_COLOR` > `color`.
This PR is a backup plan pending rust-cli/anstyle#192.
## Test Plan
Tested all cases locally for now; I didn't see existing tests for
GlobalSettings parsing.
## Summary
Given `install -e dagster`, we need to assume that the user meant
`install -e ./dagster`, even though `install dagster` should _not_ be
treated as `install ./dagster`. I suspect pip will change this in the
future (since `pip install dagster` does _not_ meant `pip install
./dagster`) but for now it's what users expect.
Closes https://github.com/astral-sh/uv/issues/3994.
This is just the result of running
./scripts/sync_scenarios.sh
From the root of the `uv` repository.
When I initially ran this, it produced some tests with snapshots that
weren't being updated. It turned out this was because the tests weren't
running, as they were gated behind the `python-patch` feature. In this
commit, we add `python-patch` to our `cargo insta` command, which should
update all relevant snapshots.
There are still some superfluous updates as a result of a spell checker
being run on generated files, but
This is a quick fix for some flaky tests where the output in the lock
file isn't stable because marker expressions can be combined in a
non-deterministic order.
I believe there is ongoing work to simplify marker expressions which
will help here, but I think some kind of normalization is still
ultimately needed to guarantee consistent output.
I first noticed the flaky test in:
https://github.com/astral-sh/uv/pull/4015
## Summary
Avoid using work-stealing Tokio workers for bytecode compilation,
favoring instead dedicated threads. Tokio's work-stealing does not
really benefit us because we're spawning Python workers and scheduling
tasks ourselves — we don't want Tokio to re-balance our workers. Because
we're doing scheduling ourselves and compilation is a primarily
compute-bound task, we can also create dedicated runtimes for each
worker and avoid some synchronization overhead.
This is part of a general desire to avoid relying on Tokio's
work-stealing scheduler and be smarter about our workload. In this case
we already had the custom scheduler in place, Tokio was just getting in
the way (though the overhead is very minor).
## Test Plan
This improves performance by ~5% on my machine.
```
$ hyperfine --warmup 1 --prepare "target/profiling/uv-dev clear-compile .venv" "target/profiling/uv-dev compile .venv" "target/profiling/uv-dev-dedicated compile .venv"
Benchmark 1: target/profiling/uv-dev compile .venv
Time (mean ± σ): 1.279 s ± 0.011 s [User: 13.803 s, System: 2.998 s]
Range (min … max): 1.261 s … 1.296 s 10 runs
Benchmark 2: target/profiling/uv-dev-dedicated compile .venv
Time (mean ± σ): 1.220 s ± 0.021 s [User: 13.997 s, System: 3.330 s]
Range (min … max): 1.198 s … 1.272 s 10 runs
Summary
target/profiling/uv-dev-dedicated compile .venv ran
1.05 ± 0.02 times faster than target/profiling/uv-dev compile .venv
$ hyperfine --warmup 1 --prepare "target/profiling/uv-dev clear-compile .venv" "target/profiling/uv-dev compile .venv" "target/profiling/uv-dev-dedicated compile .venv"
Benchmark 1: target/profiling/uv-dev compile .venv
Time (mean ± σ): 3.631 s ± 0.078 s [User: 47.205 s, System: 4.996 s]
Range (min … max): 3.564 s … 3.832 s 10 runs
Benchmark 2: target/profiling/uv-dev-dedicated compile .venv
Time (mean ± σ): 3.521 s ± 0.024 s [User: 48.201 s, System: 5.392 s]
Range (min … max): 3.484 s … 3.566 s 10 runs
Summary
target/profiling/uv-dev-dedicated compile .venv ran
1.03 ± 0.02 times faster than target/profiling/uv-dev compile .venv
```
We retry several kinds of network request failures, but it's often
unclear whether a request was retried or not
(https://github.com/astral-sh/uv/issues/3514#issuecomment-2105485773).
This PR adds a small intermediary layer that logs all transient request
failures, adding the `DEBUG Transient request failure` lines:
```
DEBUG Searching for Python interpreter in virtual environments
DEBUG Found CPython 3.12.3 at `/home/konsti/projects/uv/.venv/bin/python3` (active virtual environment)
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
DEBUG Acquired lock for `.venv`
DEBUG At least one requirement is not satisfied: tqdm
DEBUG Using registry request timeout of 30s
DEBUG Solving with target Python version 3.12.3
DEBUG Adding direct dependency: tqdm*
DEBUG No cache entry for: https://pypi.org/simple/tqdm/
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
error: Could not connect, are you offline?
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
```
I decided for multi-line logging to show the complete error trace since
only `Transient request failure for https://pypi.org/simple/tqdm/,
retrying: Request error: error sending request for url
(https://pypi.org/simple/tqdm/)` doesn't tell you the actual problem (a
dns error).
Note that running with `-v` will not show messages about retry backoff
timing, but running with `RUST_LOG=debug` now shows a complete picture:
```
DEBUG starting new connection: https://pypi.org/
DEBUG resolving host="pypi.org"
DEBUG Transient request failure for https://pypi.org/simple/tqdm/, retrying: Request error: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: error sending request for url (https://pypi.org/simple/tqdm/)
Caused by: client error (Connect)
Caused by: dns error: failed to lookup address information: Name or service not known
Caused by: failed to lookup address information: Name or service not known
WARN Retry attempt #2. Sleeping 528.728192ms before the next attempt
```
Fixes#3572
## Summary
See #4013
`uv pip ...` command loads workspace settings from pyproject.toml and
uv.toml.
Although a warning is implemented to output a warning when parsing
fails, it is not actually output.
https://github.com/astral-sh/uv/blob/main/crates/uv-workspace/src/workspace.rs#L38-L61
The reason is that the flag to display warnings is enabled after loading
the workspace settings.
This PR turns on the warning output flag before loading the workspace.
## Test Plan
pyproject.toml for test
```toml
[project]
name = "sample"
version = "0.0.0"
dependencies = ["ruff"]
[tool.uv.pip]
# originally string type.
index-url = 1
```
command output (before modification)
```bash
uv pip compile pyproject.toml
Resolved 1 package in 383ms
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml
ruff==0.4.7
# via sample (pyproject.toml)
```
command output (after modification)
```bash
uv pip compile pyproject.toml
warning: Failed to parse `pyproject.toml`: TOML parse error at line 7, column 13
|
7 | index-url = true
| ^^^^
invalid type: boolean `true`, expected a string
Resolved 1 package in 107ms
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml
ruff==0.4.7
# via sample (pyproject.toml)
```
The workspace test directories can be used both in tests and directly
for developing/debugging. In the latter, we shouldn't copy the venv and
the lockfile when running tests. Using the ignore crate over manual
recursion we exclude those files.
## Summary
Instead of checking if the target and installed version are the same, we
model the data such that the target version is only present if it was
specified by the user. This also means that we correctly say "requested
version" even if the two happen to be the same.
## Summary
I believe this is no longer necessary. Part of the problem here is that
we can't _know_ the full set of available Python versions, especially
once we start resolving against a `Requires-Python` rather than a fixed
set of two versions.
## Summary
Previously, when we locked something like `flask[dotenv]`, we created
two separate distributions in the lockfile: one for `flask`, which
included the base dependencies, and one for `flask[dotenv]`, which
included the base dependencies _and_ the `dotenv` dependencies. This was
easy to implement, but it meant that we were duplicating all of the
distribution files for every extra, and duplicating all of the base
dependencies for every extra.
This PR normalizes the data such that we now have one entry per
distribution (i.e., `ExtraName` was removed from `DistributionId`), with
an optional dependencies table with an entry per extra, like:
```toml
[[distribution]]
name = "project"
version = "0.1.0"
source = "editable+file://[TEMP_DIR]/"
sdist = { url = "file://[TEMP_DIR]/" }
[[distribution.dependencies]]
name = "anyio"
version = "3.7.0"
source = "registry+https://pypi.org/simple"
[distribution.optional-dependencies]
[[distribution.optional-dependencies.test]]
name = "iniconfig"
version = "2.0.0"
source = "registry+https://pypi.org/simple"
```
This requires a bit more work upfront, because we now need to merge
multiple packages from the `PetGraph` representation when creating the
lockfile.
Closes https://github.com/astral-sh/uv/issues/3916.
## Summary
Once we use a _range_ rather than a precise version, it won't actually
make sense to return a version here. It's no longer required, so I'm
removing it.
## Summary
Fixes a race condition in `OnceMap::wait_blocking` where the inserted
value could potentially be missed, leading to a deadlock. Fairly certain
this will resolve https://github.com/astral-sh/uv/issues/3724.
Otherwise the `uv lock` command wasn't respecting the index URL option.
This is a follow-up to #3984, and I believe should now allow #3970 to be
merged.
Seems like a recent Pull removed this, couldn't directly find out which.
I'm adding it back as we rely on this API, and I do not see another way
of accessing this, or am I mistaken?
Thanks!
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
See #3834 .
This PR adds a new namespace, `override-dependencies`, to
pyproject.toml/uv.toml.
This namespace assumes that the dependencies you want to override are
written in the form of `requirements.txt`.
a example of pyproject.toml
```toml
[project]
name = "example"
version = "0.0.0"
dependencies = [
"flask==3.0.0"
]
[tool.uv]
override-dependencies = [
"werkzeug==2.3.0"
]
```
This will improve usability by allowing you to override dependencies
without having to specify the --override option when running `uv pip
compile/install`.
## Test Plan
added test to `crates/uv/tests/pip_compile.rs`.
---------
Co-authored-by: konstin <konstin@mailbox.org>
## Summary
Running a resolution that required forking was failing due to breaking
an invariant in PubGrub. It looks like we were adding the same
incompatibility multiple times, or something like that. The issue
appears to be that when forking, we modify the current state, then clone
it as the "next state", then push to the "forked states" -- but that
means we're cloning the _modified_ state.
This PR changes the order of operations such that we clone, then modify.
It shouldn't introduce any additional clones though.
Add a `--package` option that allows switching the current project in
the workspace. Wherever you are in a workspace, you should be able to
run with any other project as root. This is the uv equivalent of `cargo
run -p`.
I don't love the `--package` name, esp. since `-p` is already taken and
in general to many things start with p already.
Part of this change is moving the workspace discovery of
`ProjectWorkspace` to `Workspace` itself.
## Usage
In albatross-virtual-workspace:
```console
$ uv venv
$ uv run --preview --package bird-feeder python -c "import albatross"
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds
Built 2 editables in 167ms
Resolved 5 packages in 4ms
Installed 5 packages in 1ms
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds)
+ sniffio==1.3.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'albatross'
$ uv venv
$ uv run --preview --package albatross python -c "import albatross"
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/albatross
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds
Built 3 editables in 173ms
Resolved 7 packages in 6ms
Installed 7 packages in 1ms
+ albatross==0.1.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/albatross)
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-virtual-workspace/packages/seeds)
+ sniffio==1.3.1
+ tqdm==4.66.4
```
In albatross-root-workspace:
```console
$ uv venv
$ uv run --preview --package bird-feeder python -c "import albatross"
Using Python 3.12.3 interpreter at: /home/konsti/.local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
Running `/home/konsti/projects/uv/target/debug/uv run --preview --package bird-feeder python -c 'import albatross'`
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds Built 2 editables in 161ms
Resolved 5 packages in 4ms
Installed 5 packages in 1ms
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds)
+ sniffio==1.3.1
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'albatross'
$ uv venv
$ cargo run run --preview --package albatross python -c "import albatross"
Using Python 3.12.3 interpreter at: /home/konsti/.local/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.13s
Running `/home/konsti/projects/uv/target/debug/uv run --preview --package albatross python -c 'import albatross'`
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder
Built file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds
Built 3 editables in 168ms
Resolved 7 packages in 5ms
Installed 7 packages in 1ms
+ albatross==0.1.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace)
+ anyio==4.4.0
+ bird-feeder==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/bird-feeder)
+ idna==3.6
+ seeds==1.0.0 (from file:///home/konsti/projects/uv/scripts/workspaces/albatross-root-workspace/packages/seeds)
+ sniffio==1.3.1
+ tqdm==4.66.4
```
## Summary
This PR ensures that if a lockfile already contains a resolved reference
(e.g., you locked with `main` previously, and it locked to a specific
commit), and you run `uv lock`, we use the same SHA, even if it's not
the latest SHA for that tag. This avoids upgrading Git dependencies
without `--upgrade`.
Closes#3920.
## Summary
This PR removes the static resolver map:
```rust
static RESOLVED_GIT_REFS: Lazy<Mutex<FxHashMap<RepositoryReference, GitSha>>> =
Lazy::new(Mutex::default);
```
With a `GitResolver` struct that we now pass around on the
`BuildContext`. There should be no behavior changes here; it's purely an
internal refactor with an eye towards making it cleaner for us to
"pre-populate" the list of resolved SHAs.
## Summary
This will help prevent bugs like #3934 by unifying the implementations
for editables and non-editable unnamed requirements. Specifically, both
of these now go through the same parsing paths and use the same struct
representations (with the exception that the editable flag is flipped in
the first case):
```
-e ./foo/bar
./foo/bar
```
We also now support PEP 508 in editable URLs. It turns out this is just
a limitation in pip, so it's correct to support it. For example, this
now works:
```
-e black[d] @ file://${PROJECT_ROOT}/scripts/packages/black_editable
```
Closes#3941.
Closes#3942.
Move `Metadata`, `MetadataLoweringError` and `ArchiveMetadata` into
their own file `metadata.rs` in `uv-distribution`, moving it out from
`lib.rs`. No functional changes.
Fixes these two warnings on nightly:
```
warning: unexpected `cfg` condition name: `codspeed`
--> crates/bench/src/lib.rs:5:15
|
5 | #[cfg(not(codspeed))]
| ^^^^^^^^ help: found config with similar value: `feature = "codspeed"`
|
= help: expected names are: `clippy`, `debug_assertions`, `doc`, `docsrs`, `doctest`, `feature`, `miri`, `overflow_checks`, `panic`, `proc_macro`, `relocation_model`, `rustfmt`, `sanitize`, `sanitizer_cfi_generalize_pointers`, `sanitizer_cfi_normalize_integers`, `target_abi`, `target_arch`, `target_endian`, `target_env`, `target_family`, `target_feature`, `target_has_atomic`, `target_has_atomic_equal_alignment`, `target_has_atomic_load_store`, `target_os`, `target_pointer_width`, `target_thread_local`, `target_vendor`, `test`, `ub_checks`, `unix`, and `windows`
= help: consider using a Cargo feature instead
= help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:
[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(codspeed)'] }
= help: or consider adding `println!("cargo::rustc-check-cfg=cfg(codspeed)");` to the top of the `build.rs`
= note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration
= note: `#[warn(unexpected_cfgs)]` on by default
warning: unexpected `cfg` condition name: `codspeed`
--> crates/bench/src/lib.rs:8:11
|
8 | #[cfg(codspeed)]
| ^^^^^^^^ help: found config with similar value: `feature = "codspeed"`
|
= help: consider using a Cargo feature instead
= help: or consider adding in `Cargo.toml` the `check-cfg` lint config for the lint:
[lints.rust]
unexpected_cfgs = { level = "warn", check-cfg = ['cfg(codspeed)'] }
= help: or consider adding `println!("cargo::rustc-check-cfg=cfg(codspeed)");` to the top of the `build.rs`
= note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration
```
```
warning: unexpected `cfg` condition value: `unix`
--> crates/uv-extract/src/tar.rs:6:16
|
6 | #[cfg_attr(not(target_os = "unix"), allow(dead_code))]
| ^^^^^^^^^^^^^^^^^^
|
= note: expected values for `target_os` are: `aix`, `android`, `cuda`, `dragonfly`, `emscripten`, `espidf`, `freebsd`, `fuchsia`, `haiku`, `hermit`, `horizon`, `hurd`, `illumos`, `ios`, `l4re`, `linux`, `macos`, `netbsd`, `none`, `nto`, `openbsd`, `psp`, `redox`, `solaris`, `solid_asp3`, `teeos`, `tvos`, `uefi`, `unknown`, `visionos`, `vita`, `vxworks`, `wasi`, `watchos`, and `windows` and 2 more
= note: see <https://doc.rust-lang.org/nightly/rustc/check-cfg/cargo-specifics.html> for more information about checking conditional configuration
= note: requested on the command line with `-W unexpected-cfgs`
```
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
With the change, we remove the special casing of workspace dependencies
and resolve `tool.uv` for all git and directory distributions. This
gives us support for non-editable workspace dependencies and path
dependencies in other workspaces. It removes a lot of special casing
around workspaces. These changes are the groundwork for supporting
`tool.uv` with dynamic metadata.
The basis for this change is moving `Requirement` from
`distribution-types` to `pypi-types` and the lowering logic from
`uv-requirements` to `uv-distribution`. This changes should be split out
in separate PRs.
I've included an example workspace `albatross-root-workspace2` where
`bird-feeder` depends on `a` from another workspace `ab`. There's a
bunch of failing tests and regressed error messages that still need
fixing. It does fix the audited package count for the workspace tests.
## Summary
In general, it's not quite right to filter preferences by `--reinstall`
-- we still want to respect existing versions, we just don't want to
respect _installed_ versions. But now that the installed versions and
preferences are decoupled, we can remove this (`--reinstall` is enforced
on the installed versions via the `Exclusions` struct that we pass to
the resolver).
While I was here, I also cleaned up the lockfile preference code to
better match the structure for `requirements.txt`.
## Summary
I believe that this is not necessary, as the installer packages are
already considered in `CandidateSelector::get_preferred`.
Firstly, note that we never pass both non-empty installed packages _and_
non-empty preferences (the installer routines pass site packages and no
preferences; the resolver routines pass no site packages but lockfile
preferences).
However, in general, if you look at `CandidateSelector::get_preferred`,
and consider what's changing, we now skip the `if let Some(version) =
preferences.version(package_name)` case for installed packages. But we
then check installed packages within that `if`, and in the `else`. So it
seems like we'll still return them in either case?
The only behavior change is in the case that you have multiple versions
of a package installed. Previously, we'd respect one of them, because
`Preferences` takes the last winner (it's a hash map, so we just replace
the package key with the last version we see); but in installed
packages, we always ignore distributions with multiple versions, since
it's indicative of a broken environment. That's a fine change IMO. We
could change `CandidateSelector::get_preferred` to support this if we
wanted to.
## Summary
We currently rely on libgit2 for most git-related functionality.
However, libgit2 has long-standing performance issues, as well as lags
significantly behind git in terms of new features. For these reasons we
now use the git CLI by default for fetching repositories
(https://github.com/astral-sh/uv/pull/1781). This PR completely drops
libgit2 in favor of the git CLI for all git-related functionality, which
should allow us to use features such as partial clones and sparse
checkouts in the future for performance.
There is also a lot of technical debt in the current git code as it's
mostly taken from Cargo. Switching to the git CLI *vastly* simplifies
the `uv-git` codebase.
Eventually we might want to look into switching to
[`gitoxide`](https://github.com/Byron/gitoxide), but it's currently too
immature for our use case.
## Summary
This PR changes the lock-file format to use inline tables for wheels and
source distributions, which currently use separate tables that make the
file harder to follow.
```diff
[[distribution]]
name = "typing-extensions"
version = "4.10.0"
source = "registry+https://pypi.org/simple"
- [distribution.sdist]
- url = "0d26ce356c7c323176620b7b483e44/typing_extensions-4.10.0.tar.gz"
- hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb"
- size = 77558
-
- [[distribution.wheel]]
- url = "dc04a3ea60b9bc8074b5d6b7ab90bb/typing_extensions-4.10.0-py3-none-any.whl"
- hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475"
- size = 33926
+ sdist = { url = "0d26ce356c7c323176620b7b483e44/typing_extensions-4.10.0.tar.gz", hash = "sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb", size = 77558 }
+ wheel = [{ url = "dc04a3ea60b9bc8074b5d6b7ab90bb/typing_extensions-4.10.0-py3-none-any.whl", hash = "sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475", size = 33926 }]
```
The downside is that the inline-tables end up quite long and TOML
doesn't support line breaks in inline tables, yet.
Part of https://github.com/astral-sh/uv/issues/3611.
We significantly regressed performance in some cases because we were
cloning the resolver state one more time than we needed to. That doesn't
sound like a lot, but in the case where there are no forks, it implies
we were cloning the state for every `get_dependencies` called when we
shouldn't have been cloning it at all.
Avoiding the clone results in somewhat tortured code. This can probably
be refactored by moving bits out to a helper routine, but that also
seemed non-trivial. So we let this suffice for now.
This addresses the lack of marker support in prior commits.
Specifically, we add them as a new field to `AnnotatedDist`, and from
there, they get added to a `Distribution` in a `Lock`.
This commit is a pretty invasive change that implements the merging
of resolutions created by each fork of the resolver.
The main idea here is that each `SolveState` is converted into a
`Resolution` (a new type) and stored on the heap after its fork
completes. When all forks complete, they are all merged into a single
`Resolution`. This `Resolution` is then used to build a `ResolutionGraph`.
Construction of `ResolutionGraph` mostly stays the same (despite the
gnarly diff due to an indent change) with one exception: the code to
extract dependency edges out of PubGrub's state has been moved to
`SolveState::into_resolution`. The idea here is that once a fork
completes, we extract what we need from the PubGrub state and then
throw it away. We store these edges in our own intermediate type which
is then converted into petgraph edges in the `ResolutionGraph`
constructor.
One interesting change we make here is that our edge
data is now a `Version` instead of a `Range<Version>`. I don't think
`Range<Version>` was actually being used anywhere, so this seems okay?
In any case, I think `Version` here is correct because a resolution
corresponds to specific dependencies of each package. Moreover, I didn't
see an easy way to make things work with `Range<Version>`. Notably,
since we no longer have the guarantee that there is only one version of
each package, we need to use `(PackageName, Version)` instead of just
`PackageName` for inverted lookups in `ResolutionGraph::from_state`.
Finally, the main resolver loop itself is changed a bit to track all
forked resolutions and then merge them at the end.
Note that we don't really have any dealings with markers in this commit.
We'll get to that in a subsequent commit.
This changes the constructor to just take an `InMemoryIndex`
directly instead of the constituent parts. No real reason other
than it seems a little simpler.
There are still some TODOs/FIXMEs here, but this makes represents a
chunk of the resolver refactoring to enable forking. We don't do any
merging of resolutions yet, so crucially, this code is broken when no
marker environment is provided. But when a marker environment is
provided, this should behave the same as a non-forking resolver. In
particular, `get_dependencies_forking` is just `get_dependencies`
whenever there's a marker environment.
This makes it so we can pass any function to determine whether an extra
is always true or not.
For example, `markers.simplify_extras_with(|_| true)` will remove all
extras in any marker expression. This wasn't possible to express
(without knowing all of the marker names) using the old API, but becomes
trivial to express with a predicate function.
While this could be done by callers since the representation
of `MarkerTree` is public, they are just annoying enough to do
that I think it makes sense to provide them on `MarkerTree`
itself.
These could also be improved in the future to do even more
flattening of conjunctions/disjunctions (or perhaps even
more robust simplification). But for now, some basic flattening
is good enough.
These routines will be used to combine marker expressions when
merging forked resolutions.
## Summary
Ensures that we avoid upgrading packages unless `--upgrade` or similar
is passed.
For now, the resolver only respects these for registry distributions.
Closes https://github.com/astral-sh/uv/issues/3918.
## Summary
This PR adds extras to the lockfile, and enables users to selectively
sync extras in `uv sync` and `uv run`. The end result here was fairly
simple, though it required a few refactors to get here. The basic idea
is that `DistributionId` now includes `extra: Option<ExtraName>`, so we
effectively treat extras as separate packages. Generating the lockfile,
and generating the resolution from the lockfile, fall out of this
naturally with no special-casing or additional changes.
The main downside here is that it bloats the lockfile significantly.
Specifically:
- We include _all_ distribution URLs and hashes for _every_ extra
variant.
- We include all dependencies for the extra variant, even though that
are dependencies of the base package.
We could normalize this representation by changing each distribution
have an `optional-dependencies` hash map that keys on extras, but we
actually don't have the information we need to create that right now
(specifically, we can't differentiate between dependencies that
_require_ the extra and dependencies on the base package).
Closes#3700.
## Summary
This PR just ensures that when running `uv lock` (or `uv run`), we lock
with all extras. When we later install, we'll also _install_ with all
extras, but that will be changed in a future PR.
## Summary
Today, we represent each package as a single node in the graph, and
combine all the extras. This is helpful for the `requirements.txt`-style
resolution, in which we want to show each a single line for each package
with the extras combined into a single array.
This PR modifies the representation to instead use a separate node for
each (package, extra) pair. We then reduce into the previous format when
printing in the `requirements.txt`-style format, so there shouldn't be
any user-facing changes here.
## Summary
This PR addresses an issue where `tool.uv` settings are not read if
`tool.uv.sources` or `tool.uv.workspaces` are present in the TOML file.
## Test Plan
Tested locally.
## Summary
Modifies `uv run` to write and read from the lockfile, rather than
resolving the project requirements as-is on each invocation.
Closes https://github.com/astral-sh/uv/issues/3891.
## Summary
Resolves https://github.com/astral-sh/uv/issues/3896. Adding progress
bars to the `MultiProgress` after configuring them seems to not
synchronize the required state fully.
## Test Plan
The repeated output is gone when testing locally.
## Summary
There are a few behavior changes in here:
- We now enforce `--require-hashes` for editables, like pip. So if you
use `--require-hashes` with an editable requirement, we'll reject it. I
could change this if it seems off.
- We now treat source tree requirements, editable or not (e.g., both `-e
./black` and `./black`) as if `--refresh` is always enabled. This
doesn't mean that we _always_ rebuild them; but if you pass
`--reinstall`, then yes, we always rebuild them. I think this is an
improvement and is close to how editables work today.
Closes#3844.
Closes#2695.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
It removes the unused result when creating the Cache with the
`from_path` constructor. I don't believe it does any io operations any
more at least.
Add workspace support when using `-r <path>/pyproject.toml` or `-e
<path>` in the pip interface. It is limited to all-editable
static-metadata workspaces, and tests only include a single main
workspace, ignoring path dependencies in another workspace. This can be
considered the MVP for workspace support: You can create a workspace,
you can install from it, but some options and conveniences are still
missing. I'll file follow-up tickets (support in lockfiles, support path
deps in other workspace, #3625)
There is also support in `uv run`, but we need
https://github.com/astral-sh/uv/issues/3700 first to properly support
using different current projects in the bluejay interface, currently the
resolution and therefore the lockfile depends on the current project.
I'd do this change first (it's big enough already), then #3700, and then
add workspace support properly to bluejay.
Fixes#3404
This bumps the versions of pep580 and pep440 to coincide with the
crates.io versions. While not strictly the same, the new types in uv us
an `Inner` struct. Practically I've found I'm still able to use the
patched versions, as can seen from the open PR here:
https://github.com/prefix-dev/pixi/pull/1436.
Would be great if this bump can be done so we can keep combining the
types :)
Follow-up to https://github.com/astral-sh/uv/pull/3797 to clean up the
test isolation in `uv-interpreter`.
I still want to expose a CLI at some point like `uv python <...>` for
discovery and test from there, hopefully this will make that transition
simpler.
Extends #3726
Moves toolchain storage out of `UV_BOOTSTRAP_DIR` (`./bin`) into the
proper user data directory as defined by #3726.
Replaces `UV_BOOTSTRAP_DIR` with `UV_TOOLCHAIN_DIR` for customization.
Installed toolchains will be discovered without opt-in, but the idea is
still that these are not yet user-facing.
## Summary
This also adds filtering for the ARM Pythons, since that needs some libc
changes; and it closes https://github.com/astral-sh/uv/issues/3854 by
way of adding an "arm" branch.
## Summary
Allows, e.g., `UV_SYSTEM_PYTHON=false uv pip install --python
.venv/bin/python`.
This was intended to work after fixing
https://github.com/astral-sh/uv/issues/3000, but I think I misdiagnosed
the scope when closing that issue, and the linked PR there only fixed
some _other_ problems around index URLs.
The only thing we really lose here is we no longer error when
`--break-system-packages` is provided without `--system`, but we can
enforce that elsewhere if we want.
Closes https://github.com/astral-sh/uv/issues/3829.
## Summary
This PR makes a variety of invalid states unrepresentable by changing
`Preference` to require a `PackageName` and `Version`, rather than
accepting a generic `Requirement`. There should be no meaningful
behavior changes.
## Summary
We actually _already_ ignore these (preferences only apply to versions,
not URLs), it just happens later on. This PR thus just avoids crashing.
The behavior is unchanged.
Closes#3822.
Closes#3784
The cache did not use an absolute path. I'm not sure this is actually a
new bug, as this code wasn't touched in #3266 but perhaps there was a
slight difference in the paths we were passing around. Note, just
canonicalizing the path as soon as we see it doesn't work because then
we jump out of the virtual environmnent into the system interpreter.
## Test plan
```
❯ uv venv
Using Python 3.12.3 interpreter at: /opt/homebrew/opt/python@3.12/bin/python3.12
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
❯ uv pip install anyio
Resolved 3 packages in 81ms
Installed 3 packages in 4ms
+ anyio==4.3.0
+ idna==3.7
+ sniffio==1.3.1
❯ mkdir uv-issue-3784 && cd uv-issue-3784
❯ uv venv
Using Python 3.12.3 interpreter at: /opt/homebrew/opt/python@3.12/bin/python3.12
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
❯ gcm
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
❯ cargo run -q -- pip list -v -p .venv
DEBUG Checking for Python interpreter in directory `.venv`
TRACE Cached interpreter info for Python 3.12.3, skipping probing: .venv/bin/python3
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
Package Version
------- -------
anyio 4.3.0
idna 3.7
sniffio 1.3.1
❯ cd uv-issue-3784
❯ cargo run -q -- pip list -v -p .venv
DEBUG Checking for Python interpreter in directory `.venv`
TRACE Cached interpreter info for Python 3.12.3, skipping probing: .venv/bin/python3
DEBUG Using Python 3.12.3 environment at /Users/zb/workspace/uv/.venv/bin/python3
Package Version
------- -------
anyio 4.3.0
idna 3.7
sniffio 1.3.1
❯ cd ..
❯ gco zb/fix-relative-venv
Switched to branch 'zb/fix-relative-venv'
❯ cargo run -q -- pip list -v -p .venv
DEBUG Checking for Python interpreter in directory `.venv`
TRACE Cached interpreter info for Python 3.12.3, skipping probing: .venv/bin/python3
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
Package Version
------- -------
anyio 4.3.0
idna 3.7
sniffio 1.3.1
❯ cd uv-issue-3784
❯ cargo run -q -- pip list -v -p .venv
DEBUG Checking for Python interpreter in directory `.venv`
TRACE Cached interpreter info for Python 3.12.3, skipping probing: .venv/bin/python3
DEBUG Using Python 3.12.3 environment at .venv/bin/python3
```
## Summary
Related to https://github.com/astral-sh/uv/issues/3818. We should
_always_ include the package name if we know it's not a file path, even
if it starts with an environment variable.
## Summary
I haven't tested on Windows yet, but the idea here is that we should use
a portable representation when printing paths.
I decided to limit the scope here to paths that we write to output
files.
Closes https://github.com/astral-sh/uv/issues/3800.
## Summary
It turns out that in the
[spec](https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-name-convention),
if a wheel filename includes a build tag, then we need to use it to
break ties. This PR implements that behavior. (Previously, we dropped
the build tag entirely.)
Closes#3779.
## Test Plan
Run: `cargo run pip install -i https://pypi.anaconda.org/intel/simple
mkl_fft==1.3.8 --python-platform linux --python-version 3.10`. This now
resolves without error. Previously, we selected build tag 63 of
`mkl_fft==1.3.8`, which led to an incompatibility with NumPy. Now, we
select build tag 70.
When parsing requirements from any source, directly parse the url parts
(and reject unsupported urls) instead of parsing url parts at a later
stage. This removes a bunch of error branches and concludes the work
parsing url parts once and passing them around everywhere.
Many usages of the assembled `VerbatimUrl` remain, but these can be
removed incrementally.
Please review commit-by-commit.
## Summary
This seems to be one of the most consistent benchmark cases we have in
terms of standard deviation:
```
➜ hyperfine "target/profiling/main pip compile scripts/requirements/airflow.in" --runs 200
Benchmark 1: target/profiling/main pip compile scripts/requirements/airflow.in
Time (mean ± σ): 292.6 ms ± 6.6 ms [User: 414.1 ms, System: 194.2 ms]
Range (min … max): 282.7 ms … 320.1 ms 200 runs
```
For smaller benchmarks, scispacy and dtlssocket seem to be a bit more
consistent than our current jupyter benchmark, but it hasn't given us
any problems so I'll leave it for now.
e.g. in `uv pip install anyio -v` this message is just noise
```
DEBUG Requirement satisfied: anyio
DEBUG Requirement satisfied: idna>=2.8
DEBUG Requirement satisfied: sniffio>=1.1
DEBUG All editables satisfied:
```
```
DEBUG Acquired lock for `.venv`
```
instead of
```
DEBUG Trying to lock if free: .venv/.lock
```
At trace level, this includes the pre-lock message as well
```
TRACE Checking lock for `.venv`
DEBUG Acquired lock for `.venv`
```
We'll still display the lock file path when something goes wrong
The venv subcommand requires a system interpreter. The tests python path
discovery would previously allow a venv interpreter, failing the venv
tests that don't have system interpreter anymore.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Allows requesting additional transitive dependencies when running a
tool.
e.g. `uv tool run -v --with anyio ruff check example.py` (Why would you
want anyio with ruff? Who knows 😄)
The motivation for doing this now is that I think the first
implementation of `uv tool install` might just shim into `uv tool run`
with pinned dependencies? Regardless this is something we need in the
long run and is a trivial addition right now.
## Summary
We now show yanks as part of the resolution diagnostics, so they now
appear for `sync`, `install`, `compile`, and any other operations.
Further, they'll also appear for cached packages (but not packages that
are _already_ installed).
Closes https://github.com/astral-sh/uv/issues/3768.
Closes#3766.
We usually infer the package the tool is pulled from to be the same name
as the tool itself, but that's not always the case. This allows users to
provide a custom package.
## Summary
This PR removes most of the code in `project/mod.rs` in favor of the
routines exposed in `pip/operations.rs`.
I think we can do a lot more to add more abstraction here and reduce the
verbosity, but for now it deduplicates a _ton_ of logic. The remaining
logic is just instantiating settings etc.
e.g. this error message is not great
```
❯ uv venv --python 3.12.2
× No interpreter found for Python 3.12.2 in provided path, search path, managed toolchains, or parent interpreter
```
e.g.
```
❯ echo "anyio" | cargo run -q -- pip compile - --python 3.9 -v
DEBUG Searching for interpreter that fulfills Python @ 3.9
DEBUG Found a virtual environment at: /Users/zb/workspace/uv/.venv
DEBUG Using Python 3.9.18 interpreter at bin/cpython-3.9.18-macos-aarch64-none/install/bin/python3 for builds
```
e.g. instead of
```
❯ uv venv --python pypy@3.10
× No interpreter found for pypy 3.10 in search path
```
we say
```
❯ uv venv --python pypy@3.10
× No interpreter found for PyPy 3.10 in search path
```
Closes#2222
Closes https://github.com/astral-sh/uv/issues/2058
Replaces https://github.com/astral-sh/uv/pull/2338
See also https://github.com/astral-sh/uv/issues/2649
We use an environment variable (`UV_INTERNAL__PARENT_INTERPRETER`) to
track the invoking interpreter when `python -m uv` is used. The parent
interpreter is preferred over all other sources (though it will be
skipped if it does not meet a `--python` request or if `--system` is
used and it belongs to a virtual environment). We warn if `--system` is
not provided and this interpreter would mutate system packages, but
allow it.
Previously, we enforced `SystemPython` outside of the interpreter
discovery exclusively with source selection. Now, we perform additional
filtering of interpreters depending on if they are a virtual
environment. This should not change any existing behavior, but will make
it much easier to have consistent behavior in ambiguous cases like
https://github.com/astral-sh/uv/pull/3736#discussion_r1610072262 where a
source could provide either a system interpreter or virtual environment
interpreter.
## Summary
This PR takes the functions used in `pip install`, moves them into a
common module, and then replaces all the `pip sync` logic with calls
into those functions. The net effect is that `pip install` and `pip
sync` share far more code and demonstrate much more consistent behavior.
Closes https://github.com/astral-sh/uv/issues/3555.
## Summary
This PR adds editables using a new source type (`editable+...`), and
then extracts the editables from the lockfile in `uv sync`.
Closes https://github.com/astral-sh/uv/issues/3695.
Otherwise `uv venv --python 3.12` can prefer `.venv/bin/python` over the
system Python (which is always used if you don't provide a `--python`
flag). I would find this confusing as a user.
Updates our executable name searches to support implementation names
i.e. `cpython` and `pypy` and adds support for PyPy.
We might want to _not_ support searching for `cpython` because that's
non-standard?
Adds `--offline` support to `uv tool run` and `uv run` because I needed
it on the airplane today.
I think we should move `--offline` to the global settings like
`--native-tls`.
## Summary
Closes https://github.com/astral-sh/uv/issues/3715.
## Test Plan
```
❯ echo "/../test" | cargo run pip compile -
error: Couldn't parse requirement in `-` at position 0
Caused by: path could not be normalized: /../test
/../test
^^^^^^^^
❯ echo "-e /../test" | cargo run pip compile -
error: Invalid URL in `-`: `/../test`
Caused by: path could not be normalized: /../test
Caused by: cannot normalize a relative path beyond the base directory
```
This is mostly a shorter version of `uv run` that infers a requirement
name from the command. The main goal here is to do the smallest amount
of work necessary to get #3560 started.
Closes#3613
e.g.
```shell
$ uv tool run -- ruff check
warning: `uv tool run` is experimental and may change without warning.
Resolved 1 package in 34ms
Installed 1 package in 2ms
+ ruff==0.4.4
error: Failed to parse example.py:1:5: Expected an expression
example.py:1:5: E999 SyntaxError: Expected an expression
Found 1 error.
```
Updates our Python interpreter discovery to conform to the rules
described in #2386, please see that issue for a full description of the
behavior. Briefly, we now will search for interpreters that satisfy a
requested version without stopping at the first Python executable.
Additionally, if retrieving information about an interpreter fails we
will continue to search for a working interpreter. We also add the
plumbing necessary to request Python implementations other than CPython,
though we do not add support for other implementations at this time.
A major internal goal of this work is to prepare for user-facing managed
toolchains i.e. fetching a requested version during `uv run`. These APIs
are not introduced, but there is some managed toolchain handling as
required for our test suite.
Some noteworthy implementation changes:
- The `uv_interpreter::find_python` module has been removed in favor of
a `uv_interpreter::discovery` module.
- There are new types to help structure interpreter requests and track
sources
- Executable discovery is implemented as a big lazy iterator and is a
central authority for source precedence
- `uv_interpreter::Error` variants were split into scoped types in each
module
- There's much more unit test coverage, but not for Windows yet
Remaining work:
- [x] Write new test cases
- [x] Determine correct behavior around executables in the current
directory
- _Future_: Combine `PythonVersion` and `VersionRequest`
- _Future_: Consider splitting `ManagedToolchain` into local and remote
variants
- _Future_: Add Windows unit test coverage
- _Future_: Explore behavior around implementation precedence (i.e.
CPython over PyPy)
Refactors split into:
- #3329
- #3330
- #3331
- #3332Closes#2386
Instead of saying
> we can conclude that you require==0a0.dev0 and
pandas-stubs==2.0.3.230814 are incompatible.
we'll say
> we can conclude that your requirements and pandas-stubs==2.0.3.230814
are incompatible.
Closes#3710
I'm not sure how to get unit test coverage for this, might look into
that. Ideally we'd skip this branch entirely?
I don't really understand why this only happens on windows clippy and
not on linux too, but as usual, boxing the error variant fixes it.
Fixup for #3585
Add minimal support for workspace discovery, only used for determining
paths in the bluejay commands.
We can now discover the workspace structure, namely that the
`pyproject.toml` of a package belongs to a workspace `pyproject.toml`
with members and exclusion. The globbing logic is inspired by cargo. We
don't resolve `workspace = true` metadata declarations yet.
Pubgrub stores incompatibilities as (package name, version range)
tuples, meaning it needs to clone the package name for each
incompatibility, and each non-borrowed operation on incompatibilities.
https://github.com/astral-sh/uv/pull/3673 made me realize that
`PubGrubPackage` has gotten large (expensive to copy), so like `Version`
and other structs, i've added an `Arc` wrapper around it.
It's a pity clippy forbids `.deref()`, it's less opaque than `&**` and
has IDE support (clicking on `.deref()` jumps to the right impl).
## Benchmarks
It looks like this matters most for complex resolutions which, i assume
because they carry larger `PubGrubPackageInner::Package` and
`PubGrubPackageInner::Extra` types.
```bash
hyperfine --warmup 5 "./uv-main pip compile -q ./scripts/requirements/jupyter.in" "./uv-branch pip compile -q ./scripts/requirements/jupyter.in"
hyperfine --warmup 5 "./uv-main pip compile -q ./scripts/requirements/airflow.in" "./uv-branch pip compile -q ./scripts/requirements/airflow.in"
hyperfine --warmup 5 "./uv-main pip compile -q ./scripts/requirements/boto3.in" "./uv-branch pip compile -q ./scripts/requirements/boto3.in"
```
```
Benchmark 1: ./uv-main pip compile -q ./scripts/requirements/jupyter.in
Time (mean ± σ): 18.2 ms ± 1.6 ms [User: 14.4 ms, System: 26.0 ms]
Range (min … max): 15.8 ms … 22.5 ms 181 runs
Benchmark 2: ./uv-branch pip compile -q ./scripts/requirements/jupyter.in
Time (mean ± σ): 17.8 ms ± 1.4 ms [User: 14.4 ms, System: 25.3 ms]
Range (min … max): 15.4 ms … 23.1 ms 159 runs
Summary
./uv-branch pip compile -q ./scripts/requirements/jupyter.in ran
1.02 ± 0.12 times faster than ./uv-main pip compile -q ./scripts/requirements/jupyter.in
```
```
Benchmark 1: ./uv-main pip compile -q ./scripts/requirements/airflow.in
Time (mean ± σ): 153.7 ms ± 3.5 ms [User: 165.2 ms, System: 157.6 ms]
Range (min … max): 150.4 ms … 163.0 ms 19 runs
Benchmark 2: ./uv-branch pip compile -q ./scripts/requirements/airflow.in
Time (mean ± σ): 123.9 ms ± 4.6 ms [User: 152.4 ms, System: 133.8 ms]
Range (min … max): 118.4 ms … 138.1 ms 24 runs
Summary
./uv-branch pip compile -q ./scripts/requirements/airflow.in ran
1.24 ± 0.05 times faster than ./uv-main pip compile -q ./scripts/requirements/airflow.in
```
```
Benchmark 1: ./uv-main pip compile -q ./scripts/requirements/boto3.in
Time (mean ± σ): 327.0 ms ± 3.8 ms [User: 344.5 ms, System: 71.6 ms]
Range (min … max): 322.7 ms … 334.6 ms 10 runs
Benchmark 2: ./uv-branch pip compile -q ./scripts/requirements/boto3.in
Time (mean ± σ): 311.2 ms ± 3.1 ms [User: 339.3 ms, System: 63.1 ms]
Range (min … max): 307.8 ms … 317.0 ms 10 runs
Summary
./uv-branch pip compile -q ./scripts/requirements/boto3.in ran
1.05 ± 0.02 times faster than ./uv-main pip compile -q ./scripts/requirements/boto3.in
```
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
This is bare-bones support for editables in `uv sync` as basis for
workspace support, notably without lockfile integration. It leverages
the existing `ResolvedEditables` infrastructure.
## Summary
This PR falls back to writing an unnamed requirement if it appears to be
a relative URL. pip is way more flexible when providing an unnamed
requirement than when providing a PEP 508 requirement. For example,
_only_ this works:
```
black @ file:///Users/crmarsh/workspace/uv/scripts/packages/black_editable
```
Any other form will fail.
Meanwhile, _all_ of these work:
```
file:///Users/crmarsh/workspace/uv/scripts/packages/black_editable
scripts/packages/black_editable
./scripts/packages/black_editable
file:./scripts/packages/black_editable
file:scripts/packages/black_editable
```
Closes https://github.com/astral-sh/uv/issues/3180.
Since we're adding a `Option<MarkerTree>` to `PubGrubPackage`, and since
we just make `PubGrubPackage` implement `Ord`, it follows that we want
`MarkerTree` to also implement `Ord`.
This makes use of the newly added `Ord` impl on `PubGrubPackage` to make
the output of `format_terms` independent of hashmap iteration order.
This was already collecting the terms into an intermediate `Vec`, so
sorting probably isn't going to add any significant overhead here.
(Plus, this is only running when formatting an error message after a
solution could not be found, so an extra sort doesn't seem like a big
deal here.)
Note that some tests are updated in this commit as a result of this
change. As far as I can tell, the semantic meaning of the output remains
the same. But the order of the listed packages does not.
Specific thing motivating this change is, in a subsequent, I added
`Option<MarkerTree>` to `PubGrubPackage::Package`, and this caused
similar changes in test output. So I backtracked and isolated this
change from the addition of `Option<MarkerTree>`.
It turns out that we use PubGrubPackage as the key in hashmaps in a fair
few places. And when we iterate over hashmaps, the order is unspecified.
This can in turn result in changes in output as a result of changes in
the PubGrubPackage definition, purely as a function of its changing
hash. This is confusing as there should be no semantic difference.
Thus, this is a precursor to introducing some more determinism to places
I found in the error reporting whose output depending on hashmap
iteration order.
It looks like the last vestiges of `Derivative` were removed in commit
7eaed07f6c, but the then rendered
superfluous `derive(Derivative)` wasn't removed.
This is split out from workspaces support, which needs editables in the
bluejay commands. It consists mainly of refactorings:
* Move the `editable` module one level up.
* Introduce a `BuiltEditableMetadata` type for `(LocalEditable,
Metadata23, Requirements)`.
* Add editables to `InstalledPackagesProvider` so we can use
`EmptyInstalledPackages` for them.
## Summary
If you have (e.g.) `extra-index-url` in your configuration file _and_
provide `--extra-index-url` on the command-line, we now merge the
options rather than ignoring those in the configuration file. As such,
merging the CLI and the persistent configuration is now semantically
identical to how we merge (project persistent configuration) with (user
persistent configuration).
Closes https://github.com/astral-sh/uv/issues/3541.
## Summary
The main motivation here is that the `.filename()` method that we
implement on `Url` will do URL decoding for the last segment, which we
were missing here.
The errors are a bit awkward, because in
`crates/uv-resolver/src/lock.rs`, we wrap in `failed to extract filename
from URL: {url}`, so in theory we want the underlying errors to _omit_
the URL? But sometimes they use `#[error(transparent)]`?
## Summary
Uncertain about this, but we don't actually need the full
`SourceDistFilename`, only the name and version -- and we often have
that information already (as in the lockfile routines). So by flattening
the fields onto `RegistrySourceDist`, we can avoid re-parsing for
information we already have.
Prompted by
https://github.com/astral-sh/uv/pull/3657#discussion_r1606041239
There's still some level of discomfort here, as the `tool` module needs
needs to import the `project` module to manage an environment. We should
probably move most of the basic operations in the `project` module root
into some sort of shared module for behind the scenes operations?
Regardless, this change should simplify that future move.
## Summary
Restore API-compatibility with pre-1.1.0 versions of the `zip` crate,
and pin the dependency to the 0.6 series, due to concerns discussed in
https://github.com/astral-sh/uv/issues/3642.
## Test Plan
```
cargo run -p uv-dev -- fetch-python
cargo test
```
## Summary
This PR adds lossless deserialization for `GitSourceDist` distributions
in the lockfile. Specifically, we now properly preserve the requested
revision, the subdirectory, and the precise Git commit SHA.
## Test Plan
`cargo test`
## Summary
This PR introduces parallelism to the resolver. Specifically, we can
perform PubGrub resolution on a separate thread, while keeping all I/O
on the tokio thread. We already have the infrastructure set up for this
with the channel and `OnceMap`, which makes this change relatively
simple. The big change needed to make this possible is removing the
lifetimes on some of the types that need to be shared between the
resolver and pubgrub thread.
A related PR, https://github.com/astral-sh/uv/pull/1163, found that
adding `yield_now` calls improved throughput. With optimal scheduling we
might be able to get away with everything on the same thread here.
However, in the ideal pipeline with perfect prefetching, the resolution
and prefetching can run completely in parallel without depending on one
another. While this would be very difficult to achieve, even with our
current prefetching pattern we see a consistent performance improvement
from parallelism.
This does also require reverting a few of the changes from
https://github.com/astral-sh/uv/pull/3413, but not all of them. The
sharing is isolated to the resolver task.
## Test Plan
On smaller tasks performance is mixed with ~2% improvements/regressions
on both sides. However, on medium-large resolution tasks we see the
benefits of parallelism, with improvements anywhere from 10-50%.
```
./scripts/requirements/jupyter.in
Benchmark 1: ./target/profiling/baseline (resolve-warm)
Time (mean ± σ): 29.2 ms ± 1.8 ms [User: 20.3 ms, System: 29.8 ms]
Range (min … max): 26.4 ms … 36.0 ms 91 runs
Benchmark 2: ./target/profiling/parallel (resolve-warm)
Time (mean ± σ): 25.5 ms ± 1.0 ms [User: 19.5 ms, System: 25.5 ms]
Range (min … max): 23.6 ms … 27.8 ms 99 runs
Summary
./target/profiling/parallel (resolve-warm) ran
1.15 ± 0.08 times faster than ./target/profiling/baseline (resolve-warm)
```
```
./scripts/requirements/boto3.in
Benchmark 1: ./target/profiling/baseline (resolve-warm)
Time (mean ± σ): 487.1 ms ± 6.2 ms [User: 464.6 ms, System: 61.6 ms]
Range (min … max): 480.0 ms … 497.3 ms 10 runs
Benchmark 2: ./target/profiling/parallel (resolve-warm)
Time (mean ± σ): 430.8 ms ± 9.3 ms [User: 529.0 ms, System: 77.2 ms]
Range (min … max): 417.1 ms … 442.5 ms 10 runs
Summary
./target/profiling/parallel (resolve-warm) ran
1.13 ± 0.03 times faster than ./target/profiling/baseline (resolve-warm)
```
```
./scripts/requirements/airflow.in
Benchmark 1: ./target/profiling/baseline (resolve-warm)
Time (mean ± σ): 478.1 ms ± 18.8 ms [User: 482.6 ms, System: 205.0 ms]
Range (min … max): 454.7 ms … 508.9 ms 10 runs
Benchmark 2: ./target/profiling/parallel (resolve-warm)
Time (mean ± σ): 308.7 ms ± 11.7 ms [User: 428.5 ms, System: 209.5 ms]
Range (min … max): 287.8 ms … 323.1 ms 10 runs
Summary
./target/profiling/parallel (resolve-warm) ran
1.55 ± 0.08 times faster than ./target/profiling/baseline (resolve-warm)
```
## Summary
Fixes a small discrepancy between the pip compile outputs for
`annotation-style=split` and `annotation-style=line` commands.
### Problem
Consider the following `pyproject.toml` file.
```sh
$ cat pyproject.toml
[project]
name = "uv_test"
dynamic = ["version"]
dependencies = ["click"]
```
Running uv pip compile with annotation-style=split on uv 0.1.44 version
yields the following:
```sh
❯ uv pip compile pyproject.toml --annotation-style=split
Resolved 1 package in 2ms
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml --annotation-style=split
click==8.1.7
# via uv-test (pyproject.toml)
```
However, running uv pip compile with annotation-style=line doesn't
include source info for root level dependencies.
```sh
❯ uv pip compile pyproject.toml --annotation-style=line
Resolved 1 package in 1ms
# This file was autogenerated by uv via the following command:
# uv pip compile pyproject.toml --annotation-style=line
click==8.1.7
```
With this PR:
```sh
❯ ../target/debug/uv pip compile --annotation-style=line pyproject.toml
Resolved 1 package in 6ms
# This file was autogenerated by uv via the following command:
# uv pip compile --annotation-style=line pyproject.toml
click==8.1.7 # via uv-test (pyproject.toml)
```
This also now matches `pip-tools` output:
```sh
❯ pip-compile --annotation-style=line pyproject.toml
#
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --annotation-style=line pyproject.toml
#
click==8.1.7 # via uv_test (pyproject.toml)
```
## Test Plan
`cargo test`
## Summary
If a user includes markers after an editable, we now ignore them (rather
than including them in the parsed URL). This matches pip's behavior. In
the future, we could further improve by respecting them, but that
_would_ be a deviation from pip.
For example, given:
```
-e ./scripts/packages/black_editable ; python_version >= "3.9" and python_ver
```
We now split at the first whitespace (just before the `;`), parse
everything before, and throw out everything after.
This logic also extends to extras. So given:
```
-e ./scripts/packages/black_editable[dev, colorama]
```
We'll now parse this as the URL
`./scripts/packages/black_editable[dev,`, and throw out ` colorama]`.
Instead, you need to do:
```
-e ./scripts/packages/black_editable[dev,colorama]
```
(I.e., remove the space.)
This _also_ matches pip's behavior. I could "fix" this but I'm unsure if
I should -- it means requirements files will be parseable by uv that
won't work with pip. Open to input. My gut reaction is that we _should_
properly support `-e ./scripts/packages/black_editable[dev, colorama]`
even if pip would reject it, but `requirements.txt` is
implementation-defined so it'd be a "deviation".
Closes https://github.com/astral-sh/uv/issues/3604.
Following from #3595, we'd like wheels to make their way into the lock
file even if the current environment selects an sdist. With #3595, this
didn't happen:
$ cargo run -p uv -- pip compile -p3.10 <(echo psycopg2)
--unstable-uv-lock-file
$ cat uv.lock
version = 1
[[distribution]]
name = "psycopg2"
version = "2.9.9"
source = "registry+https://pypi.org/simple"
[distribution.sdist]
url =
"dc6acaf46d76fce95daac5e0f0301b/psycopg2-2.9.9.tar.gz"
hash =
"sha256:d1454bde93fb1e224166811694d600e746430c006fbb031ea06ecc2ea41bf156"
The above example uses `psycopg2`, which has an sdist and wheels only on
Windows. Since I ran the above on Linux, an sdist was selected. But no
wheels appeared in the lock file.
With this PR, wheels are now correctly plumbed through:
$ cargo run -p uv -- pip compile -p3.10 <(echo psycopg2)
--unstable-uv-lock-file
$ cat uv.lock
version = 1
[[distribution]]
name = "psycopg2"
version = "2.9.9"
source = "registry+https://pypi.org/simple"
[distribution.sdist]
url =
"dc6acaf46d76fce95daac5e0f0301b/psycopg2-2.9.9.tar.gz"
hash =
"sha256:d1454bde93fb1e224166811694d600e746430c006fbb031ea06ecc2ea41bf156"
[[distribution.wheel]]
url =
"2767d96391f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl"
hash =
"sha256:38a8dcc6856f569068b47de286b472b7c473ac7977243593a288ebce0dc89516"
[[distribution.wheel]]
url =
"6572dec6831f85491a5e4dda606a98/psycopg2-2.9.9-cp310-cp310-win_amd64.whl"
hash =
"sha256:426f9f29bde126913a20a96ff8ce7d73fd8a216cfb323b1f04da402d452853c3"
[[distribution.wheel]]
url =
"1fc5b9d33c858a602868a592cdc1b0/psycopg2-2.9.9-cp311-cp311-win32.whl"
hash =
"sha256:ade01303ccf7ae12c356a5e10911c9e1c51136003a9a1d92f7aa9d010fb98372"
[[distribution.wheel]]
url =
"5133dd3183e671b278ce248810b7f7/psycopg2-2.9.9-cp311-cp311-win_amd64.whl"
hash =
"sha256:121081ea2e76729acfb0673ff33755e8703d45e926e416cb59bae3a86c6a4981"
[[distribution.wheel]]
url =
"f74ffe6b6fe119ccb8a6546c3fb893/psycopg2-2.9.9-cp312-cp312-win32.whl"
hash =
"sha256:d735786acc7dd25815e89cc4ad529a43af779db2e25aa7c626de864127e5a024"
[[distribution.wheel]]
url =
"c4a26e1918ab7ee854fb5247f16c40/psycopg2-2.9.9-cp312-cp312-win_amd64.whl"
hash =
"sha256:a7653d00b732afb6fc597e29c50ad28087dcb4fbfb28e86092277a559ae4e693"
[[distribution.wheel]]
url =
"ffeb9ac356ce0d6c4f2f34e396dbc0/psycopg2-2.9.9-cp37-cp37m-win32.whl"
hash =
"sha256:5e0d98cade4f0e0304d7d6f25bbfbc5bd186e07b38eac65379309c4ca3193efa"
[[distribution.wheel]]
url =
"0a39176d36fd7105774e57996f63cd/psycopg2-2.9.9-cp37-cp37m-win_amd64.whl"
hash =
"sha256:7e2dacf8b009a1c1e843b5213a87f7c544b2b042476ed7755be813eaf4e8347a"
[[distribution.wheel]]
url =
"86b90d30c4420cc3c0f6da2b8f3a9a/psycopg2-2.9.9-cp38-cp38-win32.whl"
hash =
"sha256:ff432630e510709564c01dafdbe996cb552e0b9f3f065eb89bdce5bd31fabf4c"
[[distribution.wheel]]
url =
"c439b378ef79997a935f10374f3c0d/psycopg2-2.9.9-cp38-cp38-win_amd64.whl"
hash =
"sha256:bac58c024c9922c23550af2a581998624d6e02350f4ae9c5f0bc642c633a2d5e"
[[distribution.wheel]]
url =
"5080c0e61ad5f08b9503e508aac116/psycopg2-2.9.9-cp39-cp39-win32.whl"
hash =
"sha256:c92811b2d4c9b6ea0285942b2e7cac98a59e166d59c588fe5cfe1eda58e72d59"
[[distribution.wheel]]
url =
"ec73fe66d4d65f5bbe54efb191d9e6/psycopg2-2.9.9-cp39-cp39-win_amd64.whl"
hash =
"sha256:de80739447af31525feddeb8effd640782cf5998e1a4e9192ebdf829717e3913"
Ref #3351
Our current flow of data from "simple registry package" to "final
resolved distribution" goes through a number of types:
* `SimpleMetadata` is the API response from a registry that includes all
published versions for a package. Each version has an assortment of
metadata
associated with it.
* `VersionFiles` is the aforementioned metadata. It is split in two: a
group of files for source distributions and a group of files for wheels.
* `PrioritizedDist` collects a subset of the files from `VersionFiles`
to form a selection of the "best" sdist and the "best" wheel for the
current environment.
* `CompatibleDist` is created from a borrowed `PrioritizedDist` that,
perhaps among other things, encapsulates the decision of whether to pick
an sdist or a wheel. (This decision depends both on compatibility and
the action being performed. e.g., When doing installation, a
`CompatibleDist` will sometimes select an sdist over a wheel.)
* `ResolvedDistRef` is like a `ResolvedDist`, but borrows a `Dist`.
* `ResolvedDist` is the almost-final-form of a distribution in a
resolution and is created from a `ResolvedDistRef`.
* `AnnotatedResolvedDist` is a new data type that is the actual final
form of a distribution that a universal lock file cares about. It
bundles a `ResolvedDist` with some metadata needed to generate a lock
file.
One of the requirements of a universal lock file is that we include all
wheels (and maybe all source distributions? but at least one if it's
present) associated with a distribution. But the above flow of data (in
the step from `VersionFiles` to `PrioritizedDist`) drops all wheels
except for the best one.
To remedy this, in this PR, we rejigger `PrioritizedDist`,
`CompatibleDist` and `ResolvedDistRef` so that all wheel data is
preserved. And when a `ResolvedDistRef` is finally turned into a
`ResolvedDist`, we copy all of the wheel data. And finally, we adjust
the `Lock` constructor to read this new data and include it in the lock
file. To make this work, we also modify `RegistryBuiltDist` so that it
can contain one or more wheels instead of just one.
One shortcoming here (called out in the code as a FIXME) is that if a
source distribution is selected as the "best" thing to use (perhaps
there are no compatible wheels), then the wheels won't end up in the
lock file. I plan to fix this in a follow-up PR.
We also aren't totally consistent on source distribution naming.
Sometimes we use `sdist`. Sometimes `source`. Sometimes `source_dist`.
I think it'd be nice to just use `sdist` everywhere, but I do prefer
the type names to be `SourceDist`. And sometimes you want function
names to match the type names (i.e., `from_source_dist`), which in turn
leads to an appearance of inconsistency. I'm open to ideas.
Closes#3351
## Summary
In `ResolutionGraph::from_state`, we have mechanisms to grab the hashes
and metadata for all distributions -- but we then throw that information
away. This PR preserves it on a new `AnnotatedDist` (yikes, open to
suggestions) that wraps `ResolvedDist` and includes (1) the hashes
(computed or from the registry) and (2) the `Metadata23`, which lets us
extract the version.
Closes https://github.com/astral-sh/uv/issues/3356.
Closes https://github.com/astral-sh/uv/issues/3357.
## Summary
Splits this into two loops that each handle independent cases, to make
the code a little easier to reason about. No behavioral or logic changes
-- just splitting the `match` across two loops.
## Summary
Closes
https://github.com/astral-sh/uv/issues/3578#issuecomment-2110675382.
## Test Plan
Verified that in the OpenSUSE test, we create both, and they're
symlinks:
```text
INFO: Creating virtual environment with `venv`...
INFO: Installing into `venv` virtual environment...
DEBUG Found a virtualenv named .venv at: /tmp/tmp4nape29h/.venv
DEBUG Cached interpreter info for Python 3.10.14, skipping probing: .venv/bin/python
DEBUG Using Python 3.10.14 environment at .venv/bin/python
DEBUG Trying to lock if free: .venv/.lock
purelib: "/tmp/tmp4nape29h/.venv/lib/python3.10/site-packages"
platlib: "/tmp/tmp4nape29h/.venv/lib64/python3.10/site-packages"
is_same_file(purelib, platlib): Ok(true)
```
## Summary
Increment the removed file counts in filters
in install_registry_source_dist_cached test, to make it work again on
Gentoo. The tested counts were updated
in 9a92a3ad37, but the filters were not.
That said, the respective count increased in Gentoo as well, so adjust
both input and output strings. I'm updating Windows as a guesswork,
though I suspect that filter may not be necessary anymore, given that CI
was passing.
## Test Plan
`cargo test` on Gentoo :-).
## Summary
Fixes a typo in a comment
## Test Plan
I assume there's no need to test comment changes, other than having a
human check they make sense. That's what this PR is for 😉
## Summary
Uses the editable handling from `pip sync`, and improves the
abstractions such that we can pass those resolved editables into the
resolver.
---------
Co-authored-by: konstin <konstin@mailbox.org>
## Summary
It's confusing that we use `constraints` here because constraints mean
something else for us (e.g., `--constraint constraints.txt`). These are
really the dependencies of a given `PubGrubPackage` -- the type is even
called `PubGrubDependencies`.
Windows does not support cloning whole directories so clone each file
instead.
closes#3547
## Test Plan
Ran ` uv pip install setuptools --link-mode=clone` manually
## Summary
I don't love this, but it turns out that setuptools is not robust to
parallel builds: https://github.com/pypa/setuptools/issues/3119. As a
result, if you run uv from multiple processes, and they each attempt to
build the same source distribution, you can hit failures.
This PR applies an advisory lock to the source distribution directory.
We apply it unconditionally, even if we ultimately find something in the
cache and _don't_ do a build, which helps ensure that we only build the
distribution once (and wait for that build to complete) rather than
kicking off builds from each thread.
Closes https://github.com/astral-sh/uv/issues/3512.
## Test Plan
Ran:
```sh
#!/bin/bash
make_venv(){
target/debug/uv venv $1
source $1/bin/activate
target/debug/uv pip install opentracing --no-deps --verbose
}
for i in {1..8}
do
make_venv ./$1/$i &
done
```
## Summary
I think this is overall good change because it explicitly encodes (in
the type system) something that was previously implicit. I'm not a huge
fan of the names here, open to input.
It covers some of https://github.com/astral-sh/uv/issues/3506 but I
don't think it _closes_ it.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Just fix typos.
While `alpha-numeric` is not really a misspelling:
- it is missing from mainstream curated dictionaries, all of them
suggest `alphanumeric`;
- it is less used than `alphanumeric` (more than ⨉10 less) according to
the Google [Ngram
Viewer](https://books.google.com/ngrams/graph?content=alpha-numeric%2Calphanumeric&year_start=1900&year_end=2019&corpus=en-2019);
- it is [missing from
SCOWL](http://app.aspell.net/lookup?dict=en_US-large;words=alpha-numeric).
## Test Plan
CI jobs.
## Summary
runpy.run_path was added in python 2.7 and 3.2 - and every python that
is not EOL supports it.
It is arguably nicer to read and the path is only given once in the
command.
At least right now, runpy - unlike exec with S102 - is not flagged by
any bandit-derived ruff check.
(I guess because it loads from a file instead of a simple string...)
Because of the import, it is also not a one-liner anymore. (But that
could be fixed with an __import__('runpy').run_path...)
## Test Plan
import runpy
runpy.run_path('/path/to/venv/bin/activate_this.py')
## Summary
If you run the script included in the linked issue, then `uv cache
clean`, we hit permissions errors on certain directories created by
`setuptools`. The permissions on those directories look like:
```
❯ sudo ls -l /Users/crmarsh/Library/Caches/uv/built-wheels-v3/pypi/opentracing/2.4.0/M-fYsaHAaQQvedmPMUl9D/opentracing-2.4.0.tar.gz/build/bdist.macosx-14.2-arm64/wheel/opentracing
Password:
total 0
drwxr-xr-x 3 crmarsh staff 96 May 11 12:51 harness
```
This PR adds logic to make those directories readable by the current
user.
Closes https://github.com/astral-sh/uv/issues/3515.
## Summary
pip passes these as positional arguments, and at least one build backend
relies on that. My personal opinion is that it's a spec violation, and
the build backend should be updated, but I'd prefer to favor
compatibility over strictness here.
Closes https://github.com/astral-sh/uv/issues/3509.
## Test Plan
`cargo run pip install cryptacular==1.6.2`
## Summary
This PR consolidates the concurrency limits used throughout `uv` and
exposes two limits, `UV_CONCURRENT_DOWNLOADS` and
`UV_CONCURRENT_BUILDS`, as environment variables.
Currently, `uv` has a number of concurrent streams that it buffers using
relatively arbitrary limits for backpressure. However, many of these
limits are conflated. We run a relatively small number of tasks overall
and should start most things as soon as possible. What we really want to
limit are three separate operations:
- File I/O. This is managed by tokio's blocking pool and we should not
really have to worry about it.
- Network I/O.
- Python build processes.
Because the current limits span a broad range of tasks, it's possible
that a limit meant for network I/O is occupied by tasks performing
builds, reading from the file system, or even waiting on a `OnceMap`. We
also don't limit build processes that end up being required to perform a
download. While this may not pose a performance problem because our
limits are relatively high, it does mean that the limits do not do what
we want, making it tricky to expose them to users
(https://github.com/astral-sh/uv/issues/1205,
https://github.com/astral-sh/uv/issues/3311).
After this change, the limits on network I/O and build processes are
centralized and managed by semaphores. All other tasks are unbuffered
(note that these tasks are still bounded, so backpressure should not be
a problem).
This only makes hashes optional for wheels/sdists that come from
registires or direct URLs. For wheels/sdists that come from other
sources, a hash should not be present.
For path dependencies, a hash should not be present because the state of
the path dependency is not intended to be tracked in the lock file. This
is consistent with how other tools deal with path dependencies, and if
it were otherwise, the hash would I believe need to be updated for every
change to the path dependency.
For git dependencies (source dists only), a hash should not be present
because the lock will contain the specific commit revision hash. This is
functionally equivalent to a hash, and so a hash is redundant.
As part of this change, we validate the presence or absence of a hash
based on the dependency source. We also add our first regression tests.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
likely necessary to resolve https://github.com/astral-sh/uv/issues/2500
made this a separate PR in an attempt to make the changes as small as
possible; let me know if it's preferred to keep them as a single PR.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
- edited the test in `interpreter.rs`
- tested manually via `println!`
```
$ cargo run --quiet pip show test
["/Users/chankang/Library/Caches/uv/.tmpKzNEPN", "/Users/chankang/.pyenv/versions/3.12.2/lib/python312.zip", "/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12", "/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12/lib-dynload", "/Users/chankang/repos/uv/.venv/lib/python3.12/site-packages"]
warning: Package(s) not found for: test
chankang@chans-Air ~/repos/uv - (syspath)
$ git diff
diff --git a/crates/uv-interpreter/src/environment.rs b/crates/uv-interpreter/src/environment.rs
index 33b785ce..8ebf0864 100644
--- a/crates/uv-interpreter/src/environment.rs
+++ b/crates/uv-interpreter/src/environment.rs
@@ -106,6 +106,7 @@ impl PythonEnvironment {
/// Some distributions also create symbolic links from `purelib` to `platlib`; in such cases, we
/// still deduplicate the entries, returning a single path.
pub fn site_packages(&self) -> impl Iterator<Item = &Path> {
+ println!("{:?}", self.interpreter.sys_path());
if let Some(target) = self.interpreter.target() {
Either::Left(std::iter::once(target.root()))
} else {
chankang@chans-Air ~/repos/uv - (syspath)
$ python -c "import sys; print(sys.path)"
['', '/Users/chankang/.pyenv/versions/3.12.2/lib/python312.zip', '/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12', '/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12/lib-dynload', '/Users/chankang/.pyenv/versions/3.12.2/lib/python3.12/site-packages']
chankang@chans-Air ~/repos/uv - (syspath)
```
<!-- How was it tested? -->
This still keeps the resolver state on the stack, but it organizes it
into a more structured representation. This is a precursor to
implementing resolver forking, where we will ultimately put this state
on the heap. The idea is that this will let us maintain multiple
independent resolver states that will all produce their own resolution
(and potentially other forked states).
Closes#3354
## Summary
I've started to refer to this as the "project" API in various places, it
seems less duplicative than the "workspace" API which is a little
different.
Now that the type is fully encapsulated, we can pretty easily
migrate to using an Arc inside of a MarkerEnvironment.
It looks like the pyo3 macros can't deal with an Arc, so we
write out the getter methods by hand.
We now use the getters and setters everywhere.
There were some places where we wanted to build a `MarkerEnvironment`
out of whole cloth, usually in tests. To facilitate those use cases, we
add a `MarkerEnvironmentBuilder` that provides a convenient constructor.
It's basically like a `MarkerEnvironment::new`, but with named
parameters. That's useful here because there are so many fields (and
they many have the same type).
This test was failing on master. I guess we don't test
this crate with the pyo3 feature enabled? I think this
regression was due to a recent change in the error reporting
of the pep440 crate.
## Summary
Ensures that we track the origins for requirements regardless of whether
they come from `pyproject.toml` or `setup.py` or `setup.cfg`.
Closes#3480.
This commit touches a lot of code, but the conceptual change here is
pretty simple: make it so we can run the resolver without providing a
`MarkerEnvironment`. This also indicates that the resolver should run in
universal mode. That is, the effect of a missing marker environment is
that all marker expressions that reference the marker environment are
evaluated to `true`. That is, they are ignored. (The only markers we
evaluate in that context are extras, which are the only markers that
aren't dependent on the environment.)
One interesting change here is that a `Resolver` no longer needs an
`Interpreter`. Previously, it had only been using it to construct a
`PythonRequirement`, by filling in the installed version from the
`Interpreter` state. But we now construct a `PythonRequirement`
explicitly since its `target` Python version should no longer be tied to
the `MarkerEnvironment`. (Currently, the marker environment is mutated
such that its `python_full_version` is derived from multiple sources,
including the CLI, which I found a touch confusing.)
The change in behavior can now be observed through the
`--unstable-uv-lock-file` flag. First, without it:
```
$ cat requirements.in
anyio>=4.3.0 ; sys_platform == "linux"
anyio<4 ; sys_platform == "darwin"
$ cargo run -qp uv -- pip compile -p3.10 requirements.in
anyio==4.3.0
exceptiongroup==1.2.1
# via anyio
idna==3.7
# via anyio
sniffio==1.3.1
# via anyio
typing-extensions==4.11.0
# via anyio
```
And now with it:
```
$ cargo run -qp uv -- pip compile -p3.10 requirements.in --unstable-uv-lock-file
x No solution found when resolving dependencies:
`-> Because you require anyio>=4.3.0 and anyio<4, we can conclude that the requirements are unsatisfiable.
```
This is expected at this point because the marker expressions are being
explicitly ignored, *and* there is no forking done yet to account for
the conflict.
We provide a new API on a `Requirement` that specifically
ignores the marker environment and only evaluates a requirement's
marker expression with respect to extras. Any marker expressions
that reference the marker environment automatically evaluate to
true.
Instead of duplicating the evaluation code, we just make a marker
environment optional on the lower level APIs. In theory, we could
just writer a separate evaluation routine that ignores everything
except extras, but the evaluator has a fair bit of other stuff in it
(such as emitting warnings) that would be good to keep DRY IMO.
This doc test seems to fail due to the recent change making
`Requirement` generic on its URL type. While the type parameter
was given a default of `VerbatimUrl`, this default doesn't always
apply. For example, the `FromStr` impl on `Requirement` is still
generic on any URL type, and so callers must indicate the type
of the URL to return. (An alternative would be to define the
`FromStr` impl for just the default URL type.)
## Summary
If a requirement is omitted due to a marker expression, we shouldn't
include it as the "source" of a package in the output.
For example, if your constraints include `pathspec ; python_version <
'3.12'`, and you're on Python 3.12, we should _not_ include the
constraint file as a "source" in the output annotations.
## Summary
Unfortunately, the `-I` flag was added in Python 3.4. So if we query a
Python version prior to 3.4 (e.g., Python 2.7), we can't run our script
at all, and lose the ability to match against our structured error.
This PR adds an additional check against the stderr output for these
cases.
Closes https://github.com/astral-sh/uv/issues/3474.
## Test Plan
Installed Python 2.7, and verified that it was skipped (and that we
instead found my `python3`).
## Summary
Fixes https://github.com/astral-sh/uv/issues/1343. This is kinda a first
draft at the moment, but does at least mostly work locally (barring some
bits of the test suite that seem to not work for me in general).
## Test Plan
Mostly running the existing tests and checking the revised output is
sane
## Outstanding issues
Most of these come down to "AFAIK, the existing tools don't support
these patterns, but `uv` does" and so I'm not sure there's an existing
good answer here! Most of the answers so far are "whatever was easiest
to build"
- [x] ~~Is "-r pyproject.toml" correct? Should it show something else or
get skipped entirely~~ No it wasn't. Fixed in
3044fa8b86
- [ ] If the requirements file is stdin, that just gets skipped. Should
it be recorded?
- [ ] Overrides get shown as "--override<override.txt>". Correct?
- [x] ~~Some of the tests (e.g.
`dependency_excludes_non_contiguous_range_of_compatible_versions`) make
assumptions about the order of package versions being outputted, which
this PR breaks. I'm not sure if the text is fairly arbitrary and can be
replaced or whether the behaviour needs fixing?~~ - fixed by removing
the custom pubgrub PartialEq/Hash
- [ ] Are all the `TrackedFromStr` et al changes needed, or is there an
easier way? I don't think so, I think it's necessary to track these sort
of things fairly comprehensively to make this feature work, and this
sort of invasive change feels necessary, but happy to be proved wrong
there :)
- [x] ~~If you have a requirement coming in from two or more different
requirements files only one turns up. I've got a closed-source example
for this (can go into more detail if needed), mostly consisting of a
complicated set of common deps creating a larger set. It's a rarer case,
but worth considering.~~ 042432b200
- [ ] Doesn't add annotations for `setup.py` yet
- This is pretty hard, as the correct location to insert the path is
`crates/pypi-types/src/metadata.rs`'s `parse_pkg_info`, which as it's
based off a source distribution has entirely thrown away such matters as
"where did this package requirement get built from". Could add "`built
package name`" as a dep, but that's a little odd.
## Summary
This PR takes a different approach to `--with` for `uv run`. Now,
instead of merging the requirements and re-resolving, we have two
phases: (1) sync the workspace requirements to the workspace
environment; then (2) sync the ephemeral `--with` requirements to an
ephemeral environment. The two environments are then layered by setting
the `PATH` and `PYTHONPATH` variables appropriately.
I think this approach simplifies a few things:
1. Once we have a lockfile, the semantics are much clearer, and we can
actually reuse it for the workspace. If we had to add arbitrary
dependencies via `--with`, then it's not really clear how the lockfile
would/should behave.
2. Caching becomes simpler, because we can just cache the ephemeral
environment based on the requirements.
The current version of this PR loses a few behaviors though that I need
to restore:
- `--python` support -- but I'm not yet sure how this is supposed to
behave within projects? It's also left unclear in `uv sync` and `uv
lock`.
- The "reuse the workspace environment if it already satisfies the
ephemeral requirements" behavior.
Closes#3411.
## Summary
This is universal environment variable used to determine the mac OS
deployment target. We now respect it in `--python-platform` -- so we
default to 12.0, but users can override it as needed.
## Summary
We already _don't_ discover a `pyproject.toml` in `~/.config/uv` -- it
must be `uv.toml`. This PR makes the same change for `--config-file` --
it _has_ to be a `uv.toml`.
I think this is reasonable and more consistent, though I'm not sure. A
`pyproject.toml` "means" something -- it defines a project itself, in
which case we should be using project configuration. But creating a
`pyproject.toml` outside the project and passing it via `--config-file`
seems like an anti-pattern.
## Summary
This PR follows Cargo's strategy for merging configuration, albeit in a
more limited way (we don't support as many configuration locations).
Specifically, we merge the user configuration with the workspace
configuration if both are present. The workspace configuration has
priority, such that we take values from the workspace configuration and
ignore those in the user configuration if both are specified for a given
setting -- with the exception of arrays and maps, which are
concatenated.
For now, if a user provides a configuration file with `--config-file`,
we _don't_ merge in the user settings.
See:
https://doc.rust-lang.org/cargo/reference/config.html#hierarchical-structure.
Closes#3420.
## Summary
This is annoying both locally in CI. If anyone wants to fuss with the
filters to fix it, that's fine too, but IMO it's better to disable than
leave it enabled on macOS for now.
When using `tool.uv.sources`, we warn that requirements have a bound,
i.e. at least a lower version constraint.
When using a library, the symbols you import were introduced in
different versions, creating an implicit lower bound. This warning makes
this explicit. This is crucial to prevent backtracking resolvers from
selecting an ancient versions that is not compatible (or worse, doesn't
build), and a performance optimization on top.
This feature is gated to `tool.uv.sources` (as it should have been to
begin with for #3263/#3443) to not unnecessarily break legacy workflows.
It is also helpful specifically when using a `tool.uv.sources` section
that contains constraints that are not published to pypi, e.g. for
workspace dependencies. We can adjust those later to e.g. not constrain
workspace dependencies with `publish = false`, but i think it's the
right setting to start with.
## Summary
These aren't intended for production use; instead, I'm just trying to
frame out the overall data flows and code-sharing for these commands. We
now have `uv sync` (sync the environment to match the lockfile, without
refreshing or resolving) and `uv lock` (generate the lockfile). Both
_require_ a virtual environment to exist (something we should change).
`uv sync`, `uv run`, and `uv lock` all share code for the underlying
subroutines (resolution and installation), so the commands themselves
are relatively small (~100 lines) and mostly consist of reading
arguments and such.
`uv lock` and `uv sync` don't actually really work yet, because we have
no way to include the project itself in the lockfile (that's a TODO in
the lockfile implementation).
Closes https://github.com/astral-sh/uv/issues/3432.
We would previously show the parsed version when erroring due to
trailing content after a valid version, which can look different than
the input. E.g. when encountering `0.1-bulbasaur`, we would display:
```
after parsing '0.1b0', found 'ulbasaur', which is not part of a valid version
```
With storing the input string instead of the input version, we now show:
```
after parsing '0.1-b', found 'ulbasaur', which is not part of a valid version
```
It turns out setuptools often uses Metadata-Version 2.1 in their
PKG-INFO:
4e766834d7/setuptools/dist.py (L64)
`Metadata23` requires Metadata-Version of at least 2.2.
This means that uv doesn't actually recognise legacy editable
installations from the most common way you'd actually get legacy
editable installations (works great for most legacy editables I make at
work though!)
Anyway, over here we only need the version and don't care about anything
else. Rather than make a `Metadata21`, I just add a version field to
`Metadata10`. The one slightly tricky thing is that only
Metadata-Version 1.2 and greater guarantee that the [version number is
PEP 440 compatible](https://peps.python.org/pep-0345/#version), so I
store the version in `Metadata10` as a `String` and only parse to
`Version` at time of use.
Also did you know that back in 2004, paramiko had a pokemon based
versioning system?
Pubgrub got a new feature where all unavailability is a custom, instead
of the reasonless `UnavailableDependencies` and our custom `String` type
previously (https://github.com/pubgrub-rs/pubgrub/pull/208). This PR
introduces a `UnavailableReason` that tracks either an entire version
being unusable, or a specific version. The error messages now also track
this difference properly.
The pubgrub commit is our main rebased onto the merged
https://github.com/pubgrub-rs/pubgrub/pull/208, i'll push
`konsti/main-rebase-generic-reason` to `main` after checking for rebase
problems.
## Summary
It's not clear to me that this should exist at all, but it's causing
errors in projects that don't use `tool.uv.sources`, so we should
definitely remove it for now.
## Summary
We already have a global `--isolated`, which means "ignore any on-disk
configuration". I think we should reuse this for the "ignore the
workspace" setting in `uv run`, rather than `--no-workspace`.
I've also merged the existing `--isolated` and `--no-workspace`
behaviors in `uv run` into a single flag. We may not need separate flags
for this, since the current intent seems to be "ignore the workspace
environment"? Though we could always re-add later.
Closes https://github.com/astral-sh/uv/issues/3421.
Resolves [#3419](https://github.com/astral-sh/uv/issues/3419)
## Summary
Add compatargs to pip install command and hint the user to create a venv
for --user arg.
## Test Plan
Tested it locally.
```bash
cargo run pip install --user flask
Compiling uv v0.1.39 (/home/ahmedilyas/uv/crates/uv)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 8.96s
Running `target/debug/uv pip install --user flask`
error: pip install's `--user` is unsupported (use a virtual environment instead).
```
This change allows switching out the url type for requirements. The
original idea was to allow different types for different requirement
origins, so that core metadata reads can ban non-pep 508 requirements
while we only allow them for requirements.txt. This didn't work out
because we expect `&Requirement`s from all sources to match.
I also tried to split `pep508_rs` into a PEP 508 compliant crate and
into our extensions, but they are to tightly coupled.
I think this change is an improvement still as it reduces the hardcoded
dependence on `VerbatimUrl`.
We now correctly emit relative paths in `uv pip compile` with
`tool.uv.sources` path inputs.
`tool.uv.sources` is mainly intended to be used with the uv lock file
over requirements.txt, but it's good to have basic `uv pip` support
working.
Fixes#3366
## Summary
All of the resolver code is run on the main thread, so a lot of the
`Send` bounds and uses of `DashMap` and `Arc` are unnecessary. We could
also switch to using single-threaded versions of `Mutex` and `Notify` in
some places, but there isn't really a crate that provides those I would
be comfortable with using.
The `Arc` in `OnceMap` can't easily be removed because of the uv-auth
code which uses the
[reqwest-middleware](https://docs.rs/reqwest-middleware/latest/reqwest_middleware/trait.Middleware.html)
crate, that seems to adds unnecessary `Send` bounds because of
`async-trait`. We could duplicate the code and create a `OnceMapLocal`
variant, but I don't feel that's worth it.
## Summary
Users often find themselves dropped into environments that contain
`.egg-info` packages. While we won't support installing these, it's not
hard to support identifying them (e.g., in `pip freeze`) and
_uninstalling_ them.
Closes https://github.com/astral-sh/uv/issues/2841.
Closes#2928.
Closes#3341.
## Test Plan
Ran `cargo run pip freeze --python
/opt/homebrew/Caskroom/miniforge/base/envs/TEST/bin/python`, with an
environment that includes `pip` as an `.egg-info`
(`/opt/homebrew/Caskroom/miniforge/base/envs/TEST/lib/python3.12/site-packages/pip-24.0-py3.12.egg-info`):
```
cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1696001825047/work
pip==24.0
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1711811537435/work
setuptools==69.5.1
wheel==0.43.0
```
Then ran `cargo run pip uninstall`, verified that `pip` was uninstalled,
and no longer listed in `pip freeze`.
Fixes#3371
It seems like uv doesn't proactively enforce 3.8+ and in most cases just
issues a warning. This PR keeps that property, only adding the new check
when it is known to fail. I checked the imports in this file and the
other ones seem fine.
## Summary
Refreshes some of the activation scripts, and fixes some bugs in
`activate_this.py` that were likely the rest of some erroneous
copy-pasting.
Closes https://github.com/astral-sh/uv/issues/3346.
## Test Plan
```
❯ python
Python 3.12.0 (main, Feb 28 2024, 09:44:16) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import httpx
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'httpx'
>>> activator = '.venv/bin/activate_this.py'
>>> with open(activator) as f:
... exec(f.read(), {'__file__': activator})
...
>>> import httpx
```
Only allow using `tool.uv.sources` with preview mode, the design isn't
finalized yet.
Not sure what to label this, do we want a preview section and label for
the release notes?
## Summary
We need to partition the editable and non-editable requirements. As-is,
`editable = true` requirements were still being installed as
non-editable.
## Summary
We were writing the build dependencies into the `--target` directory,
which both made builds fail and led to them leaking into the user's
directory.
Closes https://github.com/astral-sh/uv/issues/3349.
## Introduction
PEP 621 is limited. Specifically, it lacks
* Relative path support
* Editable support
* Workspace support
* Index pinning or any sort of index specification
The semantics of urls are a custom extension, PEP 440 does not specify
how to use git references or subdirectories, instead pip has a custom
stringly format. We need to somehow support these while still stying
compatible with PEP 621.
## `tool.uv.source`
Drawing inspiration from cargo, poetry and rye, we add `tool.uv.sources`
or (for now stub only) `tool.uv.workspace`:
```toml
[project]
name = "albatross"
version = "0.1.0"
dependencies = [
"tqdm >=4.66.2,<5",
"torch ==2.2.2",
"transformers[torch] >=4.39.3,<5",
"importlib_metadata >=7.1.0,<8; python_version < '3.10'",
"mollymawk ==0.1.0"
]
[tool.uv.sources]
tqdm = { git = "https://github.com/tqdm/tqdm", rev = "cc372d09dcd5a5eabdc6ed4cf365bdb0be004d44" }
importlib_metadata = { url = "https://github.com/python/importlib_metadata/archive/refs/tags/v7.1.0.zip" }
torch = { index = "torch-cu118" }
mollymawk = { workspace = true }
[tool.uv.workspace]
include = [
"packages/mollymawk"
]
[tool.uv.indexes]
torch-cu118 = "https://download.pytorch.org/whl/cu118"
```
See `docs/specifying_dependencies.md` for a detailed explanation of the
format. The basic gist is that `project.dependencies` is what ends up on
pypi, while `tool.uv.sources` are your non-published additions. We do
support the full range or PEP 508, we just hide it in the docs and
prefer the exploded table for easier readability and less confusing with
actual url parts.
This format should eventually be able to subsume requirements.txt's
current use cases. While we will continue to support the legacy `uv pip`
interface, this is a piece of the uv's own top level interface. Together
with `uv run` and a lockfile format, you should only need to write
`pyproject.toml` and do `uv run`, which generates/uses/updates your
lockfile behind the scenes, no more pip-style requirements involved. It
also lays the groundwork for implementing index pinning.
## Changes
This PR implements:
* Reading and lowering `project.dependencies`,
`project.optional-dependencies` and `tool.uv.sources` into a new
requirements format, including:
* Git dependencies
* Url dependencies
* Path dependencies, including relative and editable
* `pip install` integration
* Error reporting for invalid `tool.uv.sources`
* Json schema integration (works in pycharm, see below)
* Draft user-level docs (see `docs/specifying_dependencies.md`)
It does not implement:
* No `pip compile` testing, deprioritizing towards our own lockfile
* Index pinning (stub definitions only)
* Development dependencies
* Workspace support (stub definitions only)
* Overrides in pyproject.toml
* Patching/replacing dependencies
One technically breaking change is that we now require user provided
pyproject.toml to be valid wrt to PEP 621. Included files still fall
back to PEP 517. That means `pip install -r requirements.txt` requires
it to be valid while `pip install -r requirements.txt` with `-e .` as
content falls back to PEP 517 as before.
## Implementation
The `pep508` requirement is replaced by a new `UvRequirement` (name up
for bikeshedding, not particularly attached to the uv prefix). The still
existing `pep508_rs::Requirement` type is a url format copied from pip's
requirements.txt and doesn't appropriately capture all features we
want/need to support. The bulk of the diff is changing the requirement
type throughout the codebase.
We still use `VerbatimUrl` in many places, where we would expect a
parsed/decomposed url type, specifically:
* Reading core metadata except top level pyproject.toml files, we fail a
step later instead if the url isn't supported.
* Allowed `Urls`.
* `PackageId` with a custom `CanonicalUrl` comparison, instead of
canonicalizing urls eagerly.
* `PubGrubPackage`: We eventually convert the `VerbatimUrl` back to a
`Dist` (`Dist::from_url`), instead of remembering the url.
* Source dist types: We use verbatim url even though we know and require
that these are supported urls we can and have parsed.
I tried to make improve the situation be replacing `VerbatimUrl`, but
these changes would require massive invasive changes (see e.g.
https://github.com/astral-sh/uv/pull/3253). A main problem is the ref
`VersionOrUrl` and applying overrides, which assume the same
requirement/url type everywhere. In its current form, this PR increases
this tech debt.
I've tried to split off PRs and commits, but the main refactoring is
still a single monolith commit to make it compile and the tests pass.
## Demo
Adding
d1ae3b85d5/pyproject.json
as json schema (v7) to pycharm for `pyproject.toml`, you can try the IDE
support already:
[dove.webm](https://github.com/astral-sh/uv/assets/6826232/c293c272-c80b-459d-8c95-8c46a8d198a1)
In *some* places in our crates, `serde` (and `rkyv`) are optional
dependencies. I believe this was done out of reasons of "good sense,"
that is, it follows a Rust ecosystem pattern where serde integration
tends to be an opt-in crate feature. (And similarly for `rkyv`.)
However, ultimately, `uv` itself requires `serde` and `rkyv` to
function. Since our crates are strictly internal, there are limited
consumers for our crates without `serde` (and `rkyv`) enabled. I think
one possibility is that optional `serde` (and `rkyv`) integration means
that someone can do this:
cargo test -p pep440_rs
And this will run tests _without_ `serde` or `rkyv` enabled. That in
turn could lead to faster iteration time by reducing compile times. But,
I'm not sure this is worth supporting. The iterative compilation times
of
individual crates are probably fast enough in debug mode, even with
`serde` and `rkyv` enabled. Namely, `serde` and `rkyv` themselves
shouldn't need to be re-compiled in most cases. On `main`:
```
from-scratch: `cargo test -p pep440_rs --lib` 0.685
incremental: `cargo test -p pep440_rs --lib` 0.278s
from-scratch: `cargo test -p pep440_rs --features serde,rkyv --lib` 3.948s
incremental: `cargo test -p pep440_rs --features serde,rkyv --lib` 0.321s
```
So while a from-scratch build does take significantly longer, an
incremental build is about the same.
The benefit of doing this change is two-fold:
1. It brings out crates into alignment with "reality." In particular,
some crates were _implicitly_ relying on `serde` being enabled
without explicitly declaring it. This technically means that our
`Cargo.toml`s were wrong in some cases, but it is hard to observe it
because of feature unification in a Cargo workspace.
2. We no longer need to deal with the cognitive burden of writing
`#[cfg_attr(feature = "serde", ...)]` everywhere.
This PR principally adds a routine for converting a `Lock` to a
`Resolution`, where a `Resolution` is a map of package names pinned to
a specific version.
I'm not sure that a `Resolution` is ultimately what we want here (we
might need more stuff), but this was the quickest route I could find to
plug a `Lock` into our existing `uv pip install` infrastructure.
This commit also does a little refactoring of the `Lock` types. The
main thing is to permit extra state on some of the types (like a
`by_id` map on `Lock` for quick lookups of distributions) that aren't
included in the serialization format of a `Lock`. We achieve this
by defining separate `Wire` types that are automatically converted
to-and-from via `serde`.
Note that like with the lock file format types themselves, we leave a
few `todo!()` expressions around. The main idea is to get something
minimally working without spending too much effort here. (A fair bit
of refactoring will be required to generate a lock file, and it's
not clear how much this code will wind up needing to change anyway.)
In particular, we only handle the case of installing wheels from a
registry.
A demonstration of the full flow:
```
$ requirements.in
anyio
$ cargo run -p uv -- pip compile -p3.10 requirements.in --unstable-uv-lock-file
$ uv venv
$ cargo run -p uv -- pip install --unstable-uv-lock-file anyio -r requirements.in
Installed 5 packages in 7ms
+ anyio==4.3.0
+ exceptiongroup==1.2.1
+ idna==3.7
+ sniffio==1.3.1
+ typing-extensions==4.11.0
```
In order to install from a lock file, we start from the root and do a
breadth first traversal over its dependencies. We aren't yet filtering
on marker expressions (since they aren't in the lock file yet), but we
should be able to add that in the future. In so doing, the traversal
should select only the subset of distributions relevant for the current
platform.
Split out of #3266
The "selector" concept doesn't seem well enough defined as-is. For
example, `PythonVersion` belongs there but isn't present. Going for
smaller modules instead.
Split out of https://github.com/astral-sh/uv/pull/3266
If `UV_BOOTSTRAP_DIR` and `CARGO_MANIFEST_DIR` are both unset, we
currently panic. This isn't good once we start to use managed toolchains
in production. We'll need to change this more later once the toolchain
directory is more user-facing.
Moves all of `uv-toolchain` into `uv-interpreter`. We may split these
out in the future, but the refactoring I want to do for interpreter
discovery is easier if I don't have to deal with entanglement. Includes
some restructuring of `uv-interpreter`.
Part of #2386
Another split out from https://github.com/astral-sh/uv/pull/3263. This
abstracts the copy&pasted check whether an installed distribution
satisfies a requirement used by both plan.rs and site_packages.rs into a
shared module. It's less useful here than with the new requirement but
helps with reducing https://github.com/astral-sh/uv/pull/3263 diff size.
Previously, a noop `uv pip install` would only show "Audited {}
package(s)" but no details, not even with `-vv`. Now it debug logs which
requirements were met and it also debug logs which requirement was
missing to trigger the full routine, allowing it investigate caching
behaviour.
First `uv pip install -v jupyter`:
```
DEBUG At least one requirement is not satisfied: jupyter
```
Second `uv pip install -v jupyter`:
```
DEBUG Found a virtualenv named .venv at: /home/konsti/projects/uv-main/.venv
DEBUG Cached interpreter info for Python 3.12.1, skipping probing: .venv/bin/python
DEBUG Using Python 3.12.1 environment at .venv/bin/python
DEBUG Trying to lock if free: .venv/.lock
DEBUG Requirement satisfied: anyio
DEBUG Requirement satisfied: anyio>=3.1.0
DEBUG Requirement satisfied: argon2-cffi-bindings
DEBUG Requirement satisfied: argon2-cffi>=21.1
DEBUG Requirement satisfied: arrow>=0.15.0
DEBUG Requirement satisfied: asttokens>=2.1.0
DEBUG Requirement satisfied: async-lru>=1.0.0
DEBUG Requirement satisfied: attrs>=22.2.0
DEBUG Requirement satisfied: babel>=2.10
...
DEBUG Requirement satisfied: webencodings
DEBUG Requirement satisfied: webencodings>=0.4
DEBUG Requirement satisfied: websocket-client>=1.7
DEBUG Requirement satisfied: widgetsnbextension~=4.0.10
DEBUG All editables satisfied:
Audited 1 package in 12ms
```
This will clash with the `tool.uv.sources` PR, i'll rebase it on top.
## Summary
Hi! Added `UV_NO_BUILD_ISOLATION` as a boolean environment variable for
the `--no-build-isolation` command-line option.
Closes https://github.com/astral-sh/uv/issues/3309
## Test Plan
Added new test `respect_no_build_isolation_env_var` to check that the
behaviour is the same as if the ``--no-build-isolation``
command-line-option is set.
This is meant to be a base on which to build. There are some parts
which are implicitly incomplete and others which are explicitly
incomplete. The latter are indicated by TODO comments.
Here is a non-exhaustive list of incomplete things. In many cases, these
are incomplete simply because the data isn't present in a
`ResolutionGraph`. Future work will need to refactor our resolver so
that this data is correctly passed down.
* Not all wheels are included. Only the "selected" wheel for the current
distribution is included.
* Marker expressions are always absent.
* We don't emit hashes for certainly kinds of distributions (direct
URLs, git, and path).
* We don't capture git information from a dependency specification.
Right now, we just always emit "default branch."
There are perhaps also other changes we might want to make to the format
of a more cosmetic nature. Right now, all arrays are encoded using
whatever the `toml` crate decides to do. But we might want to exert more
control over this. For example, by using inline tables or squashing more
things into strings (like I did for `Source` and `Hash`). I think the
main trade-off here is that table arrays are somewhat difficult to read
(especially without indentation), where as squashing things down into a
more condensed format potentially makes future compatible additions
harder.
I also went pretty light on the documentation here than what I would
normally do. That's primarily because I think this code is going to
go through some evolution and I didn't want to spend too much time
documenting something that is likely to change.
Finally, here's an example of the lock file format in TOML for the
`anyio` dependency. I generated it with the following command:
```
cargo run -p uv -- pip compile -p3.10 ~/astral/tmp/reqs/anyio.in --unstable-uv-lock-file
```
And that writes out a `uv.lock` file:
```toml
version = 1
[[distribution]]
name = "anyio"
version = "4.3.0"
source = "registry+https://pypi.org/simple"
[[distribution.wheel]]
url = "2f20c40b45242c0b33774da0e2e34f/anyio-4.3.0-py3-none-any.whl"
hash = "sha256:048e05d0f6caeed70d731f3db756d35dcc1f35747c8c403364a8332c630441b8"
[[distribution.dependencies]]
name = "exceptiongroup"
version = "1.2.1"
source = "registry+https://pypi.org/simple"
[[distribution.dependencies]]
name = "idna"
version = "3.7"
source = "registry+https://pypi.org/simple"
[[distribution.dependencies]]
name = "sniffio"
version = "1.3.1"
source = "registry+https://pypi.org/simple"
[[distribution.dependencies]]
name = "typing-extensions"
version = "4.11.0"
source = "registry+https://pypi.org/simple"
[[distribution]]
name = "exceptiongroup"
version = "1.2.1"
source = "registry+https://pypi.org/simple"
[[distribution.wheel]]
url = "79fe92dd414cadab75055b0ae00b33/exceptiongroup-1.2.1-py3-none-any.whl"
hash = "sha256:5258b9ed329c5bbdd31a309f53cbfb0b155341807f6ff7606a1e801a891b29ad"
[[distribution]]
name = "idna"
version = "3.7"
source = "registry+https://pypi.org/simple"
[[distribution.wheel]]
url = "741d8c8280948df2ea0eda2c8b79e8/idna-3.7-py3-none-any.whl"
hash = "sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0"
[[distribution]]
name = "sniffio"
version = "1.3.1"
source = "registry+https://pypi.org/simple"
[[distribution.wheel]]
url = "75a9c94214239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl"
hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2"
[[distribution]]
name = "typing-extensions"
version = "4.11.0"
source = "registry+https://pypi.org/simple"
[[distribution.wheel]]
url = "936e2092671b43269810cd589ceaf5/typing_extensions-4.11.0-py3-none-any.whl"
hash = "sha256:c1f94d72897edaf4ce775bb7558d5b79d8126906a14ea5ed1635921406c0387a"
```
Resolves https://github.com/astral-sh/uv/issues/3313
## Summary
Add a new env variable `UV_LINK_MODE` as alias for the cli argument
--link-mode.
Updated the README env variables section.
## Test Plan
Tested manually using `UV_LINK_MODE=hardlink cargo run -p uv pip install
flask`.
When running
```
set UV_CACHE_DIR=%LOCALAPPDATA%\uv\cache-foo && uv venv venv
```
in windows CMD, the error would be just
```
error: The system cannot find the path specified. (os error 3)
```
The problem is that the first action in the cache dir is adding the tag,
and the `cachedir` crate is using `std::fs` instead of `fs_err`. I've
copied the two functions we use from the crate and changed the import
from `std::fs` to `fs_err`.
The new error is
```
error: failed to open file `C:\Users\Konstantin\AppData\Local\uv\cache-foo \CACHEDIR.TAG`
Caused by: The system cannot find the path specified. (os error 3)
```
which correctly explains the problem.
Closes#3280
## Summary
It seems like Azure might return a 401 when you request a package that
doesn't exist (even with valid credentials)? But I admittedly haven't
tested this. (We already skip 403, and this seems similar?)
Closes https://github.com/astral-sh/uv/issues/3291.
## Summary
This index strategy resolves every package to the latest possible
version across indexes. If a version is in multiple indexes, the first
available index is selected.
Implements #3137
This closely matches pip.
## Test Plan
Good question. I'm hesitant to use my certifi example here, since that
would inevitably break when torch removes this package. Please comment!
## Summary
The approach taken here is to model `--target` as an install scheme in
which all the directories are just subdirectories of the `--target`.
From there, everything else... just works? Like, upgrade, uninstalls,
editables, etc. all "just work".
Closes#1517.
## Summary
based on PEP 508 the `platform_machine` should be same as
`platform.machine()` output:
https://peps.python.org/pep-0508/#environment-markers
From my macOS M2
```python
In [1]: import platform
In [2]: platform.machine()
Out[2]: 'arm64'
```
I napari we also use `arm64` in requirements and it works as expected:
9fcf63e69a/pyproject.toml (L120)
## Test Plan
<!-- How was it tested? -->
I rebased https://github.com/astral-sh/uv/pull/2757 then realized that
we want to implement this for more than `uv venv`.
Closes https://github.com/astral-sh/uv/issues/2587
Closes https://github.com/astral-sh/uv/issues/2757
```
❯ cargo run -q -- pip install -p /Users/mz/bin/python3.7 anyio
warning: uv is only compatible with Python 3.8+, found Python 3.7.17.
Audited 1 package in 84ms
❯ cargo run -q -- venv -p /Users/mz/bin/python3.7
warning: uv is only compatible with Python 3.8+, found Python 3.7.17.
Using Python 3.7.17 interpreter at: /Users/mz/bin/python3.7
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
```
---------
Co-authored-by: Stevie Gayet <stegayet@users.noreply.github.com>
## Summary
Simplifies dependency errors of the form `you require package-a and you
require package-b` to `you require package-a and package-b`. Resolves
https://github.com/astral-sh/uv/issues/1009.
The only thing a `OnceMap` really needs to be able to do with the value
is to clone it. All extant uses benefited from having this done for them
by automatically wrapping values in an `Arc`. But this isn't necessarily
true for all things. For example, a value might have an `Arc` internally
to making cloning cheap in other contexts, and it doesn't make sense to
re-wrap it in an `Arc` just to use it with a `OnceMap`. Or
alternatively, cloning might just be cheap enough on its own that an
`Arc` isn't worth it.
Previously, this would use the "system" temporary directory.
Because we rename the resulting directory to its final destination,
and because renaming is implemented via hardlinking, and because
hardlinking doesn't work across mount points, and because /tmp is
commonly on a different mount point than where `uv` is checked out,
we "fix" this by putting the temporary directory somewhere close to
the final destination of the fetched artifact.
There are alternatives we might consider pursuing. For example,
if the `rename` fails, then we should probably do a recursive
directory copy. But this is a quick fix for now and it also
consistent with colocation of other temporary directories in `uv`.
The main downside of this change is that if a user does ^C while
`uv-dev fetch-python` is running, then there is no mechanism for
cleaning up temporary directories.
## Summary
I initially implemented this by allowing arbitrary
`x86_64-manylinux_x_y`, but it makes the Clap, Serde, and Schemars
definitions more complicated, _and_ makes it harder for the user.
Ultimately, manylinux itself only provides images for 2_17 and 2_28, so
this seems like it should be sufficient.
Closes https://github.com/astral-sh/uv/issues/3222.
## Summary
We now recursively expand any self-dependencies via extras, which lets
us detect conflicts sooner and avoid building unnecessary versions of
packages that are excluded via the extra.
Closes https://github.com/astral-sh/uv/issues/3135.
Previously, uv-auth would fail to compile due to a missing process
feature. I chose to make all tokio features we use top level features,
so we can share the tokio cache between all test invocations.
## Summary
See the diff in the tests. If you have a constraint with an extra, we
should respect it, but we shouldn't _add_ the extra to the requirements.
## Summary
macOS 11 has been EOL for 7 months, and it seems like it's common to
publish ARM-only wheels at 12.0+. I think this is a better default for
ARM macs.
Closes https://github.com/astral-sh/uv/issues/3227.
## Test Plan
`cargo run pip install scikit-learn==1.3.2 --python-platform macos
--no-build`
## Summary
The cli gives <COLOR> as value placeholder which is misleading. I
changed this to <COLOR_CHOICE> to make it obvious you don't supply a
color but a ColorChoice value.
## Test Plan
Compiled and verified --help text was correct
## Summary
This is mainly a cleanup PR to leverage uv-version in uv-virtualenv
instead of passing it via `uv`.
In #1852 I introduced the ability to pass extra cfg to `gourgeist` for
the primary purpose of passing the uv version, but since the dawn of the
uv-version crate dynamically passing more values to pyvenv.cfg is no
longer needed.
## Test Plan
Existing `uv` tests should still verify `uv = <version>` exists in the
venv and make sure no regressions were introduced.
## Summary
The function `find_in_directory` joins `uv.toml`. The initial join in
`user` is redundant.
Also a small comment fix.
## Test Plan
Not really sure how to test this. Please suggest if any tests need to be
added.
## Summary
Avoid removing quotes from markers, e.g. `numpy (>=1.19) ;
python_version >= "3.7"` should not be rewritten. Fixes
https://github.com/astral-sh/uv/issues/2551.
This PR also makes fixups a bit more flexible internally for fixes that
aren't simple to implement with a pure regex replacement, like this one.
https://github.com/astral-sh/uv/pull/1529 fixed a similar problem but
the current regex is still not smart enough to avoid all markers
completely (like `python_version`).
## Test Plan
Added a few unit tests.
Fixes the failure to lookup credentials in
https://github.com/astral-sh/uv/issues/3205
The issue is that we seed the cache with the index URL which includes a
username but no password. We did not ensure that a password was present
in the cached credentials before attempting a request with them. Now,
the cache will not return credentials when a username is provided and
the cached credentials have no password — the cached credentials are
useless in that case.
Tested with a Google Artifact Registry and keyring
```
RUST_LOG=uv=trace cargo run -q -- pip install requests --index-url https://oauth2accesstoken@us-central1-python.pkg.dev/<project>/pypi/simple/ --no-cache --keyring-provider subprocess -v
```
## Summary
I found some of these too bare (e.g., when they _just_ show a package
name with no other information). For me, this makes it easier to
differentiate error message copy from data. But open to other opinions.
Take a look at the fixture changes and LMK!
## Summary
pip supports providing a `--platform` to `pip install`, which can be
used to seed an environment (e.g., for use in a container or otherwise).
This PR adds `--python-platform` to our commands to support a similar
workflow. It has some caveats, which are documented on the CLI.
Closes#2079.
Since we're now using read timeouts and not total timeouts, we can use a
lower threshold, a single read shouldn't take 5 min (and not even 10s).
The 10s value is somewhat arbitrary.
Like #3144, this is a breaking change in some sense.
Adds hidden `--preview` / `--no-preview` flags with `UV_PREVIEW`
environment variable support. Copies the `PreviewMode` type from Ruff.
Does a little bit of extra work to port `uv run` to the new settings
model.
Note we allow `uv run` invocations without preview and only use its
presence to toggle an experimental warning.
## Test plan
```
❯ cargo run -q -- run --no-workspace -- python --version
warning: `uv run` is experimental and may change without warning.
Python 3.12.2
❯ cargo run -q -- run --no-workspace --preview -- python --version
Python 3.12.2
❯ UV_PREVIEW=1 cargo run -q -- run --no-workspace -- python --version
Python 3.12.2
```
In #2976 I made some changes that led to regressions:
- We stopped tracking URLs that we had not seen credentials for in the
cache
- This means the cache no longer returns a value to indicate we've seen
a realm before
- We stopped seeding the cache with URLs
- Combined with the above, this means we no longer had a list of
locations that we would never attempt to fetch credentials for
- We added caching of credentials found on requests
- Previously the cache was only populated from the seed or credentials
found in the netrc or keyring
- This meant that the cache was populated for locations that we
previously did not cache, i.e. GitHub artifacts(?)
Unfortunately this unveiled problems with the granularity of our cache.
We cache credentials per realm (roughly the hostname) but some realms
have mixed authentication modes i.e. different credentials per URL or
URLs that do not require credentials. Applying credentials to a URL that
does not require it can lead to a failed request, as seen in #3123 where
GitHub throws a 401 when receiving credentials.
To resolve this, the cache is expanded to supporting caching at two
levels:
- URL, cached URL must be a prefix of the request URL
- Realm, exact match required
When we don't have URL-level credentials cached, we attempt the request
without authentication first. On failure, we'll search for realm-level
credentials or fetch credentials from external services. This avoids
providing credentials to new URLs unless we know we need them.
Closes https://github.com/astral-sh/uv/issues/3123
## Summary
This PR avoids: (1) using the lookahead resolver when `--no-deps` is
specified (we'll never use those requirements), and (2) including any
transitive requirements when searching for allowed URLs, etc., when
`--no-deps` is specified.
Closes https://github.com/astral-sh/uv/issues/3183.
Previously, we got `pypi_types::DirectUrl` (the pypa spec
direct_url.json format) and `distribution_types::DirectUrl` (an enum of
all the url types we support). This lead me to confusion, so i'm
renaming the latter one to the more appropriate `ParsedUrl`.
Add a dedicated error type for direct url parsing. This change is broken
out from the new uv requirement type, which uses direct url parsing
internally.
## Summary
This PR is adding `UV_CONSTRAINT` environment variable as analogous to
`PIP_CONSTRAINT` to allow providing constraint file via environment
variable. Implementing this will simplify adoption of uv in testing
procedure in projects that I'm involved (testing using tox).
This was my motivation for opening #1841 that is closed in favor of
#1789 which was closed without implementing this feature.
In this implementation, I have used space as a separator as analogous to
`pip`. This introduces an obvious problem if the path contains space.
Another option could be to use standard separator (`:` - UNIX like, `;`
- Windows). Which one did you prefer?
## Test Plan
It is my first contribution and first rust coding experience. It will be
nice if one could point how I should implement testing this.
I can't get this to reproduce on GitHub Actions -- maybe the builds
there differ, or maybe the builds changed since we added this fix? I'll
check locally, but regardless, this is a typo.
Closes#3158.
## Summary
No behavior changes, but the idea here is that we move the argument
normalization code (e.g., create an `Upgrade` struct from `--upgrade`
and `--upgrade-package`) into the `settings.rs` file, where we build the
common settings structs.
This reduces a lot of the logic and duplication across commands in
`main.rs`.
## Summary
pip prefers somewhat-constrained over unconstrained packages... but only
if they're at equal depths in the tree. We don't have a way to track the
latter property yet (I've added a TODO), so for now, we should remove
this constraint -- it seems to be counter-productive.
I've filed https://github.com/astral-sh/uv/issues/3149 as a follow-up.
Closes https://github.com/astral-sh/uv/issues/3143
## Test Plan
- `git clone https://github.com/drivendataorg/zamba.git`
- `cat "-e .[tests]" > req.in`
- `cargo run venv && cargo run pip compile req.in --refresh -n
--python-platform linux --python-version 3.8`
I added distributions to these projects so the commit changed.
We could pin but we want to test for resolution... so we don't. These
are pretty static so this should be rare.
## Summary
This leverages the new `read_timeout` property, which ensures that (like
pip) our timeout is not applied to the _entire_ request, but rather, to
each individual read operation.
Closes: #1921.
See: #1912.
This means that a bare `uv run` invocation drops you into a REPL.
This behavior is internally controversial, and may best be served by a
dedicated `uv repl` command. I would imagine it's important to fail if
no command is given in _some_ circumstances, but those may be resolved
by _not_ doing this if we do not detect a TTY.
Regardless, I'm interested in giving this a try for a bit during this
experimental phase.
In addition to the requested requirements, we include requirements from
a `pyproject.toml` file if it exists and install the current directory.
Closes https://github.com/astral-sh/uv/issues/3104
Holy cow does installation / resolution take a ton of options. We
side-step most of them here.
If the current environment satisfies the requirements, it is used.
Otherwise, we create a new environment with the requested dependencies.
## Summary
Following up
- https://github.com/astral-sh/uv/pull/3113
- https://github.com/astral-sh/uv/pull/3115
It looks like `uv pip compile` command with `UV_SYSTEM_PYTHON` is missed
because these two PRs are close in time. And thus resulting in
```bash
$ uv --version
uv 0.1.34 (9259eceeb 2024-04-19)
$ UV_SYSTEM_PYTHON=1 uv pip compile --upgrade requirements.in -o requirements.txt
error: invalid value '1' for '--system'
[possible values: true, false]
For more information, try '--help'.
```
Signed-off-by: Jack Cherng <jfcherng@gmail.com>
## Summary
I've wanted to try this for a long time, so decided to give it a shot.
The basic idea is that you can provide a target triple (e.g.,
`--platform x86_64-pc-windows-msvc`) and resolve against that platform,
rather than the currently-running platform. It's functionally similar to
`--python-version`, though a bit simpler since there's no need to engage
with `Requires-Python`.
Our infrastructure is well-setup for this and so, in the end, it's
actually pretty straightforward: for each triple, we just need to
override the markers and platform tags.
## Summary
We weren't setting a priority for editables, so they were being visited
last.
I think there's still a problem whereby we're not aggressive enough in
visiting recursive extras (and, in fact, that's making it really hard to
write a test -- I wrote a test, but the most-reduced case still fails,
and I'd need to add a layer of indirection to make it
fail-on-main-but-pass-on-this-branch), but that problem likely already
existed on main prior to #3087, so I just want to get this quick fix out
now.
Closes https://github.com/astral-sh/uv/issues/3127.
## Test Plan
- `git clone https://github.com/cda-tum/mqt-core.git`
- `cargo run venv`
- `cargo run pip install 'scikit-build-core[pyproject]>=0.8.1'
'setuptools_scm>=7' 'pybind11>=2.12' --resolution=lowest-direct`
- `cargo run pip install --no-build-isolation
'-ve.[test,qiskit,evaluation,coverage]' --resolution=lowest-direct`
Given requirements like:
```
black==23.1.0
black[colorama]
```
The resolver will (on `main`) add a dependency on Black, and then try to
use the most recent version of Black to satisfy `black[colorama]`. For
sake of example, assume `black==24.0.0` is the most recent version. Once
the selects this most recent version, it'll fetch the metadata, then
return the dependencies for `black==24.0.0` with the `colorama` extra
enabled. Finally, it will tack on `black==24.0.0` (a dependency on the
base package). The resolver will then detect a conflict between
`black==23.1.0` and `black==24.0.0`, and throw out
`black[colorama]==24.0.0`, trying to next most-recent version.
This is both wasteful and can cause problems, since we're fetching
metadata for versions that will _never_ satisfy the resolver. In the
`apache-airflow[all]` case, I also ran into an issue whereby we were
attempting to build very old versions of `apache-airflow` due to
`apache-airflow[pandas]`, which in turn led to resolution failures.
The solution proposed here is that we create a new proxy package with
exactly two dependencies: one on `black` and one of `black[colorama]`.
Both of these packages must be at the same version as the proxy package,
so the resolver knows much _earlier_ that (in the above example) the
extra variant _must_ match `23.1.0`.
## Summary
This PR adds a test that currently leads to an error, but should
successfully resolve as of https://github.com/astral-sh/uv/pull/3100.
The core idea is that if we have a pinned package, we shouldn't try to
build other versions of that package if we have an unconstrained variant
with an extra.
## Summary
This was unintended. We ended up reverting `Option<bool>` everywhere,
but I missed this once since it's in a separate file.
(If you use `Option<bool>`, Clap requires a value, like `--no-cache
true`.)
## Test Plan
`cargo run pip install flask --no-cache`
## Summary
I think these are useful to have for consistency, though the `--system`
variant requires some new threading.
Closes: https://github.com/astral-sh/uv/issues/2242.
## Summary
Right now, we only accept _exactly `UV_NATIVE_TLS=true` and
`UV_NATIVE_TLS=false`. `BoolishValueParser` accepts a wider range of
values:
```rust
/// True values are `y`, `yes`, `t`, `true`, `on`, and `1`.
pub(crate) const TRUE_LITERALS: [&str; 6] = ["y", "yes", "t", "true", "on", "1"];
/// False values are `n`, `no`, `f`, `false`, `off`, and `0`.
pub(crate) const FALSE_LITERALS: [&str; 6] = ["n", "no", "f", "false", "off", "0"];
```
I tend to use `0` and `1` personally so this surprised me.
## Summary
resolves https://github.com/astral-sh/uv/issues/3106
## Test Plan
added a simple test where the password provided in `UV_INDEX_URL` is
hidden in the output as expected.
## Summary
With an alias for backwards compatibility. It's clearer and matches the
setting in the TOML configuration (where `compile` was deemed too
vague).
## Summary
Now that we can pick up configuration values from persistent files, we
need to enable users to _disable_ those values from the CLI. For
example, if a user has `emit_index_url = true` in the configuration
file, they should be able to do `--no-emit-index-url` on the
command-line. This PR adds support for such negations, following the
same patterns we use in Ruff.
## Summary
Enables `uv` to read configuration from (e.g.)
`/Users/crmarsh/.config/uv/uv.toml` on macOS and Linux, and
`C:\Users\Charlie\AppData\Roaming\uv\uv.toml` on Windows.
This differs slightly from Ruff, which uses the `Application Support`
directory on macOS. But I've deviated here. based on the preferences
expressed in https://github.com/astral-sh/ruff/issues/10739.
## Summary
This PR adds the structs and logic necessary to respect settings from
the workspace. It's a ton of code, but it's mostly mechanical. And,
believe it or not, I pulled out a few refactors in advance to trim down
the code and complexity.
The highlights are:
- All CLI arguments are now `Option`, so that we can detect whether they
were provided (i.e., we can't let Clap fill in the defaults).
- We now have a `*Settings` struct for each command, which merges the
CLI and workspace options (e.g., `PipCompileSettings`).
I've only implemented `PipCompileSettings` for now. If approved, I'll
implement the others prior to merging, but it's very mechanical and I
both didn't want to do the conversion prior to receiving feedback _and_
realized it would make the PR harder to review.
If a virtual environment does not exist, we will create one for the
duration of the invocation.
Adds an `--isolated` flag to force this behavior (ignoring an existing
virtual environment).
Adds `uv run` which executes a command in your current virtual
environment.
This is a simple first milestone, lots of remaining work and behavior.
The command is hidden.
Fixes https://github.com/astral-sh/uv/issues/3060
## Summary
Allows passing a virtual environment (the path to the directory, rather
than the path to the Python interpreter within the directory) to the
`--python` option of the `uv pip` command.
## Test Plan
Tested manually to confirm that the expected new functionality works.
The test suite still passes after this change.
I don't know how to add tests for a new feature like this. I would be
happy to do so if someone can give me some pointers on how to do it.
## Summary
Source distributions in the .tar.bz2 format are still relatively common
within the existing code-bases, namely, the most common examples are the
Twisted source distributions up to the version 20.3.0. As quite so often
the ability to upgrade Twisted to a more recent version is not available
for a given project, we add the support for .tar.bz2 here to still allow
`uv` to be a drop-in replacement for `pip` in these projects.
## Test Plan
The feature was tested both by adding the corresponding test coverage,
and by directly installing a package of interest under a Python version
that doesn't have the corresponding wheel:
```sh
cargo run venv -p python3.8
cargo run pip install Twisted==20.3.0 --no-cache
```
The `--no-cache` argument in the example above serves the purpose of
cleaning the cached information regarding the unsatisfiability of the
requirements, as it may have been cached during some previous attempt to
install this package by `uv` version that didn't implement this feature
yet.
## Summary
This PR adds basic struct definitions along with a "workspace" concept
for discovering settings. (The "workspace" terminology is used to match
Ruff; I did not invent it.)
A few notes:
- We discover any `pyproject.toml` or `uv.toml` file in any parent
directory of the current working directory. (We could adjust this to
look at the directories of the input files.)
- We don't actually do anything with the configuration yet; but those
PRs are large and I want this to be reviewed in isolation.
Hello! This is my first PR so do not hesitate to let me know if anything
should be done differently 🙌🏽
## Summary
This PR starts adding useful error messages and warnings when people
pass redundant or unsupported arguments to `pip list`.
For now, I've just covered `pip list --outdated`, which is currently
unsupported.
Closes https://github.com/astral-sh/uv/issues/2948
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Since the [`junction` crate](https://crates.io/crates/junction)
implements Windows-only functionality, and since the only place it is
used is guarded by `#[cfg(windows)]`,
1f626bfc73/crates/uv-fs/src/lib.rs (L65-L86)
it makes sense not to depend on this crate at all on non-Windows
platforms.
If nothing else, this makes Linux distribution packagers’ lives just a
*tiny* bit easier.
## Test Plan
<!-- How was it tested? -->
On Fedora Linux 39:
```
# To avoid an error when /tmp and the working directory are on different filesystems:
$ mkdir _tmp
$ TMPDIR="${PWD}/tmp" cargo run -p uv-dev -- fetch-python
$ cargo test
```
I don’t have access to a Windows system.
## Summary
This makes it easier to add (e.g.) JSON Schema derivations to the type.
If we have support for other dates in the future, we can generalize it
to a `UserDate` or similar.
# Avoid cache invalidation on credentials renewal
Addresses
- https://github.com/astral-sh/uv/issues/3009#issue-2239221126
## Summary
Some private package registries (e.g. AWS CodeArtifact) use short-lived
credentials. Since they are short-lived, the exact URL that is assigned
to `UV_INDEX_URL` changes frequently and with that the cache key /
hashes of these URLs. This causes the cache to be missed on token
renewal.
This PR attempts to fix this by hashing URLs for cache keys without
their user credentials.
## Test Plan
I added a test that validates that:
1. Changing user credentials returns the same hash
2. Setting no user credentials yields the same as some user credentials
## Question
I'm not sure if we should also change the `hash` implementation of
`CanonicalUrl` / `RepositoryUrl`. They also run `.hash` within.
PS. this is the first time I'm writing `rust` so if I'm wasting your
precious time, let me know and I'll try to up my skills before I ask
again. Anyway, I figured it's good to get this issue on your radar :)
## Summary
This PR adds system install tests to verify the behavior described in
#2798. It turns out this behavior _also_ affects Fedora and Amazon
Linux, we just didn't have the right conditions enabled (specifically,
you need to create the virtualenv with `python -m venv` to get these
symlinks), so the test suite was expanded to capture that.
The issue itself is also fixed by way of deduplicating the
`site-packages` entries.
Closes: https://github.com/astral-sh/uv/issues/2798
## Summary
I'm surprised we haven't hit this before, but apparently we don't allow
comments after `--index-url`, `-e` entries, etc., in the
requirements.txt parser.
Closes#3011.
## Test Plan
`cargo test`
## Summary
It turns out that if you have an environment variable set, Clap will
consider that equivalent to passing the flag, even if it's set to (e.g.)
something falsy or the default value.
So, e.g., this fails:
```shell
UV_SYSTEM=false uv pip install --python ./.venv/bin/python flask
```
Worse, this fails, because it thinks `--no-index` and `--index-url` are
conflicting:
```shell
export UV_INDEX_URL=https://google.com
uv pip install flask --no-index
```
This PR removes some of the conflicts, namely those related to
environment variables, such that:
- You _can_ pass mixes of `--no-index`, `--index-url`, etc. If
`--no-index` is provided, all the index URLs will be ignored (but we
won't error).
- Passing `--pre` will always enable prereleases, even if `--prerelease`
is also provided. (We could warn here, although honestly it's not
trivial because we'd need to make `--prerelease` take an optional, then
we'd lose the default argument from the `--help`.)
- You _can_ pass `--system` and `--python`. If `--python` is provided,
we use that, and ignore `--system`. (We could warn here.)
I guess the underlying problem here is that we can't differentiate
between arguments passed on the CLI and those set as environment
variables. But making bigger changes here seems out-of-scope.
Closes https://github.com/astral-sh/uv/issues/3000.
## Summary
It turns out that `normalize_path` (sourced from Cargo) has a subtle
bug. If you pass it a relative path that traverses beyond the root, it
silently drops components. So, e.g., passing `../foo/bar`, it will just
drop the leading `..` and return `foo/bar`.
This PR encodes that behavior as a `Result` and avoids using it in such
cases.
Closes https://github.com/astral-sh/uv/issues/3012.
Inspired by https://github.com/astral-sh/uv/issues/2964, we now properly
log hardlink failures, e.g. when the cache is a docker container but the
venv is in a bind mount, e.g.:
```
DEBUG Failed to hardlink `/code/venv/uv/lib/python3.12/site-packages/asgiref-3.8.1.dist-info/WHEEL` to `/root/.cache/uv/archive-v0/nnpkKgUoM3LMxcNDmEKJQ/asgiref-3.8.1.dist-info/WHEEL`, attempting to copy files as a fallback
```
## Summary
I don't know if this is a good change, but `main.rs` is really large.
This just moves all the Clap arguments into their own `cli.rs` module.
freethreaded python reintroduces abiflags since it is incompatible with
regular native modules and abi3.
Tests: None yet! We're lacking cpython 3.13 no-gil builds we can use in
ci.
My test setup:
```
PYTHON_CONFIGURE_OPTS="--enable-shared --disable-gil" pyenv install 3.13.0a5
cargo run -q -- venv -q -p python3.13 .venv3.13 --no-cache-dir && cargo run -q -- pip install -v psutil --no-cache-dir && .venv3.13/bin/python -c "import psutil"
```
Fixes#2429
## Summary
In all of these ID types, we pass values to `cache_key::digest` prior to
passing to `DistributionId` or `ResourceId`. The `DistributionId` and
`ResourceId` are then hashed later, since they're used as keys in hash
maps.
It seems wasteful to hash the value, then hash the hashed value? So this
PR modifies those structs to be enums that can take one of a fixed set
of types.
## Summary
If there are no hashes for a given package, we now return
`Validate(&[])` so that the policy is impossible to satisfy. Previously,
we returned `None`, which is always satisfied.
We don't really ever expect to hit this, because we detect this case in
the resolver and raise a different error. But if we have a bug
somewhere, it's better to fail with an error than silently let the
package through.
## Summary
This PR enables `--require-hashes` with unnamed requirements. The key
change is that `PackageId` becomes `VersionId` (since it refers to a
package at a specific version), and the new `PackageId` consists of
_either_ a package name _or_ a URL. The hashes are keyed by `PackageId`,
so we can generate the `RequiredHashes` before we have names for all
packages, and enforce them throughout.
Closes#2979.
When running the `uv-client` tests, i would previously get:
```
warning: field `0` is never read
--> crates/uv-configuration/src/config_settings.rs:43:27
|
43 | pub struct ConfigSettings(BTreeMap<String, ConfigSettingValue>);
| -------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| |
| field in this struct
|
= note: `ConfigSettings` has derived impls for the traits `Clone` and `Debug`, but these are intentionally ignored during dead code analysis
= note: `#[warn(dead_code)]` on by default
help: consider changing the field to be of unit type to suppress this warning while preserving the field numbering, or remove the field
|
43 | pub struct ConfigSettings(());
| ~~
warning: `uv-configuration` (lib) generated 1 warning
```
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Closes#2564
## Test Plan
1. Changed existing linehaul tests to leverage insta.
2. Ran tests in various linux distros (Debian, Ubuntu, Centos, Fedora,
Alpine) to ensure they also pass locally again.
---------
Co-authored-by: konstin <konstin@mailbox.org>
The sync scenarios script is broken, so i did the updates manually
```
$ ./scripts/sync_scenarios.sh
Setting up a temporary environment...
Using Python 3.12.1 interpreter at: /home/konsti/projects/uv/.venv/bin/python3
Creating virtualenv at: .venv
Activate with: source .venv/bin/activate
× No solution found when resolving dependencies:
╰─▶ Because docutils==0.21.post1 is unusable because the package metadata was inconsistent and you require docutils==0.21.post1, we can conclude that the requirements are unsatisfiable.
hint: Metadata for docutils==0.21.post1 was inconsistent:
Package metadata version `0.21` does not match given version `0.21.post1`
```
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
Similar to `Revision`, we now store IDs in the `Archive` entires rather
than absolute paths. This makes the cache robust to moves, etc.
Closes https://github.com/astral-sh/uv/issues/2908.
## Summary
This PR formalizes some of the concepts we use in the cache for
"pointers to things".
In the wheel cache, we have files like
`annotated_types-0.6.0-py3-none-any.http`. This represents an unzipped
wheel, cached alongside an HTTP caching policy. We now have a struct for
this to encapsulate the logic: `HttpArchivePointer`.
Similarly, we have files like `annotated_types-0.6.0-py3-none-any.rev`.
This represents an unzipped local wheel, alongside with a timestamp. We
now have a struct for this to encapsulate the logic:
`LocalArchivePointer`.
We have similar structs for source distributions too.
## Summary
This PR enables hash generation for URL requirements when the user
provides `--generate-hashes` to `pip compile`. While we include the
hashes from the registry already, today, we omit hashes for URLs.
To power hash generation, we introduce a `HashPolicy` abstraction:
```rust
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HashPolicy<'a> {
/// No hash policy is specified.
None,
/// Hashes should be generated (specifically, a SHA-256 hash), but not validated.
Generate,
/// Hashes should be validated against a pre-defined list of hashes. If necessary, hashes should
/// be generated so as to ensure that the archive is valid.
Validate(&'a [HashDigest]),
}
```
All of the methods on the distribution database now accept this policy,
instead of accepting `&'a [HashDigest]`.
Closes#2378.
## Summary
This PR modifies the distribution database to return both the
`Metadata23` and the computed hashes when clients request metadata.
No behavior changes, but this will be necessary to power
`--generate-hashes`.
## Summary
This represents a change to `--require-hashes` in the event that we
don't find a matching hash from the registry. The behavior in this PR is
closer to pip's.
Prior to this PR, if a distribution had no reported hash, or only
mismatched hashes, we would mark it as incompatible. Now, we mark it as
compatible, but we use the hash-agreement as part of the ordering, such
that we prefer any distribution with a matching hash, then any
distribution with no hash, then any distribution with a mismatched hash.
As a result, if an index reports incorrect hashes, but the user provides
the correct one, resolution now succeeds, where it would've failed.
Similarly, if an index omits hashes altogether, but the user provides
the correct one, resolution now succeeds, where it would've failed.
If we end up picking a distribution whose hash ultimately doesn't match,
we'll reject it later, after resolution.
## Summary
This PR adds support for hash-checking mode in `pip install` and `pip
sync`. It's a large change, both in terms of the size of the diff and
the modifications in behavior, but it's also one that's hard to merge in
pieces (at least, with any test coverage) since it needs to work
end-to-end to be useful and testable.
Here are some of the most important highlights:
- We store hashes in the cache. Where we previously stored pointers to
unzipped wheels in the `archives` directory, we now store pointers with
a set of known hashes. So every pointer to an unzipped wheel also
includes its known hashes.
- By default, we don't compute any hashes. If the user runs with
`--require-hashes`, and the cache doesn't contain those hashes, we
invalidate the cache, redownload the wheel, and compute the hashes as we
go. For users that don't run with `--require-hashes`, there will be no
change in performance. For users that _do_, the only change will be if
they don't run with `--generate-hashes` -- then they may see some
repeated work between resolution and installation, if they use `pip
compile` then `pip sync`.
- Many of the distribution types now include a `hashes` field, like
`CachedDist` and `LocalWheel`.
- Our behavior is similar to pip, in that we enforce hashes when pulling
any remote distributions, and when pulling from our own cache. Like pip,
though, we _don't_ enforce hashes if a distribution is _already_
installed.
- Hash validity is enforced in a few different places:
1. During resolution, we enforce hash validity based on the hashes
reported by the registry. If we need to access a source distribution,
though, we then enforce hash validity at that point too, prior to
running any untrusted code. (This is enforced in the distribution
database.)
2. In the install plan, we _only_ add cached distributions that have
matching hashes. If a cached distribution is missing any hashes, or the
hashes don't match, we don't return them from the install plan.
3. In the downloader, we _only_ return distributions with matching
hashes.
4. The final combination of "things we install" are: (1) the wheels from
the cache, and (2) the downloaded wheels. So this ensures that we never
install any mismatching distributions.
- Like pip, if `--require-hashes` is provided, we require that _all_
distributions are pinned with either `==` or a direct URL. We also
require that _all_ distributions have hashes.
There are a few notable TODOs:
- We don't support hash-checking mode for unnamed requirements. These
should be _somewhat_ rare, though? Since `pip compile` never outputs
unnamed requirements. I can fix this, it's just some additional work.
- We don't automatically enable `--require-hashes` with a hash exists in
the requirements file. We require `--require-hashes`.
Closes#474.
## Test Plan
I'd like to add some tests for registries that report incorrect hashes,
but otherwise: `cargo test`
## Summary
This lets us remove circular dependencies (in the future, e.g., #2945)
that arise from `FlatIndex` needing a bunch of resolver-specific
abstractions (like incompatibilities, required hashes, etc.) that aren't
necessary to _fetch_ the flat index entries.
See https://github.com/astral-sh/uv/issues/2617
Note this also includes:
- #2918
- #2931 (pending)
A first step towards Python toolchain management in Rust.
First, we add a new crate to manage Python download metadata:
- Adds a new `uv-toolchain` crate
- Adds Rust structs for Python version download metadata
- Duplicates the script which downloads Python version metadata
- Adds a script to generate Rust code from the JSON metadata
- Adds a utility to download and extract the Python version
I explored some alternatives like a build script using things like
`serde` and `uneval` to automatically construct the code from our
structs but deemed it to heavy. Unlike Rye, I don't generate the Rust
directly from the web requests and have an intermediate JSON layer to
speed up iteration on the Rust types.
Next, we add add a `uv-dev` command `fetch-python` to download Python
versions per the bootstrapping script.
- Downloads a requested version or reads from `.python-versions`
- Extracts to `UV_BOOTSTRAP_DIR`
- Links executables for path extension
This command is not really intended to be user facing, but it's a good
PoC for the `uv-toolchain` API. Hash checking (via the sha256) isn't
implemented yet, we can do that in a follow-up.
Finally, we remove the `scripts/bootstrap` directory, update CI to use
the new command, and update the CONTRIBUTING docs.
<img width="1023" alt="Screenshot 2024-04-08 at 17 12 15"
src="https://github.com/astral-sh/uv/assets/2586601/57bd3cf1-7477-4bb8-a8e9-802a00d772cb">
## Summary
If the user runs with `--generate-hashes`, and the lockfile doesn't
contain _any_ hashes for a package (despite being pinned), we should add
new hashes. This mirrors running `uv pip compile --generate-hashes` for
the first time with an existing lockfile.
Closes#2962.
To get more insights into test performance, allow instrumenting tests
with tracing-durations-export.
Usage:
```shell
# A single test
TRACING_DURATIONS_TEST_ROOT=$(pwd)/target/test-traces cargo test --features tracing-durations-export --test pip_install_scenarios no_binary -- --exact
# All tests
TRACING_DURATIONS_TEST_ROOT=$(pwd)/target/test-traces cargo nextest run --features tracing-durations-export
```
Then we can e.g. look at
`target/test-traces/pip_install_scenarios::no_binary.svg` and see the
builds it performs:
## Summary
If we build a source distribution from the registry, and the version
doesn't match that of the filename, we should error, just as we do for
mismatched package names. However, we should also backtrack here, which
we didn't previously.
Closes https://github.com/astral-sh/uv/issues/2953.
## Test Plan
Verified that `cargo run pip install docutils --verbose --no-cache
--reinstall` installs `docutils==0.21` instead of the invalid
`docutils==0.21.post1`.
In the logs, I see:
```
WARN Unable to extract metadata for docutils: Package metadata version `0.21` does not match given version `0.21.post1`
```
## Summary
This updates to the version of axoupdater used in cargo-dist 0.13.0's
own selfupdate command, with all relevant fixes for platforms. It also
tentatively introduces a mildly dangerous self-runtest that runs `uv
self update` and checks that the binary is installed and executable.
I *believe* some adjustments need to be made to your CI to have this new
test run, because it requires the `self-update` feature to be enabled,
and I didn't want to just start messing with how you do feature coverage
in your CI. **As a result I haven't yet had a chance to actually fully
run this in CI**, though I've locally tested it on windows (with the
guard disabled).
## Test Plan
Most of the machinery here is provided by axoupdater itself (cargo-dist
also includes a variant of these tests in its codebase). This initial
implementation has a couple major limitations:
* This is For Reals modifying the system that runs the test (so it's off
unless it detects it's running in CI, and if you want variations on this
test they'll need to be [run in
serial](5e7826f7b0/cargo-dist/tests/cli-tests.rs (L235))).
Since many of the testing issues were surrounding precise details of
Actual Deployed Executions, this seemed worth the tradeoff.
* The actual installer *script* it's ultimately invoking is the one you
last published, and *not* the one that cargo-dist will make when you
next publish.
We're already working on implementing some logic for "get cargo-dist to
generate a fresh installer script too", which is in fact the basis of a
huge amount of cargo-dist's own testsuite. Now that we're dogfooding
this stuff, it should be quite hard for this stuff to break without
cargo-dist's own codebase noticing it first.
<!-- How was it tested? -->
## Summary
The prefetcher tallies the number of times we tried a given package, and
then once we hit a threshold, grabs the version map, assuming it's
already been fetched. For direct URL distributions, though, we don't
have a version map! And there's no need to prefetch.
Closes https://github.com/astral-sh/uv/issues/2941.
## Summary
Right now, we have a `Hashes` representation that looks like:
```rust
/// A dictionary mapping a hash name to a hex encoded digest of the file.
///
/// PEP 691 says multiple hashes can be included and the interpretation is left to the client.
#[derive(Debug, Clone, Eq, PartialEq, Default, Deserialize)]
pub struct Hashes {
pub md5: Option<Box<str>>,
pub sha256: Option<Box<str>>,
pub sha384: Option<Box<str>>,
pub sha512: Option<Box<str>>,
}
```
It stems from the PyPI API, which returns a dictionary of hashes.
We tend to pass these around as a vector of `Vec<Hashes>`. But it's a
bit strange because each entry in that vector could contain multiple
hashes. And it makes it difficult to ask questions like "Is
`sha256:ab21378ca980a8` in the set of hashes"?
This PR instead treats `Hashes` as the PyPI-internal type, and uses a
new `Vec<HashDigest>` everywhere in our own APIs.
Needed to prevent circular dependencies in my toolchain work (#2931). I
think this is probably a reasonable change as we move towards persistent
configuration too?
Unfortunately `BuildIsolation` needs to be in `uv-types` to avoid
circular dependencies still. We might be able to resolve that in the
future.
Elides Python patch versions from the test suite unless the test
specifically requests a patch version.
This reduces some toil when not using our bootstrapped Python versions.
Partially addresses https://github.com/astral-sh/uv/issues/2165 though
we'll need changes to the scenario tests to really support their case.
## Summary
When you specify a source distribution via a path, it can either be a
path to an archive (like a `.tar.gz` file), or a source tree (a
directory). Right now, we handle both paths through the same methods in
the source database. This PR splits them up into separate handlers.
This will make hash generation a little easier, since we need to
generate hashes for archives, but _can't_ generate hashes for source
trees.
It also means that we can now store the unzipped source distribution in
the cache (in the case of archives), and avoid unzipping the source
distribution needlessly on every invocation; and, overall, let's un
enforce clearer expectations between the two routes (e.g., what errors
are possible vs. not), at the cost of duplicating some code.
Closes#2760 (incidentally -- not exactly the motivation for the change,
but it did accomplish it).
## Summary
I think this is a much clearer name for this concept: the set of
"versions" of a given wheel or source distribution. We also use
"Manifest" elsewhere to refer to the set of requirements, constraints,
etc., so this was overloaded.
## Summary
We have a heuristic in `File` that attempts to detect whether a URL is
absolute or relative. However, `contains("://")` is prone to false
positive. In the linked issues, the URLs look like:
```
/packages/5a/d8/4d75d1e4287ad9d051aab793c68f902c9c55c4397636b5ee540ebd15aedf/pytz-2005k.tar.bz2?hash=597b596dc1c2c130cd0a57a043459c3bd6477c640c07ac34ca3ce8eed7e6f30c&remote=4d75d1e4287636b5ee540ebd15aedf/pytz-2005k.tar.bz2#sha256=597b596dc1c2c130cd0a57a043459c3bd6477c640c07ac34ca3ce8eed7e6f30c
```
Which is relative, but includes `://`.
Instead, we should determine whether the URL has a _scheme_ which
matches the `Url` crate internally.
Closes https://github.com/astral-sh/uv/issues/2899.
## Summary
Right now, the path-based wheel cache just looks at the symlink to the
archives directory, checks the timestamp on it, and continues with that
symlink as long as the timestamp is up-to-date.
The HTTP-based wheel meanwhile, uses an intermediary `.http` file, which
includes the HTTP caching information. The `.http` file's payload is
just a path pointing to an entry in the archives directory.
This PR modifies the path-based codepaths to use a similar cache file,
which stores a timestamp along with a path to the archives directory.
The main advantage here is that we can add other data to this cache file
(namely, hashes in the future).
## Test Plan
Beyond existing tests, I also verified that this doesn't require a
version bump:
```
git checkout main
cargo run pip install ~/Downloads/zeal-0.0.1-py3-none-any.whl --cache-dir baz --reinstall
git checkout charlie/manifest
cargo run pip install ~/Downloads/zeal-0.0.1-py3-none-any.whl --cache-dir baz --reinstall
cargo run pip install ~/Downloads/zeal-0.0.1-py3-none-any.whl --cache-dir baz --reinstall --refresh
```
## Summary
I think this is kind of just an oversight. If a wheel is available via
`--find-links`, and the index is "local", we never find it in the cache.
## Test Plan
`cargo test`
## Summary
In all cases, we unzip these immediately after returning. By moving the
unzipping into the database, we can remove a bunch of code (coming in a
separate PR), and pave the way for hash-checking, since hash generation
will _also_ happen in the database, and splitting the caching layers
across the database and the unzipper creates complications.
Closes#2863.
With pubgrub being fast for complex ranges, we can now compute the next
n candidates without taking a performance hit. This speeds up cold cache
`urllib3<1.25.4` `boto3` from maybe 40s - 50s to ~2s. See docstrings for
details on the heuristics.
**Before**
**After**
---
We need two parts of the prefetching, first looking for compatible
version and then falling back to flat next versions. After we selected a
boto3 version, there is only one compatible botocore version remaining,
so when won't find other compatible candidates for prefetching. We see
this as a pattern where we only prefetch boto3 (stack bars), but not
botocore (sequential requests between the stacked bars).
The risk is that we're completely wrong with the guess and cause a lot
of useless network requests. I think this is acceptable since this
mechanism only triggers when we're already on the bad path and we should
simply have fetched all versions after some seconds (assuming a fast
index like pypi).
---
It would be even better if the pubgrub state was copy-on-write so we
could simulate more progress than we actually have; currently we're
guessing what the next version is which could be completely wrong, but i
think this is still a valuable heuristic.
Fixes#170.
## Summary
Is this, perhaps, not totally necessary? It doesn't show up in any
fixtures beyond those that I added recently.
Closes https://github.com/astral-sh/uv/issues/2846.
## Summary
Demonstrates some suboptimal behavior in how we handle invalid metadata,
which are fixed in https://github.com/astral-sh/uv/pull/2834.
The included wheels were modified by-hand to include invalid structures.
Batch prefetching needs more information from the candidate selector, so
i've split `select` into methods. Split out from #2452. No functional
changes.
## Summary
In working on `--require-hashes`, I noticed that we're missing some
incompatibility tracking for `--find-links` distributions. Specifically,
we don't respect `--no-build` or `--no-binary`, so if we select a wheel
due to `--find-links`, we then throw a hard error when trying to build
it later (if `--no-binary` is provided), rather than selecting the
source distribution instead.
Closes https://github.com/astral-sh/uv/issues/2827.
## Summary
If we're given a Git reference like `20240222`, we currently treat it as
a short commit hash. However... it _could_ be a branch or a tag. This PR
improves the Git reference logic to ensure that ambiguous references
like `20240222` are handled appropriately, by attempting to extract it
as a branch, then a tag, then a short commit hash.
Closes https://github.com/astral-sh/uv/issues/2772.
## Summary
Upgrading `rs-async-zip` enables us to support data descriptors in
streaming. This both greatly improves performance for indexes that use
data descriptors _and_ ensures that we support them in a few other
places (e.g., zipped source distributions created in Finder).
Closes#2808.
## Summary
This partially revives https://github.com/astral-sh/uv/pull/2135 (with
some modifications) to enable users to opt-in to looking for packages
across multiple indexes.
The behavior is such that, in version selection, we take _any_
compatible version from a "higher-priority" index over the compatible
versions of a "lower-priority" index, even if that means we might accept
an "older" version.
Closes https://github.com/astral-sh/uv/issues/2775.
## Summary
This was an oversight in the `-r pyproject.toml` refactor. We can't
enforce unused extras if we have a source tree. We made the correct
changes to `pip compile`, but not `pip install`. This PR just mirrors
those changes to `pip install`, and adds a few tests.
Closes https://github.com/astral-sh/uv/issues/2801.
We don't know what kind of error the OS gives us on `try_lock_exclusive`
with an already locked file, so we assume all those errors are an
already locked file and call `lock_exclusive`.
For example the windows error:
```
Os {
code: 33,
kind: Uncategorized,
message: "The process cannot access the file because another process has locked a portion of the file.",
}
```
Fixes#2767
## Summary
Rather than storing the `redirects` on the resolver, this PR just
re-uses the "convert this URL to precise" logic when we convert to a
`Resolution` after-the-fact. I think this is a lot simpler: it removes
state from the resolver, and simplifies a lot of the hooks around
distribution fetching (e.g., `get_or_build_wheel_metadata` no longer
returns `(Metadata23, Option<Url>)`).
## Summary
I noticed in #2769 that I was now stripping `.git` suffixes from Git
URLs after resolving to a precise commit. This PR cleans up the internal
caching to use a better canonical representation: a `RepositoryUrl`
along with a `GitReference`, instead of a `GitUrl` which can contain
non-canonical data. This gives us both better fidelity (preserving the
`.git`, along with any casing that the user provided when defining the
URL) and is overall cleaner and more robust.
## Summary
This PR leverages our lookahead direct URL resolution to significantly
improve the range of Git URLs that we can accept (e.g., if a user
provides the same requirement, once as a direct dependency, and once as
a tag). We did some of this in #2285, but the solution here is more
general and works for arbitrary transitive URLs.
Closes https://github.com/astral-sh/uv/issues/2614.
Originally a regression test for #2779 but we found out that there's
some weird behavior where different `anyio` versions were preferred
based on the platform.
Addresses panic introduced in #2596 and reported in
https://github.com/astral-sh/uv/issues/2763#issuecomment-2030674936
When there are multiple versions of a package available, we remove the
existing packages before installing the resolved version to "fix" the
environment. We must remove all of the package versions and reinstall
because removing _any_ of the package versions could break the others.
Since reinstalls require a pull from the remote, this broke a contract
between the resolver and the planner which must always agree on which
packages should come from the remote. This further demonstrates that we
should be constructing the install plan with more concrete knowledge
from the resolver (i.e. `ResolvedDist` instead of `Requirement`) to
avoid having to manually ensure logic matches.
## Test plan
Fails on `main` with panic succeeds on branch
```
uv venv --seed
source .venv/bin/activate
pip install anyio==3.7.0 --ignore-installed
pip install anyio==4.0.0 --ignore-installed
cargo run -- pip install anyio black -v
```
## Summary
Ensures that if we resolve any distributions before the resolver, we
cache the metadata in-memory.
_Also_ ensures that we lock (important!) when resolving Git
distributions.
## Summary
This fixes a potential bug that revealed itself in
https://github.com/astral-sh/uv/pull/2761. We don't run into this now
because we always use "allowed URLs", stores the "last" compatible URL
in the map. But the use of the "raw" URL (rather than the "canonical"
URL) means that other writers have to follow that same assumption and
iterate over dependencies in-order.
## Summary
This PR would enable us to support transitive URL requirements. The key
idea is to leverage the fact that...
- URL requirements can only come from URL requirements.
- URL requirements identify a _specific_ version, and so don't require
backtracking.
Prior to running the "real" resolver, we recursively resolve any URL
requirements, and collect all the known URLs upfront, then pass those to
the resolver as "lookahead" requirements. This means the resolver knows
upfront that if a given package is included, it _must_ use the provided
URL.
Closes https://github.com/astral-sh/uv/issues/1808.
With this change, all usages of `EXCLUDE_NEWER` are now in command
wrappers, not in the test functions themselves.
For the venv test, i refactored them into the same kind of test context
abstraction that the other test modules have in the second commit.
The third commit makes`"INSTA_FILTERS` "private", removing the last
remaining individual usage.
Pending windows CI 🤞
## Summary
We can access cache from `BuildContext`. This mirrors
`SourceDistCachedBuilder`, which doesn't accept `Cache` as an argument
and always accesses it through `BuildContext`.
## Summary
If the user provides `uv pip install pyproject.toml`, we now prompt them
to confirm that they meant the `pyproject-toml` package (as opposed to
`uv pip install -r pyproject.toml`).
## Summary
We iterate over the project "requirements" directly in a variety of
places. However, it's not always the case that an input "requirement" on
its own will _actually_ be part of the resolution, since we support
"overrides".
Historically, then, overrides haven't worked as expected for _direct_
dependencies (and we have some tests that demonstrate the current,
"wrong" behavior). This is just a bug, but it's not really one that
comes up in practice, since it's rare to apply an override to your _own_
dependency.
However, we're now considering expanding the lookahead concept to
include local transitive dependencies. In this case, it's more and more
important that overrides and constraints are handled consistently.
This PR modifies all the locations in which we iterate over requirements
directly, and modifies them to respect overrides (and constraints, where
necessary).
## Summary
This is a trimmed-down version of
https://github.com/astral-sh/uv/pull/2684 that only applies to local
source trees for now, which enables workspace-like workflows (whereby
local packages can depend on other local packages at arbitrary depth).
Closes#2699.
## Test Plan
Added new tests.
Also cloned this MRE that was shared with me
(https://github.com/timothyjlaurent/uv-poetry-monorepo-mre), and
verified that it was installed without error:
```
❯ cargo run pip install ./uv-poetry-monorepo-mre/app --no-cache
Finished dev [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/uv pip install ./uv-poetry-monorepo-mre/app --no-cache`
Resolved 4 packages in 1.28s
Built app @ file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/app
Built lib1 @ file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib1
Built lib2 @ file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib2 Downloaded 4 packages in 457ms
Installed 4 packages in 2ms
+ app==0.1.0 (from file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/app)
+ lib1==0.1.0 (from file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib1)
+ lib2==0.1.0 (from file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib2)
+ ruff==0.3.4
```
Previously, we did not consider installed distributions as candidates
while performing resolution. Here, we update the resolver to use
installed distributions that satisfy requirements instead of pulling new
distributions from the registry.
The implementation details are as follows:
- We now provide `SitePackages` to the `CandidateSelector`
- If an installed distribution satisfies the requirement, we prefer it
over remote distributions
- We do not want to allow installed distributions in some cases, i.e.,
upgrade and reinstall
- We address this by introducing an `Exclusions` type which tracks
installed packages to ignore during selection
- There's a new `ResolvedDist` wrapper with `Installed(InstalledDist)`
and `Installable(Dist)` variants
- This lets us pass already installed distributions throughout the
resolver
The user-facing behavior is thoroughly covered in the tests, but
briefly:
- Installing a package that depends on an already-installed package
prefers the local version over the index
- Installing a package with a name that matches an already-installed URL
package does not reinstall from the index
- Reinstalling (--reinstall) a package by name _will_ pull from the
index even if an already-installed URL package is present
- To reinstall the URL package, you must specify the URL in the request
Closes https://github.com/astral-sh/uv/issues/1661
Addresses:
- https://github.com/astral-sh/uv/issues/1476
- https://github.com/astral-sh/uv/issues/1856
- https://github.com/astral-sh/uv/issues/2093
- https://github.com/astral-sh/uv/issues/2282
- https://github.com/astral-sh/uv/issues/2383
- https://github.com/astral-sh/uv/issues/2560
## Test plan
- [x] Reproduction at `charlesnicholson/uv-pep420-bug` passes
- [x] Unit test for editable package
([#1476](https://github.com/astral-sh/uv/issues/1476))
- [x] Unit test for previously installed package with empty registry
- [x] Unit test for local non-editable package
- [x] Unit test for new version available locally but not in registry
([#2093](https://github.com/astral-sh/uv/issues/2093))
- ~[ ] Unit test for wheel not available in registry but already
installed locally
([#2282](https://github.com/astral-sh/uv/issues/2282))~ (seems
complicated and not worthwhile)
- [x] Unit test for install from URL dependency then with matching
version ([#2383](https://github.com/astral-sh/uv/issues/2383))
- [x] Unit test for install of new package that depends on installed
package does not change version
([#2560](https://github.com/astral-sh/uv/issues/2560))
- [x] Unit test that `pip compile` does _not_ consider installed
packages
It turns out that #2712 did _not_ fix#2711. After I put up #2712, I
started trying to track down the specific change that caused the
failure. I had assumed at first that it was related to one of our `rkyv`
types, but it actually ended up being one of our msgpack caches. I think
the failure mode is still fundamentally the same idea: the cached data
changed in a way that is still valid msgpack, but got interpreted
differently after deserializing.
The specific change that caused this was the [removal of a field] from
our
metadata type.
Ideally we would just undo the change and add the field back. But that
change has already been shipped out to users. So I believe the only
plausible choice at this point is to bump the `built-wheels` cache. This
will unfortunately mean that `uv` will need to re-build wheels.
Fixes#2711
[removal of a field]:
365c292525 (diff-e42586829f9c2cdbb909bedc5cf95691cc415247f2cbc2ebeb80d887020457bbL29)
It seems likely that we forgot to bump the version of the "simple" cache
in the 0.1.25 release. I'm still working on confirming it, but I figured
I'd get this bump up first.
The main problem here is that our "simple" cache is represented by
`rkyv`, and that in turn is tightly coupled to the representation of a
selection of data types in `uv`. Changing those data types without
bumping the cache version can result in cache deserialization errors
like this, or in the worst case, silent logic errors.
One possibility here is that the representation changed in a way that
permitted it to pass `rkyv` validation, but changed how the data itself
is interpreted. Our cache is robust with respect to `rkyv` validation
(if it fails, the cache will invalidate the entry and self-heal), but
being robust to higher level logical errors in interpretation of the
data is a much more significant challenge. Our best bet there is perhaps
some kind of checksum that we could do on top of `rkyv` validation (or
instead of it), and thus convert silent logical changes in how the data
is interpreted into failure modes that we're already robust to.
Fixes#2711
## Summary
This looks like a big change but it really isn't. Rather, I just split
`get_or_build_wheel` into separate `get_wheel` and `build_wheel` methods
internally, which made `get_or_build_wheel_metadata` capable of _not_
relying on `Tags`, which in turn makes it easier for us to use the
`DistributionDatabase` in various places without having it coupled to an
interpreter or environment (something we already did for
`SourceDistributionBuilder`).
## Summary
This PR enables the resolver to "accept" URLs, prereleases, and local
version specifiers for direct dependencies of path dependencies. As a
result, `uv pip install .` and `uv pip install -e .` now behave
identically, in that neither has a restriction on URL dependencies and
the like.
Closes https://github.com/astral-sh/uv/issues/2643.
Closes https://github.com/astral-sh/uv/issues/1853.
## Summary
This PR removes the custom `DistFinder` that we use in `pip sync`. This
originally existed because `VersionMap` wasn't lazy, and so we saved a
lot of time in `DistFinder` by reading distribution data lazily. But
now, AFAICT, there's really no benefit. Maintaining `DistFinder` means
we effectively have to maintain two resolvers, and end up fixing bugs in
`DistFinder` that don't exist in the `Resolver` (like #2688.
Closes#2694.
Closes#2443.
## Test Plan
I ran this benchmark a bunch. It's basically a wash. Sometimes one is
faster than the other.
```
❯ python -m scripts.bench \
--uv-path ./target/release/main \
--uv-path ./target/release/uv \
scripts/requirements/compiled/trio.txt --min-runs 50 --benchmark install-warm --warmup 25
Benchmark 1: ./target/release/main (install-warm)
Time (mean ± σ): 54.0 ms ± 10.6 ms [User: 8.7 ms, System: 98.1 ms]
Range (min … max): 45.5 ms … 94.3 ms 50 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 2: ./target/release/uv (install-warm)
Time (mean ± σ): 50.7 ms ± 9.2 ms [User: 8.7 ms, System: 98.6 ms]
Range (min … max): 44.0 ms … 98.6 ms 50 runs
Warning: The first benchmarking run for this command was significantly slower than the rest (77.6 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.
Summary
'./target/release/uv (install-warm)' ran
1.06 ± 0.29 times faster than './target/release/main (install-warm)'
```
I stumbled across this when writing tests for
`--emit-marker-expressions`. Namely, I observed that in CI, the `tzdata`
dependency of `pendulum` wasn't included in the `requirements.txt`
output on Windows.
@konstin [suggested] that this was a bug, so I've created a test for it.
In particular, it looks like [`tzdata` is an unconditional dependency of
`pendulum`][tzdata-unconditional].
[suggested]:
https://github.com/astral-sh/uv/pull/2651#discussion_r1539722464
[tzdata-unconditional]:
e646afbd165e58327bc5c698731107/pendulum-3.0.0-cp310-none-win_amd64.whl/pendulum-3.0.0.dist-info/METADATA#line.12
## Summary
In `pip sync`, we weren't properly handling cases in which a package
_only_ existed in `--find-links` (e.g., the user passed `--offline` or
`--no-index`).
I plan to explore removing `Finder` entirely to avoid these mismatch
bugs between `pip sync` and other commands, but this is fine for now.
Closes https://github.com/astral-sh/uv/issues/2688.
## Test Plan
`cargo test`
Unfortunately these tests are all gated on specific platforms because
the marker expressions they generate are, by design, platform specific.
I think we'll eventually want to figure out a more robust testing
strategy for multi-platform locking (of which this is just the tiniest
of first steps), but I don't think we really have the infrastructure for
that in place yet. That is, we don't yet have a way of generating a
marker expression _for_ a particular environment instead of just the one
that happens to _be_ the current environment.
When enabled, the marker expression for the pinned requirements
is written as a comment at the top of the output. It is disabled
by default *and* hidden because it's not clear whether 1) this is
useful to end users and 2) is an interface we want to commit to.
However, it is useful to expose it in some way so that it can be
tested.
For $reasons, we'll want to be able to clone a `Manifest` so
that it can be re-used to generate a marker expression.
There is likely a refactoring that could be done to avoid the
cloning, but a `Manifest` is likely to be small in practice, and
we'll only need to clone it once.
These are useful for converting lower level marker values types
to their corresponding values from a marker environment.
We'll use these for generating marker expressions based on both
the dependency graph and the current marker environment.
## Summary
Now that we're resolving metadata more aggressively for local sources,
it's worth doing this. We now pull metadata from the `pyproject.toml`
directly if it's statically-defined.
Closes https://github.com/astral-sh/uv/issues/2629.
The snapshot filtering situation has gotten way out of hand, with each
test hand-rolling it's own filters on top of copied cruft from previous
tests.
I've attempted to address this holistically:
- `TestContext.filters()` has everything you should need
- This was introduced a while ago, but needed a few more filters for it
to be generalized everywhere
- Using `INSTA_FILTERS` is **not recommended** unless you do not want
the context filters
- It is okay to extend these filters for things unrelated to paths
- If you have to write a custom path filter, please highlight it in
review so we can address it in the common module
- `TestContext.site_packages()` gives cross-platform access to the
site-packages directory
- Do not manually construct the path to site-packages from the venv
- Do not turn off tests on Windows because you manually constructed a
Unix path to site-packages
- `TestContext.workspace_root` gives access to uv's repository directory
- Use this for installing from `scripts/packages/`
- If you need coverage for relative paths, copy the test package into
the `temp_dir` don't change the working directory of the test fixture
There is additional work that can be done here, such as:
- Auditing and removing additional uses of `INSTA_FILTERS`
- Updating manual construction of `Command` instances to use a utility
- The `venv` tests are particularly frightening in their lack of a test
context and could use some love
- Improving the developer experience i.e. apply context filters to
snapshots by default
This is driving me a little crazy and is becoming a larger problem in
#2596 where I need to move more types (like `Upgrade` and `Reinstall`)
into this crate. Anything that's shared across our core resolver,
install, and build crates needs to be defined in this crate to avoid
cyclic dependencies. We've outgrown it being a single file with some
shared traits.
There are no behavioral changes here.
If you pass a `pyproject.toml` that use Hatch's context formatting API,
we currently fail because the dependencies aren't valid under PEP 508.
This PR makes the static metadata parsing a little more relaxed, so that
we appropriately fall back to PEP 517 there.
## Summary
Passing `pyproject.toml` or `setup.py` to `pip uninstall` is a bit
strange, since it will often require running a resolution to resolve the
dependencies (e.g., build the project), which means we also need to
accept `--index-url` and friends.
## Summary
Hatch allows for highly dynamic customization of metadata via hooks. In
such cases, Hatch
can't upload the PEP 517 contract, in that the metadata Hatch would
return by
`prepare_metadata_for_build_wheel` isn't guaranteed to match that of the
built wheel.
Hatch disables `prepare_metadata_for_build_wheel` entirely for pip.
We'll instead disable
it on our end when metadata is defined as "dynamic" in the
pyproject.toml, which should
allow us to leverage the hook in _most_ cases while still avoiding
incorrect metadata for
the remaining cases.
Closes: https://github.com/astral-sh/uv/issues/2130.
## Summary
When a user passes a `pyproject.toml` to `pip compile` (e.g., `uv pip
compile pyproject.toml`), we extract the requirements from the
`pyproject.toml` directly. However... that isn't always possible (as
seen in the linked issues). When it's _not_, we instead need to run the
PEP 517 build hooks to identify the metadata.
Closes https://github.com/astral-sh/uv/issues/1624.
Closes https://github.com/astral-sh/uv/issues/1644.
## Test Plan
`cargo test`
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
- Displays missing packages as single-line warnings.
- Adds support for `Editable project location` and `Required-by` fields
in `pip show`.
Part of #2526.
## Summary
`uv` was failing to install requirements defined like:
```
file://localhost/Users/crmarsh/Downloads/iniconfig-2.0.0-py3-none-any.whl
```
Closes https://github.com/astral-sh/uv/issues/2652.
While writing tests for a new flag (`--emit-marker-expression`) for `uv
pip compile`, I noticed that one of my test cases (`pendulum 3.0.0`)
was published in Dec 2023. I wanted to include this package in my
tests, but since it comes after our `EXCLUDE_NEWER` constant, it wasn't
visible to `uv`.
In this PR, I chose to resolve this by bumping `EXCLUDE_NEWER` to
`2024-03-25T00:00:00Z`. I also considered a couple other options:
* For a specific test, override and provide a custom `--exclude-newer`
flag. I felt like this would maybe be okay, but we could easily wind
up in a situation where we do this a lot and have a bunch of different
`--exclude-newer` flags in our tests. I'm not sure if this is a huge
problem in practice. Maybe it's fine.
* Find another package (or invent one) with a similarly interesting
configuration. It seemed easier to just bump `EXCLUDE_NEWER`.
The way I did this was to run `cargo insta test` after bumping
`EXCLUDE_NEWER`.
I then reviewed the snapshot diffs, and if they looked reasonable, I
accepted them.
There was only one case where I changed the test to preserve what I
thought it
was trying to test. That's isolated in its own commit.
With https://github.com/pubgrub-rs/pubgrub/pull/190, pubgrub attaches
all types to a dependency provider to reduce the number of generics. We
need a dummy dependency provider now to emulate this. On the plus side,
pep440_rs drops its pubgrub dependency.
This test was introduced in 42973cd9cb. It
looks like it compares some values against some platform specific code
that attempts to find the OS version. But the comparisons made some
assumptions about what kind of data is available. In this commit, we try
to make the test a little more flexible on Linux by not assuming that
`Option` values are `Some`.
## Summary
I don't see a great reason to allow this, and it adds a lot of
complexity, so `pyproject.toml` files are now limited to `pip compile`
and `pip install -r` -- they can't be passed as `-c` or `--override`.
## Summary
Closes Issue:
- https://github.com/astral-sh/uv/issues/2626
## Test Plan
```
cargo run -- pip install -r dev-requirements.txt -r requirements.txt
```
where both requirements files have same `--index-url`
We put a `.gitignore` with `*` at the top of our cache. When maturin was
building a source distribution inside the cache, it would walk up the
tree to find a gitignore, see that and ignore all python files. We now
add an (empty) `.git` directory one directory below, in the root of
built-wheels cache. This prevents ignore walking further up (it marks
the top level a git repository).
Deptry (from #2490) is a mid sized rust package with additional python
packages, so instead of using it in the test i've replaced it with a
small (44KB total) reproducer that uses cffi for faster building, the
entire test taking <2s on my machine.
Fixes#2490
## Summary
Ensures that (e.g.) installs from conda-forge, Homebrew, and other
distributions don't expose `uv self update` at all.
We'll still show `uv self update` for `pip install uv`, but it will fail
with a good error. Removing the `uv self update` from `pip`-installed
`uv` is more complicated, since we'd need to build separately for the
installer vs. for PyPI.
Closes#2588.
## Summary
This PR enables the source distribution database to be used with unnamed
requirements (i.e., URLs without a package name). The (significant)
upside here is that we can now use PEP 517 hooks to resolve unnamed
requirement metadata _and_ reuse any computation in the cache.
The changes to `crates/uv-distribution/src/source/mod.rs` are quite
extensive, but mostly mechanical. The core idea is that we introduce a
new `BuildableSource` abstraction, which can either be a distribution,
or an unnamed URL:
```rust
/// A reference to a source that can be built into a built distribution.
///
/// This can either be a distribution (e.g., a package on a registry) or a direct URL.
///
/// Distributions can _also_ point to URLs in lieu of a registry; however, the primary distinction
/// here is that a distribution will always include a package name, while a URL will not.
#[derive(Debug, Clone, Copy)]
pub enum BuildableSource<'a> {
Dist(&'a SourceDist),
Url(SourceUrl<'a>),
}
```
All the methods on the source distribution database now accept
`BuildableSource`. `BuildableSource` has a `name()` method, but it
returns `Option<&PackageName>`, and everything is required to work with
and without a package name.
The main drawback of this approach (which isn't a terrible one) is that
we can no longer include the package name in the cache. (We do continue
to use the package name for registry-based distributions, since those
always have a name.). The package name was included in the cache route
for two reasons: (1) it's nice for debugging; and (2) we use it to power
`uv cache clean flask`, to identify the entries that are relevant for
Flask.
To solve this, I changed the `uv cache clean` code to look one level
deeper. So, when we want to determine whether to remove the cache entry
for a given URL, we now look into the directory to see if there are any
wheels that match the package name. This isn't as nice, but it does work
(and we have test coverage for it -- all passing).
I also considered removing the package name from the cache routes for
non-registry _wheels_, for consistency... But, it would require a cache
bump, and it didn't feel important enough to merit that.
## Summary
Detects unused cache entries, which can come in a few forms:
1. Directories that are out-dated via our versioning scheme.
2. Old source distribution builds (i.e., we have a more recent version).
3. Old wheels (stored in `archive-v0`, but not symlinked-to from
anywhere in the cache).
Closes https://github.com/astral-sh/puffin/issues/1059.
Closes#2566
We were storing the username e.g. `charlie@astral.sh` as a
percent-encoded string `charlie%40astral.sh` which resulted in different
headers and broke JFrog's artifactory which apparently does not decode
usernames.
Tested with a JFrog artifactory and AWS CodeArtifact although it is
worth noting that AWS does _not_ have a username with an `@` — it'd be
nice to test another artifactory with percent-encoded characters in the
username and/or password.
## Summary
In `pip uninstall`, we shouldn't need to resolve unnamed requirements,
since we already index packages in `site-packages` by their URL.
This also changes `uninstall` to ignore editables, which matches pip's
behavior.
Part of: https://github.com/astral-sh/uv/issues/313.
## Test Plan
Run `cargo run pip install ./scripts/editable-installs/black_editable`,
followed by each of the following:
- `cargo run pip uninstall ./scripts/editable-installs/black_editable`
- `cargo run pip uninstall black`
- `cargo run pip uninstall ./scripts/editable-installs/black_editable
black`
## Summary
This PR ensures that if a package is already satisfied by the current
environment, we don't bother resolving the named requirement.
Part of: https://github.com/astral-sh/uv/issues/313.
## Test Plan
- `cargo run pip install ./scripts/editable-installs/black_editable`
- `cargo run pip install black --verbose`
## Summary
For example: `cargo run pip install .`
The strategy taken here is to attempt to extract the package name from
the distribution without executing the PEP 517 build steps. We could
choose to do that in the future if this proves lacking, but it adds
complexity.
Part of: https://github.com/astral-sh/uv/issues/313.
## Summary
This PR enables `uv pip install` to accept unnamed requirements, as long
as the requirement ends with the wheel or source distribution archive
name. For example: `cargo run pip install
~/Downloads/anyio-4.3.0.tar.gz`.
In subsequent PRs, I'll expand the scope of supported archives and
patterns.
Part of: https://github.com/astral-sh/uv/issues/313.
## Summary
First piece of https://github.com/astral-sh/uv/issues/313. In order to
support unnamed requirements, we need to be able to parse them in
`requirements-txt`, which in turn means that we need to introduce a new
type that's distinct from `pep508::Requirement`, given that these
_aren't_ PEP 508-compatible requirements.
Part of: https://github.com/astral-sh/uv/issues/313.
Scott schafer got me the idea: We can avoid repeating the path for
workspaces dependencies everywhere if we declare them in the virtual
package once and treat them as workspace dependencies from there on.
It is a common pattern to have an active conda base env (that sets
`CONDA_PREFIX`) and then create a venv on top of that (setting
`VIRTUAL_ENV`).
Previously, we would error when both `VIRTUAL_ENV` and `CONDA_PREFIX`
were set, now `VIRTUAL_ENV` takes precedence over `CONDA_PREFIX`.
Fixes#2028
## Summary
We would like to be able to configure the installer-name so that other
tools can co-exist with `uv`. In `pixi` we would like to use `pixi-uv`
as the installer name, for example, to be able to distinguish them from
packages installed by pure `uv`.
## Summary
This PR changes our user-facing representation for paths to use relative
paths, when the path is within the current working directory. This
mirrors what we do in Ruff. (If the path is _outside_ the current
working directory, we print an absolute path.)
Before:
```shell
❯ uv venv .venv2
Using Python 3.12.2 interpreter at: /Users/crmarsh/workspace/uv/.venv/bin/python3
Creating virtualenv at: .venv2
Activate with: source .venv2/bin/activate
```
After:
```shell
❯ cargo run venv .venv2
Finished dev [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/uv venv .venv2`
Using Python 3.12.2 interpreter at: .venv/bin/python3
Creating virtualenv at: .venv2
Activate with: source .venv2/bin/activate
```
Note that we still want to use the existing `.simplified_display()`
anywhere that the path is being simplified, but _still_ intended for
machine consumption (e.g., when passing to `.current_dir()`).
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This adds support for `CUSTOM_COMPILE_COMMAND` support to change the
header comment in generated requirements files.
See Issue:
- #1527
From [pip-tools docs](https://pip-tools.readthedocs.io/en/latest/):
> You might be wrapping the pip-compile command in another script. To
avoid confusing consumers of your custom script you can override the
update command generated at the top of requirements files by setting the
CUSTOM_COMPILE_COMMAND environment variable.
## Test Plan
<!-- How was it tested? -->
See unit test included
---------
Co-authored-by: konsti <konstin@mailbox.org>
## Summary
If you have a file `typing.py` in the current working directory, `python
-m` doesn't work in some Python versions:
```sh
❯ python -m foo
Could not import runpy module
Traceback (most recent call last):
File "/Users/crmarsh/.local/share/rtx/installs/python/3.9.18/lib/python3.9/runpy.py", line 15, in <module>
import importlib.util
File "/Users/crmarsh/.local/share/rtx/installs/python/3.9.18/lib/python3.9/importlib/util.py", line 2, in <module>
from . import abc
File "/Users/crmarsh/.local/share/rtx/installs/python/3.9.18/lib/python3.9/importlib/abc.py", line 17, in <module>
from typing import Protocol, runtime_checkable
ImportError: cannot import name 'Protocol' from 'typing' (/Users/crmarsh/workspace/uv/typing.py)
```
This did _not_ cause problems for us on Python 3.11 or later, because we
set `PYTHONSAFEPATH`, which avoids adding the current working directory
to `sys.path`. However, on earlier versions, we _were_ failing with the
above. (It's important that we run interpreter discovery in the current
working directory, since doing otherwise breaks pyenv shims.)
The fix implemented here uses `-I` to run Python in isolated mode, which
is even stricter. The downside of isolated mode is that we currently
rely on setting `PYTHONPATH` to find the "fake module" that we create on
disk, and `-I` means `PYTHONPATH` is totally ignored. So, instead, we
run a script directly, and that _script_ injects the path we care about
into `PYTHONSAFEPATH`.
Closes https://github.com/astral-sh/uv/issues/2547.
Add a single job for for fast lint tools. Rustfmt for rust, ruff for
python formatting and linting, prettier avoids inconsistent formatter
changes between pycharm and vscode.
## Summary
We strip extras by default, but there are some valid use-cases in which
they're required (see the linked issue). This PR doesn't change our
default, but it does add `--no-strip-extras`, which lets users preserve
extras in the output requirements.
Closes https://github.com/astral-sh/uv/issues/1595.
## Summary
We had the right fixup for `torchsde`, but a subsequent fixup was making
it invalid. In general, we should apply as few of these as we can, so
lets stop as soon as we succeed.
Closes https://github.com/astral-sh/uv/issues/2546.
## Test Plan
`cargo run pip install torchsde==0.2.5 --verbose --reinstall -n
--verbose`
## Summary
I tried out `cargo shear` to see if there are any unused dependencies
that `cargo udeps` isn't reporting. It turned out, there are a few. This
PR removes those dependencies.
## Test Plan
`cargo build`
## Summary
In reality, there's no such thing as the `site-packages` directory for a
given virtualenv. Rather, Python defines both `purelib` and `platlib`,
where the former is for pure-Python packages and the latter is for
packages that contain native code. These are almost always set to the
same thing... but they don't _have_ to be, and in fact of Fedora they
are not.
This PR changes the `site_packages` method to return an iterator of
directories.
Closes https://github.com/astral-sh/uv/issues/2527.
## Summary
When a user runs with `--output-file` and `--generate-hashes`, we should
_only_ update the hashes if the pinned version itself changes.
Closes https://github.com/astral-sh/uv/issues/1530.
This PR handles the fragment part of the URL path.
It achieves this by splitting the fragment from the path before
normalization and parsing. It then sets the fragment back after the URL
has been parsed.
Resolve#2501
Brings us in-line with Python's behavior:
1. Prioritize `none` tags _after_ all of the relevant platform tags
2. Omit `none` tags for CPython versions less than the current version
3. Prioritize major (i.e. `py3-none`) version tags over minor (i.e.
`py3x-none`) version tags less than the current version
4. Add a `none-any` tag for the current CPython version
## Test plan
Tested on my Linux machine with a script to emit tags at the desired
glibc version:
```python
from packaging import tags
import re
exclude = re.compile("_(21|22|23|24|25|26|27|28|29|30|31|32|33|34|35|36|37|38|39)_")
for tag in tags.sys_tags():
if exclude.search(str(tag)):
continue
print(tag)
```
Then performed a diff with the snapshot in `tags.rs`
## Summary
Closes#1958
This adds linehaul metadata to uv's user-agent when pep 508 markers are
provided to the RegistryClientBuilder. Thanks to #2381, we were able to
leverage most information from markers and avoid inconsistency.
Linehaul is meant to be accompanying metadata pip sends in it's user
agent when talking to registries. You can see this output by running
something like `python -c 'from pip._internal.network.session import
user_agent; print(user_agent())'`.
In PyPI, this metadata processed by the
[linehaul-cloud-function](https://github.com/pypi/linehaul-cloud-function).
More info about linehaul can be found in #1958.
Below are some examples from pip:
* Linux GHA: `pip/24.0
{"ci":true,"cpu":"x86_64","distro":{"id":"jammy","libc":{"lib":"glibc","version":"2.35"},"name":"Ubuntu","version":"22.04"},"implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL
3.0.2 15 Mar
2022","python":"3.12.2","rustc_version":"1.76.0","system":{"name":"Linux","release":"6.5.0-1016-azure"}}`
* Windows GHA: `pip/24.0
{"ci":true,"cpu":"AMD64","implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL
3.0.13 30 Jan
2024","python":"3.12.2","rustc_version":"1.76.0","system":{"name":"Windows","release":"2022Server"}}`
* OSX GHA: `pip/24.0
{"ci":true,"cpu":"arm64","distro":{"name":"macOS","version":"14.2.1"},"implementation":{"name":"CPython","version":"3.12.2"},"installer":{"name":"pip","version":"24.0"},"openssl_version":"OpenSSL
3.0.13 30 Jan
2024","python":"3.12.2","rustc_version":"1.76.0","system":{"name":"Darwin","release":"23.2.0"}}`
Here's how uv results look like (sorry for the keys not having the same
order):
* Linux GHA: `uv/0.1.21
{"installer":{"name":"uv","version":"0.1.21"},"python":"3.12.2","implementation":{"name":"CPython","version":"3.12.2"},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":"Linux","release":"6.5.0-1016-azure"},"cpu":"x86_64","openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`
* Windows GHA: `uv/0.1.21
{"installer":{"name":"uv","version":"0.1.21"},"python":"3.12.2","implementation":{"name":"CPython","version":"3.12.2"},"distro":null,"system":{"name":"Windows","release":"2022Server"},"cpu":"AMD64","openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`
* OSX GHA: `uv/0.1.21
{"installer":{"name":"uv","version":"0.1.21"},"python":"3.12.2","implementation":{"name":"CPython","version":"3.12.2"},"distro":{"name":"macOS","version":"14.2.1","id":null,"libc":null},"system":{"name":"Darwin","release":"23.2.0"},"cpu":"arm64","openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`
Distro information (such as the one pip uses `from pip._vendor import
distro` to retrieve instead of `platform` module) was not retrieved from
markers. Instead, the linux release codename/name/version uses
`sys-info` crate, adding about 50us of extra overhead on linux. The
distro osx version re-used the [mac_os version
implementation](99c992e38b/crates/platform-host/src/mac_os.rs)
from #2381 which adds about 20us of overhead on osx. I tried to use
other crates to avoid re-introducing `mac_os.rs` but most of them didn't
yield satisfactory performance (40ms-60ms~) or had the wrong values
needed (e.g. darwin version vs osx version).
I also didn't add libc retrieval or rustc retrieval as those seem to add
substantial overhead due to querying `ldd` or `rustc`. PyPy version
detection was also not added to avoid adding extra overhead to [support
PyPy for
linehaul](https://github.com/pypa/pip/blob/24.0/src/pip/_internal/network/session.py#L123).
All other behavior was kept 1-1 to match what pip's linehaul
implementation does (as of 24.0). This also aligns with what was
discussed in #1958.
## Test Plan
Added new integration test to uv-client.
---------
Co-authored-by: konstin <konstin@mailbox.org>
## Summary
By running `get_interpreter_info.py` outside of the current working
directory, we seem to have broken pyenv shims.
Closes https://github.com/astral-sh/uv/issues/2488.
## Test Plan
Without this change (resolving to the Homebrew Python, even though we
start with a shim):
```
DEBUG Starting interpreter discovery for Python @ `python3.11`
DEBUG Probing interpreter info for: /Users/crmarsh/.pyenv/shims/python3.11
DEBUG Found Python 3.11.7 for: /Users/crmarsh/.pyenv/shims/python3.11
Using Python 3.11.7 interpreter at: /opt/homebrew/opt/python@3.11/bin/python3.11
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate
```
With this change:
```
DEBUG Starting interpreter discovery for Python @ `python3.11`
DEBUG Probing interpreter info for: /Users/crmarsh/.pyenv/shims/python3.11
DEBUG Found Python 3.11.1 for: /Users/crmarsh/.pyenv/shims/python3.11
Using Python 3.11.1 interpreter at: /Users/crmarsh/.pyenv/versions/3.11.1/bin/python3.11
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate
```
## Summary
If a package uses Hatch's `root.uri` feature, we currently error:
```toml
dependencies = [
"black @ {root:uri}/../black_editable"
]
```
Even though we're using PEP 517 hooks to get the metadata, which
_should_ support this. The problem is that we load the full
`PyProjectToml`, which means we parse the requirements, which means we
reject what looks like a relative URL in dependencies.
Instead, we should only enforce a limited subset of `pyproject.toml`
(arguably none).
Closes https://github.com/astral-sh/uv/issues/2475.
## Summary
This PR adds limited support for PEP 440-compatible local version
testing. Our behavior is _not_ comprehensively in-line with the spec.
However, it does fix by _far_ the biggest practical limitation, and
resolves all the issues that've been raised on uv related to local
versions without introducing much complexity into the resolver, so it
feels like a good tradeoff for me.
I'll summarize the change here, but for more context, see [Andrew's
write-up](https://github.com/astral-sh/uv/issues/1855#issuecomment-1967024866)
in the linked issue.
Local version identifiers are really tricky because of asymmetry.
`==1.2.3` should allow `1.2.3+foo`, but `==1.2.3+foo` should not allow
`1.2.3`. It's very hard to map them to PubGrub, because PubGrub doesn't
think of things in terms of individual specifiers (unlike the PEP 440
spec) -- it only thinks in terms of ranges.
Right now, resolving PyTorch and friends fails, because...
- The user provides requirements like `torch==2.0.0+cu118` and
`torchvision==0.15.1+cu118`.
- We then match those exact versions.
- We then look at the requirements of `torchvision==0.15.1+cu118`, which
includes `torch==2.0.0`.
- Under PEP 440, this is fine, because `torch @ 2.0.0+cu118` should be
compatible with `torch==2.0.0`.
- In our model, though, it's not, because these are different versions.
If we change our comparison logic in various places to allow this, we
risk breaking some fundamental assumptions of PubGrub around version
continuity.
- Thus, we fail to resolve, because we can't accept both `torch @ 2.0.0`
and `torch @ 2.0.0+cu118`.
As compared to the solutions we explored in
https://github.com/astral-sh/uv/issues/1855#issuecomment-1967024866, at
a high level, this approach differs in that we lie about the
_dependencies_ of packages that rely on our local-version-using package,
rather than lying about the versions that exist, or the version we're
returning, etc.
In short:
- When users specify local versions upfront, we keep track of them. So,
above, we'd take note of `torch` and `torchvision`.
- When we convert the dependencies of a package to PubGrub ranges, we
check if the requirement matches `torch` or `torchvision`. If it's
an`==`, we check if it matches (in the above example) for
`torch==2.0.0`. If so, we _change_ the requirement to
`torch==2.0.0+cu118`. (If it's `==` some other version, we return an
incompatibility.)
In other words, we selectively override the declared dependencies by
making them _more specific_ if a compatible local version was specified
upfront.
The net effect here is that the motivating PyTorch resolutions all work.
And, in general, transitive local versions work as expected.
The thing that still _doesn't_ work is: imagine if there were _only_
local versions of `torch` available. Like, `torch @ 2.0.0` didn't exist,
but `torch @ 2.0.0+cpu` did, and `torch @ 2.0.0+gpu` did, and so on.
`pip install torch==2.0.0` would arbitrarily choose one one `2.0.0+cpu`
or `2.0.0+gpu`, and that's correct as per PEP 440 (local version
segments should be completely ignored on `torch==2.0.0`). However, uv
would fail to identify a compatible version. I'd _probably_ prefer to
fix this, although candidly I think our behavior is _ok_ in practice,
and it's never been reported as an issue.
Closes https://github.com/astral-sh/uv/issues/1855.
Closes https://github.com/astral-sh/uv/issues/2080.
Closes https://github.com/astral-sh/uv/issues/2328.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Resolves https://github.com/astral-sh/uv/issues/2391
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Added a few tests to make sure that the exit code returned is 0 when
there's no conflict; 1 when there's any conflict.
<!-- How was it tested? -->
## Summary
Right now, the middleware doesn't apply credentials that were
_originally_ sourced from a URL. This requires that we call
`with_url_encoded_auth` whenever we create a request to ensure that any
credentials that were passed in as part of an index URL (for example)
are respected.
This PR modifies `uv-auth` to instead apply those credentials in the
middleware itself. This seems preferable to me. As far as I can tell, we
can _only_ add in-URL credentials to the store ourselves (since in-URL
credentials are converted to headers by the time they reach the
middleware). And if we ever _didn't_ apply those credentials to new
URLs, it'd be a bug in the logic that precedes the middleware (i.e., us
forgetting to call `with_url_encoded_auth`).
## Test Plan
`cargo run pip install` with an authenticated index.
## Summary
This PR attempts to use a similar trick to that we added in
https://github.com/astral-sh/uv/pull/1878, but for post-releases.
In https://github.com/astral-sh/uv/pull/1878, we added a fake "minimum"
version to enable us to treat `< 1.0.0` as _excluding_ pre-releases of
1.0.0.
Today, on `main`, we accept post-releases and local versions in `>
1.0.0`. But per PEP 440, that should _exclude_ post-releases and local
versions, unless the specifier is itself a pre-release, in which case,
pre-releases are allowed (e.g., `> 1.0.0.post0` should allow `>
1.0.0.post1`).
To support this, we add a fake "maximum" version that's greater than all
the post and local releases for a given version. This leverages our last
remaining free bit in the compact representation.
`path_segments_mut` returns an `Err` for cannot-be-a-base URLs. These
won't be valid when we try to fetch them anyway, but we need to avoid a
panic.
Closes https://github.com/astral-sh/uv/issues/2460.
## Summary
It turns out that pip does _not_ validate the normalization of the
version specifier in the `.dist-info` directory. In particular, it seems
that some tools replace the `+` in a local version segment with a `_`.
Closes https://github.com/astral-sh/uv/issues/2424.
## Summary
The authentication middleware extracts in-URL credentials from URLs that
pass through it; however, by the time a request reaches the store, the
credentials will have already been removed, and relocated to the header.
So we were never propagating in-URL credentials.
This PR adds an explicit pass wherein we pass in-URL credentials to the
store prior to doing any work.
Closes https://github.com/astral-sh/uv/issues/2444.
## Test Plan
`cargo run pip install` against an authenticated AWS registry.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Adds basic keyring auth support for `uv` commands. Adds clone of `pip`'s
`--keyring-provider subprocess` argument (using CLI `keyring` tool).
See issue: https://github.com/astral-sh/uv/issues/1520
## Test Plan
<!-- How was it tested? -->
Hard to write full-suite unit tests due to reliance on
`process::Command` for `keyring` cli
Manually tested end-to-end in a project with GCP artifact registry using
keyring password:
```bash
➜ uv pip uninstall watchdog
Uninstalled 1 package in 46ms
- watchdog==4.0.0
➜ cargo run -- pip install --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
Finished dev [unoptimized + debuginfo] target(s) in 0.18s
Running `target/debug/uv pip install --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
error: HTTP status client error (401 Unauthorized) for url (https://<redacted>/pypi-mirror/simple/watchdog/)
➜ cargo run -- pip install --keyring-provider subprocess --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
Finished dev [unoptimized + debuginfo] target(s) in 0.17s
Running `target/debug/uv pip install --keyring-provider subprocess --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
Resolved 1 package in 2.34s
Installed 1 package in 27ms
+ watchdog==4.0.0
```
`requirements.txt`
```
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# .bin/generate-requirements
#
--index-url https://<redacted>/python/simple/
--extra-index-url https://<redacted>/pypi-mirror/simple/
...
```
```bash
➜ cargo run -- pip install --keyring-provider subprocess -r requirements.txt
Finished dev [unoptimized + debuginfo] target(s) in 0.19s
Running `target/debug/uv pip install --keyring-provider subprocess -r requirements.txt`
Resolved 205 packages in 23.52s
Built <redacted>
...
Downloaded 47 packages in 19.32s
Installed 195 packages in 276ms
+ <redacted>
...
```
---------
Co-authored-by: Thomas Gilgenast <thomas@vant.ai>
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
Django is actually pretty large (the wheel is 8MB, the source
distribution is 10MB). There's nothing specific to Django in any of
these tests, so this just replaces it with a much smaller dependency.
We should prune these down eventually since the scenarios cover a lot of
this -- this is just a bandaid.
## Summary
This may be required elsewhere, but all the traces in that issue are
related to persisting the temporary directory to our persistent cache,
so lets start there.
See: https://github.com/astral-sh/uv/issues/1491.
The architecture of uv does not necessarily match that of the python
interpreter (#2326). In cross compiling/testing scenarios the operating
system can also mismatch. To solve this, we move arch and os detection
to python, vendoring the relevant pypa/packaging code, preventing
mismatches between what the python interpreter was compiled for and what
uv was compiled for.
To make the scripts more manageable, they are now a directory in a
tempdir and we run them with `python -m` . I've simplified the
pypa/packaging code since we're still building the tags in rust. A
`Platform` is now instantiated by querying the python interpreter for
its platform. The pypa/packaging files are copied verbatim for easier
updates except a `lru_cache()` python 3.7 backport.
Error handling is done by a `"result": "success|error"` field that allow
passing error details to rust:
```console
$ uv venv --no-cache
× Can't use Python at `/home/konsti/projects/uv/.venv/bin/python3`
╰─▶ Unknown operation system `linux`
```
I've used the [maturin sysconfig
collection](855f6d2cb1/sysconfig)
as reference. I'm unsure how to test these changes across the wide
variety of platforms.
Fixes#2326
## Summary
Small follow up to https://github.com/astral-sh/uv/pull/2362 to check if
`SSL_CERT_FILE` is set to enable `--native-tls` functionality. This
maintains backwards compatibility with `0.1.17` and below users
leveraging only `SSL_CERT_FILE`.
Closes https://github.com/astral-sh/uv/issues/2400
## Test Plan
<!-- How was it tested? -->
Assuming `SSL_CERT_FILE` is already working via `--native-tls`, this is
simply a shortcut to enable `--native-tls` functionality implicitly
while still being able to let `rustls-native-certs` handle the loading
of `SSL_CERT_FILE` instead of ourselves.
Edit: Manually tested by setting up own self-signed CA certificate
bundle and set `SSL_CERT_FILE` to this and confirmed the loading happens
without having to specify `--native-tls`.
## Summary
Per [PEP 508](https://peps.python.org/pep-0508/), `python_version` is
just major and minor:
Right now, we're using the provided version directly, so if it's, e.g.,
`-p 3.11.8`, we'll inject the wrong marker. This was causing `pandas` to
omit `numpy` when `-p 3.11.8` was provided, since its markers look like:
```
Requires-Dist: numpy<2,>=1.22.4; python_version < "3.11"
Requires-Dist: numpy<2,>=1.23.2; python_version == "3.11"
Requires-Dist: numpy<2,>=1.26.0; python_version >= "3.12"
```
Closes https://github.com/astral-sh/uv/issues/2392.
## Summary
This PR ensures that we expand environment variables _before_ sniffing
for the URL scheme (e.g., `file://` vs. `https://` vs. something else).
Closes https://github.com/astral-sh/uv/issues/2375.
## Summary
This is a more robust fix for
https://github.com/astral-sh/uv/issues/2300.
The basic issue is:
- When we resolve, we attempt to pre-fetch the distribution metadata for
candidate packages.
- It's possible that the resolution completes _without_ those pre-fetch
responses. (In the linked issue, this was mainly because we were running
with `--no-deps`, but the pre-fetch was causing us to attempt to build a
package to get its dependencies. The resolution would then finish before
the build completed.)
- In that case, the `Index` will be marked as "waiting" for that
response -- but it'll never come through.
- If there's a subsequent call to the `Index`, to see if we should fetch
or are waiting for that response, we'll end up waiting for it forever,
since it _looks_ like it's in-flight (but isn't). (In the linked issue,
we had to build the source distribution for the install phase of `pip
install`, but `setuptools` was in this bad state from the _resolve_
phase.)
This PR modifies the resolver to ensure that we flush the stream of
requests before returning. Specifically, we now `join` rather than
`select` between the resolution and request-handling futures.
This _could_ be wasteful, since we don't _need_ those requests, but it
at least ensures that every `.wait` is followed by ` .done`. In
practice, I expect this not to have any significant effect on
performance, since we end up using the pre-fetched distributions almost
every time.
## Test Plan
I ran through the test plan from
https://github.com/astral-sh/uv/pull/2373, but ran the build 10 times
and ensured it never crashed. (I reverted
https://github.com/astral-sh/uv/pull/2373, since that _also_ fixes the
issue in the proximate case, by never fetching `setuptools` during the
resolve phase.)
I also added logging to verify that requests are being handled _after_
the resolution completes, as expected.
I also introduced an arbitrary error in `fetch` to ensure that the error
was immediately propagated.
## What
Adds a `--dry-run` flag that ejects out of the installation process
early (but after resolution) and displays only what *would have*
installed
## Closes
Closes#1244
## Out of Scope
I think it may be nice to include a `dry-run` flag for `uninstall` even
though `pip` doesn't implement this... thinking `Would uninstall X
packages: ...`
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
Follow-up for
395be442fc
adds `Requires` field to pip show output.
I've aimed to make it behave exactly the same as `pip` does for now, but
there seem to be subtle issues that may require some discussion going
forward:
- Should `uv pip show` support extras? `pip` has an open issue for it,
but currently does not support https://github.com/pypa/pip/issues/4824.
- Relatedly, `Requred-by` field (not implemented in this PR) in `pip
show` currently doesn't take the extras into account transparently, i.e.
when `PySocks` has been installed as an extra for `requests[socks]`,
`pip show PySocks` doesn't have `requests` or `requests[socks]` under
`Requred-by` field. Should `uv pip show` for now just replicate `pip`'s
behavior for now for simplicity and parity or try to cover the extras
for completeness?
## Test Plan
Added a couple of tests:
1. `requests==2.31.0` has four dependencies that would be ordered
differently unless sorted. Additionally, it has two dependencies that
are optionally included for extras.
2. `pandas==2.1.3` depends on different versions of `numpy` depending on
the python version used.
## Summary
It turns out that on macOS, reading the native certificates can add
hundreds of milliseconds to client initialization. This PR makes
`--native-tls` a command-line flag, to toggle (at runtime) the choice of
the `webpki` roots or the native system roots.
You can't accomplish this kind of configuration with the `reqwest`
builder API, so instead, I pulled out the heart of that logic from the
crate
(e319263851/src/async_impl/client.rs (L498)),
and modified it to allow toggling a choice of root.
Note that there's an open PR for this in reqwest
(https://github.com/seanmonstar/reqwest/pull/1848), along with an issue
(https://github.com/seanmonstar/reqwest/issues/1843), which I may ping,
but it's been around for a while and I believe reqwest is focused on its
next major release.
Closes https://github.com/astral-sh/uv/issues/2346.
## Summary
When running under `--no-deps`, we don't need to pre-fetch, because
pre-fetching fetches the _distribution_ metadata. But with `--no-deps`,
we only need the package metadata for the top-level requirements. We
never need distribution metadata.
Incidentally, this will fix https://github.com/astral-sh/uv/issues/2300.
## Test Plan
- `cargo test`
- `./target/debug/uv pip install --verbose --no-cache-dir --no-deps
--reinstall ddtrace==2.6.2 debugpy==1.8.1 ecdsa==0.18.0
editorconfig==0.12.4 --verbose` in a Python 3.10 Docker contain
repeatedly.
## Summary
This is analogous to #669, but for cases in which the package name is a
filesystem path. In such cases, we'll fail when parsing the _package
name_, since it doesn't start with a valid character, as opposed to
failing when we go to parse the remaining version specifier.
Inspired by https://github.com/astral-sh/uv/issues/2356.
## Summary
Add a new env variable `UV_SYSTEM` as alias for the cli argument
`--system`.
Use cases:
- No need to specify on each uv call inside the docker container the
`--system` flag
- It allows installing and configuring uv in a base container and in the
child containers nobody needs to know if you need the `--system` cli
flag
- The Home Assistant development env can be set up via devcontainer or a
venv. Both use some common scripts. Instead of adding duplicate or
special code to identify the dev container to set the `--system` flag,
it would be nicer to set it via an env variable.
I'm unfamiliar with Rust and tried to add the support by looking at the
code.
## Test Plan
I did test it manually
`UV_SYSTEM_PYTHON=true uv pip install requests`
## Summary
It turns out that when we iterate over the incompatibilities of a
package, PubGrub will _also_ show us the inverse dependencies. I suspect
this was rare, because we have a version check at the bottom... So, this
specifically required that you had some dependency that didn't end up
appearing in the output resolution, but that matched the version
constraints of the package you care about.
In this case, `langchain-community` depends on `langchain-core`. So we
were seeing an incompatibility like:
```rust
FromDependencyOf(Package(PackageName("langchain-community"), None, None), Range { segments: [(Included("0.0.10"), Included("0.0.10")), (Included("0.0.11"), Included("0.0.11"))] }, Package(PackageName("langchain-core"), None, None), Range { segments: [(Included("0.1.8"), Excluded("0.2"))] })
```
Where we were iterating over `langchain-core`, and looking for version
`0.0.11`... which happens to match `langchain-community`.
(`langchain-community was omitted from the resolution; hence, it didn't
exist in the map.)
Closes https://github.com/astral-sh/uv/issues/2358.
## Summary
We now initialize an HTTP client in advance for remote requirements
files. It turns out this adds a significant overhead, even for
operations like auditing the environment (at least on macOS).
This PR makes initialization lazy. After a lot of evaluation, I took the
easiest route, which is: we just pass in `Connectivity`, and then use
the default HTTP client. So we won't respect netrc files and anything
else that we get from our registry client. If we want to keep using the
registry client, we _can_, it's just way more ceremony to pass down a
closure.
See: https://github.com/astral-sh/uv/issues/2346.
## Test Plan
- Verified that `cargo run pip compile
https://raw.githubusercontent.com/ansible/ansible/f1ded0f41759235eb15a7d13dbc3c95dce5d5acd/requirements.txt`
completed without error.
- Verified that `cargo run pip compile
https://raw.githubusercontent.com/ansible/ansible/f1ded0f41759235eb15a7d13dbc3c95dce5d5acd/requirements.txt
--offline` failed with an error.
- Verified that `./target/release/uv pip install requests` completed in
0-2ms, rather than hundreds.
## Summary
In #2000, I shipped a regression whereby we stopped writing relative
paths for scripts within `data` directories. The net effect here is that
we aren't _uninstalling_ binaries in all cases. (This does _not_ apply
to entrypoints, only scripts in `data` directories.)
Closes https://github.com/astral-sh/uv/issues/2330.
## Test Plan
Most Python packages ship entrypoints, not binaries, so I don't know how
to test this cheaply. But I did test it locally by verifying that `uv`
is now removed from the `bin` directory after an uninstall.
## Summary
Some zip files can't be streamed; in particular, `rs-async-zip` doesn't
support data descriptors right now (though it may in the future). This
PR adds a fallback path for such zips that downloads the entire zip file
to disk, then unzips it from disk (which gives us `Seek`).
Closes https://github.com/astral-sh/uv/issues/2216.
## Test Plan
`cargo run pip install --extra-index-url https://buf.build/gen/python
hashb_foxglove_protocolbuffers_python==25.3.0.1.20240226043130+465630478360
--force-reinstall -n`
## Summary
The netrc middleware we added in
https://github.com/astral-sh/uv/pull/2241 has a slight problem. If you
include credentials in your index URL, _and_ in the netrc file, the
crate blindly adds the netrc credentials as a header. And given the
`ReqwestBuilder` API, this means you end up with _two_ `Authorization`
headers, which always leads to an invalid request, though the exact
failure can take different forms.
This PR removes the middleware crate in favor of our own middleware.
Instead of using the `RequestInitialiser` API, we have to use the
`Middleware` API, so that we can remove the header on the request
itself.
Closes https://github.com/astral-sh/uv/issues/2323.
## Test Plan
- Verified that running against a private index with credentials in the
URL (but no netrc file) worked without error.
- Verified that running against a private index with credentials in the
netrc file (but not the URL) worked without error.
- Verified that running against a private index with a mix of
credentials in both _also_ worked without error.
## Summary
We write this a few lines down with a value passed in by the caller. I
suspect I missed that this was already here (with a less accurate value)
when adding `INSTALLER`.
`packaging==24.0` came out which broke this test. It has to run without
`--exclude-newer` since it's testing an index that doesn't support it.
Instead, though, we can just disable dependencies, since the test still
exercises the same logic.
Preparing for #2058, i found it hard to follow where which discovery
function gets called. I moved all the discovery functions to a
`find_python` module (some exposed through `PythonEnvironment`) and
documented which subcommand uses which python discovery strategy.
No functional changes.
`install_extra_index_url_has_priority` started failing because
`packaging` had a new release. I'm not sure if this preserves the index
order check as intended, but it does unblock CI.
## Summary
No behavioral changes; just taking code that's duplicated between two
branches in `distribution_database.rs` and pulling it into its own
method.
## Summary
Addressing the extremely slow performance detailed in
https://github.com/astral-sh/uv/issues/2220. There are two changes to
increase download performance:
1. setting `accept-encoding: identity`, in the spirit of
https://github.com/pypa/pip/pull/1688
2. increasing buffer from 8KiB to 128KiB.
### 1. accept-encoding: identity
I think this related `pip` PR has a good explanation of what's going on:
https://github.com/pypa/pip/pull/1688
```
# We use Accept-Encoding: identity here because requests
# defaults to accepting compressed responses. This breaks in
# a variety of ways depending on how the server is configured.
# - Some servers will notice that the file isn't a compressible
# file and will leave the file alone and with an empty
# Content-Encoding
# - Some servers will notice that the file is already
# compressed and will leave the file alone and will add a
# Content-Encoding: gzip header
# - Some servers won't notice anything at all and will take
# a file that's already been compressed and compress it again
# and set the Content-Encoding: gzip header
```
The `files.pythonhosted.org` server is the 1st kind. Example debug log I
added in `uv` when installing against PyPI:
<img width="1459" alt="image"
src="https://github.com/astral-sh/uv/assets/12058921/ef10d758-46aa-4c8e-9dba-47f33437401b">
(there is no `content-encoding` header in this response, the `whl`
hasn't been compressed, and there is a content-length header)
Our internal mirror is the third case. It does seem sensible that our
mirror should be modified to act like the 1st kind. But `uv` should
handle all three cases like `pip` does.
### 2. buffer increase
In https://github.com/astral-sh/uv/issues/2220 I observed that `pip`'s
downloading was causing up-to 128KiB flushes in our mirror.
After fix 1, `uv` was still only causing up-to 8KiB flushes, and was
slower to download than `pip`. Increasing this buffer from the default
8KiB led to a download performance improvement against our mirror and
the expected observed 128KiB flushes.
## Test Plan
Ran benchmarking as instructed by @charliermarsh
<img width="1447" alt="image"
src="https://github.com/astral-sh/uv/assets/12058921/840d9c8d-4b98-4bfa-89f3-073a2dec1f23">
No performance improvement or regression.
## Summary
In #1813, we were failing to install `scikit-image==0.19.3` from source
in Python 3.11. Confusingly, though, the trace showed that the build
command exited with status 0...
The issue is that we get results from the PEP 517 hooks by reading from
`stdout` -- that is, we `print` at the end of the script, and parse the
printed output on the other side.
It turns out that for `scikit-image`, in this case, there was output
_after_ the wheel filename:
```
...
no previously-included directories found matching 'doc/gh-pages'
adding license file 'LICENSE.txt'
writing manifest file 'scikit_image.egg-info/SOURCES.txt'
Copying scikit_image.egg-info to build/bdist.macosx-12.6-arm64/wheel/scikit_image-0.19.3-py3.11.egg-info
running install_scripts
scikit_image-0.19.3-cp311-cp311-macosx_14_0_arm64.whl
INFO:
########### EXT COMPILER OPTIMIZATION ###########
INFO: Platform :
Architecture: aarch64
Compiler : clang
CPU baseline :
Requested : 'min'
Enabled : NEON NEON_FP16 NEON_VFPV4 ASIMD
Flags : none
Extra checks: none
CPU dispatch :
Requested : 'max -xop -fma4'
Enabled : ASIMDHP ASIMDDP ASIMDFHM
Generated : none
INFO: CCompilerOpt.cache_flush[864] : write cache to path -> /private/var/folders/nt/6gf2v7_s3k13zq_t3944rwz40000gn/T/.tmp5ZPIbv/built-wheels-v0/pypi/scikit-image/0.19.3/hLW_f7wWeGDOPRlSazQXw/scikit-image-0.19.3.tar.gz/build/temp.macosx-12.6-arm64-3.11/ccompiler_opt_cache_ext.py
```
We need the `scikit_image-0.19.3-cp311-cp311-macosx_14_0_arm64.whl`
line, but we were failing to find it due to all the extra output at the
end (presumedly, some kind of `atexit` logging).
This PR modifies the hooks to instead write their results to files that
are passed in by the parent. On the other end, we then read the results
back from disk. This makes it much more robust to "other" output in the
script.
Closes https://github.com/astral-sh/uv/issues/1813.
## Test Plan
Ran `cargo run pip install scikit-image==0.19.3 --reinstall
--no-cache-dir` on Python 3.11.
## Summary
It turns out that setuptools includes a shim to patch distutils. I'll
admit that I don't fully understand why or how it's different, but this
is the trick `pip` uses to ensure that it gets the "original" distutils.
We actually use distutils in two places: once for the system Python
scheme, and once for virtual environments. In virtualenv, they _do_ use
the patched distutils, so this could deviate in ways I don't understand.
Closes#2302.
Extends the "compatibility" types introduced in #1293 to apply to source
distributions as well as wheels.
- We now track the most-relevant incompatible source distribution
- Exclude newer, Python requirements, and yanked versions are all
tracked as incompatibilities in the new model (this lets us remove
`DistMetadata`!)
Since Python 3.7, deterministic pycs are possible (see [PEP
552](https://peps.python.org/pep-0552/))
To select the bytecode invalidation mode explicitly by env var:
PYC_INVALIDATION_MODE=UNCHECKED_HASH uv pip install --compile ...
Valid values are TIMESTAMP (default), CHECKED_HASH, and UNCHECKED_HASH.
The latter options are useful for reproducible builds.
---------
Co-authored-by: konstin <konstin@mailbox.org>
## Summary
PyPI now supports Metadata 2.2, which means distributions with Metadata
2.2-compliant metadata will start to appear. The upside is that if a
source distribution includes a `PKG-INFO` file with (1) a metadata
version of 2.2 or greater, and (2) no dynamic fields (at least, of the
fields we rely on), we can read the metadata from the `PKG-INFO` file
directly rather than running _any_ of the PEP 517 build hooks.
Closes https://github.com/astral-sh/uv/issues/2009.
This PR tweaks uv to support reading `requirements.txt` regardless of
whether it is encoded as UTF-8 or UTF-16. This is particularly relevant
on Windows where `uv pip freeze > requirements.txt` will likely write a
UTF-16 encoded `requirements.txt` file.
There is some discussion on #1666 where it's suggested that perhaps
we should explicitly not support this. I didn't see that until I
had already put this PR together, but even so, I think it's worth
considering this. UTF-16 is predominant on Windows. It is very easy
to produce a UTF-16 encoded file. Moreover, there is an easy and well
specified way to recognize and transcode UTF-16 encoded data to UTF-8.
I think the downside of this is that it could encourage the use UTF-16
encoded `requirements.txt` files *in addition* to UTF-8 encoded
files, and it would probably be nice to converge and standardize on
one encoding. One possible alternative to this PR is that we provide
a better error message. Another alternative is to ensure that a
`-o/--output` flag exists for all commands (neither `uv pip freeze` nor
`pip freeze` have such a flag) so that users can always write output
to a file without relying on their environment's piping behavior.
(Although this last alternative seems a little sad to me.)
It's also worth noting the [PEP-0508] doesn't seem to mention file
encoding at all. So I think from a "do the standards allow this"
perspective, this change is OK.
Finally, `pip` itself seems to work with UTF-16 encoded
`requirements.txt` files.
I think I personally overall lean towards supporting UTF-16 for
`requirements.txt` files. In part because I think it smoothes out the
UX a little bit, in part because there is no obvious specification
(that I'm aware of) that mandates that these files are UTF-8, and
finally in part because `pip` supports it too.
Fixes#1666, Fixes#2276
[PEP-0508]: https://peps.python.org/pep-0508/
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
refactor pip_list function to trim end of joined elements
fixed: https://github.com/astral-sh/uv/issues/2296
<!-- How was it tested? -->
## Summary
If a user provides a source distribution via a direct path, it can
either be an archive (like a `.tar.gz` or `.zip` file) or a directory.
If the former, we need to extract (e.g., unzip) the contents at some
point. Previously, this extraction was in `uv-build`; this PR lifts it
up to the distribution database.
The first benefit here is that various methods that take the
distribution are now simpler, as they can assume a directory.
The second benefit is that we no longer extract _multiple times_ when
working with a source distribution. (Previously, if we tried to get the
metadata, then fell back and built the wheel, we'd extract the wheel
_twice_.)
## Summary
This PR removes the URL conflict errors when the output of a `uv pip
compile` is used as a constraint to a subsequent `uv pip compile`.
If you run `uv pip compile`, the output file will contain your Git
dependencies, but pinned to a specific commit, like:
```
git+https://github.com/pallets/werkzeug@32e69512134c2f8183c6438b2b2e13fd24e9d19f
```
If you then use the output as a constraint to a subsequent resolution
(e.g., perhaps you require
`git+https://github.com/pallets/werkzeug@main`), we currently fail. I
think this is a reasonable workflow to support when all of these
requirements are coming from _your own_ dependencies. So we now assume
when resolving that the former is a more precise variant of the latter.
Closes https://github.com/astral-sh/uv/issues/1903.
Closes https://github.com/astral-sh/uv/issues/2266.
## Test Plan
`cargo test`
## Summary
This PR enables use of the Windows Store Pythons even with `py` is not
installed. Specifically, we need to ensure that the `python.exe` and
`python3.exe` executables installed into the
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApp` directory _are_ used
when they're not "App execution aliases" (which merely open the Windows
Store, to help you install Python).
When `py` is installed, this isn't strictly necessary, since the
"resolved" executables are discovered via `py`. These look like
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbs5n2kfra8p0\python.exe`.
Closes https://github.com/astral-sh/uv/issues/2264.
## Test Plan
- Removed all Python installations from my Windows machine.
- Uninstalled `py`.
- Enabled "App execution aliases".
- Verified that for both `cargo run venv --python python.exe` and `cargo
run venv --python python3.exe`, `uv` exited with a failure that no
Python could be found.
- Installed Python 3.10 via the Windows Store.
- Verified that the above commands succeeded without error.
- Verified that `cargo run venv --python python3.10.exe` _also_
succeeded.
- Now that `packse` is being published to PyPI we can install it from
there.
- Tweaks the tooling around scenario updates to manage a temporary
virtual environment for you.
- Makes use of a new index URL
- Includes local version segment scenarios (supersedes
https://github.com/astral-sh/uv/pull/2022)
Sometimes, the first time we read from the stdout of the bytecode
compiler python subprocess, we get an empty string back (no newline). If
we try to write to stdin, it will often be a broken pipe (#2245). After
we got an empty string the first time, we will get the same empty string
if we read a line again.
The details of this behavior are mysterious to me, but it seems that it
can be identified by the first empty string. We check by inserting
starting with a `Ready` message on the Python side. When we encounter
the broken state, we discard the interpreter and try again.
We have to introduce a third timeout check for the interpreter launch
itself.
Minimized test script:
```bash
#!/usr/bin/env bash
set -euo pipefail
while true; do
date --iso-8601=seconds # Progress indicator
rm -rf testenv
target/profiling/uv venv testenv -q --python 3.12
VIRTUAL_ENV=$PWD/testenv target/profiling/uv pip install -q --compile wheel==0.42.0
done
```
Run as
```
cargo build --profile profiling && bash compile_bug.sh
```
Fixes#2245
## Summary
In #2102, I did some refactor, and changed a method to return the Python
executable path rather than the parent directory path. But I missed this
one codepath for Conda on Windows.
Closes https://github.com/astral-sh/uv/issues/2269.
## Test Plan
- Installed micromamba on my Windows machine.
- Reproduced the failure in the linked issue.
- Verified that `python.exe` exists at `${CONDA_PREFIX}\python.exe`.
- Ran with this change; installed successfully.
## Summary
In the list of tags produced by `Tags::from_env`, higher-priority tags
are expected to come earlier in the list. Right now, though, we push
tags like `py38` before `py312`. So if you run `cargo run pip install
multiprocess -n --reinstall --verbose` on Python 3.12, you get the
`py38` wheel rather than the `py32` wheel.
Closes https://github.com/astral-sh/uv/issues/2261.
## Summary
This PR adds support for pip's `--no-build-isolation`. When enabled,
build requirements won't be installed during PEP 517-style builds, but
the source environment _will_ be used when executing the build steps
themselves.
Closes https://github.com/astral-sh/uv/issues/1715.
Fix parsing `pytest;'4.0'>=python_version`, where previously the
operator and the variable were incorrectly tokenized as one invalid
operator.
Fixes#2247
Running the pep508_rs tests was failing due to uv-fs depending on
`fs_err::tokio` even when not selected. But the function that used it is
unused anyway, so i removed it.
## Summary
It turns out that by keeping the `RECORD` file open, older versions of
Windows mark it for deletion, but don't allow it to be deleted until
it's closed. As such, we end up leaving the `.dist-info` directory
around, since it appears non-empty; but once the program terminates, we
_do_ delete `RECORD`, leaving it empty. This then creates the impression
that a package exists where it does not.
Closes https://github.com/astral-sh/uv/issues/2074.
## Summary
If we fallback to streaming the wheel (because the registry doesn't
support range requests), we currently don't cache the metadata at all.
This PR fixes that, ensuring that we cache based on the same HTTP
policies, etc.
## Summary
We're seeing reports that Sonatype Nexus isn't working with cached data.
Users are reporting 304 responses that show "Found modified response..."
path in the logs. I can't reproduce this on latest Sonatype Nexus, but
my best guess is that there's a 304 response that is failing our
validators, and we try to use that as if it's a "complete" response?
Closes https://github.com/astral-sh/uv/issues/1754.
## Summary
Add netrc support to the uv-client.
closes#1405
## Test Plan
I've added a corresponding test case to validate the correct header.
Furthermore a tested it against a real world private repository.
## Summary
Per the
[`EXTERNALLY-MANAGED`](https://packaging.python.org/en/latest/specifications/externally-managed-environments/)
spec, installers SHOULD add a `--break-system-packages` flag to allow
users to override the package manager warnings raised by
`EXTERNALLY-MANAGED`. This PR adds the flag to comply with the spec, and
enable system Python installs on newer versions of certain
distributions.
While this flag feels kind of bad, it's not necessarily a change in
behavior. We _already_ allow installing into these system distributions
-- it's just that `EXTERNALLY-MANAGED` doesn't exist for distributions
that were packaged prior to the spec, so we don't run into this problem.
Closes https://github.com/astral-sh/uv/issues/2234.
## Summary
This is no longer necessary as `AsyncHttpRangeReader` now accepts
`ClientWithMiddleware` -- which is good, because it means all relevant
middleware will be enforced (like offline, or `.netrc` in the future).
## Summary
Fix computing replacements pattern for pip_list tests to count
characters in the original directory string rather than the
regex::escape'd string. The latter yields incorrect results if the
workspace path contains characters such as `-` or `.`.
Fixes#2232
## Test Plan
`cargo test --test pip_list` in a directory named `uv-test` to provoke
the bug.
## Summary
Allow using http(s) urls for constraints and requirements files handed
to the CLI, by handling paths starting with `http://` or `https://`
differently. This allows commands for such as: `uv pip install -c
https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.8.txt
requests`.
closes#1332
## Test Plan
Testing install using a `constraints.txt` file hosted on github in the
airflow repository:
fbdc2eba8e/crates/uv/tests/pip_install.rs (L1440-L1484)
## Advice Needed
- filesystem/http dispatch is implemented at a relatively low level (at
`crates/uv-fs/src/lib.rs#read_to_string`). Should I change some naming
here so it is obvious that the function is able to dispatch?
- I kept the CLI argument for -c and -r as a PathBuf, even though now it
is technically either a path or a url. We could either keep this as is
for now, or implement a new enum for this case? The enum could then
handle dispatch to files/http.
- Using another abstraction layer like
https://docs.rs/object_store/latest/object_store/ for the
files/urls/[s3] could work as well, though I ran into a bug during
testing which I couldn't debug
## Summary
Follow up from discussion in https://github.com/astral-sh/uv/pull/2223
Detect CMD.exe by checking if `PROMPT` env var is set on windows,
otherwise assume it's PowerShell.
Note, this will not work if user modifies their system env vars to
include `PROMPT` by default or if they launch nested PowerShell from
Command Prompt (e.g. `Developer PowerShell for VS 2022`).
## Test Plan
Only tested locally, although we try to add some CI tests that
specifically use CMD.exe
Command Prompt
```
Microsoft Windows [Version 10.0.19044.3086]
(c) Microsoft Corporation. All rights reserved.
Z:\Users\samypr100\dev\uv>Z:\Users\samypr100\.cargo\bin\cargo.exe +stable run --color=always -- venv "Foo Bar"
Finished dev [unoptimized + debuginfo] target(s) in 0.69s
Running `target\debug\uv.exe venv "Foo Bar"`
Using Python 3.12.2 interpreter at: Z:\Users\samypr100\AppData\Local\Programs\Python\Python312\python.exe
Creating virtualenv at: Foo Bar
Activate with: "Foo Bar\Scripts\activate"
```
Power Shell
```
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS Z:\Users\samypr100\dev\uv>Z:\Users\samypr100\.cargo\bin\cargo.exe +stable run --color=always -- venv "Foo Bar"
Finished dev [unoptimized + debuginfo] target(s) in 0.63s
Running `target\debug\uv.exe venv "Foo Bar"`
Using Python 3.12.2 interpreter at: Z:\Users\samypr100\AppData\Local\Programs\Python\Python312\python.exe
Creating virtualenv at: Foo Bar
Activate with: & "Foo Bar\Scripts\activate"
```
## Summary
Implementation for https://github.com/astral-sh/uv/issues/1594
The output will contain only the name, version and location of the
packages for now but it should be extendable to include other
information in the future.
Quite inexperienced with Rust, so please forgive me if there are things
that obviously don't make sense 😭
## Test Plan
Added a bunch of unit tests. The exit code behavior matches `pip`'s
behavior:
- When the package is found -> exit code 0
- When the package isn't found -> exit code 1
- When one package is found but another isn't -> exit code 0
Implements `pip list --format=freeze` and `pip list --format=json`
Closes https://github.com/astral-sh/uv/issues/1970
## Test Plan
Extended existing `pip list` tests to match output.
Need to look at escaping in the Windows test 🪟
## Summary
`pip` uses `sysconfig` for Python 3.10 and later by default; however, it
falls back to `distutils` for earlier Python versions, and distros can
actually tell `pip` to continue falling back to `distutils` via the
`_PIP_USE_SYSCONFIG` variable.
By _always_ using `sysconfig`, we're doing the wrong then when
installing into some system Pythons, e.g., on Debian prior to Python
3.10.
This PR modifies our logic to mirror `pip` exactly, which is what's been
recommended to me as the right thing to do.
Closes https://github.com/astral-sh/uv/issues/2113.
## Test Plan
Most notably, the new Debian tests pass here (which fail on main:
https://github.com/astral-sh/uv/pull/2144).
I also added Pyston as a second stress-test.
## Summary
This PR migrates our virtualenv creation from a setup that assumes prior
knowledge of the correct paths, to a technique borrowed from
`virtualenv` whereby we use `sysconfig` and `distutils` to determine the
paths. The general trick is to grab the expected paths with `sysconfig`,
then make them all relative, then make them absolute for a given
directory.
Closes#2095.
Closes#2153.
## Summary
This makes `--python python3` and `--python 3.10` more consistent on
Windows.
Closes https://github.com/astral-sh/uv/issues/2213.
## Test Plan
Ran `cargo run venv --python python3.12` with the Windows Store Python.
## Summary
We have logic in `python_query.rs` to filter out Windows Store shims
when you use invocations like `-p 3.10`, but not `--python python3`,
which is uncommon but allowed on Windows.
Closes#2211.
## Summary
Our Windows shim detection wasn't catching shims like `python3.12.exe`.
Closes#2208.
## Test Plan
Installed Python 3.12 via the Windows Store; verified that `cargo run
venv --python 3.12` failed before but passes after this change.
## Summary
When determining "direct" dependencies, we need to ensure that we
respect markers. In the linked issue, the user had an optional
dependency like:
```toml
[project.optional-dependencies]
dev = [
"setuptools>=64",
"setuptools_scm>=8"
]
```
By not respecting markers, we tried to resolve `setuptools` to the
lowest-available version. However, since `setuptools>=64` _isn't_
enabled (since it's optional), we won't respect _that_ constraint.
To be consistent, we need to omit optional dependencies just as we will
at resolution time.
Closes https://github.com/astral-sh/uv/issues/2203.
## Test Plan
`cargo test`
Follow-up to #2086: Don't use timeouts for the entire workers, but only
for the section that's about communicating with the (potentially broken)
`python` subprocess. I've also raised the timeout to 60s.
Add a `--compile` option to `pip install` and `pip sync`.
I chose to implement this as a separate pass over the entire venv. If we
wanted to compile during installation, we'd have to make sure that
writing is exclusive, to avoid concurrent processes writing broken
`.pyc` files. Additionally, this ensures that the entire site-packages
are bytecode compiled, even if there are packages that aren't from this
`uv` invocation. The disadvantage is that we do not update RECORD and
rely on this comment from [PEP 491](https://peps.python.org/pep-0491/):
> Uninstallers should be smart enough to remove .pyc even if it is not
mentioned in RECORD.
If this is a problem we can change it to run during installation and
write RECORD entries.
Internally, this is implemented as an async work-stealing subprocess
worker pool. The producer is a directory traversal over site-packages,
sending each `.py` file to a bounded async FIFO queue/channel. Each
worker has a long-running python process. It pops the queue to get a
single path (or exists if the channel is closed), then sends it to
stdin, waits until it's informed that the compilation is done through a
line on stdout, and repeat. This is fast, e.g. installing `jupyter
plotly` on Python 3.12 it processes 15876 files in 319ms with 32 threads
(vs. 3.8s with a single core). The python processes internally calls
`compileall.compile_file`, the same as pip.
Like pip, we ignore and silence all compilation errors
(https://github.com/astral-sh/uv/issues/1559). There is a 10s timeout to
handle the case when the workers got stuck. For the reviewers, please
check if i missed any spots where we could deadlock, this is the hardest
part of this PR.
I've added `uv-dev compile <dir>` and `uv-dev clear-compile <dir>`
commands, mainly for my own benchmarking. I don't want to expose them in
`uv`, they almost certainly not the correct workflow and we don't want
to support them.
Fixes#1788Closes#1559Closes#1928
## Summary
We have at least one reported case of this happening. It's preferable
IMO to move on rather than fail hard despite sub-pbar registry behavior.
Closes https://github.com/astral-sh/uv/issues/2099.
## Summary
This will make it easier to use the paths returned by `distutils.py`
(for some cases). No code or behavior changes; just removing some fields
we don't need.
## Summary
Closes#1977
This allows us to send uv's version in the `uv-client` User Agent
header.
Here's how request headers look like to a server now:
```
...
Accept: application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html;q=0.2, text/html;q=0.01
User-Agent: uv/0.1.13
...
```
~~I went for a mix of Option 1 and 2 from #1977.~~ Open to alternative
naming as well, not tied too strongly here to the names picked.
~~Another possibility for this new crate is that we can use it to
consolidate metadata that exists across crates to ultimately be able to
create linehaul information described in #1958, but I haven't looked
into what those changes might look like.~~
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
<!-- How was it tested? -->
Added initial tests in the new crate to exercise its public API and
added a new test to uv-client to validate the headers using a 1-time
disposable server.
## Summary
Ensures that local dependencies function similarly to editables, in that
if they're `uv pip install`ed, we invalidate them.
Closes https://github.com/astral-sh/uv/issues/1651.
## Summary
Internal-only refactor to consolidate multiple codepaths we have for
checking whether a cached or installed entry is up-to-date with a local
requirement.
Error for `uv pip compile scripts/requirements/jupyter.in` without
internet:
**Before**
```
error: error sending request for url (https://pypi.org/simple/jupyter/): error trying to connect: dns error: failed to lookup address information: No such host is known. (os error 11001)
Caused by: error trying to connect: dns error: failed to lookup address information: No such host is known. (os error 11001)
Caused by: dns error: failed to lookup address information: No such host is known. (os error 11001)
Caused by: failed to lookup address information: No such host is known. (os error 11001)
```
**After**
```
error: Could not connect, are you offline?
Caused by: error sending request for url (https://pypi.org/simple/django/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
Caused by: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
Caused by: dns error: failed to lookup address information: Temporary failure in name resolution
Caused by: failed to lookup address information: Temporary failure in name resolution
```
On linux, it would be "Temporary failure in name resolution" instead of
"No such host is known. (os error 11001)".
The implementation checks for "dne error" stringly as hyper errors are
opaque. The danger is that this breaks with a hyper update. We still get
the complete error trace since reqwest eagerly inlines errors
(https://github.com/seanmonstar/reqwest/issues/2147).
No test since i wouldn't know how to simulate this in cargo test.
Fixes#1971
## Summary
This PR expands environment variables in `-r` and `-c` paths _within_
requirements files. We already do this for `@` URL references and
others.
Closes https://github.com/astral-sh/uv/issues/1473.
## Summary
After https://github.com/astral-sh/uv/pull/2121, the only remaining
issue is that calling `canonicalize` on these Pythons returns an error.
Closes https://github.com/astral-sh/uv/issues/2105.
## Test Plan
Uninstalled all python.org Pythons on my Windows machine, then created a
virtualenv. The resulting config file:
```
Using Python 3.11.8 interpreter at: C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe
Creating virtualenv at: .venv
Activate with: .venv\Scripts\activate
PS C:\Users\crmar\workspace\puffin> cat .\.venv\pyvenv.cfg
home = C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0
implementation = CPython
version_info = 3.11.8
include-system-site-packages = false
uv = 0.1.13
prompt = puffin
```
Prior to this PR, it would fail with a canonicalization error.
Prior to #2121, it would leave a "bad" Python in the config file:
```
Using Python 3.11.8 interpreter at: C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe
Creating virtualenv at: .venv
Activate with: .venv\Scripts\activate
PS C:\Users\crmar\workspace\puffin> cat .\.venv\pyvenv.cfg
home = C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2288.0_x64__qbz5n2kfra8p0
implementation = CPython
version_info = 3.11.8
include-system-site-packages = false
uv = 0.1.13
prompt = puffin
```
Which, once activated, would fail with:
```
(venv) PS C:\Users\crmar\workspace\puffin> python
No Python at '"C:\Users\crmar\AppData\Local\Programs\Python\Python312\python.exe'
```
## Summary
When I install via the Windows Store, `interpreter.base_prefix` contains
a bunch of resolved information that leads to a broken environment.
Instead, we now use `sys._base_executable` on Windows by default,
falling back to `sys.base_prefix` if it doesn't exist. (There are some
issues with `sys.base_executable` that lead to complexity in
`virtualenv`, but they only affect POSIX.) Admittedly, I don't know when
`sys._base_executable` wouldn't exist. It exists in all the environments
I've tested.
Additionally, we use the system interpreter directly if we're outside of
a virtualenv.
## Summary
Closes#2012
This changes `RequirementsTxtParserError::Parser` to take a line, column
instead of cursor location to improve reporting of parser errors. A new
function was added to compute the line and column based on the content
and cursor location when a parser error occurs for simplicity.
Given `uv pip compile .\requirements.txt` of below
```
numpy>=1,<2
--borken
tqdm
```
Before:
```
error: Unexpected '-', expected '-c', '-e', '-r' or the start of a requirement in `.\requirements.txt` at position 14
```
After:
```
error: Unexpected '-', expected '-c', '-e', '-r' or the start of a requirement in `.\requirements.txt` at position 2:3
```
Open Question: Do we want to support `line:column` for other types of
errors? I didn't look dig other potential error types where this might
be desired.
## Test Plan
New test was added to `requirements-txt` crate with this example.
## Summary
It looks like these have been included since the very first gourgeist
commit, but `virtualenv` and `venv` don't include them, and they only
add complexity AFAICT.
## Summary
I think Camino is nice but it makes it much harder to work in
`uv-virtualenv`, since it's the _only_ crate that uses it. If we want to
use Camino, we should use it everywhere IMO.
## Summary
Right now, we have virtualenv construction encoded in a few different
places. Namely, it happens in both `gourgeist` and
`virtualenv_layout.rs` -- _and_ `interpreter.rs` also encodes some
knowledge about how they work, by way of reconstructing the
`SysconfigPaths`.
Instead, `gourgeist` now returns the complete layout, enumerating all
the directories it created. So, rather than returning a root directory,
and re-creating all those paths in `uv-interpreter`, we pass the data
directly back to it.
## Summary
Adds support for `--system-site-packages`. Unlike `pip`, we won't take
the system site packages into account in subsequent commands. I think
this is ok.
Closes https://github.com/astral-sh/uv/issues/1483.
Update #1918 to handle https://github.com/pypa/pip/pull/12536, where pip
removed their python minor entrypoint. The pip test is semi-functional
since it builds pip from source instead of using a wheel with the wrong
entrypoint, we have to update it when this pip version has a release.
Closes#1593.
## Summary
This is based on Pradyun's installer branch
(d01624e5f2/src/installer/scripts.py (L54)),
which is itself based on pip
(0ad4c94be7/src/pip/_vendor/distlib/scripts.py (L136)).
The gist of it is: on Posix platforms, if a path contains a space (or is
too long), we wrap the shebang in a `/bin/sh` invocation.
Closes https://github.com/astral-sh/uv/issues/2076.
## Test Plan
```
❯ cargo run venv "foo"
Finished dev [unoptimized + debuginfo] target(s) in 0.14s
Running `target/debug/uv venv foo`
Using Python 3.12.0 interpreter at: /Users/crmarsh/.local/share/rtx/installs/python/3.12.0/bin/python3
Creating virtualenv at: foo
Activate with: source foo/bin/activate
❯ source "foo bar/bin/activate"
❯ which black
black not found
❯ cargo run pip install black
Resolved 6 packages in 177ms
Installed 6 packages in 17ms
+ black==24.2.0
+ click==8.1.7
+ mypy-extensions==1.0.0
+ packaging==23.2
+ pathspec==0.12.1
+ platformdirs==4.2.0
❯ which black
/Users/crmarsh/workspace/uv/foo bar/bin/black
❯ black
Usage: black [OPTIONS] SRC ...
One of 'SRC' or 'code' is required.
❯ cat "foo bar/bin/black"
#!/bin/sh
'''exec' '/Users/crmarsh/workspace/uv/foo bar/bin/python' "$0" "$@"
' '''
# -*- coding: utf-8 -*-
import re
import sys
from black import patched_main
if __name__ == "__main__":
sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
sys.exit(patched_main())
```
I'm not at all sure whether this is a correct fix or not, but it does
seem to make `pypy` work in at least some cases with `uv`. Previously,
I couldn't get it to work at all. Namely the virtualenv was created
with a `lib/python3.10/site-packages`, but whenever I did a `uv
pip install` in that virtualenv, it was looking for a non-existent
`lib/pypy3.10/site-packages` directory.
With this PR, the workflow reported as not working in #1488 now works
for me:
```
$ pypy3 --version
Python 3.10.13 (fc59e61cfbff, Jan 17 2024, 05:35:45)
[PyPy 7.3.15 with GCC 13.2.1 20230801]
$ uv venv --python $(which pypy3) --seed
Using Python 3.10.13 interpreter at: /usr/bin/pypy3
Creating virtualenv at: .venv
+ pip==24.0
+ setuptools==69.1.1
+ wheel==0.42.0
Activate with: source .venv/bin/activate
$ uv pip install 'alembic==1.0.11'
Resolved 9 packages in 8ms
Installed 9 packages in 14ms
+ alembic==1.0.11
+ greenlet==3.0.3
+ mako==1.3.2
+ markupsafe==2.1.5
+ python-dateutil==2.8.2
+ python-editor==1.0.4
+ six==1.16.0
+ sqlalchemy==2.0.27
+ typing-extensions==4.10.0
```
Where as previously (current `main`), I was hitting this error:
```
$ uv venv --python $(which pypy3) --seed
Using Python 3.10.13 interpreter at: /usr/bin/pypy3
Creating virtualenv at: .venv
+ pip==24.0
+ setuptools==69.1.1
+ wheel==0.42.0
Activate with: source .venv/bin/activate
$ uv pip install 'alembic==1.0.11'
error: Failed to list installed packages
Caused by: failed to read directory `/home/andrew/astral/issues/uv/i1488/.venv/lib/pypy3.10/site-packages`
Caused by: No such file or directory (os error 2)
```
Notice though that neither outcome above matches the error reported in #1488,
so this is likely not a complete fix. There are perhaps other lurking
issues.
Ref #1488
## Summary
This was a missed find-and-replace. We shouldn't assume `layout.platlib`
here, since `RECORD` will be written to `site_packages` (which could be
`layout.purelib`).
This is hard to reproduce. You need a _fresh_ environment where
`purelib` and `platlib` differ (which isn't the case for virtualenvs, at
least typically), and you need to be installing a new package that is a
purelib. I tested it by manually changing `platlib` to point to a
different path.
Closes https://github.com/astral-sh/uv/issues/2064.
Previously, `uv` would always prioritize the index given by
`--index-url`. It would then try any indexes after that given by zero
or more `--extra-index-url` flags. This differed from `pip` in that any
priority was given at all, where `pip` doesn't guarantee any priority
ordering of indexes.
We could go in the direction of mimicing `pip`'s behavior here, but it
at present has issues with dependency confusion attacks where packages
may get installed from indexes you don't control. More specifically,
there is an issue of different trust levels. See discussion in #171 and
[PEP-0708] for more on the security impact.
In contrast, `uv` will only select versions for a package from a single
index. That is, even if `foo` is in indexes `a` and `b`, it will
only consider the versions from the index that it checks first. This
probably helps with respect to dependency confusion attacks, but also
means that `uv` doesn't quite cover all of the same use cases as `pip`.
In this PR, we retain the notion of prioritizing indexes, but
tweak it so that PyPI is preferred last as opposed to first. Or
more precisely, the `--index-url` flag specifies a fallback index,
not the primary index, and is deprioritized beneath every index
specified by `--extra-index-url`. The ordering among indexes given by
`--extra-index-url` remains the same: earlier indexes are prioritized
over later indexes.
While this tweak likely won't hit all use cases, I believe it will
resolve some of the most common pain points without exacerbating
dependency confusion problems.
Ref #171, Fixes#1377, Fixes#1451, Fixes#1600
[PEP-0708]: https://peps.python.org/pep-0708/
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Expose the uv_normalize types from pep508, so that these can be used
with a crates.io version.
---------
Co-authored-by: konsti <konstin@mailbox.org>
`uv --system` is failing in GitHub Actions, because `py --list-paths`
returns all the pre-cached Pythons:
```
-V:3.12 * C:\hostedtoolcache\windows\Python\3.12.2\x64\python.exe
-V:3.12-32 C:\hostedtoolcache\windows\Python\3.12.2\x86\python.exe
-V:3.11 C:\hostedtoolcache\windows\Python\3.11.8\x64\python.exe
-V:3.11-32 C:\hostedtoolcache\windows\Python\3.11.8\x86\python.exe
-V:3.10 C:\hostedtoolcache\windows\Python\3.10.11\x64\python.exe
-V:3.10-32 C:\hostedtoolcache\windows\Python\3.10.11\x86\python.exe
-V:3.9 C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe
-V:3.9-32 C:\hostedtoolcache\windows\Python\3.9.13\x86\python.exe
-V:3.8 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
-V:3.8-32 C:\hostedtoolcache\windows\Python\3.8.10\x86\python.exe
-V:3.7 C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe
-V:3.7-32 C:\hostedtoolcache\windows\Python\3.7.9\x86\python.exe
```
So, our default selector returns the first entry here. But none of these
are actually in `PATH` except the one that the user installed via
`actions/setup-python@v5` -- that's the point of the action, that it
puts the correct versions in `PATH`.
It seems to me like we should prioritize `PATH` over `py --list-paths`.
Is there a good reason not to do this?
Closes: https://github.com/astral-sh/uv/issues/2056
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
With this PR I've added the option environment variables to the wheel
building process, through the `BuildDispatch`. When integrating uv with
our project pixi (https://github.com/prefix-dev/pixi/pull/863). We ran
into this missing requirement, I've made a rough version here, could
maybe use some refinement.
### Why do we need this?
Because pixi allow the user to use a conda activated prefix for wheel
building, this comes with a number of environment variables, like `PATH`
but also `CONDA_PREFIX` amongst others. This allows the user to use
system dependencies from conda-forge to use during an sdist build.
Because we use `uv` as a library we need to pass in the options
programatically. Additionally, in general there is nothing holding a
python sdist back from actually depending on an environment variable,
see
e.g the test package: https://pypi.org/project/env-test-package/
### What about `ConfigSettings`
I think `ConfigSettings` does not suffice because e.g. CMake could
function differently when the `CONDA_PREFIX` is set. Also, we do not
know if the user supplied backend actually support these settings.
### Path handling
Because the user can now also supply a PATH in the environment map, the
logic I had was the following, I format the path so that it has the
following precedence
1. venv scripts dir.
2. user supplied path.
3. system path.
### Improvements
There is some path modification and copying happening everytime we use
the `run_python_script` function, I think we could improve this but
would like some pointers where to best put the maybe split and cached
version, we might also want to use some types to split these things up.
### Finally
I did not add any of these options to the uv executables, I first would
like to know if this is a direction we would want to go in. I'm happy to
do this or make any changes that you feel would benefit this project.
Also tagging @wolfv to keep track of this as well.
## Test Plan
<!-- How was it tested? -->
---------
Co-authored-by: konsti <konstin@mailbox.org>
## Summary
We shouldn't be resolving symlinks on the provided interpreter;
otherwise we break `pyenv`, since running `cargo run pip install mypy
--python .venv/bin/python` will immediately resolve to (e.g.)
`/Users/crmarsh/.pyenv/versions/3.10.2/bin/python3.10`, and pyenv relies
on the path to do its lookups.
Instead, the canonicalizing happens when we query the interpreter
metadata.
Closes https://github.com/astral-sh/uv/issues/2068.
## Test Plan
Ran `cargo run pip install mypy --python .venv/bin/python -v -n` with a
virtualenv created using a pyenv Python; verified that Mypy was
installed into the virtual environment, rather than into the global
environment.
## Summary
If a pre-release marker is present on a requirement in a constraint
file, we should allow pre-releases for that package.
Closes https://github.com/astral-sh/uv/issues/2063.
## Summary
Temporarily disabling `install_git_private_https_pat_and_username` since
running this test can break your local Git authentication for other
projects. I experienced this today and keep finding myself needing to
ignore it locally.
See: https://github.com/astral-sh/uv/issues/1980.
## Summary
Internal refactor to `PrioritizedDistribution` that I think should
reduce the size? Although the motivation here is simplicity, not perf.
Instead of storing:
```rust
/// The highest-priority, installable wheel for the package version.
compatible_wheel: Option<(DistMetadata, TagPriority)>,
/// The most-relevant, incompatible wheel for the package version.
incompatible_wheel: Option<(DistMetadata, IncompatibleWheel)>,
```
We now store:
```rust
wheel: Option<(DistMetadata, WheelCompatibility)>,
```
Where `WheelCompatibility` is an enum of `TagPriority` or
`IncompatibleWheel`.
## Summary
This is essentially a wrapper around something like `--python $(which
python3)`, but gives users a portable and streamlined way to solve the
common pain point of using `uv` in GitHub Actions or a Docker container.
See: https://github.com/astral-sh/uv/issues/1526.
## Summary
This also preserves the environment variables in the output file, e.g.:
```
Resolved 1 package in 216ms
# This file was autogenerated by uv via the following command:
# uv pip compile requirements.in --emit-index-url
--index-url https://test.pypi.org/${SUFFIX}
requests==2.5.4.1
```
I'm torn on whether that's correct or undesirable here.
Closes#2035.
## Summary
`PythonPlatform` only exists to format paths to directories within
virtual environments based on a root and an OS, so it's now
`VirtualenvLayout`.
`Virtualenv` is now used for non-virtual environment Pythons, so it's
now `PythonEnvironment`.
## Summary
Now that we have the ability to introspect the installed packages for
arbitrary Pythons, we can allow `pip freeze` and `pip list` to fall back
to the "default" Python, if no virtualenv is present.
Closes https://github.com/astral-sh/uv/issues/2005.
## Summary
This PR aligns the `uv pip install --python` flag with the `uv venv
--python` flag, such that the former now accepts binary names and Python
versions by way of using the same `find_requested_python` method under
the hood.
## Summary
This PR adds a `--python` flag that allows users to provide a specific
Python interpreter into which `uv` should install packages. This would
replace the `VIRTUAL_ENV=` workaround that folks have been using to
install into arbitrary, system environments, while _also_ actually being
correct for installing into non-virtual environments, where the bin and
site-packages paths can differ.
The approach taken here is to use `sysconfig.get_paths()` to get the
correct paths from the interpreter, and then use those for determining
the `bin` and `site-packages` directories, rather than constructing them
based on hard-coded expectations for each platform.
Closes https://github.com/astral-sh/uv/issues/1396.
Closes https://github.com/astral-sh/uv/issues/1779.
Closes https://github.com/astral-sh/uv/issues/1988.
## Test Plan
- Verified that, on my Windows machine, I was able to install `requests`
into a global environment with: `cargo run pip install requests --python
'C:\\Users\\crmarsh\\AppData\\Local\\Programs\\Python\\Python3.12\\python.exe`,
then `python` and `import requests`.
- Verified that, on macOS, I was able to install `requests` into a
global environment installed via Homebrew with: `cargo run pip install
requests --python $(which python3.8)`.
In this context, we already know (as the comment says) that `self` does
not have a local segment, so we don't need to strip it.
This change isn't motivated by anything other than making the code and
comment in sync. For example, when I first looked at it, I wondered
whether the extra stripping was somehow necessary. But it isn't.
## Summary
When a `pyproject.toml` is provided directly to `uv pip compile`, we
were failing to resolve recursive extras. The solution I settled on here
is to flatten them recursively when determining the requirements
upfront.
Closes https://github.com/astral-sh/uv/issues/1987.
## Test Plan
`cargo test`
## Summary
For a venv created by `virtualenv`, the `pyvenv.cfg` file specifies the
full version string in the `version_info` field:
```
home = /Users/x/.rye/py/cpython@3.12.1/install/bin
implementation = CPython
version_info = 3.12.1.final.0
virtualenv = 20.25.0
include-system-site-packages = false
base-prefix = /Users/x/.rye/py/cpython@3.12.1/install
base-exec-prefix = /Users/x/.rye/py/cpython@3.12.1/install
base-executable = /Users/x/.rye/py/cpython@3.12.1/install/bin/python3
```
The relevant code can be found here:
4ca8a20c17/src/virtualenv/create/creator.py (L167)
This PR changes `pyvenv.cfg` created by uv for better compatibility with
`virtualenv`.
## Test Plan
```sh
uv venv
cat .venv/pyvenv.cfg
```
Thank you for writing `uv`! We're already using it internally on some
container image builds and finding that it's noticeably faster 💯
## Summary
I was attempting to use `uv` alongside [modal](https://modal.com/)'s
internal PyPi mirror and ran into some issues. The first issue was the
following error:
```
error: Failed to download: nltk==3.8.1
Caused by: content-length header is missing from response
```
This error was coming from within
`RegistryClient::wheel_metadata_no_pep658`. By logging requests on the
client (uv) and server (internal mirror) sides I've concluded that it's
occurring because `uv` is sending a header suggesting that it can accept
a gzip'd response, but decompressing the gzip'd response strips the
`content-length` header:
https://github.com/seanmonstar/reqwest/issues/294.
**Logged request, client-side:**
```
0.981664s 0ms INFO uv_client::registry_client JONO, REQ: Request { method: HEAD, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(172.21.0.1)), port: Some(5555), path: "/simple/joblib/joblib-1.3.2-py3-none-any.whl", query: None, fragment: None }, headers: {} }
```
No headers set explicitly by `uv`.
**Logged request, server-side:**
```
2024-02-26T03:45:08.598272Z DEBUG pypi_mirror: origin request = Request { method: HEAD, uri: /simple/joblib/joblib-1.3.2-py3-none-any.whl, version: HTTP/1.1, headers: {"accept": "*/*", "user-agent": "uv", "accept-encoding": "gzip, br", "host": "172.21.0.1:5555"}, body: Body(Empty) }
```
Server receives `"accept-encoding": "gzip, br",`.
My change adding the header to the request fixed this issue. But our
internal mirror is just passing through PyPI responses and PyPI
responses do contain PEP 658 data, and so `wheel_metadata_no_pep658`
shouldn't execute.
The issue there is that the PyPi response field has _dashes_ not
_underscores_ (https://peps.python.org/pep-0691/).
<img width="1261" alt="image"
src="https://github.com/astral-sh/uv/assets/12058921/35230f27-441a-457a-827b-870a1a16c16a">
After changing the `alias` the PEP 658 codepath now runs correctly :)
## Test Plan
I tested by installing against both our mirror and against PyPi:
```
RUST_LOG="uv=trace" UV_NO_CACHE=true UV_INDEX_URL="http://172.21.0.1:5555/simple" target/release/uv pip install -v nltk
RUST_LOG="uv=trace" UV_NO_CACHE=true UV_INDEX_URL="http://localhost:5555/simple" target/release/uv pip uninstall -v nltk
```
```
target/release/uv pip install -v nltk
target/release/uv pip uninstall -v nltk
```
## Summary
When you invoke `python -c`, an empty string is prepended to `sys.path`,
which allows loading modules in the current directory
(https://docs.python.org/3/using/cmdline.html#cmdoption-P). However, in
PEP 517 builds, the current directory should _not_ be part of the path.
There's a flag we can use to disable this behavior (`-P`), but it's only
available in Python 3.11 and later, so instead, I'm doing something
similar to pip's `__main__.py`, which avoids this for `python -m pip`
invocations.
Closes https://github.com/astral-sh/uv/issues/1972.
## Summary
Closes#1922
When a timeout occurs, it hints to the user to configure the
`UV_HTTP_TIMEOUT` env var.
Before
```
error: Failed to download distributions
Caused by: Failed to fetch wheel: torch==2.2.0
Caused by: Failed to extract source distribution
Caused by: request or response body error: operation timed out
Caused by: operation timed out
```
After
```
error: Failed to download distributions
Caused by: Failed to fetch wheel: torch==2.2.0
Caused by: Failed to extract source distribution
Caused by: Failed to download distribution due to network timeout. Try increasing UV_HTTP_TIMEOUT.
```
## Test Plan
<!-- How was it tested? -->
Wasn't sure if we'd want a test. If we do, is there a existing mechanism
or preferred approach to force a timeout to occur in tests? Maybe set
the timeout to 1 and add torch as an install check (although it's
possible that could become flaky)?
Hi, love your work on `uv` 👋!
Opening a Draft PR early to check if there are any existing rust table
formatting libs that I am unaware of (either already in `uv`/`ruff`, or
the rust ecosystem) before spending much time on inventing the wheel
myself and cleaning it up. Any other pointers are also welcome (e.g. on
the editable filtering).
Editable project locations in `uv pip list` include the file scheme
(`file://`), where they are omitted in `pip list`. Is this desired, or
should it replicate pip?
## Summary
Implementation for #1401
`--editable` flag is implemented.
`--outdated` and `--uptodate` out of scope for this PR (requires latest
version information, and type wheel/sdist)
## Test Plan
Not yet implemented as I couldn't locate the tests for `uv pip freeze`.
We can compare to `pip` in
`scripts/compare_with_pip/compare_with_pip.py`?
Address a few pedantic lints
lints are separated into separate commits so they can be reviewed
individually.
I've not added enforcement for any of these lints, but that could be
added if desirable.
## Summary
When the user provides an output file, avoid writing the `pip compile`
output to `stdout` when `-q` is specified. (We still write to `stdout`
if no output file is provided, since otherwise, the resolution won't be
printed _anywhere_.)
## Summary
Instead of looking at _either_ `pyproject.toml` or `setup.py`, we should
just be conservative and take the most-recent timestamp out of
`pyproject.toml`, `setup.py`, and `setup.cfg`. That will help prevent
staleness issues like those described in
https://github.com/astral-sh/uv/issues/1913#issuecomment-1961544084.
## Summary
Even when pre-releases are "allowed", per PEP 440, `pydantic<2.0.0`
should _not_ include pre-releases. This PR modifies the specifier
translation to treat `pydantic<2.0.0` as `pydantic<2.0.0.min0`, where
`min` is an internal-only version segment that's invisible to users.
Closes https://github.com/astral-sh/uv/issues/1641.
## Summary
We were applying every constraint to every dependency. This is
"harmless" in practice since this is just an optimization, but we thus
had false negatives ~every time which could lead to wasted work.
## Summary
If a `pyproject.toml` or similar is changed within an editable, we
should avoid passing our audit check (and thus re-install the package).
Closes https://github.com/astral-sh/uv/issues/1913.
## Summary
For context, we have three extraction paths:
- untar (async) - used for any `.tar.gz`, local or remote.
- unzip (async) - used to unzip remote wheels, or local or remote source
distributions.
- unzip (sync) - used to untar locally-available wheels into the cache.
We use three different crates for these:
- [`tokio-tar`](https://github.com/vorot93/tokio-tar)
- [`async-zip`](https://github.com/Majored/rs-async-zip)
- [`zip-rs`](https://github.com/zip-rs/zip)
These all seem to have different support for symlinks:
- `tokio-tar` tries to create a symlink (which works fine on Unix but
errors on Windows, since we typically don't have elevated permissions).
- `async-zip` _seems_ to write the target contents directly to the file
(which is what we want).
- `zip-rs` _apparently_ writes the _name_ of the target to the file
(which isn't what we want).
Thankfully, symlinks are not allowed in wheels
(https://github.com/pypa/pip/issues/5919,
https://discuss.python.org/t/symbolic-links-in-wheels/1945), so we can
ignore `zip-rs`.
For `tokio-tar`, we now _skip_ (and warn) if we see a symlink on
Windows. We could do what pip does, and recursively copy, but it's
difficult because we don't have `Seek` on the file. (Alternatively, we
could use hard links and junctions, though those also might need to
exist already.) Let's see how far this gets us.
(We also no longer attempt to set permissions on symlinks on Unix, which
caused another failure.)
Closes https://github.com/astral-sh/uv/issues/1858.
## Summary
We're printing the `Display` representation of `InstalledDist`, which
isn't guaranteed to be (and in fact isn't) a valid PEP 508 requirement,
making it impossible to use the `freeze` output as an input to an
install.
Closes https://github.com/astral-sh/uv/issues/1931.
## Summary
Like `platform.python_implementation`, we should support the
`python_implementation` "alias" for `platform_python_implementation`.
Closes https://github.com/astral-sh/uv/issues/1906.
## Test Plan
```shell
❯ cargo run pip install "pynacl==1.4.0"
Finished dev [unoptimized + debuginfo] target(s) in 7.02s
Running `target/debug/uv pip install pynacl==1.4.0`
Resolved 4 packages in 9ms
Built pynacl==1.4.0 Downloaded 1 package in 31.51s
Installed 4 packages in 3ms
+ cffi==1.16.0
+ pycparser==2.21
+ pynacl==1.4.0
+ six==1.16.0
```
Similar to https://github.com/astral-sh/ruff/pull/8034
Adds more version information so it's clear what revision the user is on
```
❯ cargo run -q -- --version
uv 0.1.10 (daa8565a7 2024-02-23)
❯ cargo run -q -- -V
uv 0.1.10
❯ cargo run -q -- version
uv 0.1.10 (daa8565a7 2024-02-23)
❯ cargo run -q -- version --output-format json
{
"version": "0.1.10",
"commit_info": {
"short_commit_hash": "daa8565a7",
"commit_hash": "daa8565a75249305821fdc34ace085060c082ba3",
"commit_date": "2024-02-23",
"last_tag": null,
"commits_since_last_tag": 0
}
}
```
## Summary
When a virtual environment contains multiple packages with the same
name, we no longer throw a hard error. Instead:
- In `uv pip freeze`, we list all versions.
- In `uv pip uninstall`, we uninstall all versions.
- In `uv pip install`, we uninstall all versions prior to installing a
new version.
Closes https://github.com/astral-sh/uv/issues/1848.
## Summary
In uv, we're going to use `--no-emit-package` for this, to convey that
the package will be included in the resolution but not in the output
file. It also mirrors flags like `--emit-index-url`.
We're also including an `--unsafe-package` alias.
Closes https://github.com/astral-sh/uv/issues/1415.
Users expect pip to have `pip`, `pip3` and `pip3.x` entrypoints. But pip
is a universal wheel, so it contains the `pip3.x` entrypoint where it
was built on. To fix this, pip special cases itself when installing
(3898741e29/src/pip/_internal/operations/install/wheel.py (L283)),
replacing the wheel entrypoint with one for the current version. We now
do the same.
Fixes#1593
Read the key read for uv from `pyenv.cfg` from `gourgeist` instead of
`uv`. I missed that we're also reading pyenv.cfg when reviewing #1852.
We could check for gourgeist for backwards compatibility, but i think
it's fine this way.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
On Windows `10.0.19045` the `py` command prints to `stderr` even when
working correctly. This means that uv should not treat this as a
failure.
Fixes https://github.com/astral-sh/uv/issues/1904
## Test Plan
I ran the modified code and it worked. I expect the pull request to run
automated tests.
## Summary
Fixes https://github.com/astral-sh/uv/issues/1693
`uv` currently fails when a user has `python` 2 or older installed on
their system without a `python3` or `python3.exe` on their path because
the `get_interpreter_info.py` script fails executing (it uses some
Python 3+ APIs).
This PR fixes this by:
* Returning an explicit error code in `get_interpreter_info` if the
Python version isn't supported
* Skipping over this error in `python_query` if the user requested ANY
python version or a version >= 3.
* Error if the user requested a Python 2 version.
## Test Plan
Error if the user requests a legacy python version.
```
uv venv -p 2
× Python 2 or older is not supported. Please use Python 3 or newer.
```
Ignore any python 2 installation when querying newer python
installations (using v4 here because I have python3 on the path and that
takes precedence over querying python)
```
uv_interpreter::python_query::find_python selector=Major(4)
0.005541s 0ms DEBUG uv_interpreter::interpreter Detecting markers for: /home/micha/.pyenv/shims/python
0.059730s 54ms DEBUG uv_interpreter::python_query Found a Python 2 installation that isn't supported by uv, skipping.
0.059983s 54ms DEBUG uv_interpreter::interpreter Using cached markers for: /usr/bin/python
× No Python 4 In `PATH`. Is Python 4 installed?
```
## Summary
This matches `pip-compile` and is, I think, intuitive. If you want to
suppress output, you can always pipe it away.
Closes https://github.com/astral-sh/uv/issues/1895.
## Summary
Hey guys! The motivation described in #1834
## Test Plan
Changed snapshot of the existing tests. `--index-url` and
`--extra-index-url` occur pretty often, so no extra testing is required,
imo.
I'm confused that we have this separate specification of `reqwests`? I'm
not sure this has any effect, but it seems like it should be done for
correctness.
Follows #1512
Closes https://github.com/astral-sh/uv/issues/1709
Closes https://github.com/astral-sh/uv/issues/1371
Tested with the reproduction provided in #1709 which gets past the HTTP
401.
Reuses the same copying logic we introduced in
https://github.com/astral-sh/uv/pull/1874 to ensure authentication is
attached to file URLs with a realm that matches that of the index. I had
to move the authentication logic into a new crate so it could be used in
`distribution-types`.
We will want to something more robust in the future, like track all
realms with authentication in a central store and perform lookups there.
That's what `pip` does and it allows consolidation of logic like netrc
lookups. That refactor feels significant though, and I'd like to get
this fixed ASAP so this is a minimal fix.
A couple moons ago, I introduced an optimization for version comparisons
by devising a format where *most* versions would be represented by a
single `u64`. This in turn meant most comparisons (of which many are
done during resolution) would be extremely cheap.
Unfortunately, when I did that, I screwed up the preservation of
ordering as defined by the [Version Specifiers spec]. I think I messed
it up because I had originally devised the representation so that we
could pack things like `1.2.3.dev1.post5`, but later realized it would
be better to limit ourselves to a single suffix. However, I never
updated the binary encoding to better match "up to 4 release versions
and up to precisely 1 suffix." Because of that, there were cases where
versions weren't ordered correctly. For example, this fixes a bug where
`1.0a2 < 1.0dev2`, even though all dev releases should order before
pre-releases.
We also update a test so that it catches these kinds of bugs in the
future. (By testing all pairs of versions in a sequence instead of just
the adjacent versions.)
[Version Specifiers spec]:
https://packaging.python.org/en/latest/specifications/version-specifiers/#summary-of-permitted-suffixes-and-relative-ordering
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
This modifies `gourgeist` to allow passing additional k,v pairs to add
to the `pyvenv.cfg` file as proposed in #1697.
I made it allow an arbitrary set of pairs (to decouple from `uv` since
this is mainly a change to `gourgeist`) , but I can slim it down to just
allow just a name and version strings if that's desired.
The `pyvenv.cfg` will also have a `uv = <uv-crate-version>` when a venv
is created via `uv venv` ~~and `uv-build = <uv-build-crate-version>`
when it's created via `SourceBuild::setup`~~.
Example below via `uv venv`:
```ini
home = ...
implementation = CPython
version_info = 3.12
include-system-site-packages = false
base-prefix = ...
base-exec-prefix = ...
base-executable = ...
uv = 0.1.6
prompt = uv
```
Open to any suggestions, thanks!
Closes#1697
## Test Plan
Added new test in `tests/venv.rs` called `verify_pyvenv_cfg` to verify
that it contains the right uv version string. I didn't see tests
configured in `gourgeist` itself, so I didn't add any there.
We don't have test coverage for this, but a term can reference an
incompatibility with root and then we'll display the internal 'root'
package to the user.
Raised in https://github.com/astral-sh/uv/issues/1855
Closes https://github.com/astral-sh/uv/issues/1860
In https://github.com/astral-sh/uv/pull/1816, we started using the URL
attached to a response instead of the request URL for subsequent
requests — this fixes various bugs but has the side-effect of dropping
credentials from the URL. Here, we transfer credentials from the request
URL to the response URL. We perform RFC compliant checks for safety.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
To integrate `uv` into `pixi` I need to specify a custom
`ResolverProvider` to be able to specify that some packages are already
installed by conda and should not be touched. However, some of the types
required to implement your own `ResolverProvider` were not accessible
through the public API. This PR basically adds them.
## Test Plan
I didnt add an explicit test for this.
## Summary
Fixes#1444.
In situations where the installer fails to perform a reflink, a regular
file copy is also attempted, as a fallback. This circumvents issues with
linking files across filesystems or volumes.
## Test Plan
N/A
## Summary
This revives a PR from long ago
(https://github.com/astral-sh/uv/pull/383 and
https://github.com/zanieb/pubgrub/pull/24) that modifies how we deal
with dependencies that are declared multiple times within a single
package.
To quote from the originating PR:
> Uses an experimental pubgrub branch (#370) that allows us to handle
multiple version ranges for a single dependency to the solver which
results in better error messages because the derivation tree contains
all of the relevant versions. Previously, the version ranges were merged
(by us) in the resolver before handing them to pubgrub since only one
range could be provided per package. Since we don't merge the versions
anymore, we no longer give the solver an empty range for conflicting
requirements; instead the solver comes to that conclusion from the
provided versions. You can see the improved error message for direct
dependencies in [this
snapshot](https://github.com/astral-sh/puffin/pull/383/files#diff-a0437f2c20cde5e2f15199a3bf81a102b92580063268417847ec9c793a115bd0).
The main issue with that PR was around its handling of URL dependencies,
so this PR _also_ refactors how we handle those. Previously, we stored
URL dependencies on `PubGrubPackage`, but they were omitted from the
hash and equality implementations of `PubGrubPackage`. This led to some
really careful codepaths wherein we had to ensure that we always visited
URLs before non-URL packages, so that the URL-inclusive versions were
included in any hashmaps, etc. I considered preserving this approach,
but it would require us to rely on lots of internal details of PubGrub
(since we'd now be relying on PubGrub to merge those packages in the
"right" order).
So, instead, we now _always_ set the URL on a given package, whenever
that package was _given_ a URL upfront. I think this is easier to reason
about: if the user provided a URL for `flask`, then we should just
always add the URL for `flask`. If we see some other URL for `flask`, we
error, like before. If we see some unknown URL for `flask`, we error,
like before.
Closes https://github.com/astral-sh/uv/issues/1522.
Closes https://github.com/astral-sh/uv/issues/1821.
Closes https://github.com/astral-sh/uv/issues/1615.
## Summary
We need to take care to keep wheel tags in "priority order" (e.g., we
should prefer ARM wheels over universal wheels). However... it looks
like we've had a `.sort()` in here all along, that risks throwing off
the ordering?
Closes https://github.com/astral-sh/uv/issues/1840.
## Test Plan
ensure that `rlax` uses the ARM wheel rather than the universal wheel:
- `cargo run venv`
- `cargo run pip install rlax`
- `import rlax`
Closes https://github.com/astral-sh/uv/issues/1775
Closes https://github.com/astral-sh/uv/issues/1452
Closes https://github.com/astral-sh/uv/issues/1514
Follows https://github.com/astral-sh/uv/pull/1717
libgit2 does not support host names with extra identifiers during SSH
lookup (e.g. [`github.com-some_identifier`](
https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#using-multiple-repositories-on-one-server))
so we use the `git` command instead for fetching. This is required for
`pip` parity.
See the [Cargo
documentation](https://doc.rust-lang.org/nightly/cargo/reference/config.html#netgit-fetch-with-cli)
for more details on using the `git` CLI instead of libgit2. We may want
to try to use libgit2 first in the future, as it is more performant
(#1786).
We now support authentication with:
```
git+ssh://git@<hostname>/...
git+ssh://git@<hostname>-<identifier>/...
```
Tested with a deploy key e.g.
```
cargo run -- \
pip install uv-private-pypackage@git+ssh://git@github.com-test-uv-private-pypackage/astral-test/uv-private-pypackage.git \
--reinstall --no-cache -v
```
and
```
cargo run -- \
pip install uv-private-pypackage@git+ssh://git@github.com/astral-test/uv-private-pypackage.git \
--reinstall --no-cache -v
```
with a ssh config like
```
Host github.com
Hostname github.com
IdentityFile=/Users/mz/.ssh/id_ed25519
Host github.com-test-uv-private-pypackage
Hostname github.com
IdentityFile=/Users/mz/.ssh/id_ed25519
```
It seems quite hard to add test coverage for this to the test suite, as
we'd need to add the SSH key and I don't know how to isolate that from
affecting other developer's machines.
Previously, we were only checking /bin/sh. While that works in most
cases, it seems like there are still scenarios where /bin/sh isn't an
executable itself, and is instead just a shell script that calls
/bin/dash. (See #1810 for example.)
In this PR, we make the `ld` detection a bit more robust by trying
multiple paths. As with previous changes, we emit copious logs to help
debug this in the future.
It's not totally clear how to test this. I'm not sure how to reproduce
the environment mentions in #1810 specifically since it seems like an
internal variant of WSL Ubuntu.
Fixes#1810
## Summary
We still need to wait for the distribution metadata (for direct
dependencies), even when resolving with `--no-deps`, since we rely on it
to report diagnostics to the user.
Closes https://github.com/astral-sh/uv/issues/1801.
## Summary
We currently maintain separate untar methods for sync and async, but we
only use the sync version when the user provides a local source
distribution. (Otherwise, we untar as we download the distribution.) In
my testing, this is actually slower anyway:
```
❯ python -m scripts.bench \
--uv-path ./target/release/main \
--uv-path ./target/release/uv \
./requirements.in --benchmark resolve-cold --min-runs 50
Benchmark 1: ./target/release/main (resolve-cold)
Time (mean ± σ): 835.2 ms ± 107.4 ms [User: 346.0 ms, System: 151.3 ms]
Range (min … max): 639.2 ms … 1051.0 ms 50 runs
Benchmark 2: ./target/release/uv (resolve-cold)
Time (mean ± σ): 750.7 ms ± 91.9 ms [User: 345.7 ms, System: 149.4 ms]
Range (min … max): 637.9 ms … 905.7 ms 50 runs
Summary
'./target/release/uv (resolve-cold)' ran
1.11 ± 0.20 times faster than './target/release/main (resolve-cold)'
```
## Summary
Allows the corresponding `pypi_types` struct to use any URL, since other
installers can put those into the environment, and Poetry seems to write
invalid URLs.
If we see a distribution with an invalid URL, we just treat it as a
registry distribution, which isn't ideal, but is better than (1)
erroring, and (2) changing `Url` to `String` everywhere internally. (I'm
torn on this second option.)
Closes https://github.com/astral-sh/uv/issues/1744.
## Test Plan
- Added `flask = { git = "git@github.com:pallets/flask.git", rev =
"b90a4f1f4a370e92054b9cc9db0efcb864f87ebe" }` to
`scripts/editable-installs/poetry_editable/pyproject.toml`.
- Ran `poetry install`.
- Ran `cargo pip freeze`. Verified that it errored on `main`, but passed
here.
- Ran `cargo run pip install "flask==3.0.0"`. Verified that it
uninstalled the existing Flask, and installed a new version from the
registry.
Add a `UV_BOOTSTRAP_DIR` option to configure the python bootstrap
directory. This is helpful when working across multiple platforms in a
single IDE session.
## Summary
If a registry doesn't support range requests, then today, we download
the entire wheel to disk and then read the metadata from the downloaded
archive. This PR instead modifies the registry client to stream the
zipfile and stop as soon as it's seen the metadata, which should be more
efficient.
Closes https://github.com/astral-sh/uv/issues/1596.
## Test Plan
Made this the _only_ path for downloading metadata; verified that the
test suite passed.
## Summary
Hello there! The motivation for this feature is described here #1678
## Test Plan
I've added unit tests and also tested this manually on my work project
by comparing it to the original `pip-compile` output - it looks much
like the `pip-compile` generated lock file.
Fixes handling of GitHub PATs in HTTPS URLs, which were otherwise
dropped. We now supporting the following authentication schemes:
```
git+https://<user>:<token>/...
git+https://<token>/...
```
On Windows, the username is required. We can consider adding a
special-case for this in the future, but this just matches libgit2's
behavior.
I tested with fine-grained tokens, OAuth tokens, and "classic" tokens.
There's test coverage for fine-grained tokens in CI where we use a real
private repository and PAT. Yes, the PAT is committed to make this test
usable by anyone. It has read-only permissions to the single repository,
expires Feb 1 2025, and is in an isolated organization and GitHub
account.
Does not yet address SSH authentication.
Related:
- https://github.com/astral-sh/uv/issues/1514
- https://github.com/astral-sh/uv/issues/1452
## Summary
The generated `pyvenv.cfg` file hardcodes `implementation = CPython`
even for PyPy venvs, created with `uv venv venv --python pypy3.10`, for
example.
```ini
home = /path/to/.pyenv/versions/pypy3.10-7.3.15/bin
implementation = CPython
version_info = 3.10
gourgeist = 0.0.4
include-system-site-packages = false
base-prefix = /path/to/.pyenv/versions/pypy3.10-7.3.15
base-exec-prefix = /path/to/.pyenv/versions/pypy3.10-7.3.15
base-executable = /path/to/.pyenv/versions/pypy3.10-7.3.15/bin/pypy3.10
```
## Test Plan
Manually verified that `pyvenv.cfg` now contains `implementation =
PyPy`. I can try refactoring `create_bare_venv` to make it more easily
testable, though.
## Summary
Some packages encode logic to embed the current commit SHA in the
version tag, when built within a Git repo. This typically results in an
invalid (non-compliant) version. Here's an example from `pylzma`:
ccb0e7cff3/version.py (L45).
This PR adds a phony, empty `.git` to the cache root, to ensure that any
`git` commands fail.
Closes https://github.com/astral-sh/uv/issues/1768.
## Test Plan
- Create a tag on the current commit, like `v0.5.0`.
- Build `pylzma`, using a cache _within_ the repo:
```
rm -rf foo
cargo run venv
cargo run pip install "pylzma @ 10ef072c3c3b9ea77ebe9546499975/pylzma-0.5.0.tar.gz" --verbose --cache-dir bar
```
(This PR message is mostly copied from the comment in the code.)
For local builds of Python, at time of writing, the version numbers end
with
a `+`. This makes the version non-PEP-440 compatible since a `+`
indicates
the start of a local segment which must be non-empty. Thus, `uv` chokes
on it
and [spits out an error][1] when trying to create a venv using a "local"
build
of Python. Arguably, the right fix for this is for [CPython to use a
PEP-440
compatible version number][2].
However, as a work-around for now, [as suggested by pradyunsg][3] as one
possible direction forward, we strip the `+`.
This fix does unfortunately mean that one [cannot specify a Python
version
constraint that specifically selects a local version][4]. But at the
time of
writing, it seems reasonable to block such functionality on this being
fixed
upstream (in some way).
Another alternative would be to treat such invalid versions as strings
(which
is what PEP-508 suggests), but this leads to undesirable behavior in
this
case. For example, let's say you have a Python constraint of `>=3.9.1`
and
a local build of Python with a version `3.11.1+`. Using string
comparisons
would mean the constraint wouldn't be satisfied:
>>> "3.9.1" < "3.11.1+"
False
So in the end, we just strip the trailing `+`, as was done in the days
of old
for [legacy version numbers][5].
I tested this fix by manually confirming that
uv venv --python local/python
failed before it and succeeded after it.
Fixes#1357
[1]: https://github.com/astral-sh/uv/issues/1357
[2]: https://github.com/python/cpython/issues/99968
[3]:
https://github.com/pypa/packaging/issues/678#issuecomment-1436033646
[4]: https://github.com/astral-sh/uv/issues/1357#issuecomment-1947645243
[5]:
085ff41692/packaging/version.py (L168-L193)
## Summary
Add `installer` method to `InstalledDist` to distinguish between
different installers. Might be nice to add an enum for all possible
installers, but this might be too hard to keep up to date :).
The `INSTALLER` file is a file that can be added to the `.dist-info`
folder with the installer name.
Closes: #1759
## Test Plan
Not sure if there is a place I can automatically test it, if you have a
pointer I would be happy to add a test.
PEP 508 requires a space between a URL and the semicolon separating it
from the markers to disambiguate it from a url ending with a semicolon.
This is easy to get wrong because the space is not required after a
plain name of PEP 440 specifier. The new error message explicitly points
out the missing space.
Fixes#1637
## Summary
The `DefaultResolverProvider` struct was not public. This PR exposes it
so we can build our own and use this as a fallback.
## Test Plan
I did not explicitly test this trivial change.
A WARN log was being emitted for a "broken cache entry" in the case
where the cache entry simply doesn't exist. But this is totally fine and
expected. So we detect the kind of error that occurred and emit a TRACE
if the file simply didn't exist.
A file in a zip can set arbitrary unix permissions, but we, like pip,
want to preserve only the executable bit and otherwise use the OS
defaults.
This should be faster for wheels with many files since we now avoid the
blocking fs call to set the permissions in most cases.
Fixes#1740.
## Summary
Don't preserve mtime to work around alexcrichton/tar-rs#349. Same as
#634 except for the streaming unzip.
Fixes#1748.
## Test Plan
Added the tomli source dist as test case.
## Summary
I am looking to instantiate a `RegistryClient`. However, when using the
`RegistryClientBuilder` a new reqwest client is always constructed. I
would like to pass in a custom `reqwest::Client` to be able to share the
http resources with other parts of my application.
## Test Plan
The uv codebase does not use my addition to the builder and all tests
still succeed. And in my code I can pass a custom Client.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Add the environment variable `UV_REQUEST_TIMEOUT` to allow control over
pip timeouts.
Closes#1549
## Test Plan
I built uv in the repository top Dockerfile, set the timeout to 3
seconds, and ran `uv pip install torch`.
I measured the execution time with the time command and confirmed that
the process finished at a value close to the timeout we set.
```bash
root@037c69228cdc:~# time UV_REQUEST_TIMEOUT=3 /uv pip install torch
Resolved 22 packages in 25ms
error: Failed to download distributions
Caused by: Failed to fetch wheel: nvidia-cusolver-cu12==11.4.5.107
Caused by: Failed to extract source distribution
Caused by: request or response body error: operation timed out
Caused by: operation timed out
real 0m3.064s
user 0m0.225s
sys 0m0.240s
```
## Summary
This opens up space to add other cache-related commands. (`uv clean`
continues to work for backwards compatibility but is hidden from the
CLI.)
## Summary
We don't control these, so it seems preferable _not_ to fail on them,
but rather, to just ignore them entirely. (I considered adding a long
allow-list, but then questioned the point of it? We'd end up having to
extend it if more invalid extras were published in the future.)
Closes https://github.com/astral-sh/uv/issues/1633.
## Summary
The main change is that we need to have an explicit list of protocols we
_do_ support (like `https`), so that when we see a Windows absolute path
(`C:\...`), we don't treat the `C` as a protocol itself.
Closes https://github.com/astral-sh/uv/issues/1539.
## Summary
When we read `--index-url` from a `requirements.txt`, we attempt to
respect the `--index-url` provided by the CLI if it exists.
Unfortunately, `--index-url` from the CLI has a default value... so we
_never_ respect the `--index-url` in the requirements file.
This PR modifies the CLI to use `None`, and moves the default into logic
in the `IndexLocations `struct.
Closes https://github.com/astral-sh/uv/issues/1692.
Uses `--find-links` to discover vendored scenario build dependencies and
allows us to use `--index-url` instead of `--extra-index-url` to avoid
hitting the real PyPI in scenario tests.
## Summary
This fixes https://github.com/astral-sh/uv/issues/1704 by removing the
version from the produced header.
## Test Plan
Checked with clippy, and tests are updated too.
## Summary
This PR adds the `--prompt` option to `venv` subcommand.
The default behavior for `uv venv` is to create a virtual environment in
the current directory with `.venv` name. This is different from `venv` /
`virtualenv` where a user always needs to provide the virtual
environment path. This allows us to define our own behavior in the
default scenario (`uv venv`). We've decided to use the current
directory's name in that case.
Workflows:
| Command | Virtual Environment Name | Prompt |
|--------|--------|--------|
| `uv venv` | `.venv` (default) | Current directory name |
| `uv venv project` | `project` | `project` |
| `uv venv --prompt .` | `.venv` | Current directory name |
| `uv venv --prompt foobar` | `.venv` | `foobar` |
| `uv venv project --prompt foobar` | `project` | `foobar` |
Fixes#1445
## Test Plan
This is my first Rust code and I don't know how to write tests yet.
I just checked the behavior manually:
```
$ cargo build
$ mkdir t
$ cd t
$ ../target/debug/uv venv -p 3.11
$ rg -w t .venv/bin/acti*
.venv/bin/activate.csh
13:setenv VIRTUAL_ENV '/Users/inada-n/work/uv/t/.venv'
20:if ('t' != "") then
21: setenv VIRTUAL_ENV_PROMPT 't'
23: setenv VIRTUAL_ENV_PROMPT "$VIRTUAL_ENV:t:q"
38: # in which case, $prompt is undefined and we wouldn't
.venv/bin/activate
48:VIRTUAL_ENV='/Users/inada-n/work/uv/t/.venv'
59: VIRTUAL_ENV_PROMPT="t"
.venv/bin/activate.fish
61:set -gx VIRTUAL_ENV '/Users/inada-n/work/uv/t/.venv'
73:if test -n 't'
74: set -gx VIRTUAL_ENV_PROMPT 't'
.venv/bin/activate.ps1
40:if ("t" -ne "") {
41: $env:VIRTUAL_ENV_PROMPT = "t"
.venv/bin/activate.nu
6:# but then simply `deactivate` won't work because it is just an alias to hide
35: let virtual_env = '/Users/inada-n/work/uv/t/.venv'
50: let virtual_env_prompt = (if ('t' | is-empty) {
53: 't'
```
---------
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
This PR introduces more robust cache healing when `uv` fails to
deserialize an existing cache entry.
("Cache healing" in this context means that if `uv` fails to
deserialize a cache entry, then it will automatically invalidate that
entry and re-generate the data. Typically by sending an HTTP request.)
Previous to some optimizations I made around deserialization, we were
already doing this. After those optimizations, deserializing a cache
policy and the payload were split into two steps. While deserializing
a cache policy retained its cache healing behavior, deserializing the
payload did not. This became an issue when #1556 landed, which changed
one of our `rkyv` data types. This in turn made our internal types
incompatible with existing cache entries. One could work-around this
by clearing `uv`'s cache with `uv clean`, but we should just do it
automatically on a cache entry by entry basis.
This does technically introduce a new cost by pessimistically cloning
the HTTP request so that we can re-send it if necessary (see the commit
messages for the knot pushing me toward this approach). So I re-ran my
favorite ad-hoc benchmark:
```
$ hyperfine -w10 --runs 50 "uv-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null" "uv-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null" ; A bart
Benchmark 1: uv-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
Time (mean ± σ): 114.4 ms ± 3.2 ms [User: 149.4 ms, System: 221.5 ms]
Range (min … max): 106.7 ms … 122.0 ms 50 runs
Benchmark 2: uv-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
Time (mean ± σ): 114.0 ms ± 3.0 ms [User: 146.0 ms, System: 223.3 ms]
Range (min … max): 105.3 ms … 121.4 ms 50 runs
Summary
uv-test pip compile --cache-dir ~/astral/tmp/cache-test ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null ran
1.00 ± 0.04 times faster than uv-main pip compile --cache-dir ~/astral/tmp/cache-main ~/astral/tmp/reqs/home-assistant-reduced.in -o /dev/null
```
Which is about what I expected.
We should endeavor to have a better testing strategy for these kinds of
bugs, but I think it might be a little tricky to do. I created
https://github.com/astral-sh/uv/issues/1699 to track that.
Fixes#1571
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Adds cli command / flag (`generate-shell-completion <SHELL>` /
`--generate-shell-completion <SHELL>`) to generate the completion script
for the given shell. Implemented in exactly the same way as it is done
in ruff
(https://github.com/astral-sh/ruff/blob/main/crates/ruff/src/lib.rs#L197)
Closes https://github.com/astral-sh/uv/issues/1654
## Test Plan
I've normally tested the generated script manually only for bash shell
on Ubuntu 22.04.3
```bash
$ uv --generate-shell-completion bash > /usr/share/bash-completion/completions/uv
$ uv # <TAB>
-q -h --verbose --no-cache --version clean
-v -V --no-color --cache-dir pip generate-shell-completion
-n --quiet --color --help venv help
$ uv pip # <TAB>
-q -n -V --verbose --color --cache-dir --version sync uninstall help
-v -h --quiet --no-color --no-cache --help compile install freeze
```
Resolves#1292.
## Summary
Move the yanked warnings for `uv pip sync` and `uv pip install` to the
end of the commands, as per #1292.
## Test Plan
I ran the unit tests: `cargo nextest run`
## Summary
Just as we mark virtualenvs as `gitignore`d by default, we should also
mark them as `CACHEDIR.TAG`, to ensure that they aren't included in
backups, etc.
Closes https://github.com/astral-sh/uv/issues/1648.
## Test Plan
Ran `cargo run venv` and:
```
❯ ls .venv
CACHEDIR.TAG bin lib pyvenv.cfg
```
## Summary
Added `uv` to the list of the preserved packages when building the
installer plan. In that case `uv` is not going to be removed when, for
example, using `python -m uv pip sync requirements.txt` when
requirements.txt does not contain `uv`, but `uv` is installed in that
venv.
Closes#1631
## Test Plan
Got through the example attached to
https://github.com/astral-sh/uv/issues/1631 and did see the uv deletion
in the output
```
$ python -m uv pip sync requirements.txt
Installed 1 package in 20ms
+ ruff==0.2.2
```
## Sumamry
This PR adds the `activation.bat`, `deactivation.bat` and `pyenv.bat`
files to add support for using uv from CMD.
This PR further fixes an issue with our trampoline implementation where
calling an executable like `black` failed:
```
(venv) C:\Users\Micha\astral\test>where black
C:\Users\Micha\astral\test\.venv\Scripts\black.exe
(venv) C:\Users\Micha\astral\test>black
C:\Users\Micha\AppData\Local\Programs\Python\Python312\python.exe: can't open file 'C:\\Users\\Micha\\astral\\test\\black': [Errno 2] No such file or directory
```
The issue was that CMD doesn't extend `black` to its full path before
passing it to the trampoline and our trampoline generated the command
`<python> black` instead of `<python> .venv/Scripts/black`, and Python
can't find `black` in the project directory.
This PR fixes this by using the full executable name (that we already
parsed out to discover the Python version). This adds one complication,
we need to preserve the arguments without repeating the executable name
that is the first argument.
One option is to use
[`CommandLineToArgvW`](https://learn.microsoft.com/de-de/windows/win32/api/shellapi/nf-shellapi-commandlinetoargvw)
and then serialize the arguments 1.. to a string again. I decided
against that. Win32 API calls are easy to get wrong. That's why I
implemented the parsing rules specified in
[`CommandLineToArgvW`](https://learn.microsoft.com/de-de/windows/win32/api/shellapi/nf-shellapi-commandlinetoargvw)
to skip the first argument.
Fixes https://github.com/astral-sh/uv/issues/1471
## Test Plan
https://github.com/astral-sh/uv/assets/1203881/bdb537b6-97c8-4f7e-bb4a-3a614eb5e0f6
Powershell continues to work
https://github.com/astral-sh/uv/assets/1203881/6c806477-a7c6-4047-9ffc-5ed91c6f1c84
I haven't been able to test the aarch binaries.
## Summary
If an editable package declares a direct URL requirement, we currently
error since it's not considered an "allowed" requirement. We need to add
those URLs to the allow-list.
Closes https://github.com/astral-sh/uv/issues/1603.
## Summary
It's incorrect to pass the resolution and dependency mode down to the
`BuildDispatch`, since it means that we'll use `--no-deps` when building
source distributions. If you set resolution to `lowest`, it also means
we end up using (e.g.) the lowest version of `wheel`, which also doesn't
make sense.
It's fine to pass `--exclude-newer`.
Closes https://github.com/astral-sh/uv/issues/1355.
Closes https://github.com/astral-sh/uv/issues/1563.
This PR fixes the bug where the `BIN_NAME` replacement field wasn't
being used in the activator scripts.
fixes: #1518
## Test plan
As I don't have a Windows machine, I switched the `bin_name` value here
to point to `Scripts` on `unix` platform:
2a76c59084/crates/gourgeist/src/bare.rs (L99-L105)
<details><summary>Code diff</summary>
<p>
```diff
```diff
diff --git a/crates/gourgeist/src/bare.rs b/crates/gourgeist/src/bare.rs
index 4c7808d3..0e0b41cf 100644
--- a/crates/gourgeist/src/bare.rs
+++ b/crates/gourgeist/src/bare.rs
@@ -97,9 +97,9 @@ pub fn create_bare_venv(location: &Utf8Path,
interpreter: &Interpreter) -> io::R
// TODO(konstin): I bet on windows we'll have to strip the prefix again
let location = location.canonicalize_utf8()?;
let bin_name = if cfg!(unix) {
- "bin"
- } else if cfg!(windows) {
"Scripts"
+ } else if cfg!(windows) {
+ "bin"
} else {
unimplemented!("Only Windows and Unix are supported")
};
```
</p>
</details>
I then created the virtual environment as usual and tested out that the path modifications were correct:
```console
$ cargo run --bin uv -- venv
Finished dev [unoptimized + debuginfo] target(s) in 0.13s
Running `target/debug/uv venv`
Using Python 3.12.1 interpreter at
/Users/dhruv/.pyenv/versions/3.12.1/bin/python3.12
Creating virtualenv at: .venv
$ source .venv/Scripts/activate
$ echo $PATH
/Users/dhruv/work/astral/uv/.venv/Scripts:[...]
$ which python
/Users/dhruv/work/astral/uv/.venv/Scripts/python
```
I'm not sure how else to test this without having access to a Windows machine
## Summary#1562
It turns out that `hexdump` uses an invalid source distribution format
whereby the contents aren't nested in a top-level directory -- instead,
they're all just flattened at the top-level. In looking at pip's source
(51de88ca64/src/pip/_internal/utils/unpacking.py (L62)),
it only strips the top-level directory if all entries have the same
directory prefix (i.e., if it's the only thing in the directory). This
PR accommodates these "invalid" distributions.
I can't find any history on this method in `pip`. It looks like it dates
back over 15 years ago, to before `pip` was even called `pip`.
Closes https://github.com/astral-sh/uv/issues/1376.
## Summary
This was just a missing line -- we have `dependencies.remove(&package);`
in the ~identical branch above, but it must've been an oversight to omit
it here.
Closes https://github.com/astral-sh/uv/issues/1467.
## Test Plan
`cargo test`
## Summary
It turns out that it's not uncommon to end up with repeated packages in
requirements files when running `pip-sync`, e.g., you might have
`anyio==4.0.0` specified multiple times. This PR relaxes our assertions
in the install plan to allow such repeated packages, as long as the
requirement markers are exactly the same (i.e., they are truly
duplicates).
Closes https://github.com/astral-sh/uv/issues/1552.
## Summary
If you're developing on a package like `attrs` locally, and it has a
recursive extra like `attrs[dev]`, it turns out that we then try to find
the `attrs` in `attrs[dev]` from the registry, rather than recognizing
that it's part of the editable.
This PR fixes the issue by making editables slightly more first-class
throughout the resolver. Instead of mocking metadata, we explicitly
check for extras in various places. Part of the problem here is that we
treated editables as URL dependencies, but when we saw an _extra_ like
`attrs[dev]`, we didn't map that back to the URL. So now, we treat them
as registry dependencies, but with the appropriate guardrails
throughout.
Closes https://github.com/astral-sh/uv/issues/1447.
## Test Plan
- Cloned `attrs`.
- Ran `cargo run venv && cargo run pip install -e ".[dev]" -v`.
## Summary
This _could_ fix https://github.com/astral-sh/uv/issues/1454, but I'm
not sure. I was able to replicate by forcing a bunch of error states.
But, in short, if we fail to hardlink on the initial copy due to a file
existing, and then fail _again_, we fallback to copying. But if we copy,
then the tempfile doesn't exist, and so the `fs_err::rename(&tempfile,
&out_path)?;` will fail with "File not found".
This PR just ensures that the cases are explicitly mutually exclusive:
we only attempt to rename if the hardlink succeeded.
This PR fixes the OS detection for Alpine Linux such that the version
of musl available is correctly determined. The issue boiled down to
a regex that required 2 digits for each version component. But a
valid musl version is 1.2.4, which only has a single digit for each
component.
It's unclear how this was working for musl before this change. My
theory is that our other methods of OS detection were somehow working.
The first commit in this PR cleans up our Linux detection logic and adds
lots of tracing calls to make debugging issues like this easier in the
future. To do so, one can run:
$ RUST_LOG=trace uv pip install -v whatever
The second commit has the actual fix.
Fixes#1427
## Summary
By using the display representation of `Version` to form a `PackageId`,
we run the risk (as seen in the linked issue) of thinking that versions
like `2021.1` and `2021.1.0` are not equivalent.
Closes https://github.com/astral-sh/uv/issues/1536
This fixes a bug where `uv pip install` failed to install `polars`:
```
$ uv pip install polars==0.14.0
error: Failed to download: polars==0.14.0
Caused by: Couldn't parse metadata of polars-0.14.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl from 749022b096cb7c1c2cc32b7f433c4f/polars-0.14.0-cp37-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
Caused by: Operator >= cannot be used with a wildcard version specifier
pyarrow>=4.0.*; extra == 'pyarrow'
^^^^^^^
```
Since `pyarrow>=4.0.*; extra == 'pyarrow'` is invalid *and* it comes
from the metadata of a dependency (that isn't under the control of the
end user), we actually attempt to "fix" it. Namely, wildcard
dependency specifications are only allowed with `==` and `!=`, as per
the [Version Specifiers spec]. (They aren't explicitly forbidden in
these cases, but instead only have specified behavior for the `==` and
`!=` operators.)
This is all fine, but it turns out that when we fix the `>=4.0.*`
component, we also strip the quotes around `pyarrow`. (Because some
dependency specifications include stray quotes.) We fix this by making
our quote stripping a bit more selective. (We require that it appear
adjacent to a digit or a `*`.)
Note that #1477 also reports this error:
```
$ uv pip install 'requests>=2.30.*'
error: Failed to parse `requests>=2.30.*`
Caused by: Operator >= cannot be used with a wildcard version specifier
requests>=2.30.*
```
However, we specifically keep that error message since it's something
under the end user's control. And similarly for a dependency
specification in a `requirements.txt` file.
Fixes#1477
[Version Specifiers spec]:
https://packaging.python.org/en/latest/specifications/version-specifiers/
It turns out that /bin/ls can sometimes be plain text file. For
example, in Rocky Linux 9:
```
$ cat /bin/ls
#!/usr/bin/coreutils --coreutils-prog-shebang=ls
```
However, `/bin/sh` is an ELF binary:
```
$ file /bin/sh
/bin/sh: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=7acbb41bf6f1b7d977f1b44675bf3ed213776835, for GNU/Linux 3.2.0, stripped
```
In a related issue (#1433), @zanieb fixed#1395 where, on NixOS,
`/bin/ls` doesn't exist but `/bin/sh` does. However, the fix attempts
`/bin/ls` first and only tries `/bin/sh` if `/bin/ls` doesn't exist. If
`/bin/ls` exists but isn't a valid ELF file, then the entire enterprise
gives up and `uv` fails to detect the version of `libc` that is
installed.
Instead of tweaking the logic to keep trying `/bin/ls` and then
`/bin/sh` after even if parsing `/bin/ls` fails, we just switch over to
reading `/bin/sh` only. It seems like a more fundamental thing to sniff
and likely less error prone.
We can adjust this heuristic as needed if it provdes to be problematic.
I tested this fix manually on Rocky Linux 9 via Docker:
```
$ cross b -r -p uv --target x86_64-unknown-linux-musl
$ cp target/x86_64-unknown-linux-musl/release/uv ~/astral/issues/uv/i1486/uv
$ docker run --rm -it --mount type=bind,src=/home/andrew/astral/issues/uv/i1486,dst=/host rockylinux:9 bash
[root@df2baa65d2f8 /]# /host/uv venv
Using Python 3.9.18 interpreter at /usr/bin/python3.9
Creating virtualenv at: .venv
[root@df2baa65d2f8 /]#
```
Fixes#1486, Ref #1433
Andrew Gallant
Zanie Blue
Michał Górny
Charlie Marsh
Ibraheem Ahmed
samypr100
konsti
Brian Mego
Nicolas Delaby
Nyakku Shigure
Di-Is
Tim de Jager
Matthieu Darbois
Ahmed Ilyas
Tim de Jager
Ofek Lev
renovate[bot]
Ben Beasley
Adolfo Ochagavía
Bas Schoenmaeckers
Dimitri Papadopoulos Orfanos
Fred Stober
Chan Kang
Tom Parker-Shemilt
Shantanu
DaniPopes
Godefroid Chapelle
Alexander Gherm
Ahmed Ilyas
Yorick
Grzegorz Bokota
Michael Ritsema
haarisr
哇呜哇呜呀咦耶
Jack Cherng
Paul Moore
Sergey Kolosov
elbaro
Alvise Vianello
Alex Waygood
Jos van de Wolfshaar
Aria Beingessner
Sławomir Ehlert
Ikko Eltociear Ashimine
Zander
eth3lbert
Hans Baker
Wolf Vollprecht
veryyet
Hans Baker
Micha Reiser
John
Jacob Coffee
Robert Resch
Jonathon Belotti
wim glenn
Zhan Rongrui
jannisko
Simon Brugman
dependabot[bot]
Taniguchi Yasufumi
Jo
danieleades
Jonathan Newnham
Evgeniy Dubovskoy
Bas Zalmstra
Jane Lewis
Edgar Ramírez Mondragón
Bas Zalmstra
Viktor Szépe
обоо
Di-Is
Olivier Le Floch
Henry Schreiner
Inada Naoki
markmmm
Alexander Gherm
Daniel Reeves
Arjun Munji
Alexander Gherm
Aarni Koskela
Dhruv Manilawala
David Szotten