This is a bit of a weird request, but in [pixi](https://pixi.sh) we are
making use of this function to lazily instantiate a conda environment.
Well, in actuality we are using a shim to the `BuildDispatch` to
actually to only create a conda prefix, if some package needs to be
built during the resolution phase. Otherwise we can resolve everything
without an enviroment containing a python intepreter.
We are using a method now - that uses the runtime to run async code
inside this function, as `interpreter` is the first method called on a
`BuildContext` when running a source build - using
`tokio::Handle::block_on`.
However was causing a deadlock in very specific situations, me and
@baszalmstra + @wolfv have investigated this thoroughly, but have not
been able to find the root cause. It would hang in a part of the uv code
that hits the index, but that is **after** all of our initialization
*and the blocking call* was completed.
Changing this to be fully async fixes the problem, this requires this
method to be async though.
We get that this is not necessarily required, and we might find a
workaround, but I wanted to try it this way first.
Thanks!
Replaces https://github.com/astral-sh/uv/pull/14092
Adds `tool.uv.extra-build-dependencies = {package = [dependency, ...]}`
which extends `build-system.requires` during package builds.
These are lowered via workspace sources, are applied to transitive
dependencies, and are included in the wheel cache shard hash.
There are some features we need to follow-up on, but are out of scope
here:
- Preferring locked versions for build dependencies
- Settings for requiring locked versions for build depencies
There are some quality of life follow-ups we should also do:
- Warn on `extra-build-dependencies` that do not apply to any packages
- Add test cases and improve error messaging when the
`extra-build-dependencies` resolve fails
-------
There ~are~ were a few open decisions to be made here
1. Should we resolve these dependencies alongside the
`build-system.requires` dependencies? Or should we resolve separately?
(I think the latter is more powerful? because you can override things?
but it opens the door to breaking your build)
2. Should we install these dependencies into the same environment? Or
should we layer it on top as we do elsewhere? (I think it's fine to
install into the same environment)
3. Should we respect sources defined in the parent project? (I think
yes, but then we need to lower the dependencies earlier — I don't think
that's a big deal, but it's not implemented)
4. Should we respect sources defined in the child project? (I think no,
this gets really complicated and seems weird to allow)
5. Should we apply this to transitive dependencies? (I think so)
---------
Co-authored-by: Aria Desires <aria.desires@gmail.com>
Co-authored-by: konstin <konstin@mailbox.org>
## Summary
This should give us some performance and error message improvements.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
This was just an oversight on my part in the initial implementation.
Closes https://github.com/astral-sh/uv/issues/14719.
## Test Plan
With:
```toml
[project]
name = "foo"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.13.2"
dependencies = [
]
[[tool.uv.index]]
url = "https://download.pytorch.org/whl/cpu"
cache-control = { api = "max-age=600" }
```
Ran `cargo run lock -vvv` and verified that the PyTorch index response
was cached (whereas it typically returns `cache-control:
no-cache,no-store,must-revalidate`).
We currently treat path sources as virtual if they do not specify a
build system, which is surprising behavior. This PR updates the behavior
to treat path sources as packages unless the path source is explicitly
marked as `package = false` or its own `tool.uv.package` is set to
`false`.
Closes#12015
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
If a user specifies `-e /path/to/dir` and `/path/to/dir` in a `uv pip
install` command, we want the editable to "win" (rather than erroring
due to conflicting URLs). Unfortunately, this behavior meant that when
you requested a package as editable and non-editable in conflicting
groups, the editable version was _always_ used. This PR modifies the
requisite types to use `Option<bool>` rather than `bool` for the
`editable` field, so we can determine whether a requirement was
explicitly requested as editable, explicitly requested as non-editable,
or not specified (as in the case of `/path/to/dir` in a
`requirements.txt` file). In the latter case, we allow editables to
override the "unspecified" requirement.
If a project includes a path dependency twice, once with `editable =
true` and once without any `editable` annotation, those are now
considered conflicting URLs, and lead to an error, so I've marked this
change as breaking.
Closes https://github.com/astral-sh/uv/issues/14139.
## Summary
The basic idea here is that we can (should) reuse a build environment
across resolution (`prepare_metadata_for_build_wheel`) and installation.
This also happens to solve the build-PyTorch-from-source problem, since
we use a consistent build environment between the invocations.
Since `SourceDistributionBuilder` is stateless, we instead store the
builds on `BuildContext`, and we key them by various properties: the
underlying interpreter, the configuration settings, etc. This just
ensures that if we build the same package twice within a process, we
don't accidentally reuse an incompatible build (virtual) environment.
(Note that still drop build environments at the end of the command, and
don't attempt to reuse them across processes.)
Closes#14269.
## Summary
This PR ensures that we avoid cleaning up build directories until the
end of a resolve-and-install cycle. It's not bulletproof (since we could
still run into issues with `uv lock` followed by `uv sync` whereby a
build directory gets cleaned up that's still referenced in the `build`
artifacts), but it at least gets PyTorch building without error with `uv
pip install .`, which is a case that's been reported several times.
Closes https://github.com/astral-sh/uv/issues/14269.
Don't log that we resolved a reference through the GitHub fast path if
we didn't use GitHub at all but used the cached revision. This avoids
stating that the fast path works when it's blocked due to unrelated
reasons (e.g. rate limits).
This PR is a combination of #12920 and #13754. Prior to these changes,
following a redirect when searching indexes would bypass our
authentication middleware. This PR updates uv to support propagating
credentials through our middleware on same-origin redirects and to
support netrc credentials for both same- and cross-origin redirects. It
does not handle the case described in #11097 where the redirect location
itself includes credentials (e.g.,
`https://user:pass@redirect-location.com`). That will be addressed in
follow-up work.
This includes unit tests for the new redirect logic and integration
tests for credential propagation. The automated external registries test
is also passing for AWS CodeArtifact, Azure Artifacts, GCP Artifact
Registry, JFrog Artifactory, GitLab, Cloudsmith, and Gemfury.
Using a companion change in the middleware
(https://github.com/TrueLayer/reqwest-middleware/pull/235, forked&tagged
pending review), we can check and show retries for HTTP status core
errors, to consistently report retries again.
We fix two cases:
* Show retries for status code errors for cache client requests
* Show retries for status code errors for Python download requests
Not handled:
* Show previous retries when a distribution download fails mid-streaming
* Perform retries when a distribution download fails mid-streaming
* Show previous retries when a Python download fails mid-streaming
* Perform retries when a Python download fails mid-streaming
(or legacy tool.uv.workspace).
This cleaves out a dedicated SourcedDependencyGroups type based on
RequiresDist but with only the DependencyGroup handling implemented.
This allows `uv pip` to read `dependency-groups` from pyproject.tomls
that only have that table defined, per PEP 735, and as implemented by
`pip`.
However we want our implementation to respect various uv features when
they're available:
* `tool.uv.sources`
* `tool.uv.index`
* `tool.uv.dependency-groups.mygroup.requires-python` (#13735)
As such we want to opportunistically detect "as much as possible" while
doing as little as possible when things are missing. The issue with the
old RequiresDist path was that it fundamentally wanted to build the
package, and if `[project]` was missing it would try to desperately run
setuptools on the pyproject.toml to try to find metadata and make a hash
of things.
At the same time, the old code also put in a lot of effort to try to
pretend that `uv pip` dependency-groups worked like `uv`
dependency-groups with defaults and non-only semantics, only to separate
them back out again. By explicitly separating them out, we confidently
get the expected behaviour.
Note that dependency-group support is still included in RequiresDist, as
some `uv` paths still use it. It's unclear to me if those paths want
this same treatment -- for now I conclude no.
Fixes#13138
This allows you to specify requires-python on individual dependency-groups,
with the intended usecase being "oh my dev-dependencies have a higher
requires-python than my actual project".
This includes a large driveby move of the RequiresPython type to
uv-distribution-types to allow us to generate the appropriate markers at
this point in the code. It also migrates RequiresPython from
pubgrub::Range to version_ranges::Ranges, and makes several pub(crate)
items pub, as it's no longer defined in uv_resolver.
Fixes#11606
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [tempfile](https://stebalien.com/projects/tempfile-rs/)
([source](https://redirect.github.com/Stebalien/tempfile)) |
workspace.dependencies | minor | `3.19.1` -> `3.20.0` |
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>Stebalien/tempfile (tempfile)</summary>
###
[`v3.20.0`](https://redirect.github.com/Stebalien/tempfile/blob/HEAD/CHANGELOG.md#3200)
[Compare
Source](https://redirect.github.com/Stebalien/tempfile/compare/v3.19.1...v3.20.0)
This release mostly unifies the behavior/capabilities around "keeping"
temporary files:
- Rename `Builder::keep(bool)` (via deprecation) to
`Builder::disable_cleanup(bool)` to make it clear that behaves
differently from `NamedTempFile::keep()`. The former disables automatic
cleanup while the latter *consumes* the `NamedTempFile` object entirely
and unsets the "temporary file" attribute (on Windows).
- Rename `TempDir::into_path` (via deprecation) to `TempDir::keep` to
mirror `NamedTempFile::keep`.
- Add `TempDir::disable_cleanup`, `NamedTempFile::disable_cleanup`, and
`TempPath::disable_cleanup` making it possible to disable automatic
cleanup in-place *after* creating a temporary file/directory (equivalent
to calling `Builder::disable_cleanup` before creating the
file/directory).
Additionally, it adds a few spooled temporary file features:
- Add `SpooledTempFile::into_file` for turning a `SpooledTempFile` into
a regular unnamed temporary file, writing it to the backing storage
("rolling" it) if it was still stored in-memory.
- Add `spooled_tempfile_in` and `SpooledTempFile::new_in` methods for
creating spooled temporary files in a specific directory. This makes it
possible to choose the backing device for your spooled temporary file
which is rather important on Linux where the default temporary directory
is likely backed by memory (defeating the entire point of having a
spooled temporary file).
Finally, this release improves documentation, especially the top-level
documentation explaining which temporary file type to use.
**BREAKING** for those with `deny(warnings)`:
- `Builder::keep` deprecated in favor of `Builder::disable_cleanup`.
- `TempDir::into_path` is deprecated in favor of `TempDir::keep`.
**BREAKING**:
</details>
---
### Configuration
📅 **Schedule**: Branch creation - "before 4am on Monday" (UTC),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/astral-sh/uv).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MC4zMy42IiwidXBkYXRlZEluVmVyIjoiNDAuMzMuNiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiaW50ZXJuYWwiXX0=-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: konstin <konstin@mailbox.org>
Prior to this PR, there were numerous places where uv would leak
credentials in logs. We had a way to mask credentials by calling methods
or a recently-added `redact_url` function, but this was not secure by
default. There were a number of other types (like `GitUrl`) that would
leak credentials on display.
This PR adds a `DisplaySafeUrl` newtype to prevent leaking credentials
when logging by default. It takes a maximalist approach, replacing the
use of `Url` almost everywhere. This includes when first parsing config
files, when storing URLs in types like `GitUrl`, and also when storing
URLs in types that in practice will never contain credentials (like
`DirectorySourceUrl`). The idea is to make it easy for developers to do
the right thing and for the compiler to support this (and to minimize
ever having to manually convert back and forth). Displaying credentials
now requires an active step. Note that despite this maximalist approach,
the use of the newtype should be zero cost.
One conspicuous place this PR does not use `DisplaySafeUrl` is in the
`uv-auth` crate. That would require new clones since there are calls to
`request.url()` that return a `&Url`. One option would have been to make
`DisplaySafeUrl` wrap a `Cow`, but this would lead to lifetime
annotations all over the codebase. I've created a separate PR based on
this one (#13576) that updates `uv-auth` to use `DisplaySafeUrl` with
one new clone. We can discuss the tradeoffs there.
Most of this PR just replaces `Url` with `DisplaySafeUrl`. The core is
`uv_redacted/lib.rs`, where the newtype is implemented. To make it
easier to review the rest, here are some points of note:
* `DisplaySafeUrl` has a `Display` implementation that masks
credentials. Currently, it will still display the username when there is
both a username and password. If we think is the wrong choice, it can
now be changed in one place.
* `DisplaySafeUrl` has a `remove_credentials()` method and also a
`.to_string_with_credentials()` method. This allows us to use it in a
variety of scenarios.
* `IndexUrl::redacted()` was renamed to
`IndexUrl::removed_credentials()` to make it clearer that we are not
masking.
* We convert from a `DisplaySafeUrl` to a `Url` when calling `reqwest`
methods like `.get()` and `.head()`.
* We convert from a `DisplaySafeUrl` to a `Url` when creating a
`uv_auth::Index`. That is because, as mentioned above, I will be
updating the `uv_auth` crate to use this newtype in a separate PR.
* A number of tests (e.g., in `pip_install.rs`) that formerly used
filters to mask tokens in the test output no longer need those filters
since tokens in URLs are now masked automatically.
* The one place we are still knowingly writing credentials to
`pyproject.toml` is when a URL with credentials is passed to `uv add`
with `--raw`. Since displaying credentials is no longer automatic, I
have added a `to_string_with_credentials()` method to the `Pep508Url`
trait. This is used when `--raw` is passed. Adding it to that trait is a
bit weird, but it's the simplest way to achieve the goal. I'm open to
suggestions on how to improve this, but note that because of the way
we're using generic bounds, it's not as simple as just creating a
separate trait for that method.
Rustfmt introduces a lot of formatting changes in the 2024 edition. To
not break everything all at once, we split out the set of formatting
changes compatible with both the 2021 and 2024 edition by first
formatting with the 2024 style, and then again with the currently used
2021 style.
Notable changes are the formatting of derive macro attributes and lines
with overly long strings and adding trailing semicolons after statements
consistently.
#5577 fixed a bug on macos due to dynamically linking lzma/xz through
static linking. In #7686, this feature was moved to the performance
category.
This PR moves the `xz2/static` back to the general default features,
and, inspired by https://github.com/Homebrew/homebrew-core/pull/222211,
it structures and documents the feature flags cleaner.
We need to take care that this feature does not accidentally disable
features we want.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
This PR restores #13041 and integrates two PRs from @zanieb:
* #13038
* #13040
It also adds tests for relative URI and fragment handling.
Closes#13037.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
uv was failing to authenticate on 302 redirects when credentials were
available. This was because it was relying on `reqwest_middleware`'s
default redirect behavior which bypasses the middleware pipeline when
trying the redirect request (and hence bypasses our authentication
middleware). This PR updates uv to retrigger the middleware pipeline
when handling a 302 redirect, correctly using credentials from the URL,
the keyring, or `.netrc`.
Closes#5595Closes#11097
## Summary
closes#12234
[fetch_with_cli](e0f81f0d4a/crates/uv-git/src/git.rs (L573))
doesn't respect the registry client's [connectivity
setting](e0f81f0d4a/crates/uv-client/src/registry_client.rs (L1009))
- this pr updates `fetch_with_cli` to set `GIT_ALLOW_PROTOCOL=file` when
the client's connectivity setting is `Connectivity::Offline`
## Test Plan
E2E
```sh
cargo run add "pycurl @ git+https://github.com/pycurl/pycurl.git" --directory ~/src/offline-test/ --offline
```
```sh
Compiling uv-cli v0.0.1 (/Users/justinchapman/src/uv/crates/uv-cli)
Compiling uv v0.6.11 (/Users/justinchapman/src/uv/crates/uv)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.47s
Running `target/debug/uv add 'pycurl @ git+https://github.com/pycurl/pycurl.git' --directory /Users/justinchapman/src/offline-test/ --offline`
Updating https://github.com/pycurl/pycurl.git (HEAD) × Failed to download and build `pycurl @ git+https://github.com/pycurl/pycurl.git`
├─▶ Git operation failed
├─▶ failed to fetch into: /Users/justinchapman/.cache/uv/git-v0/db/9a596e5213c3162d
╰─▶ process didn't exit successfully: `/usr/bin/git fetch --force --update-head-ok 'https://github.com/pycurl/pycurl.git' '+HEAD:refs/remotes/origin/HEAD'` (exit status: 128)
--- stderr
fatal: transport 'https' not allowed
help: If you want to add the package regardless of the failed resolution, provide the `--frozen` flag to skip locking and syncing.
```
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
This PR extends `[[tool.uv.index]]` to support `--find-links`-style
"flat" indexes, so that users can point to such indexes without using
`--find-links` _and_ get access to the full functionality of
`[[tool.uv.index]]` (e.g., they can now pin packages to
`--find-links`-style indexes).
Note that, at present, `--find-links` indexes actually have some quirky
behavior, in that we combine them into a single entity and then merge
the discovered distributions into each Simple API-style index. The
motivation here, IIRC, was to match pip's behavior quite closely. I'm
interested in _removing_ that behavior, but it'd be breaking (and may
also be inconvenient for some use-cases). So, the behavior for indexes
passed in via `--find-links` remains completely unchanged. However,
`[[tool.uv.index]]` entries with `format = "flat"` are now treated
identically to those defined with `format = "simple"` (the default), in
that we stop after we find the first-matching index, etc.
Closes https://github.com/astral-sh/uv/issues/11634.
## Summary
I don't know if I actually want to commit this, but I did it on the
plane last time and just polished it off (got it to compile) while
waiting to board.
## Summary
This PR modifies the requirement source entities to store a (new)
container struct that wraps `IndexUrl`. This will allow us to store
user-defined metadata alongside `IndexUrl`, and propagate that metadata
throughout resolution.
Specifically, I need to store the "kind" of the index (Simple API vs.
`--find-links`), but I also ran into this problem when I tried to add
support for overriding `Cache-Control` headers on a per-index basis: at
present, we have no way to passing around metadata alongside an
`IndexUrl`.
## Summary
This crate is for standards-compliant types, but this is explicitly a
type that's custom to uv. It's also strange because we kind of want to
reference `IndexUrl` on the registry type, but that's in a crate that
_depends_ on `uv-pypi-types`, which to me is a sign that this is off.
## Summary
This ended up being more involved than expected. The gist is that we
setup all the packages we want to reinstall upfront (they're passed in
on the command-line); but at that point, we don't have names for all the
packages that the user has specified. (Consider, e.g., `uv pip install
.` -- we don't have a name for `.`, so we can't add it to the list of
`Reinstall` packages.)
Now, `Reinstall` also accepts paths, so we can augment `Reinstall` based
on the user-provided paths.
Closes#12038.
Reduce the overhead of `uv run` in large workspaces. Instead of
re-discovering the entire workspace each time we resolve the metadata of
a member, we can the discovered set of workspace members. Care needs to
be taken to not cache the discovery for `uv init`, `uv add` and `uv
remove`, which change the definitions of workspace members.
Below is apache airflow e3fe06382df4b19f2c0de40ce7c0bdc726754c74 `uv run
python` with a minimal payload. With this change, we avoid a ~350ms
overhead of each `uv run` invocation.
```
$ hyperfine --warmup 2 \
"uv run --no-dev python -c \"print('hi')\"" \
"uv-profiling run --no-dev python -c \"print('hi')\""
Benchmark 1: uv run --no-dev python -c "print('hi')"
Time (mean ± σ): 492.6 ms ± 7.0 ms [User: 393.2 ms, System: 97.1 ms]
Range (min … max): 482.3 ms … 501.5 ms 10 runs
Benchmark 2: uv-profiling run --no-dev python -c "print('hi')"
Time (mean ± σ): 129.7 ms ± 2.5 ms [User: 105.4 ms, System: 23.2 ms]
Range (min … max): 126.0 ms … 136.1 ms 22 runs
Summary
uv-profiling run --no-dev python -c "print('hi')" ran
3.80 ± 0.09 times faster than uv run --no-dev python -c "print('hi')"
```
The profile after those change below. We still spend a large chunk in
toml parsing (both `uv.lock` and `pyproject.toml`), but it's not
excessive anymore.
## Summary
In https://github.com/astral-sh/uv/issues/11998, a user is attempting to
vendor `pydantic-core`. But when they add `pydantic-core = { path =
"src/foo/vendor/pydantic-core" } `, we're installing it as a virtual
package, since `pydantic-core/pyproject.toml` contains `package =
false`.
This PR allows users to mark dependencies as "explicitly a package" or
"explicitly not a package" (i.e., virtual), as a workaround.
Closes https://github.com/astral-sh/uv/issues/11998.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Closes#2410
<!-- What's the purpose of the change? What does it do, and why? -->
This changes the name of files in `wheels` bucket to use a hash instead
of the wheel name as to not exceed maximum file length limit on various
systems.
This only addresses the primary concern of #2410. It still does _not_
address:
- Path limit of 260 on windows:
https://github.com/astral-sh/uv/issues/2410#issuecomment-2062020882
To solve this we need to opt-in to longer path limits on windows
([ref](https://github.com/astral-sh/uv/issues/2410#issuecomment-2150532658)),
but I think that is a separate issue and should be a separate MR.
- Exceeding filename limit while building a wheel from source
distribution
As per my understanding, this is out of uv's control. Name of the output
wheel will be decided by build-backend used by the project. For wheels
built from source distribution, pip also uses the wheel names in cache.
So I have not touched `sdists` cache.
I have added a `filename: WheelFileName` field in `Archive`, so we can
use it while indexing instead of relying on the filename on disk.
Another way to do this was to read `.dist-info/WHEEL` and
`.dist-info/METADATA` and build `WheelFileName` but that seems less
robust and will be slower.
## Test Plan
<!-- How was it tested? -->
Tested by installing `yt-dlp`, `httpie` and `sqlalchemy` and verifying
that cache files in `wheels` bucket use hash.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
* Upgrade the rust toolchain to 1.85.0. This does not increase the MSRV.
* Update windows trampoline to 1.86 nightly beta (previously in 1.85
nightly beta).
## Test Plan
Existing tests
## Summary
I believe these are not necessary... They're currently used in two
places:
1. When building wheels. But that's already wrapped in an in-flight map,
which does the same thing.
2. When fetching source distribution metadata. But every route there
uses it's own `flock` to coordinate across processes, so this seems
redundant?
Initially, we were limiting Git schemes to HTTPS and SSH as only
supported schemes. We lost this validation in #3429. This incidentally
allowed file schemes, which apparently work with Git out of the box.
A caveat for this is that in tool.uv.sources, we parse the git field
always as URL. This caused a problem with #11425: repo = { git =
'c:\path\to\repo', rev = "xxxxx" } was parsed as a URL where c: is the
scheme, causing a bad error message down the line.
This PR:
* Puts Git URL validation back in place. It bans everything but HTTPS,
SSH, and file URLs. This could be a breaking change, if users were using
a git transport protocol were not aware of, even though never
intentionally supported.
* Allows file: URL in Git: This seems to be supported by Git and we were
supporting it albeit unintentionally, so it's reasonable to continue to
support it.
* It does not allow relative paths in the git field in tool.uv.sources.
Absolute file URLs are supported, whether we want relative file URLs for
Git too should be discussed separately.
Closes#3429: We reject the input with a proper error message, while
hinting the user towards file:. If there's still desire for relative
path support, we can keep it open.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
We want to build `uv-build` without depending on the network crates. In
preparation for that, we split uv-git into uv-git and uv-git-types,
where only uv-git depends on reqwest, so that uv-build can use
uv-git-types.
We added this to help with resolving some specific packages, and for
parity with Poetry. But in some cases, this metadata is just wrong, and
at the very least it's unreliable.
Closes https://github.com/astral-sh/uv/issues/8989.
Closes#10945.
Instead of using junctions, we can just write files that contain (as the
file contents) the target path. This requires a little more finesse in
that, as readers, we need to know where to expect these. But it also
means we get to avoid junctions, which have led to a variety of
confusing behaviors. Further, `replace_symlink` should now be on atomic
on Windows.
Closes#11263.
We've never bumped the version of this bucket, and we may never do so...
But it's still incorrect for us to omit it from these serialized structs
in the cache. Specifically, these structs include a pointer into the
archive bucket (namely, the ID). But we don't include the bucket
version! So, in theory, we could end up pointing to archives that don't
match the current bucket version expected in the code.
## Summary
Just a logic issue... If we see a dynamic field that isn't `"version"`,
we end up _not_ propagating the fact that `"version"` is also dynamic.
Closes https://github.com/astral-sh/uv/issues/11460.
## Summary
The environment is located at a stable path within the cache, based on
the script's absolute path.
If a lockfile exists for the script, then we use our standard lockfile
semantics (i.e., update the lockfile if necessary, etc.); if not, we
just do a `uv pip sync` (roughly).
Example usage:
```
❯ uv init --script hello.py
Initialized script at `hello.py`
❯ uv add --script hello.py requests
Updated `hello.py`
❯ cargo run sync --script hello.py
Using script environment at: /Users/crmarsh/.cache/uv/environments-v1/hello-84e289fe3f6241a0
Resolved 5 packages in 3ms
Installed 5 packages in 12ms
+ certifi==2025.1.31
+ charset-normalizer==3.4.1
+ idna==3.10
+ requests==2.32.3
+ urllib3==2.3.0
```
Closes https://github.com/astral-sh/uv/issues/6637.
## Summary
We currently enforce that if you do `uv pip install
./dist/iniconfig-1.0.0.tar.gz`, the build _must_ produce a wheel like
`iniconfig-1.0.0-py3-none-any.whl` (i.e., the name and version must
match). It turns out some packages produce a wheel that has a local
suffix on it, like `vllm`. This PR makes the check a little more
permissive in that we now accept `1.0.0` or that version with a local
suffix (e.g., `1.0.0+cpu`). I don't love this practice, but we already
relaxed this check when _installing_ a wheel, so this seems reasonable:
5e15881dcc/crates/uv-install-wheel/src/install.rs (L50-L52)
Note that this is _still_ stricter than pip. pip seems to only require
that the package name is the same (i.e., `iniconfig` matches
`iniconfig`; but they'll happily install a wheel like
`iniconfig-2.0.0-py3-none-any.whl` given
`./dist/iniconfig-1.0.0.tar.gz`).
Closes https://github.com/astral-sh/uv/issues/11038.
## Motivation
No-op `uv lock` in apache airflow
(891c67f210ab7c877d1f00ea6ea3d3cdbb0e96ef) is slow, which makes `uv run`
slow, too.
Reference project:
```
$ hyperfine "uv run python -c \"print('hi')\""
Benchmark 1: uv run python -c "print('hi')"
Time (mean ± σ): 16.3 ms ± 1.5 ms [User: 9.8 ms, System: 6.4 ms]
Range (min … max): 13.0 ms … 20.0 ms 186 runs
```
Apache airflow before:
```
$ hyperfine "uv run python -c \"print('hi')\""
Benchmark 1: uv run python -c "print('hi')"
Time (mean ± σ): 161.0 ms ± 5.2 ms [User: 135.3 ms, System: 24.1 ms]
Range (min … max): 155.0 ms … 176.3 ms 18 runs
```
## Optimization
`FlatRequiresDist::from_requirements` is taking 50% of main thread
runtime.
Before:
After both commits:
Apache airflow after the first commit:
```
$ hyperfine "uv-profiling run python -c \"print('hi')\""
Benchmark 1: uv-profiling run python -c "print('hi')"
Time (mean ± σ): 122.3 ms ± 5.4 ms [User: 96.1 ms, System: 24.7 ms]
Range (min … max): 114.0 ms … 133.2 ms 23 runs
```
Apache airflow after the second commit:
```
$ hyperfine "uv-profiling run python -c \"print('hi')\""
Benchmark 1: uv-profiling run python -c "print('hi')"
Time (mean ± σ): 108.5 ms ± 3.4 ms [User: 83.2 ms, System: 24.2 ms]
Range (min … max): 103.6 ms … 119.9 ms 28 runs
```
These are noisy relative to the effect they have on the user. It seems
better to prioritize hints on poor resolutions. Notably, it seems hard
to make these "not noisy" ref #11091.
Does not include the "lowest" resolution mode, in which lower bounds are
critical.
With the parallel simple index fetching, we would only acquire one
download concurrency token, meaning that we could in the worst case make
times the number of indexes more requests than the user requested limit.
We fix this by passing the semaphore down to the simple API method.
## Summary
If we fail to deserialize cached metadata in the cache, we should just
ignore it, rather than failing.
Ideally, this never happens. If it does, it means we missed a cache
version bump. But if it does happen, it should still be non-fatal.
Closes https://github.com/astral-sh/uv/issues/11043.
Closes https://github.com/astral-sh/uv/issues/11101.
## Test Plan
Prior to this PR, the following would fail:
- `uvx uv@0.5.25 venv --python 3.12 --cache-dir foo`
- `uvx uv@0.5.25 pip install ./scripts/packages/hatchling_dynamic
--no-deps --python 3.12 --cache-dir foo`
- `uvx uv@0.5.18 venv --python 3.12 --cache-dir foo`
- `uvx uv@0.5.18 pip install ./scripts/packages/hatchling_dynamic
--no-deps --python 3.12 --cache-dir foo`
We can't go back and fix 0.5.18, but this will prevent such regressions
in the future.
## Summary
The issue here boils down to: when we write metadata that came from
building the wheel itself, we aren't setting the `dynamic` field.
We now _always_ set the dynamic field when reading, even when we read
cached data.
Closes https://github.com/astral-sh/uv/issues/11047.
## Summary
On Windows, we have a lot of issues with atomic replacement and such.
There are a bunch of different failure modes, but they generally
involve: trying to persist a fail to a path at which the file already
exists, trying to replace or remove a file while someone else is reading
it, etc.
This PR adds locks to all of the relevant database paths. We already use
these advisory locks when building source distributions; now we use them
when unzipping wheels, storing metadata, etc.
Closes#11002.
## Test Plan
I ran the following script:
```shell
# Define the cache directory path
$cacheDir = "C:\Users\crmar\workspace\uv\cache"
# Clear the cache directory if it exists
if (Test-Path $cacheDir) {
Remove-Item -Recurse -Force $cacheDir
}
# Create the cache directory again
New-Item -ItemType Directory -Force -Path $cacheDir
# Define the command to run with --cache-dir flag
$command = {
param ($venvPath)
# Create a virtual environment in the specified path with --python
uv venv $venvPath
# Run the pip install command with --cache-dir flag
C:\Users\crmar\workspace\uv\target\profiling\uv.exe pip install flask==1.0.4 --no-binary flask --cache-dir C:\Users\crmar\workspace\uv\cache -v --python $venvPath
}
# Define the paths for the different virtual environments
$venv1 = "C:\Users\crmar\workspace\uv\venv1"
$venv2 = "C:\Users\crmar\workspace\uv\venv2"
$venv3 = "C:\Users\crmar\workspace\uv\venv3"
$venv4 = "C:\Users\crmar\workspace\uv\venv4"
$venv5 = "C:\Users\crmar\workspace\uv\venv5"
# Start the command in parallel five times using Start-Job, each with a different venv
$job1 = Start-Job -ScriptBlock $command -ArgumentList $venv1
$job2 = Start-Job -ScriptBlock $command -ArgumentList $venv2
$job3 = Start-Job -ScriptBlock $command -ArgumentList $venv3
$job4 = Start-Job -ScriptBlock $command -ArgumentList $venv4
$job5 = Start-Job -ScriptBlock $command -ArgumentList $venv5
# Wait for all jobs to complete
$jobs = @($job1, $job2, $job3, $job4, $job5)
$jobs | ForEach-Object { Wait-Job $_ }
# Retrieve the results (optional)
$jobs | ForEach-Object { Receive-Job -Job $_ }
# Clean up the jobs
$jobs | ForEach-Object { Remove-Job -Job $_ }
```
And ensured it succeeded in five straight invocations (whereas on
`main`, it consistently fails with a variety of different traces).
## Summary
We should only be ignoring changes in `version` for dynamic projects;
for static projects, it should still be enforced. We should also be
invalidating the lockfile if a project goes from static to dynamic or
vice versa.
Closes#10852.
## Summary
For example: in the linked issue, the user has a symlink at
`pyproject.toml`. The GitHub CDN doesn't give us any way to determine
whether a file is a symlink, so we should just log the error and move on
to the slow path.
Closes https://github.com/astral-sh/uv/issues/10857
## Summary
I noticed that we're only handling `Error::WheelMetadataNameMismatch`
here; but `Error::WheelMetadataVersionMismatch` should also be treated
as non-fatal.
## Summary
I needed this for https://github.com/astral-sh/uv/pull/10794, but it
makes sense as a standalone change, since it's much more testable. We
can also reuse this in at least one more place.
## Summary
When resolving Git metadata, we may be able to fetch the metadata from
GitHub directly in some cases. This is _way_ faster, since we don't need
to perform many Git operations and, in particular, don't need to clone
the repo.
This only works in the following cases:
- The Git repository is public. Otherwise, I believe you need an access
token, which we don't have.
- The `pyproject.toml` has static metadata.
- The `pyproject.toml` has no `tool.uv.sources`. Otherwise, we need to
lower them... And, if there are any paths or workspace sources, that
requires an install path (i.e., we need the content on-disk).
- The project is in the repo root. If it's in a subdirectory, it could
be a workspace member. And if it's a workspace member, there could be
sources defined in the workspace root. But we can't know without
fetching the workspace root -- and we need the workspace in order to
find the root...
Closes#10568.
## Summary
This PR modifies the lockfile to omit versions for source trees that use
`dynamic` versioning, thereby enabling projects to use dynamic
versioning with `uv.lock`.
Prior to this change, dynamic versioning was largely incompatible with
locking, especially for popular tools like `setuptools_scm` -- in that
case, every commit bumps the version, so every commit invalidates the
committed lockfile.
Closes https://github.com/astral-sh/uv/issues/7533.
## Summary
Sort of undecided on this. These are already stored as `dyn Reporter` in
each struct, so we're already using dynamic dispatch in that sense. But
all the methods take `impl Reporter`. This is sometimes nice (the
callsites are simpler?), but it also means that in practice, you often
_can't_ pass `None` to these methods that accept `Option<impl
Reporter>`, because Rust can't infer the generic type.
Anyway, this adds more consistency and simplifies the setup by using
`Arc<dyn Reporter>` everywhere.
Build failures are one of the most common user facing failures that
aren't "obivous" errors (such as typos) or resolver errors. Currently,
they show more technical details than being focussed on this being an
error in a subprocess that is either on the side of the package or -
more likely - in the build environment, e.g. the user needs to install a
dev package or their python version is incompatible.
The new error message clearly delineates the part that's important (this
is a build backend problem) from the internals (we called this hook) and
is consistent about which part of the dist building stage failed. We
have to calibrate the exact wording of the error message some more. Most
of the implementation is working around the orphan rule, (this)error
rules and trait rules, so it came out more of a refactoring than
intended.
Example:
## Summary
This has been bothering me a bit: `uv pip install "foo @
https://github.com/user/foo"` fails, telling you that it doesn't end in
a supported extension. But we should be able to tell you that it looks
like a Git repo.
## Summary
This optimization isn't quite right, because we can successfully extract
metadata without having to build from source. (The builder itself will
error if we reach the point at which we need to build, but builds are
disabled.)
Closes https://github.com/astral-sh/uv/issues/9776.
## Summary
On `main`, if you ask for a source but name a missing subdirectory, you
just get:
```
{source} does not appear to be a Python project, as neither `pyproject.toml` nor `setup.py` are present in the directory
```
But, in reality, the directory doesn't exist at all.
## Summary
We were reading an `.egg-info` file from the root directory that didn't
apply to the root member -- it was for another workspace member. I think
this is driven from some idiosyncracies in the `setuptools` setup for
that workspace member, but it's still wrong to fail.
This PR adds a few measures to fix this:
1. We validate the `egg-info` filename against the package metadata.
2. We skip, rather than fail, if we see incorrect metadata in an
`egg-info` file or similar. This is an optimization anyway; worst case,
we try to build the package, then fail there.
Closes https://github.com/astral-sh/uv/issues/9743.
## Summary
This PR allows users to specify a source both in `project.dependencies`
("production") and `tool.uv.sources` ("development"). It's not intended
as a holistic fix for "production" vs. "development" dependencies, but
in some cases this is good enough with `--no-sources`, and I don't see a
great reason for enforcing it right now.
Closes: https://github.com/astral-sh/uv/issues/9682
Ref: https://github.com/astral-sh/uv/issues/7945 (but I'll leave this
open?)
## Summary
Small thing I noticed while working on another change: if we error when
extracting `requires-dist`, we go through the full metadata build. We
need to distinguish between fatal errors and "the data isn't static".
This is like #9556, but at the level of all other builds, including the
resolver and installer. Going through PEP 517 to build a package is
slow, so when building a package with the uv build backend, we can call
into the uv build backend directly instead: No temporary virtual env, no
temp venv sync, no python subprocess calls, no uv subprocess calls.
This fast path is gated through preview. Since the uv wheel is not
available at test time, I've manually confirmed the feature by comparing
`uv venv && cargo run pip install . -v --preview --reinstall .` and `uv
venv && cargo run pip install . -v --reinstall .`. When hacking the
preview so that the python uv build backend works without the setting
the direct build also (wheel built with `maturin build --profile
profiling`), we can see the perfomance difference:
```
$ hyperfine --prepare "uv venv" --warmup 3 \
"UV_PREVIEW=1 target/profiling/uv pip install --no-deps --reinstall scripts/packages/built-by-uv --preview" \
"target/profiling/uv pip install --no-deps --reinstall scripts/packages/built-by-uv --find-links target/wheels/"
Benchmark 1: UV_PREVIEW=1 target/profiling/uv pip install --no-deps --reinstall scripts/packages/built-by-uv --preview
Time (mean ± σ): 33.1 ms ± 2.5 ms [User: 25.7 ms, System: 13.0 ms]
Range (min … max): 29.8 ms … 47.3 ms 73 runs
Benchmark 2: target/profiling/uv pip install --no-deps --reinstall scripts/packages/built-by-uv --find-links target/wheels/
Time (mean ± σ): 115.1 ms ± 4.3 ms [User: 54.0 ms, System: 27.0 ms]
Range (min … max): 109.2 ms … 123.8 ms 25 runs
Summary
UV_PREVIEW=1 target/profiling/uv pip install --no-deps --reinstall scripts/packages/built-by-uv --preview ran
3.48 ± 0.29 times faster than target/profiling/uv pip install --no-deps --reinstall scripts/packages/built-by-uv --find-links target/wheels/
```
Do we need a global option to disable the fast path? There is one for
`uv build` because `--force-pep517` moves `uv build` much closer to a
`pip install` from source that a user of a library would experience (See
discussion at #9610), but uv overall doesn't really make guarantees
around the build env of dependencies, so I consider the direct build a
valid option.
Best reviewed commit-by-commit, only the last commit is the actual
implementation, while the preview mode introduction is just a
refactoring touching too many files.
Charlie Marsh
Tim de Jager
Zanie Blue
Ibraheem Ahmed
John Mumm
konsti
Aria Desires
renovate[bot]
justin
Ankit Saini
samypr100