When we generate conflict markers for each resolution after the
resolver runs, it turns out that generating them just from exclusion
rules is not sufficient.
For example, if `foo` and `bar` are declared as conflicting extras, then
we end up with the following forks:
A: extra != 'foo'
B: extra != 'bar'
C: extra != 'foo' and extra != 'bar'
Now let's take an example where these forks don't share the same version
for all packages. Consider a case where `idna==3.9` is in forks A and C,
but where `idna==3.10` is in fork B. If we combine the markers in forks
A and C through disjunction, we get the following:
idna==3.9: extra != 'foo' or (extra != 'foo' and extra != 'bar')
idna==3.10: extra != 'bar'
Which simplifies to:
idna==3.9: extra != 'foo'
idna==3.10: extra != 'bar'
But these are clearly not disjoint. Both dependencies could be selected,
for example, when neither `foo` nor `bar` are active. We can remedy this
by keeping around the inclusion rules for each fork:
A: extra != 'foo' and extra == 'bar'
B: extra != 'bar' and extra == 'foo'
C: extra != 'foo' and extra != 'bar'
And so for `idna`, we have:
idna==3.9: (extra != 'foo' and extra == 'bar') or (extra != 'foo' and extra != 'bar')
idna==3.10: extra != 'bar' and extra == 'foo'
Which simplifies to:
idna==3.9: extra != 'foo'
idna==3.10: extra != 'bar' and extra == 'foo'
And these *are* properly disjoint. There is no way for them both to be
active. This also correctly accounts for fork C where neither `foo` nor
`bar` are active, and yet, `idna==3.9` is still enabled but `idna==3.10`
is not. (In the [motivating example], this comes from `baz` being enabled.)
That is, this captures the idea that for `idna==3.10` to be installed,
there must actually be a specific extra that is enabled. That's what
makes it disjoint from `idna==3.9`.
We aren't quite done yet, because this does add *too many* conflict
markers to dependency edges that don't need it. In the next commit,
we'll add in our world knowledge to simplify these conflict markers.
[motivating example]: https://github.com/astral-sh/uv/issues/9289
I think Ibraheem had this routine at some point in the past, but
we ended up dropping it because we didn't have a use for it. Well,
now we do!
It turns out that when we generate "conflict markers," they don't
actually take "world knowledge" into account. In particular, there
is "world knowledge" that a particular set of extras cannot be
enabled simultaneously. This in turn allows us to simplify most
conflict markers. If we didn't do this, it's likely that lock files
would become littered with conflict markers whenever any conflicts
are declared.
This is somewhat (although not completely) analogous to how we
"simplify" markers with respect to `requires-python`. That is,
`requires-python` reflects world knowledge that enables markers
to be written more simply than they otherwise would be without
world knowledge.
Previously, we had copied the behavior of `try_markers` to return
`None` in the case where the marker was always true. I believe this
was done because it somewhat implies that there is no forking
happening. But I find this somewhat strange personally, and instead
flipped this around so that it still returns a marker in that case.
The one call site that is impacted by this is the resolution
graph construction. If we left it as-is, it would end up with
a list of one marker that is always true in some cases. And this
in turn results in writing an empty `resolution-markers` to the
lock file. Probably the output logic should be tweaked instead,
but we leave it alone for now.
## Summary
If we're installing with `--target` or `--prefix`, then it's not a
mutable operation, so we should be allowed to discover system Pythons. I
suspect this was hard to special-case in the past but is now trivial
after @zanieb's various refactors.
Closes https://github.com/astral-sh/uv/issues/9356.
## Summary
Aligns the description of `UV_NO_PROGRESS` with other env vars that also
have a related flag.
`--no-progress` is a "global option" and exists in every command.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
This effectively combines a PEP 508 marker and an as-yet-specified
marker for expressing conflicts among extras and groups.
This just defines the type and threads it through most of the various
points in the code that previously used `MarkerTree` only. Some parts
do still continue to use `MarkerTree` specifically, e.g., when dealing
with non-universal resolution or exporting to `requirements.txt`.
This doesn't change any behavior.
This doesn't change any behavior. But this makes it a bit
clearer in the code that `uv pip compile` does not support
specifying conflicts. Indeed, we always pass an empty set of
conflicts to the resolver.
This is because there is no way to encode the conditional
logic of conflicts into the `requirements.txt` format. This
is unlike markers.
This doesn't change any behavior. My guess is that this code was
a casualty of refactoring. But basically, it was doing redundant
case analysis and iterating over all resolutions (even though it's
in the branch that can only occur when there is only one
resolution).
This filtering is now redundant, since forking now avoids these
degenerate cases by construction.
The main change to forking that enables skipping over "always
false" forks is that forking now starts with the parent's markers
instead of starting with MarkerTree::TRUE and trying to combine
them with the parent's markers later. This in turn leads to
skipping over anything that "can't" happen when combined with the
parents markers. So we never hit the case of generating a fork
that, when combined with the parent's markers, results in a
marker that is always false. We just avoid it in the first place.
This test demonstrates the difference between `extra != "foo"` and
`sys_platform != "foo"`.
I wrote this test down to test the extra simplification logic was
correct. And I also wanted to test whether we could somehow hackily
encode `group` (as opposed to just `extra`) logic into marker
expressions by reusing another field. But I don't think we can.
_get_glibc_version() can after #9005 return either (0, 0) if glibc
string is missing or (-1, -1) if the string can't be parsed. There was
no need to change missing string to (0, 0).
Also, move back indentation to make it easier to understand.
Currently, user display returns an empty path if the current dir is the
directory we are printing. This leads to odd messages such as
> Including project.license-files at `` with `LICENSE*`
or
> Not a license files match: ``
Instead, we display the current path as a dot.
## Summary
`--upgrade` isn't useful, since it's the default. So it's now hidden,
but continues to warn if you enable it.
`--no-upgrade` isn't useful, since it panics. So it's now removed
entirely. This isn't breaking, since it already didn't work.
`--upgrade-package` actually _is_ useful, because it turns out it allows
things like: `uv tool upgrade babel --upgrade-package "babel<0.2.14"` to
constrain the upgrade.
I left this in place but hid it... I think we should provide a better
workflow for this, like `uv tool upgrade "babel<0.2.14"`? It's strange
to specify the package twice, and that `uv tool upgrade` has an
`--upgrade-package` flag.
Closes https://github.com/astral-sh/uv/issues/9317.
## Summary
On Termux, uv currently fails to find any interpreter because it can't
find a glibc version, because there isn't one. But the Python
interpreter is still functional nonetheless.
So, when glibc cannot be found, simply return 0 for the version numbers
and mark the interpreter as being incompatible with manylinux
I really don't know if this is the right way to address this, but I can
attest that manual testing shows uv appears to be fully functional, at
least for pip and virtualenvs.
Fixes#7373
## Test Plan
I tried running the test suite, and after some tweaks, a good portion of
the test suite passes as well. A significant number of tests fail, but
this appears to be due to minor differences in output, like warnings
about hard links not working (hard links are completely disallowed on
Android), differences in the number of files removed, etc. The test
suite seems to be very sensitive to minor variations in output.
This PR contains three smaller improvements:
* Improve the include/exclude logging. We're still showing the current
directory as empty backticks, not sure what to do about that
* Add early stopping to license file globbing, so we don't traverse the
whole directory recursively when license files can only be in few places
* Support explicit wheel excludes. These are still not entirely right,
but at least we're correctly excluding compiled python files now. The
next step is to make sure that the wheel excludes contain all pattern
from source dist excludes, to make sure source tree -> wheel can't have
more files than source tree -> source dist -> wheel.
## Summary
The issue here is fairly complex. Consider the following:
```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12.0"
dependencies = []
[project.optional-dependencies]
cpu = [
"torch>=2.5.1",
"torchvision>=0.20.1",
]
cu124 = [
"torch>=2.5.1",
"torchvision>=0.20.1",
]
[tool.uv]
conflicts = [
[
{ extra = "cpu" },
{ extra = "cu124" },
],
]
[tool.uv.sources]
torch = [
{ index = "pytorch-cpu", extra = "cpu", marker = "platform_system != 'Darwin'" },
]
torchvision = [
{ index = "pytorch-cpu", extra = "cpu", marker = "platform_system != 'Darwin'" },
]
[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
```
When solving this project, we first pick a PyTorch version from PyPI, to
solve the `cu124` extra, selecting `2.5.1`.
Later, we try to solve the `cpu` extra. In solving that extra, we look
at the PyTorch CPU index. Ideally, we'd select `2.5.1+cpu`... But
`2.5.1` is already a preference. So we choose that.
Now, we only respect preferences for explicit indexes if they came from
the same index.
Closes https://github.com/astral-sh/uv/issues/9295.
## Summary
The new `--index` and `--default-index` flags are being omitted in the
`uv pip compile` header, unintentionally.
Closes https://github.com/astral-sh/uv/issues/9287.
## Summary
I find myself messing this up with `--build-constraint` vs.
`--build-constraints`, and it turns out our own CLI isn't fully
consistent here either.
When building only a single crate in the workspace to run its tests, we
often recompile a lot of other, unrelated crates. Whenever cargo has a
different set of crate features, it needs to recompile. By moving some
features (non-exhaustive for now) to the workspace level, we always
activate them an avoid recompiling.
The cargo docs mismatch the behavior of cargo around default-deps, so I
filed that upstream and left most `default-features` mismatches:
https://github.com/rust-lang/cargo/issues/14841.
Reference script:
```python
import tomllib
from collections import defaultdict
from pathlib import Path
uv = Path("/home/konsti/projects/uv")
skip_list = ["uv-trampoline", "uv-dev", "uv-performance-flate2-backend", "uv-performance-memory-allocator"]
root_feature_map = defaultdict(set)
root_default_features = defaultdict(bool)
cargo_toml = tomllib.loads(uv.joinpath("Cargo.toml").read_text())
for dep, declaration in cargo_toml["workspace"]["dependencies"].items():
root_default_features[dep] = root_default_features[dep] or declaration.get("default-features", True)
root_feature_map[dep].update(declaration.get("features", []))
feature_map = defaultdict(set)
default_features = defaultdict(bool)
for crate in uv.joinpath("crates").iterdir():
if crate.name in skip_list:
continue
if not crate.joinpath("Cargo.toml").is_file():
continue
cargo_toml = tomllib.loads(crate.joinpath("Cargo.toml").read_text())
for dep, declaration in cargo_toml.get("dependencies", {}).items():
# If any item uses default features, they are used everywhere
default_features[dep] = default_features[dep] or declaration.get("default-features", True)
feature_map[dep].update(declaration.get("features", []))
for dep, features in sorted(feature_map.items()):
features = features - root_feature_map.get(dep, set())
if not features and default_features[dep] == root_default_features[dep]:
continue
print(dep, default_features[dep], sorted(features))
```
## Summary
Just trying to unify the retry handling, as in
https://github.com/astral-sh/uv/pull/9274 and elsewhere. Right now, the
publish handler doesn't use any backoff and always retries three times
regardless of settings.
## Summary
This was an oversight in the initial implementation. We shouldn't
validate sources for the `build-system.requires` field, since extras and
groups can _never_ be active.
Closes https://github.com/astral-sh/uv/issues/9259.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
In uv-globfilter, use the workspace `fs-err` in `dev-dependencies`.
This fixes an unnecessary dev-dependency on `fs-err` 2.x even after the
workspace fs-err was updated to 3.x in
https://github.com/astral-sh/uv/pull/8625.
The `Cargo.lock` file still has `fs-err v2.11.0` after this PR, but it
is via `tracing-durations-export v0.3.0` rather than directly required
by any `uv` crate.
## Test Plan
<!-- How was it tested? -->
```
$ cd crates/uv-globfilter/
$ cargo test
```
## Summary
It turns out that `WrappedReqwestError` skips the `reqwest::Error`
itself in order to hack the display. This PR adds it to the list of
variants we check when retrying transient errors.
Closes https://github.com/astral-sh/uv/issues/9246.
## Test Plan
Patched `reqwest` locally to return an error in `bytes()`. Verified that
it was _not_ caught prior to this PR, but was caught afterwards.
- Adds a collapsible section for the project concept
- Splits the project concept document into several child documents.
- Moves the workspace and dependencies documents to under the project
section
- Adds a mkdocs plugin for redirects, so links to the moved documents
still work
I attempted to make the minimum required changes to the contents of the
documents here. There is a lot of room for improvement on the content of
each new child document. For review purposes, I want to do that work
separately. I'd prefer if the review focused on this structure and idea
rather than the content of the files.
I expect to do this to other documentation pages that would otherwise be
very nested.
The project concept landing page and nav (collapsed by default) looks
like this now:
<img width="1507" alt="Screenshot 2024-11-14 at 11 28 45 AM"
src="https://github.com/user-attachments/assets/88288b09-8463-49d4-84ba-ee27144b62a5">
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
PR #4965 added `*-manylinux_2_31` as a target triple, and issue #4966
described the need for a more general solution.
In lieu of a general solution, this PR adds further explicit manylinux
target triples for different glibc version up to the one used by the
latest Ubuntu release (glibc 2.40 used in Ubuntu 24.10).
## Test Plan
<!-- How was it tested? -->
Local, manual testing with a Python wheel targeting
`x86_64-manylinux_2_35`.
Allow including data files in wheels, configured through
`pyproject.toml`. This configuration is currently only read in the build
backend. We'd only start using it in the frontend when we're adding a
fast path.
Each data entry is a directory, whose contents are copied to the
matching directory in the wheel in
`<name>-<version>.data/(purelib|platlib|headers|scripts|data)`. Upon
installation, this data is moved to its target location, as defined by
<https://docs.python.org/3.12/library/sysconfig.html#installation-paths>:
- `data`: Installed over the virtualenv environment root. Warning: This
may override existing files!
- `scripts`: Installed to the directory for executables, `<venv>/bin` on
Unix or `<venv>\Scripts` on Windows. This directory is added to PATH
when the virtual environment is activated or when using `uv run`, so
this data type can be used to install additional binaries. Consider
using `project.scripts` instead for starting Python code.
- `headers`: Installed to the include directory, where compilers
building Python packages with this package as built requirement will
search for header files.
- `purelib` and `platlib`: Installed to the `site-packages` directory.
It is not recommended to uses these two options.
For simplicity, for now we're just defining a directory to be copied for
each data directory, while using the glob based include mechanism in the
background. We thereby introduce a third mechanism next to the main
includes and the PEP 639 mechanism, which is not what we should finalize
on.
## Summary
The reqwest middleware doesn't retry errors that occur "after" the
request completes -- but in some cases, these do include spurious errors
that we want to retry. See https://github.com/astral-sh/uv/issues/8144
for examples. This PR adds a second retry layer during the response
_handler_, which should help with some of the spurious failures we see
in the linked issue.
Closes https://github.com/astral-sh/uv/issues/8144.
## Summary
We missed the case in which the user has a legacy non-`[project]` root
-- we were always installing all members.
Closes https://github.com/astral-sh/uv/issues/9214.
## Summary
This PR enables something like the "final boss" of PyTorch setups --
explicit support for CPU vs. GPU-enabled variants via extras:
```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.13.0"
dependencies = []
[project.optional-dependencies]
cpu = [
"torch==2.5.1+cpu",
]
gpu = [
"torch==2.5.1",
]
[tool.uv.sources]
torch = [
{ index = "torch-cpu", extra = "cpu" },
{ index = "torch-gpu", extra = "gpu" },
]
[[tool.uv.index]]
name = "torch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
[[tool.uv.index]]
name = "torch-gpu"
url = "https://download.pytorch.org/whl/cu124"
explicit = true
[tool.uv]
conflicts = [
[
{ extra = "cpu" },
{ extra = "gpu" },
],
]
```
It builds atop the conflicting extras work to allow sources to be marked
as specific to a dedicated extra being enabled or disabled.
As part of this work, sources now have an `extra` field. If a source has
an `extra`, it means that the source is only applied to the requirement
when defined within that optional group. For example, `{ index =
"torch-cpu", extra = "cpu" }` above only applies to
`"torch==2.5.1+cpu"`.
The `extra` field does _not_ mean that the source is "enabled" when the
extra is activated. For example, this wouldn't work:
```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.13.0"
dependencies = ["torch"]
[tool.uv.sources]
torch = [
{ index = "torch-cpu", extra = "cpu" },
{ index = "torch-gpu", extra = "gpu" },
]
[[tool.uv.index]]
name = "torch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
[[tool.uv.index]]
name = "torch-gpu"
url = "https://download.pytorch.org/whl/cu124"
explicit = true
```
In this case, the sources would effectively be ignored. Extras are
really confusing... but I think this is correct? We don't want enabling
or disabling extras to affect resolution information that's _outside_ of
the relevant optional group.
## Summary
These were moved as part of a broader refactor to create a single
integration test module. That "single integration test module" did
indeed have a big impact on compile times, which is great! But we aren't
seeing any benefit from moving these tests into their own files (despite
the claim in [this blog
post](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html),
I see the same compilation pattern regardless of where the tests are
located). Plus, we don't have many of these, and same-file tests is such
a strong Rust convention.
## Summary
I moved this to a separate test. The packages may or may not be
downloaded already, since the previous command fails -- it just depends
on timing.
## Summary
The distributions used to be stored in a `BTreeMap`, keyed by name.
They're now stored in a graph... so iteration isn't guaranteed to
produce a deterministic hash!
This fixes a "flaky" test, though it's actually a real bug. The test was
right!
Closes#9137.
Fixes#9164
Using clap's `default_value_t` makes the `flag` function unhappy, so
just set the default when we unwrap. Tested with no flags,
`--verify-hashes`, `--no-verify-hashes` and setting in uv.toml
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
When building source distributions, we need to include the readme, so it
can become part the METADATA body when building the wheel. We also need
to support the license files from PEP 639. When building the source
distribution, we copy those file relative to their origin, when building
the wheel, we copy them to `.dist-info/licenses`.
The test for idempotence in wheel building is merged into the file
listing test, which also covers that source tree -> source dist -> wheel
is equivalent to source tree -> wheel, though we do need to check for
file inclusion stronger here.
Best reviewed commit-by-commit
## Summary
Align uv's workspace discovery with red knots (see
https://github.com/astral-sh/ruff/pull/14308#issuecomment-2474296308)
* Detect nested workspace inside the current workspace rather than
testing if the current workspace is a member of any outer workspace.
* Detect packages with identical names.
## Test Plan
I added two integration tests. I also back ported the tests to main to
verify that both these invalid workspaces aren't catched by uv today.
That makes this, technically, a breaking change but I would consider the
change a bug fix.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
I was wrongly using `.name()` to detect if a package was "not root", but
in `pip compile`, the root can have a name -- so we were failing to find
the derivation chain.
## Summary
This PR adds context to our error messages to explain _why_ a given
package was included, if we fail to download or build it.
It's quite a large change, but it motivated some good refactors and
improvements along the way.
Closes https://github.com/astral-sh/uv/issues/8962.
## Summary
This PR should not contain any user-visible changes, but the goal is to
refactor the `Resolution` type to retain a dependency graph. We want to
be able to explain _why_ a given package was excluded on error (see:
https://github.com/astral-sh/uv/issues/8962), which in turn requires
that at install time, we can go back and figure out the dependency
chain. At present, `Resolution` is just a map from package name to
distribution; this PR remodels it as a graph in which each node is a
package, and the edges contain markers plus extras or dependency groups.
A first milestone: source tree -> source dist -> wheel -> install works.
This PR adds a test for this.
There's obviously a lot still missing, including basics such as the
Readme inclusion.
## Summary
As discussed in Discord... This struct has evolved to include a lot of
information apart from the `petgraph::Graph`. And I want to add a graph
to the simplified `Resolution` type. So I think this name makes more
sense.
When doing a directory traversal for source dist inclusion, we want to
offer the user include and exclude options, and we want to avoid
traversing irrelevant directories. The latter is important for
performance, especially on network file systems, but also with large
data directories, or (not-included) directories with other permissions.
To support this, we introduce `GlobDirFilter`, which uses a DFA from
regex_automata to determine whether any children of a directory can be
included and skips the directory if not.
The globs are based on PEP 639. The syntax is more restricted than glob
or globset, but it's standardized. I chose it over glob or globset
because we're already using this syntax for `project.license-files` a
required by PEP 639, so it makes sense to use the same globs for all
includes (see e.g.
4f52a3bb62/pyproject.toml (L36-L48)
for example with same semantics for include and exclude)
### Semantics
Glob semantics are complex due to mixing directories and files,
expectations around simplicity and our need to exclude most of the tree
in the project from traversal. The current draft uses a syntax that
optimizes for simple default use cases for the start.
#### includes
Glob expressions which files and directories to include in the source
distribution.
Includes are anchored, which means that `pyproject.toml` includes only
`<project root>/pyproject.toml`. Use for example `assets/**/sample.csv`
to include for all
`sample.csv` files in `<project root>/assets` or any child directory. To
recursively include
all files under a directory, use a `/**` suffix, e.g. `src/**`. For
performance and
reproducibility, avoid unanchored matches such as `**/sample.csv`.
The glob syntax is the reduced portable glob from
[PEP 639](https://peps.python.org/pep-0639/#add-license-FILES-key).
#### excludes
Glob expressions which files and directories to exclude from the
previous source
distribution includes.
Excludes are not, which means that `__pycache__` excludes all
directories named
`__pycache__` and it's children anywhere. To anchor a directory, use a
`/` prefix, e.g.,
`/dist` will exclude only `<project root>/dist`.
The glob syntax is the reduced portable glob from
[PEP 639](https://peps.python.org/pep-0639/#add-license-FILES-key).
Surprisingly, this seems to be all that's necessary.
Previously, we were only extracting an extra from a
PubGrubPackage to test for conflicts. But now we extract
either an extra or a group. The surrounding code all
remains the same.
We do need to add some extra checking for groups
specifically, but I believe that's it.
Andrew Gallant
Charlie Marsh
Matthijs Kok
Jon Åslund
konsti
Zanie Blue
Skyler Hawthorne
Jo
Ben Beasley
Ed Morley
Ahmed Ilyas
Hristo Filaretov
renovate[bot]
Shantanu
Micha Reiser