Commit Graph

763 Commits

Author SHA1 Message Date
Charlie Marsh 8126a5ed32
Make `MarkerTree` `Copy` (#9542)
## Summary

It's just a `usize`. It seems simpler and perhaps even more performant
(?) to make it `Copy`.
2024-11-30 14:07:07 -05:00
Charlie Marsh 6bebf79ac3
Remove uses of `Option<MarkerTreeContents>` in `PubGrubPackage` (#9541)
## Summary

Just use `MarkerTree::TRUE` instead of `None`.
2024-11-30 10:36:18 -05:00
Charlie Marsh 22fd9f7ff1
Pass extra when evaluating lockfile markers (#9539)
## Summary

When we serialize and deserialize the lockfile, we remove the conflict
markers. So in the linked case, the edges for the `tqdm` entries are
like:

```
complexified_marker: UniversalMarker {
    pep508_marker: python_full_version >= '3.9.0',
    conflict_marker: true,
},
```

However... when we evaluate in-memory, the conflict markers are still
there...

```
complexified_marker: UniversalMarker {
    pep508_marker: true,
    conflict_marker: extra == 't1' and extra != 't2',
},
```

So if `uv run` creates the lockfile, we evaluate this as `false`.

We should make this consistent, and I expect @BurntSushi is aware. But
for now, it's reasonable / correct to pass the extra when evaluating at
this specific point, since we know the dependency was enabled by the
marker.

Closes
https://github.com/astral-sh/uv/issues/9533#issuecomment-2508908591.
2024-11-30 14:22:18 +00:00
Charlie Marsh 53fe301b1d
Avoid using IDs across PubGrub states (#9538)
## Summary

This isn't safe, because the prefetcher is global but the IDs could come
from different PubGrub states (i.e., different forks).
2024-11-30 13:59:21 +00:00
Charlie Marsh 2aca623691
Remove unused specifier field from `PubGrub` structs (#9536) 2024-11-30 03:59:34 +00:00
Charlie Marsh 69811837e5
Support recursive extras with marker in `pip compile -r pyproject.toml` (#9535)
## Summary

Closes https://github.com/astral-sh/uv/issues/9530.
2024-11-30 03:40:22 +00:00
Charlie Marsh 950855877a
Use constraints in trace rather than irrelevant requires-python (#9529) 2024-11-29 19:25:43 +00:00
Charlie Marsh cf20673197
Upgrade to Rust 1.83 (#9511)
## Summary

A lot of good new lints, and most importantly, error stabilizations. I
tried to find a few usages of the new stabilizations, but I'm sure there
are more.

IIUC, this _does_ require bumping our MSRV.
2024-11-29 12:04:22 -05:00
Charlie Marsh 590294a0ac
Propagate markers for recursive extras in resolver (#9509)
## Summary

Closes
https://github.com/astral-sh/uv/issues/9499#issuecomment-2506491209.
2024-11-28 12:40:00 -05:00
konsti b80bd87b99
Don't emit unpinned warning for proxy packages (#9497)
Whenever we see an unpinned proxy package, we know that its base package
is also unpinned and emitted a warning.

Part 1 of 2 for #8155
2024-11-28 12:39:38 +00:00
Charlie Marsh 8a27d4d340
Allow file: URLs to include package names (#9493)
## Summary

Closes https://github.com/astral-sh/uv/issues/9489.
2024-11-27 22:27:07 -05:00
Charlie Marsh f1ccbcb065
Consistently enforce requested-vs.-built metadata when retrieving wheels (#9484)
## Summary

We were missing a bunch of edge cases, e.g., the wheel exists in the
cache already.

Closes https://github.com/astral-sh/uv/issues/9480.
2024-11-27 16:51:01 -05:00
Charlie Marsh ee84620e90
Use consistent formatting for build system errors (#9340)
## Summary

These look pretty different from the help / hint messages we show on
resolver failure. I've added color, backticks, and a "hint:" prefix.

Before:

![Screenshot 2024-11-21 at 5 45
40 PM](https://github.com/user-attachments/assets/71ee2551-8fc6-4715-a9d7-68055cd34547)

After:

![Screenshot 2024-11-21 at 6 05
31 PM](https://github.com/user-attachments/assets/24e7fb46-49aa-4b6f-a442-115a71a3414b)
2024-11-27 14:22:53 +00:00
Charlie Marsh 916d5d7778
Migrate to PubGrub's arena for package names (#9448)
## Summary

There's more we can do here, i.e., to leverage the IDs more widely, but
this is a start.
2024-11-26 15:05:39 -05:00
Charlie Marsh 8aeaf98f59
Rename from `Lowered` to `Canonical` (#9447)
By request:
https://github.com/astral-sh/uv/pull/9341#pullrequestreview-2460979421.
2024-11-26 13:34:43 -05:00
Charlie Marsh d18753527f
Remove `python_version` from lowered marker representation (#9343)
## Summary

We never construct these -- they should be impossible, since we always
translate to `python_full_version`. This PR encodes that impossibility
in the types.
2024-11-26 14:39:36 +00:00
Charlie Marsh 106863069d
Report marker diagnostics during parsing, rather than evaluation (#9338)
## Summary

I want to move towards a more normalized marker representation within
the marker tree, which means that the things we warn against will
disappear by the time we get to evaluation. I think it makes more sense
to show these warnings when we create the tree, rather than when we
evaluate it.
2024-11-26 14:15:33 +00:00
Charlie Marsh c30a314e0e
Allow dependency groups to include the containing package (#9385)
## Summary

Closes https://github.com/astral-sh/uv/issues/9383.
2024-11-25 21:56:14 -05:00
Li-Lun Lin e485dfd7f1
feat: add support for `--no-extra` flag and setting (#9387)
<!--  
Thank you for contributing to uv! To help us review effectively, please
ensure that:
- The pull request includes a summary of the change.  
- The title is descriptive and concise.  
- Relevant issues are referenced where applicable.  
-->

## Summary

Resolves #9333  

This pull request introduces support for the `--no-extra` command-line
flag and the corresponding `no-extra` UV setting.

### Behavior
- When `--all-extras` is supplied, the specified extras in `--no-extra`
will be excluded from the installation.
- If `--all-extras` is not supplied, `--no-extra` has no effect and is
safely ignored.

## Test Plan

Since `ExtrasSpecification::from_args` and
`ExtrasSpecification::extra_names` are the most important parts in the
implementation, I added the following tests in the
`uv-configuration/src/extras.rs` module:

- **`test_no_extra_full`**: Verifies behavior when `no_extra` includes
the entire list of extras.
- **`test_no_extra_partial`**: Tests partial exclusion, ensuring only
specified extras are excluded.
- **`test_no_extra_empty`**: Confirms that no extras are excluded if
`no_extra` is empty.
- **`test_no_extra_excessive`**: Ensures the implementation ignores
`no_extra` values that don't match any available extras.
- **`test_no_extra_without_all_extras`**: Validates that `no_extra` has
no effect when `--all-extras` is not supplied.
- **`test_no_extra_without_package_extras`**: Confirms correct behavior
when no extras are available in the package.
- **`test_no_extra_duplicates`**: Verifies that duplicate entries in
`pkg_extras` or `no_extra` do not cause errors.

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2024-11-24 02:25:09 +00:00
Skyler Hawthorne e5f5bd63cf
feat: export --prune (#9389)
## Summary

This adds a `--prune` flag to the `export` command to correspond with
the `--prune` flag of the `tree` command.

The purpose is for generating a `requirements.txt` that omits a package
and all of that package's unique dependencies. This is useful for cases
where the project has a dependency on a common core package, but where
that package does not need to be installed in the target environment.

For example, a pyspark job needs spark for development, but when
installing into a cluster that already has pyspark installed, it is
desirable to omit pyspark's whole dependency tree so that only the
unique dependencies that your job needs get installed, and do not risk
breaking the pyspark dependencies with something incompatible.

Dev groups cannot always cover this case because there are other
projects where this common dependency occurs as a transitive. One
example is Airflow providers, which include Airflow itself as a
dependency, but it is unnecessary and undesirable to include Airflow's
dependency tree in the `requirements.txt` for your DAGs.

Partly related to #7214, though I'm not sure it covers the ask in that
one of having this functionality extend to the project's actual
published metadata.


## Test Plan

An integration test was added, and some manual testing. Let me know if
more would be better.

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2024-11-24 02:11:54 +00:00
Andrew Gallant f7bed37a4e uv-resolver: add "include" rules to `ResolverEnvironment`
When we generate conflict markers for each resolution after the
resolver runs, it turns out that generating them just from exclusion
rules is not sufficient.

For example, if `foo` and `bar` are declared as conflicting extras, then
we end up with the following forks:

    A: extra != 'foo'
    B: extra != 'bar'
    C: extra != 'foo' and extra != 'bar'

Now let's take an example where these forks don't share the same version
for all packages. Consider a case where `idna==3.9` is in forks A and C,
but where `idna==3.10` is in fork B. If we combine the markers in forks
A and C through disjunction, we get the following:

     idna==3.9: extra != 'foo' or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar'

But these are clearly not disjoint. Both dependencies could be selected,
for example, when neither `foo` nor `bar` are active. We can remedy this
by keeping around the inclusion rules for each fork:

    A: extra != 'foo' and extra == 'bar'
    B: extra != 'bar' and extra == 'foo'
    C: extra != 'foo' and extra != 'bar'

And so for `idna`, we have:

     idna==3.9: (extra != 'foo' and extra == 'bar') or (extra != 'foo' and extra != 'bar')
    idna==3.10: extra != 'bar' and extra == 'foo'

Which simplifies to:

     idna==3.9: extra != 'foo'
    idna==3.10: extra != 'bar' and extra == 'foo'

And these *are* properly disjoint. There is no way for them both to be
active. This also correctly accounts for fork C where neither `foo` nor
`bar` are active, and yet, `idna==3.9` is still enabled but `idna==3.10`
is not. (In the [motivating example], this comes from `baz` being enabled.)
That is, this captures the idea that for `idna==3.10` to be installed,
there must actually be a specific extra that is enabled. That's what
makes it disjoint from `idna==3.9`.

We aren't quite done yet, because this does add *too many* conflict
markers to dependency edges that don't need it. In the next commit,
we'll add in our world knowledge to simplify these conflict markers.

[motivating example]: https://github.com/astral-sh/uv/issues/9289
2024-11-23 13:14:27 -05:00
Andrew Gallant 5cdac563e5 uv-resolver: remove `ResolverEnvironment::try_markers`
This was almost not used any more with the refactor toward
'UniversalMarker', so this just removes the final uses.
2024-11-23 13:14:27 -05:00
Andrew Gallant b71f5a7f02 uv-resolver: fix Display impl for UniversalMarker
It was missing a closing parenthesis.
2024-11-23 13:14:27 -05:00
Andrew Gallant d1f0ee7a47 uv-resolver: slight tweek to try_universal_markers
Previously, we had copied the behavior of `try_markers` to return
`None` in the case where the marker was always true. I believe this
was done because it somewhat implies that there is no forking
happening. But I find this somewhat strange personally, and instead
flipped this around so that it still returns a marker in that case.

The one call site that is impacted by this is the resolution
graph construction. If we left it as-is, it would end up with
a list of one marker that is always true in some cases. And this
in turn results in writing an empty `resolution-markers` to the
lock file. Probably the output logic should be tweaked instead,
but we leave it alone for now.
2024-11-23 13:14:27 -05:00
Andrew Gallant dae584d49b uv-resolver: introduce new UniversalMarker type
This effectively combines a PEP 508 marker and an as-yet-specified
marker for expressing conflicts among extras and groups.

This just defines the type and threads it through most of the various
points in the code that previously used `MarkerTree` only. Some parts
do still continue to use `MarkerTree` specifically, e.g., when dealing
with non-universal resolution or exporting to `requirements.txt`.

This doesn't change any behavior.
2024-11-22 08:21:23 -05:00
Andrew Gallant a66b0eb931 uv-resolver: remove 'package_markers'
This was completely unused. I noticed this while trying to read
and understand the code. It's unclear when or how this happened.
2024-11-22 08:21:23 -05:00
Andrew Gallant 085fde8955 uv-resolver: simplify `fork_markers` construction
This doesn't change any behavior. My guess is that this code was
a casualty of refactoring. But basically, it was doing redundant
case analysis and iterating over all resolutions (even though it's
in the branch that can only occur when there is only one
resolution).
2024-11-22 08:21:23 -05:00
Andrew Gallant 2b6d9b2289 uv-resolver: remove 'fork.is_false()' filtering
This filtering is now redundant, since forking now avoids these
degenerate cases by construction.

The main change to forking that enables skipping over "always
false" forks is that forking now starts with the parent's markers
instead of starting with MarkerTree::TRUE and trying to combine
them with the parent's markers later. This in turn leads to
skipping over anything that "can't" happen when combined with the
parents markers. So we never hit the case of generating a fork
that, when combined with the parent's markers, results in a
marker that is always false. We just avoid it in the first place.
2024-11-22 08:21:23 -05:00
Charlie Marsh b9e9b60232
Remove `Option` from preference marker (#9304)
## Summary

This... _should_ be equivalent?
2024-11-21 09:36:24 -05:00
Charlie Marsh 3143494ddb
Make marker enums `Copy` (#9305) 2024-11-21 04:45:25 +00:00
Charlie Marsh 5e48819dbb
Only respect preferences across the same indexes (#9302)
## Summary

The issue here is fairly complex. Consider the following:

```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12.0"
dependencies = []

[project.optional-dependencies]
cpu = [
  "torch>=2.5.1",
  "torchvision>=0.20.1",
]
cu124 = [
  "torch>=2.5.1",
  "torchvision>=0.20.1",
]

[tool.uv]
conflicts = [
  [
    { extra = "cpu" },
    { extra = "cu124" },
  ],
]

[tool.uv.sources]
torch = [
  { index = "pytorch-cpu", extra = "cpu", marker = "platform_system != 'Darwin'" },
]
torchvision = [
  { index = "pytorch-cpu", extra = "cpu", marker = "platform_system != 'Darwin'" },
]

[[tool.uv.index]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
```

When solving this project, we first pick a PyTorch version from PyPI, to
solve the `cu124` extra, selecting `2.5.1`.

Later, we try to solve the `cpu` extra. In solving that extra, we look
at the PyTorch CPU index. Ideally, we'd select `2.5.1+cpu`... But
`2.5.1` is already a preference. So we choose that.

Now, we only respect preferences for explicit indexes if they came from
the same index.

Closes https://github.com/astral-sh/uv/issues/9295.
2024-11-20 22:26:43 -05:00
Charlie Marsh a0de83001c
Parallelize network requests in `uv tree --outdated` (#9280)
## Summary

Closes https://github.com/astral-sh/uv/issues/9266.
2024-11-20 16:45:14 +00:00
Jo 23cc9b0322
Add `--all-groups` to `uv sync|run|export|tree` (#8892)
## Summary

Closes #8594

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2024-11-20 16:07:36 +00:00
Ahmed Ilyas 0454c3b3ec
Sort hashes in `uv export` output (#9237)
## Summary

Closes #9225 

## Test Plan

`cargo test`
2024-11-19 19:54:27 +00:00
Charlie Marsh 821f3de095
Automatically retry body errors when processing response (#9213)
## Summary

The reqwest middleware doesn't retry errors that occur "after" the
request completes -- but in some cases, these do include spurious errors
that we want to retry. See https://github.com/astral-sh/uv/issues/8144
for examples. This PR adds a second retry layer during the response
_handler_, which should help with some of the spurious failures we see
in the linked issue.

Closes https://github.com/astral-sh/uv/issues/8144.
2024-11-19 04:14:58 +00:00
Charlie Marsh e4fc875afa
Allow conflicting extras in explicit index assignments (#9160)
## Summary

This PR enables something like the "final boss" of PyTorch setups --
explicit support for CPU vs. GPU-enabled variants via extras:

```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.13.0"
dependencies = []

[project.optional-dependencies]
cpu = [
    "torch==2.5.1+cpu",
]
gpu = [
    "torch==2.5.1",
]

[tool.uv.sources]
torch = [
    { index = "torch-cpu", extra = "cpu" },
    { index = "torch-gpu", extra = "gpu" },
]

[[tool.uv.index]]
name = "torch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

[[tool.uv.index]]
name = "torch-gpu"
url = "https://download.pytorch.org/whl/cu124"
explicit = true

[tool.uv]
conflicts = [
    [
        { extra = "cpu" },
        { extra = "gpu" },
    ],
]
```

It builds atop the conflicting extras work to allow sources to be marked
as specific to a dedicated extra being enabled or disabled.

As part of this work, sources now have an `extra` field. If a source has
an `extra`, it means that the source is only applied to the requirement
when defined within that optional group. For example, `{ index =
"torch-cpu", extra = "cpu" }` above only applies to
`"torch==2.5.1+cpu"`.

The `extra` field does _not_ mean that the source is "enabled" when the
extra is activated. For example, this wouldn't work:

```toml
[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.13.0"
dependencies = ["torch"]

[tool.uv.sources]
torch = [
    { index = "torch-cpu", extra = "cpu" },
    { index = "torch-gpu", extra = "gpu" },
]

[[tool.uv.index]]
name = "torch-cpu"
url = "https://download.pytorch.org/whl/cpu"
explicit = true

[[tool.uv.index]]
name = "torch-gpu"
url = "https://download.pytorch.org/whl/cu124"
explicit = true
```

In this case, the sources would effectively be ignored. Extras are
really confusing... but I think this is correct? We don't want enabling
or disabling extras to affect resolution information that's _outside_ of
the relevant optional group.
2024-11-19 01:06:25 +00:00
Charlie Marsh d08bfee718
Remove separate test files in favor of same-file `mod tests` (#9199)
## Summary

These were moved as part of a broader refactor to create a single
integration test module. That "single integration test module" did
indeed have a big impact on compile times, which is great! But we aren't
seeing any benefit from moving these tests into their own files (despite
the claim in [this blog
post](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html),
I see the same compilation pattern regardless of where the tests are
located). Plus, we don't have many of these, and same-file tests is such
a strong Rust convention.
2024-11-18 20:11:46 +00:00
konsti a07d5a1d18
Update pubgrub to `Ranges::from_iter` (#9145)
Co-authored-by: Zanie Blue <contact@zanie.dev>
2024-11-18 12:28:17 +00:00
Charlie Marsh 12266f8f81
Add a dedicated error for `include = "dev"` with `tool.uv.dev-dependencies` (#9173)
## Summary

This isn't really spec-compliant, so we already don't allow it -- this
just adds a better error message and an explicit test for it.
2024-11-17 13:22:04 -05:00
Charlie Marsh 5bff2ba243
Include extras and dependency groups in derivation chains (#9113)
## Summary

Displays extras and dependency groups when explaining inclusion.
2024-11-15 15:37:18 -05:00
Charlie Marsh 8dd095cab8
Include version constraints in derivation chains (#9112)
## Summary

Derivation chains can now include the versions at which a package was
requested.
2024-11-15 15:06:24 -05:00
Charlie Marsh 2b08d767cd
Show derivation markers for resolutions with project name (#9136)
## Summary

I was wrongly using `.name()` to detect if a package was "not root", but
in `pip compile`, the root can have a name -- so we were failing to find
the derivation chain.
2024-11-14 19:25:25 -05:00
Charlie Marsh 92735ced9a
Misc. touch-ups to target-to-resolution routine (#9131) 2024-11-14 16:24:03 -05:00
Charlie Marsh fe477c3417
Show full derivation chain when encountering build failures (#9108)
## Summary

This PR adds context to our error messages to explain _why_ a given
package was included, if we fail to download or build it.

It's quite a large change, but it motivated some good refactors and
improvements along the way.

Closes https://github.com/astral-sh/uv/issues/8962.
2024-11-14 15:48:26 -05:00
Charlie Marsh a552f74308
Refactor `Resolution` type to retain dependency graph (#9106)
## Summary

This PR should not contain any user-visible changes, but the goal is to
refactor the `Resolution` type to retain a dependency graph. We want to
be able to explain _why_ a given package was excluded on error (see:
https://github.com/astral-sh/uv/issues/8962), which in turn requires
that at install time, we can go back and figure out the dependency
chain. At present, `Resolution` is just a map from package name to
distribution; this PR remodels it as a graph in which each node is a
package, and the edges contain markers plus extras or dependency groups.
2024-11-14 15:25:34 -05:00
Charlie Marsh b37170df94
Rename `ResolutionGraph` to `ResolverOutput` (#9103)
## Summary

As discussed in Discord... This struct has evolved to include a lot of
information apart from the `petgraph::Graph`. And I want to add a graph
to the simplified `Resolution` type. So I think this name makes more
sense.
2024-11-14 14:51:11 +00:00
Andrew Gallant 277e7f8dd0 uv-resolver: support conflicting groups
Surprisingly, this seems to be all that's necessary.

Previously, we were only extracting an extra from a
PubGrubPackage to test for conflicts. But now we extract
either an extra or a group. The surrounding code all
remains the same.

We do need to add some extra checking for groups
specifically, but I believe that's it.
2024-11-14 08:02:46 -05:00
Andrew Gallant 06943ca870 uv-pypi-types: make room for group names in addition to extras
This adds support for providing conflicting group names in addition to
extra names to `Conflicts`.

This merely makes "room" for it in the types while keeping everything
working. We'll add proper support for it in the next commit.

Note that one interesting trick we do here is depend directly on
`hashbrown` so that we can make use of its `Equivalent` trait. This in
turn lets us use things like `ConflictItemRef` as a lookup key for a
hashset that contains `ConflictItem`. This mirrors using a `&str` as a
lookup key for a hashset that contains `String`, but works for arbitrary
types. `std` doesn't support this, but `hashbrown` does. This trick in
turn lets us simplify some of our data structures.

This also rejiggers some of the serde-interaction with the conflicting
types. We now use a wire type to represent our conflicting items for
more flexibility. i.e., Support `extra` XOR `group` fields.
2024-11-14 08:02:46 -05:00
Andrew Gallant cda8b3276a uv-resolver: add to `diverging_packages` when forking based on conflicts
This addresses Konsti's comment about it being empty:
https://github.com/astral-sh/uv/pull/8976#discussion_r1837951568
2024-11-14 08:02:46 -05:00
Andrew Gallant bb78e00a87 *: update "conflicting groups" terminology everywhere else 2024-11-14 08:02:46 -05:00
Andrew Gallant 19a044d4db uv-pypi-types: rename "conflicting group" types to more generic "conflicts"
Since this is intended to support _both_ groups and extras, it doesn't
make sense to just name it for groups. And since there isn't really a
word that encapsulates both "extra" and "group," we just fall back to
the super general "conflicts."

We'll rename the variables and other things in the next commit.
2024-11-14 08:02:46 -05:00
Charlie Marsh 9339e55a11
Add `version` to `ResolvedDist` (#9102)
## Summary

I need this for the derivation chain work
(https://github.com/astral-sh/uv/issues/8962), but it just seems
generally useful. You can't always get a version from a `Dist` (it could
be URL-based!), but when we create a `ResolvedDist`, we _do_ know the
version (and not just the URL). This PR preserves it.
2024-11-13 19:06:16 -05:00
Charlie Marsh 35549de62d
Defer reporting of build failures in resolver (#9098)
## Summary

In https://github.com/astral-sh/uv/issues/9078, resolution fails because
we fail to build `jsmin`. However... if you look at what's actually
happening, `jsmin` fails to build during _prefetching_. And we never
actually attempt to access its metadata later on.

This PR modifies the metadata result handling such that we don't raise
these errors until the resolver actually asks for the metadata, so
https://github.com/astral-sh/uv/issues/9078 now succeeds.

I actually had to make this change anyway in pursuing
https://github.com/astral-sh/uv/issues/8962, so I've decided to carve it
out here.

Closes https://github.com/astral-sh/uv/issues/9078.
2024-11-13 15:49:08 -05:00
Andrew Gallant 15ef807c80
add support for specifying conflicting extras (#8976)
This PR adds support for conflicting extras. For example, consider
some optional dependencies like this:

```toml
[project.optional-dependencies]
project1 = ["numpy==1.26.3"]
project2 = ["numpy==1.26.4"]
```

These dependency specifications are not compatible with one another.
And if you ask uv to lock these, you'll get an unresolvable error.

With this PR, you can now add this to your `pyproject.toml` to get
around this:

```toml
[tool.uv]
conflicting-groups = [
    [
      { package = "project", extra = "project1" },
      { package = "project", extra = "project2" },
    ],
]
```

This will make the universal resolver create additional forks
internally that keep the dependencies from the `project1` and
`project2` extras separate. And we make all of this work by reporting
an error at **install** time if one tries to install with two or more
extras that have been declared as conflicting. (If we didn't do this,
it would be possible to try and install two different versions of the
same package into the same environment.)

This PR does *not* add support for conflicting **groups**, but it is
intended to add support in a follow-up PR.

Closes #6981

Fixes #8024

Ref #6729, Ref #6830

This should also hopefully unblock
https://github.com/dagster-io/dagster/pull/23814, but in my testing, I
did run into other problems (specifically, with `pywin`). But it does
resolve the problem with incompatible dependencies in two different
extras once you declare `test-airflow-1` and `test-airflow-2` as
conflicting for `dagster-airflow`.

NOTE: This PR doesn't make `conflicting-groups` public yet. And in a
follow-up PR, I plan to switch the name to `conflicts` instead of
`conflicting-groups`, since it will be able to accept conflicting extras
_and_ conflicting groups.
2024-11-13 09:52:28 -05:00
Charlie Marsh b3c660c58a
Rename `Fetch` to `Download` in build errors (#9039)
## Summary

We're inconsistent with these -- sometimes it's `Error::Fetch` and
sometimes it's `Error::Download`. The message says download, so let's
just use that?
2024-11-12 02:30:20 +00:00
Charlie Marsh 59f3ec2b6c
Remove duplicate `Lock::to_resolution` (#9029)
## Summary

This got moved to `InstallTarget`! Must've been an oversight not to
delete. I verified that no code was changed here since the date that we
moved it to `InstallTarget`.
2024-11-11 20:30:00 +00:00
Charlie Marsh dd4b27bfbd
Allow `--no-build` to validate lock (#9024)
## Summary

Just as we don't enforce tag compliance, we shouldn't enforce
`--no-build` when validating the lockfile. If we end up building from
source, the distribution database will correctly error.

Closes https://github.com/astral-sh/uv/issues/9016.
2024-11-11 14:02:37 -05:00
Charlie Marsh ac9ce853cf
Remove some unused public methods (#8993)
## Summary

Notice these in an unrelated refactor.
2024-11-10 22:55:22 +00:00
Charlie Marsh 744a9091a2
Allow default indexes to be marked as explicit (#8990)
## Summary

Closes https://github.com/astral-sh/uv/issues/8985.
2024-11-10 18:05:39 +00:00
Charlie Marsh 8a3e5d43e6
Fix references to `--resolution-strategy` in error message output (#8971)
Closes https://github.com/astral-sh/uv/issues/8967.
2024-11-09 13:54:49 +00:00
Charlie Marsh 363c589f76
Prefer compatible to incompatible distributions when packages exist on multiple indexes (#8961)
## Summary

At time of writing, `markupsafe==3.0.2` exists on the PyTorch index, but
there's
only a single wheel:


`MarkupSafe-3.0.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`

Meanwhile, there are a large number of wheels on PyPI for the same
version. If the
user is on Python 3.12, and we return the incompatible PyTorch wheel
without
considering the PyPI wheels, PubGrub will mark 3.0.2 as an incompatible
version,
even though there are compatible wheels on PyPI.

Closes https://github.com/astral-sh/uv/issues/8922.
2024-11-08 19:48:48 -05:00
Charlie Marsh 75c26229a2
Support multiple extras in universal pip compile output (#8960)
## Summary

We were making some incorrect assumptions in the extra-merging code for
universal `pip compile`. This PR corrects those assumptions and adds a
bunch of additional tests.

Closes https://github.com/astral-sh/uv/issues/8915.
2024-11-08 18:22:29 -05:00
Charlie Marsh 90653e1f5e Remove all special-casing for local version identifiers (#8818)
After https://github.com/astral-sh/uv/pull/8797, we have spec-compliant
handling for local version identifiers and can completely remove all the
special-casing around it.
2024-11-07 14:29:54 -06:00
Eric Mark Martin c49c7bdf97 Implement PEP 440-compliant local version semantics (#8797)
Implement a full working version of local version semantics. The (AFAIA)
major move towards this was implemented in #2430. This added support
such that the version specifier `torch==2.1.0+cpu` would install
`torch@2.1.0+cpu` and consider `torch@2.1.0+cpu` a valid way to satisfy
the requirement `torch==2.1.0` in further dependency resolution.

In this feature, we more fully support local version semantics. Namely,
we now allow `torch==2.1.0` to install `torch@2.1.0+cpu` regardless of
whether `torch@2.1.0` (no local tag) actually exists.

We do this by adding an internal-only `Max` value to local versions that
compare greater to all other local versions. Then we can translate
`torch==2.1.0` into bounds: greater than 2.1.0 with no local tag and
less than 2.1.0 with the `Max` local tag.

Depends on https://github.com/astral-sh/packse/pull/227.
2024-11-07 14:29:54 -06:00
Zanie Blue 4874b32d85 do not imply pre-release when `!=` operator is used (#7974)
closes #6640

Could you suggest how I should test it?

(already tested locally)

---------

Co-authored-by: konstin <konstin@mailbox.org>
Co-authored-by: Charles Tapley Hoyt <cthoyt@gmail.com>
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2024-11-07 14:29:54 -06:00
Charlie Marsh 29e1b15473
Add `uv tree --outdated` (#8893)
## Summary

Similar to `pip list --outdated`, but for `uv tree`.

## Test Plan

Looks like:

```
foo v0.1.0
└── flask v2.0.0 (latest: v3.0.3)
    ├── click v8.1.7
    ├── itsdangerous v2.2.0
    ├── jinja2 v3.1.4
    │   └── markupsafe v3.0.2
    └── werkzeug v3.1.2
        └── markupsafe v3.0.2
```

With `(latest: v3.0.3)` in bold cyan.
2024-11-07 14:10:46 -06:00
Charlie Marsh 987d778631
Respect dynamic version updates in `uv lock` (#8867)
## Summary

Closes https://github.com/astral-sh/uv/issues/8866.
2024-11-06 11:40:23 -05:00
Charlie Marsh 26e3511ebd
Respect fork markers in `--resolution-mode=lowest-direct` (#8839)
## Summary

Previously, given:

```toml
dependencies = [
    "pycountry >= 22.1.10",
    "setuptools >= 50.0.0; python_version>='3.12'"
]
```

We'd solve for the lowest version of setuptools (with _no_ lower-bound
constraint) in the `python_version < '3.12'` complement.

Closes https://github.com/astral-sh/uv/issues/8819.
2024-11-05 21:09:38 +00:00
Andrew Gallant acaed763b7 uv: use ResolverEnvironment instead of ResolverMarkers
This updates the surrounding code to use the new ResolverEnvironment
type. In some cases, this simplifies caller code by removing case
analysis. There *shouldn't* be any behavior changes here. Some test
snapshots were updated to account for some minor tweaks to error
messages.

I didn't split this up into separate commits because it would have been
too difficult/costly.
2024-11-04 11:09:06 -05:00
Andrew Gallant 44c9ef6aea uv-resolver: add new ResolverEnvironment type
This type is intended to replace `ResolverMarkers`. The main difference
between them is that this type encapsulates more decision making by
un-exporting the different cases. So instead of callers needing to do
explicit case analysis depending on the type of resolver environment,
callers instead use methods that know how to do the right thing. In the
next commit, there are at least a few cases where this greatly
simplifies case analysis on the caller side.

The motivation for this type is to centralize decision making about
forking. In particular, we want to expand forking to include conflicting
groups instead of just `MarkerTree`. So to a certain extent, the
refactor here is about removing bare use of `MarkerTree` in favor of a
more purpose built type that encapsulates the forking logic.

The encapsulation is not quite perfect here. I expect to improve on it a
bit once we add support for conflicting groups.

This is split off from the subsequent commit (that makes use of
`ResolverEnvironment`) so that it's a bit easier to review the addition
in isolation.
2024-11-04 11:09:06 -05:00
Tim de Jager ef8724c979
Add `From` for `FlatDistributions` struct (#8800)
Co-authored-by: konsti <konstin@mailbox.org>
2024-11-04 08:41:39 +00:00
Charlie Marsh bf79d985ee
Allow incompatible `requires-python` for source distributions with static metadata (#8768)
## Summary

At present, when we have a Python requirement and we see a wheel, we
verify that the Python requirement is compatible with the wheel. For
source distributions, though, we verify that both the Python requirement
_and_ the currently-installed version are compatible, because we assume
that we'll need to build the source distribution in order to get
metadata. However, we can often extract source distribution metadata
_without_ building (e.g., if there's a `pyproject.toml` with no dynamic
keys).

This PR thus modifies the source distribution handling to defer that
incompatibility ("We couldn't get metadata for this project, because it
has no static metadata and requires a higher Python version to run /
build") until we actually try to build the package. As a result, you can
now resolve source distribution-only packages using Python versions
below their `requires-python`, as long as they include static metadata.

Closes https://github.com/astral-sh/uv/issues/8767.
2024-11-03 19:03:55 +00:00
Charlie Marsh d3e50a2376
Enable `--all-packages --frozen` in `uv sync` and `uv export` (#8760)
## Summary

This PR improves the interaction of `--frozen` such that we reduce the
dependency on the `pyproject.toml` and increase the dependency on the
`uv.lock`. Specifically, we now read the list of workspace members from
the `uv.lock` rather than the `pyproject.toml`, which means we don't
need to discover the member `pyproject.toml` files in order to perform a
`uv sync --frozen --all-packages`.
2024-11-02 02:48:55 +00:00
Charlie Marsh b36ae6d5ae
Add support for `uv export --all-packages` (#8742)
## Summary

Same as the other PRs, but for `uv export`.
2024-11-02 02:25:29 +00:00
Charlie Marsh 3c9dd97fe9
Add support for `uv sync --all-packages` (#8739)
## Summary

This PR enables `uv sync --all-packages` to sync all packages in a
workspace. It removes a common use-case for the legacy non-`[project]`
packages that we're trying to move away from.

Closes https://github.com/astral-sh/uv/issues/8724.
2024-11-02 01:55:08 +00:00
Andrew Gallant 58a9811881
fix bug where `python_version < '0'` could appear in a final resolution (#8759)
This PR fixes a bug where it was possible for dependencies to be
included in a final resolution with markers that always evaluate to
false. Specifically, `python_version < '0'`.

While we do filter based on Python markers during forking, it turns out
that the markers for each fork are "combined" *after* this filtering
step. But the process of combination can result in a more specific
marker that is always false for the configured Python requirement. This
could result in dependencies with markers that are always false (like
`python_version < '0'`) appearing in the resolution.

The first commit in this PR adds a regression test (with an undesirable
result), and the second commit fixes the regression and updates the
test.

Fixes #8676
2024-11-01 20:11:40 +00:00
Charlie Marsh 24ad43e79c
Avoid showing dependency group annotations on workspace members in tree (#8730)
## Summary

By default, `uv tree` shows the full workspace, not _just_ the root. If
the root depended on a workspace member as a dev dependency, then we'd
still show it as `(group: dev)` in `uv tree` even if you passed
`--no-dev`, because we weren't filtering the edges in the right place.

This is still somewhat confusing, because if `root` depends on workspace
member `child` as a dev dependency, `uv tree --no-dev` still shows both.

Closes https://github.com/astral-sh/uv/issues/8719.
2024-10-31 16:09:36 -04:00
Charlie Marsh bed47d512a
Use portable paths for subdirectories in lock URLs (#8707)
## Summary

We're writing different URLs on Windows vs. Linux because the
subdirectory path is being serialized with a back vs. forward slash.
2024-10-30 15:37:40 -04:00
konsti c1a0fb35e8
Simplify pep440 -> version ranges conversion (#8683) 2024-10-30 13:10:48 +01:00
Charlie Marsh cd408cbc51
Clarify relationship between specifiers and `requires-python` range (#8688) 2024-10-29 22:30:01 +01:00
Charlie Marsh dc5e35ee5d
Fix outdated documentation on `Requires-Python` (#8679)
## Summary

Closes https://github.com/astral-sh/uv/issues/8675.
2024-10-29 20:18:43 +00:00
konsti e5b8cdba70
Merge uv-pubgrub into uv-pep440 (#8669) 2024-10-29 20:15:18 +01:00
Andrew Gallant 9fcc21236e
uv-resolver: simplify case analysis in ForkMap::get (#8672)
Previously, when doing a `uv pip` resolution, we would only return the
first entry in the map. But there should only ever be one entry, or else
we would have incompatible dependencies. So we can collapse the case
with the "one universal fork" case.

(I found this while doing some refactoring of how we handle forking, and
collapsing these cases simplifies some of that refactoring work.)
2024-10-29 13:38:21 -04:00
Andrew Gallant a56e521179
uv-resolver: use disjointness checks instead of marker equality (#8661)
When this code was written, we didn't have "proper" disjointness checks,
and so simple equality was used instead. Arguably disjointness checks
are
more correct, and this would also simplify some case analysis in an
ongoing
refactor.
2024-10-29 11:25:18 -04:00
Charlie Marsh bf14b6a282
Respect dependency group markers in `uv export` (#8659)
## Summary

Closes https://github.com/astral-sh/uv/issues/8658.
2024-10-29 13:41:27 +00:00
Charlie Marsh debe67ffdb
Use `dev-dependencies` and `requires-dev` for lockfile compatibility (#8599)
## Summary

Unfortunately, it looks like we lost
https://github.com/astral-sh/uv/pull/8501 somewhere in a bad rebase.
This PR re-adds the change, with compatibility for those lockfiles
created in v0.4.27. I'm not certain we should actually merge this. It
might be less painful and confusing to just bite the bullet on the
change.

---------

Co-authored-by: Zanie Blue <contact@zanie.dev>
2024-10-28 22:08:33 +00:00
Charlie Marsh bfa84cd1ab
Fork when minimum Python version increases (#8628)
## Summary

This is a re-implementation of
https://github.com/astral-sh/uv/pull/4712, though is now seemingly much
simpler. This issue keeps coming up, and users have a workaround with
`tool.uv.environments`, but it's really a bug in the resolver.

Closes https://github.com/astral-sh/uv/issues/4668.
2024-10-28 09:48:04 -04:00
Charlie Marsh 001d00ba26
Remove inverse-node tracking in `uv tree` (#8610)
## Summary

Just simplifying some of the representations.
2024-10-27 20:44:32 -04:00
Charlie Marsh f23d9c1a21
Support cyclic dependencies in `uv tree` (#8564)
## Summary

Closes https://github.com/astral-sh/uv/issues/8551.
2024-10-26 13:30:42 -04:00
Charlie Marsh 5ef0e6c838
Ignore empty groups when validating lock (#8598)
## Summary

It turns out we were omitting empty dependency groups from the lockfile
metadata, which was then causing us to reject locks when empty groups
were defined.

We now include them (that section of the lock is meant to be a true
representation of the metadata, and an empty-but-defined group is
different from an absent group), though we can ignore them for
validation, since it doesn't affect any behavior.

Closes https://github.com/astral-sh/uv/issues/8581.
2024-10-26 17:29:07 +00:00
Charlie Marsh dd0f696695
Allow `[dependency-groups]` in non-`[project]` projects (#8574)
## Summary

We already support `tool.uv.dev-dependencies` in the legacy
non-`[project]` projects. This adds equivalent support for
`[dependency-groups]`, e.g.:

```toml
[tool.uv.workspace]

[dependency-groups]
lint = ["ruff"]
```
2024-10-25 13:57:06 -05:00
Charlie Marsh 291c4c496d Add support for `default-groups` (#8471)
This PR adds support for `tool.uv.default-groups`, which defaults to
`["dev"]` for backwards-compatibility. These represent the groups we
sync by default.
2024-10-25 13:27:37 -05:00
Charlie Marsh 2e028cd3b6 Rewrite some references to "optional groups" (#8454)
## Summary

We generally want to avoid references to "optional groups" now that
dependency groups are a first-class, standardized concept.
2024-10-25 13:27:37 -05:00
Charlie Marsh 5ab860be20 Rename dev dependencies to dependency groups in lockfile (#8391)
This is backwards compatible (we respect `dev-dependencies` as an
alias).

Part of https://github.com/astral-sh/uv/pull/8272.
2024-10-25 13:27:37 -05:00
Zanie Blue 1bee82329c Add `--group`, `--only-group`, and `--only-dev` support to `uv tree` (#8338)
Part of #8090

Most of the heavy lifting is done in #8309

Includes `--only-dev` which appears to be missing as an oversight.
2024-10-25 13:27:37 -05:00
Zanie Blue d2e1f180ef Refactor development dependency configuration (#8309)
Part of #8090
Unblocks https://github.com/astral-sh/uv/pull/8274

Refactors `DevMode` and `DevSpecification` into a shared type
`DevGroupsSpecification` that allows us to track if `--dev` was
implicitly or explicitly provided.
2024-10-25 13:27:37 -05:00
Zanie Blue 4c0590ff6f Add `--group` support to `uv add` and `uv remove` (#8108)
Part of #8090

Adds the ability to add and remove dependencies from arbitrary groups
using `uv add` and `uv remove`. Does not include resolving with the new
dependencies — tackling that in #8110.

Additionally, this does not yet resolve interactions with the existing
`dev` group — we'll tackle that separately as well. I probably won't
merge the stack until that design is resolved.
2024-10-25 13:27:37 -05:00
Charlie Marsh f4a3f97fe0
Fix dangling non-platform dependencies in `uv tree` (#8532)
## Summary

We were including dependencies that were only included by a dependency
that isn't relevant on the current platform (i.e., we were enforcing the
"current environment" at one level, but not transitively).

Closes https://github.com/astral-sh/uv/issues/8516.
2024-10-24 16:39:00 +00:00
Charlie Marsh 2651aee33f
Enforce lockfile schema versions (#8509)
## Summary

Historically, we haven't enforced schema versions. This PR adds a
versioning policy such that, if a uv version writes schema v2, then...

- It will always reject lockfiles with schema v3 or later.
- It _may_ reject lockfiles with schema v1, but can also choose to read
them, if possible.

(For example, the change we proposed to rename `dev-dependencies` to
`dependency-groups` would've been backwards-compatible: newer versions
of uv could still read lockfiles that used the `dev-dependencies` field
name, but older versions should reject lockfiles that use the
`dependency-groups` field name.)

Closes https://github.com/astral-sh/uv/issues/8465.
2024-10-24 12:23:56 -04:00
Charlie Marsh e9c08b1639
Rewrite `uv tree` to allow arbitrary `--package` includes (#8507)
## Summary

Previously, `uv tree --package` had some strange behavior due to how we
were computing the root nodes. This PR refactors the entire
implementation to use `petgraph` so we can do proper operations on a
graph structure.

Closes https://github.com/astral-sh/uv/issues/8382.
2024-10-23 18:43:26 +00:00