Commit Graph

1357 Commits

Author SHA1 Message Date
Charlie Marsh ffd4b6fcac
Remove unused `download_and_extract_archive` method (#2793)
## Summary

This was introduced at some point but then removed in favor of accessing
the database directly.
2024-04-03 02:34:14 +00:00
Charlie Marsh 29492b7754
Fix references to "WerkZeug" in "anyio" tests (#2791)
## Summary

Also fixes up the descriptions (some were wrong).
2024-04-02 22:28:12 -04:00
Zander 4b2e67955f
fixed uv can't create .venv for cpython-x86 on Windows (#2707)
Adaptation to the win32 platform is added.

https://docs.python.org/3/library/sysconfig.html#sysconfig.get_platform


## Summary

fixed uv can't create .venv for cpython-x86 on Windows 

[uv can't create .venv for cpython-x86 on Windows
](https://github.com/astral-sh/rye/issues/952)

---------

Co-authored-by: Nashan <34827878+zhuang1234@users.noreply.github.com>
2024-04-02 20:45:53 -05:00
Charlie Marsh 684f790d5d
Preserve `.git` suffixes and casing in Git dependencies (#2789)
## Summary

I noticed in #2769 that I was now stripping `.git` suffixes from Git
URLs after resolving to a precise commit. This PR cleans up the internal
caching to use a better canonical representation: a `RepositoryUrl`
along with a `GitReference`, instead of a `GitUrl` which can contain
non-canonical data. This gives us both better fidelity (preserving the
`.git`, along with any casing that the user provided when defining the
URL) and is overall cleaner and more robust.
2024-04-03 00:24:29 +00:00
Charlie Marsh c30a65ee0c
Allow conflicting Git URLs that refer to the same commit SHA (#2769)
## Summary

This PR leverages our lookahead direct URL resolution to significantly
improve the range of Git URLs that we can accept (e.g., if a user
provides the same requirement, once as a direct dependency, and once as
a tag). We did some of this in #2285, but the solution here is more
general and works for arbitrary transitive URLs.

Closes https://github.com/astral-sh/uv/issues/2614.
2024-04-02 23:36:35 +00:00
Zanie Blue 20d4762776
Move scenario sync script to top-level scripts (#2788)
Same idea as https://github.com/astral-sh/uv/pull/2787
2024-04-02 23:27:05 +00:00
Zanie Blue 0046a0d596
Bump version to 0.1.28 (#2785) 2024-04-02 21:39:35 +00:00
Charlie Marsh 03b709a0e7
Detect Fish via `FISH_VERSION` (#2781)
## Summary

Apparently `SHELL` isn't always set to Fish:
https://github.com/fish-shell/fish-shell/issues/374.

## Test Plan

`uv venv` after running `fish` in macOS.
2024-04-02 19:50:48 +00:00
Zanie Blue 1ac9672b95
Resolve non-determistic behavior in preferences due to site-packages ordering (#2780)
Originally a regression test for #2779 but we found out that there's
some weird behavior where different `anyio` versions were preferred
based on the platform.
2024-04-02 13:48:33 -05:00
Zanie Blue 119d753cfe
Exclude installed distributions with multiple versions from consideration in the resolver (#2779)
Addresses panic introduced in #2596 and reported in
https://github.com/astral-sh/uv/issues/2763#issuecomment-2030674936

When there are multiple versions of a package available, we remove the
existing packages before installing the resolved version to "fix" the
environment. We must remove all of the package versions and reinstall
because removing _any_ of the package versions could break the others.
Since reinstalls require a pull from the remote, this broke a contract
between the resolver and the planner which must always agree on which
packages should come from the remote. This further demonstrates that we
should be constructing the install plan with more concrete knowledge
from the resolver (i.e. `ResolvedDist` instead of `Requirement`) to
avoid having to manually ensure logic matches.

## Test plan

Fails on `main` with panic succeeds on branch

```
uv venv --seed
source .venv/bin/activate
pip install anyio==3.7.0 --ignore-installed
pip install anyio==4.0.0 --ignore-installed
cargo run -- pip install anyio black -v
```
2024-04-02 12:10:52 -05:00
Charlie Marsh b73cfbb0de
Use `user_display` for source tree error (#2768) 2024-04-02 00:42:11 +00:00
Charlie Marsh ccd457a37e
Use distribution database and index for all pre-resolution phases (#2766)
## Summary

Ensures that if we resolve any distributions before the resolver, we
cache the metadata in-memory.

_Also_ ensures that we lock (important!) when resolving Git
distributions.
2024-04-02 00:34:13 +00:00
Charlie Marsh dfdcce68fd
Split `get_or_build_wheel_metadata` into distinct methods (#2765)
## Summary

Internal refactor which makes `DistributionDatabase` for unnamed
requirements (or, at least, source distributions).
2024-04-01 23:52:21 +00:00
Charlie Marsh f2c9e88f3e
Populate the in-memory index when resolving lookahead URLs (#2761)
## Summary

Just ensures that we don't have to go back to the cache when resolving.
2024-04-01 22:07:54 +00:00
Charlie Marsh 999d653404
Use canonical URL to key redirect map (#2764)
## Summary

This fixes a potential bug that revealed itself in
https://github.com/astral-sh/uv/pull/2761. We don't run into this now
because we always use "allowed URLs", stores the "last" compatible URL
in the map. But the use of the "raw" URL (rather than the "canonical"
URL) means that other writers have to follow that same assumption and
iterate over dependencies in-order.
2024-04-01 17:57:13 -04:00
Charlie Marsh d2bbc07b76
Rename `resolver.rs` to `unnamed.rs` in `uv-requirements` (#2762)
## Summary

We have a few different "resolvers" in here now so the name is
confusing.
2024-04-01 21:40:03 +00:00
Charlie Marsh a48bcaecb1
Recursively resolve direct URL references upfront (#2684)
## Summary

This PR would enable us to support transitive URL requirements. The key
idea is to leverage the fact that...

- URL requirements can only come from URL requirements.
- URL requirements identify a _specific_ version, and so don't require
backtracking.

Prior to running the "real" resolver, we recursively resolve any URL
requirements, and collect all the known URLs upfront, then pass those to
the resolver as "lookahead" requirements. This means the resolver knows
upfront that if a given package is included, it _must_ use the provided
URL.

Closes https://github.com/astral-sh/uv/issues/1808.
2024-04-01 21:16:20 +00:00
Charlie Marsh f60e7495ca
Bump version to v0.1.27 (#2758) 2024-04-01 14:54:50 -04:00
konsti 05ed7ac64b
Don't use exclude newer manually in test functions (#2697)
With this change, all usages of `EXCLUDE_NEWER` are now in command
wrappers, not in the test functions themselves.

For the venv test, i refactored them into the same kind of test context
abstraction that the other test modules have in the second commit.

The third commit makes`"INSTA_FILTERS` "private", removing the last
remaining individual usage.

Pending windows CI 🤞
2024-04-01 16:07:25 +02:00
Charlie Marsh e68cdb1049
Rename to `SourceDistributionBuilder` (#2750)
## Summary

This is more consistent with `DistributionDatabase`. The order of the
arguments is also now consistent between the two structs.
2024-04-01 02:37:43 +00:00
Charlie Marsh 8596ff3470
Remove `Cache` argument from `DistributionDatabase` (#2749)
## Summary

We can access cache from `BuildContext`. This mirrors
`SourceDistCachedBuilder`, which doesn't accept `Cache` as an argument
and always accesses it through `BuildContext`.
2024-03-31 22:22:25 -04:00
Charlie Marsh ffdb283ec9
Add `pyproject.toml` et al to list of prompted packages (#2746)
## Summary

If the user provides `uv pip install pyproject.toml`, we now prompt them
to confirm that they meant the `pyproject-toml` package (as opposed to
`uv pip install -r pyproject.toml`).
2024-04-01 02:02:06 +00:00
Charlie Marsh c669542a9e
Respect overrides in all direct-dependency iterators (#2742)
## Summary

We iterate over the project "requirements" directly in a variety of
places. However, it's not always the case that an input "requirement" on
its own will _actually_ be part of the resolution, since we support
"overrides".

Historically, then, overrides haven't worked as expected for _direct_
dependencies (and we have some tests that demonstrate the current,
"wrong" behavior). This is just a bug, but it's not really one that
comes up in practice, since it's rare to apply an override to your _own_
dependency.

However, we're now considering expanding the lookahead concept to
include local transitive dependencies. In this case, it's more and more
important that overrides and constraints are handled consistently.

This PR modifies all the locations in which we iterate over requirements
directly, and modifies them to respect overrides (and constraints, where
necessary).
2024-03-31 14:03:49 -04:00
eth3lbert f65f013066
Add `--exclude-editable` support to `pip-freeze` (#2740)
## Summary

Closes #2738
2024-03-31 17:59:30 +00:00
Charlie Marsh 6d5b5ae9a2
Use distinct `Constraints` and `Overrides` types more widely (#2741)
## Summary

No functional changes.
2024-03-31 13:46:37 -04:00
Charlie Marsh ff6aea3f5c
Respect subdirectories when reading static metadata (#2728)
## Summary

This was just an oversight (and a lack of test coverage).

Closes https://github.com/astral-sh/uv/issues/2727.
2024-03-29 22:36:10 -04:00
Charlie Marsh 1f31350d46
Recursively allow URL requirements for local dependencies (#2702)
## Summary

This is a trimmed-down version of
https://github.com/astral-sh/uv/pull/2684 that only applies to local
source trees for now, which enables workspace-like workflows (whereby
local packages can depend on other local packages at arbitrary depth).

Closes #2699.

## Test Plan

Added new tests.

Also cloned this MRE that was shared with me
(https://github.com/timothyjlaurent/uv-poetry-monorepo-mre), and
verified that it was installed without error:

```
❯ cargo run pip install ./uv-poetry-monorepo-mre/app --no-cache
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
     Running `target/debug/uv pip install ./uv-poetry-monorepo-mre/app --no-cache`
Resolved 4 packages in 1.28s
   Built app @ file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/app
   Built lib1 @ file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib1
   Built lib2 @ file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib2                                        Downloaded 4 packages in 457ms
Installed 4 packages in 2ms
 + app==0.1.0 (from file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/app)
 + lib1==0.1.0 (from file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib1)
 + lib2==0.1.0 (from file:///Users/crmarsh/workspace/uv/uv-poetry-monorepo-mre/lib2)
 + ruff==0.3.4
```
2024-03-28 22:53:57 +00:00
Charlie Marsh ce5df77ecb
Add `UV_RESOLUTION` environment variable for `--resolution` (#2720)
Closes https://github.com/astral-sh/uv/issues/2710.
2024-03-28 17:08:16 -04:00
Zanie Blue 27b96211a2
Filter flake message for `install_git_public_https_missing_commit` on win32 (#2717) 2024-03-28 13:52:17 -05:00
Zanie Blue e1878c8359
Consider installed packages during resolution (#2596)
Previously, we did not consider installed distributions as candidates
while performing resolution. Here, we update the resolver to use
installed distributions that satisfy requirements instead of pulling new
distributions from the registry.

The implementation details are as follows:

- We now provide `SitePackages` to the `CandidateSelector`
- If an installed distribution satisfies the requirement, we prefer it
over remote distributions
- We do not want to allow installed distributions in some cases, i.e.,
upgrade and reinstall
- We address this by introducing an `Exclusions` type which tracks
installed packages to ignore during selection
- There's a new `ResolvedDist` wrapper with `Installed(InstalledDist)`
and `Installable(Dist)` variants
- This lets us pass already installed distributions throughout the
resolver

The user-facing behavior is thoroughly covered in the tests, but
briefly:

- Installing a package that depends on an already-installed package
prefers the local version over the index
- Installing a package with a name that matches an already-installed URL
package does not reinstall from the index
- Reinstalling (--reinstall) a package by name _will_ pull from the
index even if an already-installed URL package is present
- To reinstall the URL package, you must specify the URL in the request

Closes https://github.com/astral-sh/uv/issues/1661

Addresses:

- https://github.com/astral-sh/uv/issues/1476
- https://github.com/astral-sh/uv/issues/1856
- https://github.com/astral-sh/uv/issues/2093
- https://github.com/astral-sh/uv/issues/2282
- https://github.com/astral-sh/uv/issues/2383
- https://github.com/astral-sh/uv/issues/2560

## Test plan

- [x] Reproduction at `charlesnicholson/uv-pep420-bug` passes
- [x] Unit test for editable package
([#1476](https://github.com/astral-sh/uv/issues/1476))
- [x] Unit test for previously installed package with empty registry
- [x] Unit test for local non-editable package
- [x] Unit test for new version available locally but not in registry
([#2093](https://github.com/astral-sh/uv/issues/2093))
- ~[ ] Unit test for wheel not available in registry but already
installed locally
([#2282](https://github.com/astral-sh/uv/issues/2282))~ (seems
complicated and not worthwhile)
- [x] Unit test for install from URL dependency then with matching
version ([#2383](https://github.com/astral-sh/uv/issues/2383))
- [x] Unit test for install of new package that depends on installed
package does not change version
([#2560](https://github.com/astral-sh/uv/issues/2560))
- [x] Unit test that `pip compile` does _not_ consider installed
packages
2024-03-28 13:49:17 -05:00
Andrew Gallant 7b685a8158
uv-cache: bump built-wheels cache version (#2715)
It turns out that #2712 did _not_ fix #2711. After I put up #2712, I
started trying to track down the specific change that caused the
failure. I had assumed at first that it was related to one of our `rkyv`
types, but it actually ended up being one of our msgpack caches. I think
the failure mode is still fundamentally the same idea: the cached data
changed in a way that is still valid msgpack, but got interpreted
differently after deserializing.

The specific change that caused this was the [removal of a field] from
our
metadata type.

Ideally we would just undo the change and add the field back. But that
change has already been shipped out to users. So I believe the only
plausible choice at this point is to bump the `built-wheels` cache. This
will unfortunately mean that `uv` will need to re-build wheels.

Fixes #2711

[removal of a field]:
365c292525 (diff-e42586829f9c2cdbb909bedc5cf95691cc415247f2cbc2ebeb80d887020457bbL29)
2024-03-28 13:24:47 -05:00
Zanie Blue ffa1b01176
Bump version to 0.1.26 (#2713) 2024-03-28 12:55:36 -05:00
Andrew Gallant c72d3d73d5
uv-cache: bump simple cache version (#2712)
It seems likely that we forgot to bump the version of the "simple" cache
in the 0.1.25 release. I'm still working on confirming it, but I figured
I'd get this bump up first.

The main problem here is that our "simple" cache is represented by
`rkyv`, and that in turn is tightly coupled to the representation of a
selection of data types in `uv`. Changing those data types without
bumping the cache version can result in cache deserialization errors
like this, or in the worst case, silent logic errors.

One possibility here is that the representation changed in a way that
permitted it to pass `rkyv` validation, but changed how the data itself
is interpreted. Our cache is robust with respect to `rkyv` validation
(if it fails, the cache will invalidate the entry and self-heal), but
being robust to higher level logical errors in interpretation of the
data is a much more significant challenge. Our best bet there is perhaps
some kind of checksum that we could do on top of `rkyv` validation (or
instead of it), and thus convert silent logical changes in how the data
is interpreted into failure modes that we're already robust to.

Fixes #2711
2024-03-28 12:29:29 -05:00
Zanie Blue 4e59bfd16e
Bump version to 0.1.25 (#2709)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2024-03-28 10:17:52 -05:00
Charlie Marsh f8f7f848f5
Remove `Tags` from `tracing` (#2704)
Oops, these show in `tracing` now, and they're huge.
2024-03-28 02:18:54 +00:00
Charlie Marsh 4cc91cc6bb
Add convenience methods to `Manifest` to iterate over requirements (#2701)
## Summary

These are repeated a bunch. It's nice to DRY them up and ensure the
ordering is consistent.
2024-03-28 01:03:44 +00:00
Charlie Marsh b6ab919945
Make tags non-required for fetching wheel metadata (#2700)
## Summary

This looks like a big change but it really isn't. Rather, I just split
`get_or_build_wheel` into separate `get_wheel` and `build_wheel` methods
internally, which made `get_or_build_wheel_metadata` capable of _not_
relying on `Tags`, which in turn makes it easier for us to use the
`DistributionDatabase` in various places without having it coupled to an
interpreter or environment (something we already did for
`SourceDistributionBuilder`).
2024-03-28 00:06:25 +00:00
Charlie Marsh cf30932831
Allow prereleases, locals, and URLs in non-editable path requirements (#2671)
## Summary

This PR enables the resolver to "accept" URLs, prereleases, and local
version specifiers for direct dependencies of path dependencies. As a
result, `uv pip install .` and `uv pip install -e .` now behave
identically, in that neither has a restriction on URL dependencies and
the like.

Closes https://github.com/astral-sh/uv/issues/2643.
Closes https://github.com/astral-sh/uv/issues/1853.
2024-03-27 22:17:09 +00:00
Charlie Marsh 4b69ad4281
Remove some unused dependencies (#2698) 2024-03-27 17:50:55 -04:00
Charlie Marsh f8fa887c0b
Use `Resolver` in `pip sync` (#2696)
## Summary

This PR removes the custom `DistFinder` that we use in `pip sync`. This
originally existed because `VersionMap` wasn't lazy, and so we saved a
lot of time in `DistFinder` by reading distribution data lazily. But
now, AFAICT, there's really no benefit. Maintaining `DistFinder` means
we effectively have to maintain two resolvers, and end up fixing bugs in
`DistFinder` that don't exist in the `Resolver` (like #2688.

Closes #2694.

Closes #2443.

## Test Plan

I ran this benchmark a bunch. It's basically a wash. Sometimes one is
faster than the other.

```
❯ python -m scripts.bench \
        --uv-path ./target/release/main \
        --uv-path ./target/release/uv \
        scripts/requirements/compiled/trio.txt --min-runs 50 --benchmark install-warm --warmup 25
Benchmark 1: ./target/release/main (install-warm)
  Time (mean ± σ):      54.0 ms ±  10.6 ms    [User: 8.7 ms, System: 98.1 ms]
  Range (min … max):    45.5 ms …  94.3 ms    50 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: ./target/release/uv (install-warm)
  Time (mean ± σ):      50.7 ms ±   9.2 ms    [User: 8.7 ms, System: 98.6 ms]
  Range (min … max):    44.0 ms …  98.6 ms    50 runs

  Warning: The first benchmarking run for this command was significantly slower than the rest (77.6 ms). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

Summary
  './target/release/uv (install-warm)' ran
    1.06 ± 0.29 times faster than './target/release/main (install-warm)'
```
2024-03-27 17:36:38 -04:00
Andrew Gallant d41ab0ef4d
uv/tests: add regression test for `echo pendulum | uv pip compile -` (#2693)
I stumbled across this when writing tests for
`--emit-marker-expressions`. Namely, I observed that in CI, the `tzdata`
dependency of `pendulum` wasn't included in the `requirements.txt`
output on Windows.

@konstin [suggested] that this was a bug, so I've created a test for it.
In particular, it looks like [`tzdata` is an unconditional dependency of
`pendulum`][tzdata-unconditional].

[suggested]:
https://github.com/astral-sh/uv/pull/2651#discussion_r1539722464
[tzdata-unconditional]:
e646afbd165e58327bc5c698731107/pendulum-3.0.0-cp310-none-win_amd64.whl/pendulum-3.0.0.dist-info/METADATA#line.12
2024-03-27 13:04:50 -04:00
Charlie Marsh dc957d7322
Respect `--no-index` with `--find-links` in `pip sync` (#2692)
## Summary

In `pip sync`, we weren't properly handling cases in which a package
_only_ existed in `--find-links` (e.g., the user passed `--offline` or
`--no-index`).

I plan to explore removing `Finder` entirely to avoid these mismatch
bugs between `pip sync` and other commands, but this is fine for now.

Closes https://github.com/astral-sh/uv/issues/2688.

## Test Plan

`cargo test`
2024-03-27 16:15:14 +00:00
Andrew Gallant 384355bb57 uv: add tests for --emit-marker-expressions
Unfortunately these tests are all gated on specific platforms because
the marker expressions they generate are, by design, platform specific.

I think we'll eventually want to figure out a more robust testing
strategy for multi-platform locking (of which this is just the tiniest
of first steps), but I don't think we really have the infrastructure for
that in place yet. That is, we don't yet have a way of generating a
marker expression _for_ a particular environment instead of just the one
that happens to _be_ the current environment.
2024-03-27 11:22:29 -04:00
Andrew Gallant c21e759567 uv: add new hidden --emit-marker-expression flag
When enabled, the marker expression for the pinned requirements
is written as a comment at the top of the output. It is disabled
by default *and* hidden because it's not clear whether 1) this is
useful to end users and 2) is an interface we want to commit to.
However, it is useful to expose it in some way so that it can be
tested.
2024-03-27 11:22:29 -04:00
Andrew Gallant 980dcc3324 uv-resolver: add new 'marker_tree' method on ResolutionGraph
This routine generates a marker expression that, when evaluated
to true, implies that the resolution graph is guaranteed to be
valid.
2024-03-27 11:22:29 -04:00
Andrew Gallant 50e48847f1 uv-resolver: add Clone impls
For $reasons, we'll want to be able to clone a `Manifest` so
that it can be re-used to generate a marker expression.

There is likely a refactoring that could be done to avoid the
cloning, but a `Manifest` is likely to be small in practice, and
we'll only need to clone it once.
2024-03-27 11:22:29 -04:00
Andrew Gallant b177bd026d pep508: expose lower level marker env routines
These are useful for converting lower level marker values types
to their corresponding values from a marker environment.

We'll use these for generating marker expressions based on both
the dependency graph and the current marker environment.
2024-03-27 11:22:29 -04:00
Charlie Marsh 236a4163be
Unhide `--emit-index-url` and `--emit-find-links` (#2691)
## Summary

I don't know why I hid these in the first place. Maybe I copied these
over from the `compat.rs` version.

Closes https://github.com/astral-sh/uv/issues/1502.
2024-03-27 14:55:25 +00:00
Charlie Marsh 15c9a5c980
Use Scripts folder for virtualenv activation prompt (#2690)
## Summary

We shouldn't assume that you're on Unix when using Nushell.

Closes https://github.com/astral-sh/uv/issues/2687.
2024-03-27 10:46:38 -04:00
Charlie Marsh 8c7a6038fd
Update Rust to v1.77 (#2591) 2024-03-27 14:41:27 +00:00
Charlie Marsh 365c292525
Read package metadata from `pyproject.toml` when statically defined (#2676)
## Summary

Now that we're resolving metadata more aggressively for local sources,
it's worth doing this. We now pull metadata from the `pyproject.toml`
directly if it's statically-defined.

Closes https://github.com/astral-sh/uv/issues/2629.
2024-03-27 14:34:18 +00:00
Zanie Blue 248d6f89ef
Bring harmony to the test snapshot filtering situation (#2678)
The snapshot filtering situation has gotten way out of hand, with each
test hand-rolling it's own filters on top of copied cruft from previous
tests.

I've attempted to address this holistically:

- `TestContext.filters()` has everything you should need 
- This was introduced a while ago, but needed a few more filters for it
to be generalized everywhere
- Using `INSTA_FILTERS` is **not recommended** unless you do not want
the context filters
    - It is okay to extend these filters for things unrelated to paths
- If you have to write a custom path filter, please highlight it in
review so we can address it in the common module
- `TestContext.site_packages()` gives cross-platform access to the
site-packages directory
    - Do not manually construct the path to site-packages from the venv
- Do not turn off tests on Windows because you manually constructed a
Unix path to site-packages
- `TestContext.workspace_root` gives access to uv's repository directory
    - Use this for installing from `scripts/packages/`
- If you need coverage for relative paths, copy the test package into
the `temp_dir` don't change the working directory of the test fixture

There is additional work that can be done here, such as:

- Auditing and removing additional uses of `INSTA_FILTERS`
- Updating manual construction of `Command` instances to use a utility
- The `venv` tests are particularly frightening in their lack of a test
context and could use some love
- Improving the developer experience i.e. apply context filters to
snapshots by default
2024-03-27 14:10:12 +00:00
Charlie Marsh ffd78d0821
Add an in-memory cache for Git references (#2682)
## Summary

Ensures that, even if we try to resolve the same Git reference twice
within an invocation, it always returns a (cached) consistent result.

Closes https://github.com/astral-sh/uv/issues/2673.

## Test Plan

```
❯ cargo run pip install git+https://github.com/pallets/flask.git --reinstall --no-cache
   Compiling uv-distribution v0.0.1 (/Users/crmarsh/workspace/uv/crates/uv-distribution)
   Compiling uv-resolver v0.0.1 (/Users/crmarsh/workspace/uv/crates/uv-resolver)
   Compiling uv-installer v0.0.1 (/Users/crmarsh/workspace/uv/crates/uv-installer)
   Compiling uv-dispatch v0.0.1 (/Users/crmarsh/workspace/uv/crates/uv-dispatch)
   Compiling uv-requirements v0.1.0 (/Users/crmarsh/workspace/uv/crates/uv-requirements)
   Compiling uv v0.1.24 (/Users/crmarsh/workspace/uv/crates/uv)
    Finished dev [unoptimized + debuginfo] target(s) in 3.95s
     Running `target/debug/uv pip install 'git+https://github.com/pallets/flask.git' --reinstall --no-cache`
 Updated https://github.com/pallets/flask.git (b90a4f1)
Resolved 7 packages in 280ms
   Built flask @ git+https://github.com/pallets/flask.git@b90a4f1f4a370e92054b9cc9db0efcb864f87ebe                                                                                                                                            Downloaded 7 packages in 212ms
Installed 7 packages in 9ms
```
2024-03-27 01:39:01 +00:00
Charlie Marsh 32d8ee8ba3
Add `--no-binary` and `--only-binary` support to `requirements.txt` (#2680)
## Summary

Closes https://github.com/astral-sh/uv/issues/1461.
2024-03-27 01:12:29 +00:00
Zanie Blue 0b08ba1e67
Rename `uv-traits` and split into separate modules (#2674)
This is driving me a little crazy and is becoming a larger problem in
#2596 where I need to move more types (like `Upgrade` and `Reinstall`)
into this crate. Anything that's shared across our core resolver,
install, and build crates needs to be defined in this crate to avoid
cyclic dependencies. We've outgrown it being a single file with some
shared traits.

There are no behavioral changes here.
2024-03-26 15:39:43 -05:00
Charlie Marsh 39769d82a0
Fall back to PEP 517 hooks for non-compliant PEP 621 metadata (#2662)
If you pass a `pyproject.toml` that use Hatch's context formatting API,
we currently fail because the dependencies aren't valid under PEP 508.
This PR makes the static metadata parsing a little more relaxed, so that
we appropriately fall back to PEP 517 there.
2024-03-26 02:28:39 +00:00
Charlie Marsh 12846c2c85
Disallow `pyproject.toml` from `pip uninstall` (#2663)
## Summary

Passing `pyproject.toml` or `setup.py` to `pip uninstall` is a bit
strange, since it will often require running a resolution to resolve the
dependencies (e.g., build the project), which means we also need to
accept `--index-url` and friends.
2024-03-26 01:35:43 +00:00
Charlie Marsh 5270624b11
Avoid `prepare_metadata_for_build_wheel` calls for Hatch packages with dynamic dependencies (#2645)
## Summary

Hatch allows for highly dynamic customization of metadata via hooks. In
such cases, Hatch
can't upload the PEP 517 contract, in that the metadata Hatch would
return by
`prepare_metadata_for_build_wheel` isn't guaranteed to match that of the
built wheel.

Hatch disables `prepare_metadata_for_build_wheel` entirely for pip.
We'll instead disable
it on our end when metadata is defined as "dynamic" in the
pyproject.toml, which should
allow us to leverage the hook in _most_ cases while still avoiding
incorrect metadata for
the remaining cases.

Closes: https://github.com/astral-sh/uv/issues/2130.
2024-03-25 22:26:51 +00:00
Charlie Marsh ae35a215c6
Accept `setup.py` and `setup.cfg` files in compile (#2634)
## Summary

This costs us ~nothing now that we support using PEP 517 hooks for
`pyproject.toml`.

## Test Plan

`cargo test`
2024-03-25 20:39:35 +00:00
Charlie Marsh 71428f7d74
Use PEP 517 to extract non-static `pyproject.toml` metadata (#2633)
## Summary

When a user passes a `pyproject.toml` to `pip compile` (e.g., `uv pip
compile pyproject.toml`), we extract the requirements from the
`pyproject.toml` directly. However... that isn't always possible (as
seen in the linked issues). When it's _not_, we instead need to run the
PEP 517 build hooks to identify the metadata.

Closes https://github.com/astral-sh/uv/issues/1624.

Closes https://github.com/astral-sh/uv/issues/1644.

## Test Plan

`cargo test`
2024-03-25 20:27:41 +00:00
Charlie Marsh f3753347e0
Add `rustc-hash` to `uv` crate (#2660)
## Summary

This is required as of https://github.com/astral-sh/uv/pull/2589, but
isn't a direct dependency. I'm not sure how that PR passed CI exactly.
2024-03-25 20:21:43 +00:00
eth3lbert 6d7b2c7ca4
`Editable project location` and `Required-by` for `pip show` (#2589)
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

- Displays missing packages as single-line warnings.
- Adds support for `Editable project location` and `Required-by` fields
in `pip show`.

Part of #2526.
2024-03-25 14:45:05 -05:00
Charlie Marsh 8587c440fe
Support `file://localhost/` schemes (#2657)
## Summary

`uv` was failing to install requirements defined like:

```
file://localhost/Users/crmarsh/Downloads/iniconfig-2.0.0-py3-none-any.whl
```

Closes https://github.com/astral-sh/uv/issues/2652.
2024-03-25 19:23:26 +00:00
Andrew Gallant 7a5571fa5c
bump EXCLUDE_NEWER to more recent datetime (#2650)
While writing tests for a new flag (`--emit-marker-expression`) for `uv
pip compile`, I noticed that one of my test cases (`pendulum 3.0.0`)
was published in Dec 2023. I wanted to include this package in my
tests, but since it comes after our `EXCLUDE_NEWER` constant, it wasn't
visible to `uv`.

In this PR, I chose to resolve this by bumping `EXCLUDE_NEWER` to
`2024-03-25T00:00:00Z`. I also considered a couple other options:

* For a specific test, override and provide a custom `--exclude-newer`
flag. I felt like this would maybe be okay, but we could easily wind
up in a situation where we do this a lot and have a bunch of different
`--exclude-newer` flags in our tests. I'm not sure if this is a huge
problem in practice. Maybe it's fine.
* Find another package (or invent one) with a similarly interesting
configuration. It seemed easier to just bump `EXCLUDE_NEWER`.

The way I did this was to run `cargo insta test` after bumping
`EXCLUDE_NEWER`.
I then reviewed the snapshot diffs, and if they looked reasonable, I
accepted them.
There was only one case where I changed the test to preserve what I
thought it
was trying to test. That's isolated in its own commit.
2024-03-25 11:29:05 -04:00
konsti 33bde826a0
Update pubgrub to use a dependency provider (#2648)
With https://github.com/pubgrub-rs/pubgrub/pull/190, pubgrub attaches
all types to a dependency provider to reduce the number of generics. We
need a dummy dependency provider now to emulate this. On the plus side,
pep440_rs drops its pubgrub dependency.
2024-03-25 15:51:31 +01:00
Andrew Gallant 8cc69a723d
uv-client: fix linehaul test (#2647)
This test was introduced in 42973cd9cb. It
looks like it compares some values against some platform specific code
that attempts to find the OS version. But the comparisons made some
assumptions about what kind of data is available. In this commit, we try
to make the test a little more flexible on Linux by not assuming that
`Option` values are `Some`.
2024-03-25 10:12:00 -04:00
Charlie Marsh 1601ea08e6
Rename `scripts/editable-installs` to `scripts/packages` (#2644)
## Summary

This was discussed in a prior PR.
2024-03-25 03:56:00 +00:00
Shantanu a6602ad416
Do not force recompile pyc files (#2642)
Helps with #2637
2024-03-24 20:02:53 -04:00
Charlie Marsh a7b251f65e
Limit overrides and constraints to `requirements.txt` format (#2632)
## Summary

I don't see a great reason to allow this, and it adds a lot of
complexity, so `pyproject.toml` files are now limited to `pip compile`
and `pip install -r` -- they can't be passed as `-c` or `--override`.
2024-03-23 09:33:29 -04:00
Hans Baker a632d2485b
Don't error on multiple matching index URLs (#2627)
## Summary

Closes Issue:
- https://github.com/astral-sh/uv/issues/2626

## Test Plan

```
cargo run -- pip install -r dev-requirements.txt -r requirements.txt
```
where both requirements files have same `--index-url`
2024-03-22 23:37:02 +00:00
Zanie Blue d0cb301d59
Use dense formatting for requirement version specifiers in diagnostics (#2601)
For consistency with the output in "no solution" errors
2024-03-22 16:35:49 -05:00
Charlie Marsh 90a1f89ac4
Extract local versions from direct URL requirements (#2624)
## Summary

Closes https://github.com/astral-sh/uv/issues/2623.

## Test Plan

`cargo run pip install -r requirements.txt`, with:

```
 whisperX @ befe2b242e.zip
 pyannote-audio @ git+https://github.com/pyannote/pyannote-audio@11b56a137a578db9335efc00298f6ec1932e6317
 torch @ https://download.pytorch.org/whl/cu118/torch-2.2.1%2Bcu118-cp311-cp311-linux_x86_64.whl#sha256=84328a35621cc6a67a182a327baaab67e5f5869981c4b1677ed05f92c15cceb1
 torchaudio @ https://download.pytorch.org/whl/cu118/torchaudio-2.2.1%2Bcu118-cp311-cp311-linux_x86_64.whl#sha256=cd3b1c3582b17792c6d7a367dea0459b123e54d7a4242809ea87ccc10fa220e5
 pytorch_triton @ https://download.pytorch.org/whl/nightly/pytorch_triton-2.1.0%2B7d1a95b046-cp311-cp311-linux_x86_64.whl
```
2024-03-22 21:34:14 +00:00
Zanie Blue a5cae02922
Bump version to 0.1.24 (#2621) 2024-03-22 14:58:10 -05:00
konsti fca2883864
Avoid ignore crate finding cache gitignore (#2615)
We put a `.gitignore` with `*` at the top of our cache. When maturin was
building a source distribution inside the cache, it would walk up the
tree to find a gitignore, see that and ignore all python files. We now
add an (empty) `.git` directory one directory below, in the root of
built-wheels cache. This prevents ignore walking further up (it marks
the top level a git repository).

Deptry (from #2490) is a mid sized rust package with additional python
packages, so instead of using it in the test i've replaced it with a
small (44KB total) reproducer that uses cffi for faster building, the
entire test taking <2s on my machine.

Fixes #2490
2024-03-22 14:36:07 -05:00
Wolf Vollprecht 68432e77e5
fix: expose `AllowedYanks` (#2608)
## Summary

If I see correctly, this is used in the interface of
`DefaultResolverProvider` but not accessible outside.
2024-03-22 09:15:22 -05:00
konsti 48bd02b8a8
Update miette v7, pubgrub and small Cargo.toml cleanup (#2610)
I was going through the output of `cargo tree --duplicate -p uv`, not
much success except these small cleanups.
2024-03-22 10:42:48 +00:00
konsti 375b33ff0c
Remove unused deps with cargo machete (#2609) 2024-03-22 10:01:17 +00:00
Charlie Marsh 9986710a53
Make self-update an opt-in Cargo feature (#2606)
## Summary

Ensures that (e.g.) installs from conda-forge, Homebrew, and other
distributions don't expose `uv self update` at all.

We'll still show `uv self update` for `pip install uv`, but it will fail
with a good error. Removing the `uv self update` from `pip`-installed
`uv` is more complicated, since we'd need to build separately for the
installer vs. for PyPI.

Closes #2588.
2024-03-22 00:23:09 -04:00
Charlie Marsh 0ea51d43f5
Add progress reporting for named requirement resolution (#2605) 2024-03-22 04:01:10 +00:00
Charlie Marsh 31743f2bd8
Use PEP 517 build hooks to resolve unnamed requirements (#2604) 2024-03-22 03:20:40 +00:00
Charlie Marsh 5d7d7dce24
Enable PEP 517 builds for unnamed requirements (#2600)
## Summary

This PR enables the source distribution database to be used with unnamed
requirements (i.e., URLs without a package name). The (significant)
upside here is that we can now use PEP 517 hooks to resolve unnamed
requirement metadata _and_ reuse any computation in the cache.

The changes to `crates/uv-distribution/src/source/mod.rs` are quite
extensive, but mostly mechanical. The core idea is that we introduce a
new `BuildableSource` abstraction, which can either be a distribution,
or an unnamed URL:

```rust
/// A reference to a source that can be built into a built distribution.
///
/// This can either be a distribution (e.g., a package on a registry) or a direct URL.
///
/// Distributions can _also_ point to URLs in lieu of a registry; however, the primary distinction
/// here is that a distribution will always include a package name, while a URL will not.
#[derive(Debug, Clone, Copy)]
pub enum BuildableSource<'a> {
    Dist(&'a SourceDist),
    Url(SourceUrl<'a>),
}
```

All the methods on the source distribution database now accept
`BuildableSource`. `BuildableSource` has a `name()` method, but it
returns `Option<&PackageName>`, and everything is required to work with
and without a package name.

The main drawback of this approach (which isn't a terrible one) is that
we can no longer include the package name in the cache. (We do continue
to use the package name for registry-based distributions, since those
always have a name.). The package name was included in the cache route
for two reasons: (1) it's nice for debugging; and (2) we use it to power
`uv cache clean flask`, to identify the entries that are relevant for
Flask.

To solve this, I changed the `uv cache clean` code to look one level
deeper. So, when we want to determine whether to remove the cache entry
for a given URL, we now look into the directory to see if there are any
wheels that match the package name. This isn't as nice, but it does work
(and we have test coverage for it -- all passing).

I also considered removing the package name from the cache routes for
non-registry _wheels_, for consistency... But, it would require a cache
bump, and it didn't feel important enough to merit that.
2024-03-21 22:46:39 -04:00
Charlie Marsh 12192dd872
Remove unused `requirements.rs` file (#2603)
This was accidentally duplicated in a rebase.
2024-03-22 02:21:36 +00:00
Wolf Vollprecht 7d285148b2
fix: do not error when there are warnings on stderr (#2599)
## Summary

We had some users report bugs because the Python querying failed due to
warnings in `stderr`. I don't think this should fail on any `stderr`
output.

E.g.

```
  × Querying Python at `USER/.pixi/envs/default/bin/python3.10` failed with status exit status: 0 with exit status: 0
  │ --- stdout:
  │ {"markers": {"implementation_name": "cpython", "implementation_version": "3.10.0", "os_name": "posix", "platform_machine": "x86_64", "platform_python_implementation": "CPython", "platform_release": "5.15.146.1-microsoft-standard-WSL2",
  │ "platform_system": "Linux", "platform_version": "#1 SMP Thu Jan 11 04:09:03 UTC 2024", "python_full_version": "3.10.0", "python_version": "3.10", "sys_platform": "linux"}, "base_prefix": "USER/.pixi/
  │ envs/default", "base_exec_prefix": "USER/.pixi/envs/default", "prefix": "USER/.pixi/envs/default", "base_executable": "USER/.pixi/envs/default/
  │ bin/python3.10", "sys_executable": "USER/.pixi/envs/default/bin/python3.10", "stdlib": "USER/.pixi/envs/default/lib/python3.10", "scheme": {"platlib": "/home/mvanniekerk/
  │ code/vice-python/.pixi/envs/default/lib/python3.10/site-packages", "purelib": "USER/.pixi/envs/default/lib/python3.10/site-packages", "include": "USER/.pixi/envs/default/
  │ include/python3.10", "scripts": "USER/.pixi/envs/default/bin", "data": "USER/.pixi/envs/default"}, "virtualenv": {"purelib": "lib/python3.10/site-packages", "platlib": "lib/
  │ python3.10/site-packages", "include": "include/site/python3.10", "scripts": "bin", "data": ""}}
  │ --- stderr:
  │ [03/21/24 15:59:48] WARNING  pyproject.toml does not contain a setuptools.py:119
  │                              tool.setuptools_scm section
  │ ---
```
2024-03-21 19:23:41 -04:00
Zanie Blue c6e181d233
Respect HTTP client options when reading remote requirements files (#2434)
Uses the base client introduced in #2431 to restore use of our fully
configured client when reading remote requirements files.

Closes https://github.com/astral-sh/uv/issues/2357

## Test plan

```
npx http-server --username user --password password
cargo run -- pip install -r http://user:password@127.0.0.1:8080/requirements.txt
```

Fails on main succeeds on branch.
2024-03-21 13:48:57 -05:00
Charlie Marsh 0f96386032
Add a garbage collection mechanism to the CLI (#1217)
## Summary

Detects unused cache entries, which can come in a few forms:

1. Directories that are out-dated via our versioning scheme.
2. Old source distribution builds (i.e., we have a more recent version).
3. Old wheels (stored in `archive-v0`, but not symlinked-to from
anywhere in the cache).

Closes https://github.com/astral-sh/puffin/issues/1059.
2024-03-21 18:07:48 +00:00
Zanie Blue 7ee90dc71f
Fix authentication with JFrog artifactories (#2592)
Closes #2566 

We were storing the username e.g. `charlie@astral.sh` as a
percent-encoded string `charlie%40astral.sh` which resulted in different
headers and broke JFrog's artifactory which apparently does not decode
usernames.

Tested with a JFrog artifactory and AWS CodeArtifact although it is
worth noting that AWS does _not_ have a username with an `@` — it'd be
nice to test another artifactory with percent-encoded characters in the
username and/or password.
2024-03-21 12:10:43 -05:00
konsti 2375008cc1
Use c-string literals and update trampolines (#2590)
Rust 1.77 has stabilized c-string literals (`c"<string>"`):
https://doc.rust-lang.org/nightly/edition-guide/rust-2021/c-string-literals.html.
This PR replaces the usages of the custom c-string literal macro in the
trampoline with the new syntax.
2024-03-21 15:36:00 +00:00
Charlie Marsh f91ce521c5
Move requirements resolution into its own crate (#2579)
## Summary

No functional changes, but this was a lot of core logic in the `uv`
crate, which is mostly meant to be the CLI.
2024-03-21 13:52:47 +00:00
Charlie Marsh 2979918320
Add support for unnamed Git and HTTP requirements (#2578)
## Summary

Enables, e.g., `uv pip install
git+https://github.com/pallets/flask.git`.

Part of: https://github.com/astral-sh/uv/issues/313.
2024-03-21 13:44:54 +00:00
Tim de Jager fc9761d020
feat: make direct_url public (#2584)
## Summary

Made the `direct_url` method public because it was not :) Can be merged
instead of in addition to: https://github.com/astral-sh/uv/pull/2510
2024-03-21 09:37:15 -04:00
veryyet d6dad57fab
chore: fix some typos (#2581) 2024-03-21 04:09:37 +00:00
Charlie Marsh 7c728a2e77
Support unnamed requirements directly in pip uninstall (#2577)
## Summary

In `pip uninstall`, we shouldn't need to resolve unnamed requirements,
since we already index packages in `site-packages` by their URL.

This also changes `uninstall` to ignore editables, which matches pip's
behavior.

Part of: https://github.com/astral-sh/uv/issues/313.

## Test Plan

Run `cargo run pip install ./scripts/editable-installs/black_editable`,
followed by each of the following:

- `cargo run pip uninstall ./scripts/editable-installs/black_editable`
- `cargo run pip uninstall black`
- `cargo run pip uninstall ./scripts/editable-installs/black_editable
black`
2024-03-21 00:02:06 -04:00
Charlie Marsh 34bef37072
Enable install audits without resolving named requirements (#2575)
## Summary

This PR ensures that if a package is already satisfied by the current
environment, we don't bother resolving the named requirement.

Part of: https://github.com/astral-sh/uv/issues/313.

## Test Plan

- `cargo run pip install ./scripts/editable-installs/black_editable`
- `cargo run pip install black --verbose`
2024-03-20 23:54:45 -04:00
Charlie Marsh e5b0cf7f89
Add support for unnamed local directory requirements (#2571)
## Summary

For example: `cargo run pip install .`

The strategy taken here is to attempt to extract the package name from
the distribution without executing the PEP 517 build steps. We could
choose to do that in the future if this proves lacking, but it adds
complexity.

Part of: https://github.com/astral-sh/uv/issues/313.
2024-03-21 03:46:11 +00:00
Charlie Marsh 4d96255ade
Enable unnamed requirements for direct URLs (#2569)
## Summary

This PR enables `uv pip install` to accept unnamed requirements, as long
as the requirement ends with the wheel or source distribution archive
name. For example: `cargo run pip install
~/Downloads/anyio-4.3.0.tar.gz`.

In subsequent PRs, I'll expand the scope of supported archives and
patterns.

Part of: https://github.com/astral-sh/uv/issues/313.
2024-03-21 03:39:02 +00:00
Charlie Marsh ee211b35bc
Add support for parsing unnamed URL requirements (#2567)
## Summary

First piece of https://github.com/astral-sh/uv/issues/313. In order to
support unnamed requirements, we need to be able to parse them in
`requirements-txt`, which in turn means that we need to introduce a new
type that's distinct from `pep508::Requirement`, given that these
_aren't_ PEP 508-compatible requirements.

Part of: https://github.com/astral-sh/uv/issues/313.
2024-03-21 03:28:58 +00:00
Charlie Marsh 0af6a3d41d
Bump version to v0.1.23 (#2580) 2024-03-21 02:51:21 +00:00
konsti 70e0967dbd
Avoid repeating paths of workspace packages (#2573)
Scott schafer got me the idea: We can avoid repeating the path for
workspaces dependencies everywhere if we declare them in the virtual
package once and treat them as workspace dependencies from there on.
2024-03-20 16:16:02 -04:00
konsti 7111fdd637
`VIRTUAL_ENV` takes precedence over `CONDA_PREFIX` (#2574)
It is a common pattern to have an active conda base env (that sets
`CONDA_PREFIX`) and then create a venv on top of that (setting
`VIRTUAL_ENV`).

Previously, we would error when both `VIRTUAL_ENV` and `CONDA_PREFIX`
were set, now `VIRTUAL_ENV` takes precedence over `CONDA_PREFIX`.

Fixes #2028
2024-03-20 16:14:44 -04:00
Wolf Vollprecht 1255c981a7
feat: allow setting installer name other than `uv` (#2561)
## Summary

We would like to be able to configure the installer-name so that other
tools can co-exist with `uv`. In `pixi` we would like to use `pixi-uv`
as the installer name, for example, to be able to distinguish them from
packages installed by pure `uv`.
2024-03-20 10:36:49 -04:00