Commit Graph

576 Commits

Author SHA1 Message Date
Bojan Serafimov b3c6915828 no unwrap 2024-01-11 16:14:50 -05:00
Bojan Serafimov d4b21dcf50 use debug log 2024-01-11 16:05:11 -05:00
Bojan Serafimov 26edd03b05 Merge branch 'main' into bojan/perf-metrics 2024-01-11 15:27:24 -05:00
bojanserafimov 4c047f858f
Remove InMemoryWheel and dead code (#879) 2024-01-11 10:11:07 -05:00
bojanserafimov 10227a74f8
Unzip while downloading (#856) 2024-01-11 09:41:46 -05:00
konsti 0dfbddd275
Shorten resolve many dev output (#885) 2024-01-11 13:53:13 +00:00
konsti 8c2b7d55af
Cleanup deps and docs (#882)
Fix warnings from `cargo +nightly udeps` and `cargo doc`.

Removes all mentions of regex from pep440_rs.
2024-01-11 10:43:40 +00:00
Bojan Serafimov d3bbb110a8 fmt 2024-01-10 20:34:58 -05:00
Bojan Serafimov 0910d153c8 Add perf metrics 2024-01-10 20:21:10 -05:00
Zanie Blue d6fa628e11
Fix failing test (#880) 2024-01-11 00:41:37 +00:00
Bojan Serafimov b4b3ed3c59 fmt 2024-01-10 19:06:03 -05:00
Bojan Serafimov 35d1e54ea3 Remove InMemoryWheel and dead code 2024-01-10 19:04:42 -05:00
Bojan Serafimov 2d04469d1a add extract error 2024-01-10 16:24:56 -05:00
Zanie Blue 811332eacc
Improve handling of "full" version ranges (#868)
Reduces the number of implementation branches handling `Range:full`,
deferring it to `PackageRange`.
Improves some user-facing messages, e.g. saying `all versions of
<package>` instead of `<package>*`.
Changes the member names of the `PackageRangeKind` enum — they were not
very clear.
2024-01-10 21:03:55 +00:00
Bojan Serafimov c970160e20 clippy 2024-01-10 15:53:20 -05:00
Zanie Blue a65c55ff4a
Say "cannot be used" and "must be used" instead of "forbidden" and "mandatory" (#867)
Closes #858
2024-01-10 20:49:40 +00:00
Bojan Serafimov a01097a061 fmt 2024-01-10 15:45:37 -05:00
Bojan Serafimov 6f2ce53311 comment 2024-01-10 15:43:13 -05:00
Bojan Serafimov 024db936c8 comments 2024-01-10 15:42:06 -05:00
Zanie Blue 845ba6801d
Improve formatting of incompatible terms when there are two items (#866) 2024-01-10 20:36:54 +00:00
Bojan Serafimov bb65144f79 comment 2024-01-10 15:16:26 -05:00
Zanie Blue 93d3093a2a
Improve formatting of package ranges in error messages (#864)
Closes #810
Closes https://github.com/astral-sh/puffin/issues/812
Requires https://github.com/zanieb/pubgrub/pull/19 and
https://github.com/zanieb/pubgrub/pull/18

- Always pair package ranges with names e.g. `... of a matching a<1.0`
instead of `... of a matching <1.0`
- Split range segments onto multiple lines when not a singleton as
suggested in
[#850](https://github.com/astral-sh/puffin/pull/850#discussion_r1446419610)
- Improve formatting when ranges are split across multiple lines e.g. by
avoiding extra spaces and improving wording

Note review will require expanding the hidden files as there are
significant changes to the report formatter and snapshots.

Bear with me here as these are definitely not perfect still.

The following changes build on top of this independently for further
improvements:
- #868 
- #867 
- #866 
- #871
2024-01-10 14:16:23 -06:00
Bojan Serafimov 9276af2bf6 comment 2024-01-10 15:14:52 -05:00
Bojan Serafimov 88d0aaade2 less unwrapping 2024-01-10 15:10:47 -05:00
Bojan Serafimov 4fec040750 comments 2024-01-10 15:03:26 -05:00
konsti 4d8bfd7f61
Split source dist error type into error and kind (#872)
It's a better, less redundant error type. It will come in handy when
adding a second parse function.
2024-01-10 17:42:54 +00:00
Charlie Marsh fbb57b24dd
Add `--seed` flag to `venv` to allow seed package environments (#865)
## Summary

Installs the seed packages you get with `virtualenv`, but opt-in rather
than opt-out.

Closes https://github.com/astral-sh/puffin/issues/852.

## Test Plan

```
❯ ./scripts/benchmarks/venv.sh
+ hyperfine --runs 20 --warmup 3 --prepare 'rm -rf .venv' './target/release/puffin venv' --prepare 'rm -rf .venv' 'virtualenv --without-pip .venv' --prepare 'rm -rf .venv' 'python -m venv --without-pip .venv'
Benchmark 1: ./target/release/puffin venv
  Time (mean ± σ):       4.6 ms ±   0.2 ms    [User: 2.4 ms, System: 3.6 ms]
  Range (min … max):     4.3 ms …   4.9 ms    20 runs

  Warning: Command took less than 5 ms to complete. Note that the results might be inaccurate because hyperfine can not calibrate the shell startup time much more precise than this limit. You can try to use the `-N`/`--shell=none` option to disable the shell completely.

Benchmark 2: virtualenv --without-pip .venv
  Time (mean ± σ):      73.3 ms ±   0.3 ms    [User: 57.4 ms, System: 14.2 ms]
  Range (min … max):    72.8 ms …  74.0 ms    20 runs

Benchmark 3: python -m venv --without-pip .venv
  Time (mean ± σ):      22.5 ms ±   0.3 ms    [User: 17.0 ms, System: 4.9 ms]
  Range (min … max):    22.0 ms …  23.2 ms    20 runs

Summary
  './target/release/puffin venv' ran
    4.92 ± 0.20 times faster than 'python -m venv --without-pip .venv'
   16.00 ± 0.63 times faster than 'virtualenv --without-pip .venv'
+ hyperfine --runs 20 --warmup 3 --prepare 'rm -rf .venv' './target/release/puffin venv --seed' --prepare 'rm -rf .venv' 'virtualenv .venv' --prepare 'rm -rf .venv' 'python -m venv .venv'
Benchmark 1: ./target/release/puffin venv --seed
  Time (mean ± σ):      20.2 ms ±   0.4 ms    [User: 8.6 ms, System: 15.7 ms]
  Range (min … max):    19.7 ms …  21.2 ms    20 runs

Benchmark 2: virtualenv .venv
  Time (mean ± σ):     135.1 ms ±   2.4 ms    [User: 66.7 ms, System: 65.7 ms]
  Range (min … max):   133.2 ms … 142.8 ms    20 runs

Benchmark 3: python -m venv .venv
  Time (mean ± σ):      1.656 s ±  0.014 s    [User: 1.447 s, System: 0.186 s]
  Range (min … max):    1.641 s …  1.697 s    20 runs

Summary
  './target/release/puffin venv --seed' ran
    6.67 ± 0.17 times faster than 'virtualenv .venv'
   81.79 ± 1.70 times faster than 'python -m venv .venv'
```
2024-01-09 20:45:56 -05:00
Charlie Marsh 55f2be72e2
Default to PEP 517-based builds (#843)
## Summary

Our current setup uses the legacy `setup.py`-based builds if a
`pyproject.toml` file isn't present. This matches pip's behavior.
However, `pypa/build` uses PEP 517-based builds in such cases, and it
looks like pip plans to make that the default
(https://github.com/pypa/pip/issues/9175), with the limiting factor
being performance issues related to isolated builds.

This is now the default behavior, but the `--legacy-setup-py` flag
allows users to opt-in to using `setup.py` directly for distributions
that lack a `pyproject.toml`.
2024-01-10 01:27:06 +00:00
Charlie Marsh e26dc8e33d
Add support for `prepare_metadata_for_build_wheel` (#842)
## Summary

This PR adds support for `prepare_metadata_for_build_wheel`, which
allows us to determine source distribution metadata without building the
source distribution. This represents an optimization for the resolver,
as we can skip the expensive build phase for build backends that support
it.

For reference, `prepare_metadata_for_build_wheel` seems to be supported
by:

- `hatchling` (as of
[1.0.9](https://hatch.pypa.io/latest/history/hatchling/#hatchling-v1.9.0)).
- `flit`
- `setuptools`

In fact, it seems to work for every backend _except_ those using legacy
`setup.py`.

Closes #599.
2024-01-10 00:07:37 +00:00
konsti 858d5584cc
Use `Dist` in `VersionMap` (#851)
Refactoring split out from find links support: Find links files can be
represented as `Dist`, but not really as `File`, they don't have url nor
hashes.

`DistRequiresPython` is somewhat odd as an in between type.
2024-01-10 00:14:42 +01:00
konsti 1203f8f9e8
Gourgeist updates (#862)
* Use caching again
* Make clap feature only required for the cli/bin optional
2024-01-09 23:04:15 +00:00
Bojan Serafimov fc9b547d0b Merge branch 'main' into bojan/unzip-async 2024-01-09 16:23:26 -05:00
Zanie Blue 34d548de21
Improve error messages when there are no versions of a singleton range (#855) 2024-01-09 15:09:52 -06:00
Charlie Marsh 33982efb25
Remove a TOCTOU read in build (#860)
We should just read and handle the not-found case, rather than checking
if the file doesn't exist first.
2024-01-09 20:33:08 +00:00
Charlie Marsh 31139aa88d
Add derive feature to `gourgeist` (#854)
Needed to build `gourgeist` directly, probably dropped during a
refactor.
2024-01-09 17:46:16 +00:00
konsti ee6d809b60
Remove unused `Result` (#849)
Remove some dead code, seems to be a refactoring oversight
2024-01-09 16:35:10 +00:00
konsti 643e5e4a49
Use pdm for black editable as PEP 621 test case (#848)
This gives us a PEP 621 test package in tree and increases the diversity
for the editable tests a bit.
2024-01-09 16:33:05 +00:00
konsti 5b0b072e3c
Allow files >4GB on 32-bit platforms (#847)
Changes `File::size` from a `usize` to a `u64`.

The motivations are that with tensorflow wheels being 475 MB
(https://pypi.org/project/tensorflow/2.15.0.post1/#files), we're already
only one order of magnitude away and to avoid target dependent failures.
2024-01-09 17:31:49 +01:00
Bojan Serafimov 1adb85830f Unzip while downloading 2024-01-09 11:06:08 -05:00
Charlie Marsh ee3a6431c7
Show available pre-releases in error hints (#844)
## Summary

If pre-releases are available for a package that we otherwise couldn't
resolve, we now show a hint that includes one of the example versions.

Closes https://github.com/astral-sh/puffin/issues/811.
2024-01-09 09:58:38 -05:00
konsti b1edecdf1f
Filter out files with invalid requires python specifiers (#775)
Instead of trying to fixup _all_ the invalid version specifiers on pypi
and elsewhere, this filters out distributions with invalid
`requires-python` version specifiers that even
`LenientVersionSpecifiers` couldn't parse, as opposed to failing
entirely, which we currently do.

I would be nicer to model through an invalid distribution pubgrub type,
together with e.g. source dists with an unknown extension, so that the
version itself still shows up in the error trace.

At the same time, we reduce the log level for fixups from warning to
trace, as they are not actionable for the user.
2024-01-09 02:46:27 +00:00
Zanie Blue 64da1f0306
Always pair package names with ranges in error messages (#838)
Adjusts display of "no versions available" in error messages to be
consistent with other package/range pairings i.e. we usually display
"<package-name><range>".
2024-01-08 22:11:10 +00:00
Charlie Marsh 19c6d655b5
Avoid duplicated source distribution handling in url (#841)
## Summary

Right now, both the callback _and_ the "We have no compatible wheel"
paths have a lot of repeated code. This PR changes the callback to
_just_ remove all the wheels and handle the download, and the rest of
the method following the callback is responsible for finding and
building any wheels.
2024-01-08 16:19:54 -05:00
Charlie Marsh cc9140643e
Rename `metadata` to `built_wheel` in `source/mod.rs` (#840) 2024-01-08 19:20:20 +00:00
Charlie Marsh df254087d9
Break `source_dist.rs` into a module (#839)
## Summary

Finding this file hard to edit and work in since it's gotten quite
large.
2024-01-08 19:14:45 +00:00
Zanie Blue 2b0c2e294b
Fix formatting of negated singleton versions in error messages (#836)
Closes #805 
Requires https://github.com/zanieb/pubgrub/pull/17
2024-01-08 12:33:01 -06:00
Charlie Marsh aeefe65227
Fix `tracing-duration-export` compilation (#835)
## Summary

I'm unable to run `puffin-cli` on `main` as the
`tracing-durations-export` is marked as optional, but the crate actually
depends on it to compile. Further, without `tracing-durations-export`,
there are `Option` types that can't resolve to a concrete type.

This PR fixes compilation with and without the feature.
2024-01-08 18:04:23 +00:00
Charlie Marsh c06bf658bb
Remove some filesystem calls from the installer (#834)
Noticed these when working on something unrelated. Generally:

- Prefer `entry.file_type()` over `entry.path().is_file()` or similar,
as the former is almost always free on Unix.
- Call `entry.path()` once, since it allocates internally (returns a
`PathBuf`).
2024-01-08 12:59:01 -05:00
konsti 004147d441
Add tracing_durations_export feature to puffin-cli (#830)
The optional `tracing-durations-export` feature allows creating
parallelism plots from all puffin-cli commands without affecting
production builds.

Usage:

```
virtualenv --clear -p 3.10 .venv310 && TRACING_DURATIONS_FILE=target/traces/jupyter-no-cache.ndjson RUST_LOG=puffin=info VIRTUAL_ENV=.venv310 cargo run --bin puffin --profile profiling --features tracing-durations-export -- pip-install -v --no-cache jupyter
virtualenv --clear -p 3.10 .venv310 && TRACING_DURATIONS_FILE=target/traces/jupyter.ndjson RUST_LOG=puffin=info VIRTUAL_ENV=.venv310 cargo run --bin puffin --profile profiling --features tracing-durations-export -- pip-install -v jupyter
 ```

Output, plotted in collapsed mode for readability:

Cached jupyter:

![jupyter](https://github.com/astral-sh/puffin/assets/6826232/f7e03c68-0438-4cf4-bceb-9a4a146cc506)

Uncached jupyter:

![image](https://github.com/astral-sh/puffin/assets/6826232/cfdd3383-7a9d-43d6-b8d0-201f64611596)
2024-01-08 16:20:45 +01:00
konsti b6338b5e4a
Use tracing-durations-export to visualize parallelism bottlenecks (dev commands) (#816)
Example usage:

```
# Cached
TRACING_DURATIONS_FILE=target/traces/black.ndjson RUST_LOG=puffin=info cargo run --bin puffin-dev --profile profiling -- resolve black
TRACING_DURATIONS_FILE=target/traces/meine_stadt_transparent.ndjson RUST_LOG=puffin=info cargo run --bin puffin-dev --profile profiling -- resolve meine_stadt_transparent
TRACING_DURATIONS_FILE=target/traces/jupyter.ndjson RUST_LOG=puffin=info cargo run --bin puffin-dev --profile profiling -- resolve jupyter

# No cache
TRACING_DURATIONS_FILE=target/traces/black-no-cache.ndjson RUST_LOG=puffin=info cargo run --bin puffin-dev --profile profiling -- resolve --no-cache black
TRACING_DURATIONS_FILE=target/traces/meine_stadt_transparent-no-cache.ndjson RUST_LOG=puffin=info cargo run --bin puffin-dev --profile profiling -- resolve --no-cache meine_stadt_transparent
TRACING_DURATIONS_FILE=target/traces/jupyter-no-cache.ndjson RUST_LOG=puffin=info cargo run --bin puffin-dev --profile profiling -- resolve --no-cache jupyter
```

Uncached black output example:


![black-no-cache](https://github.com/astral-sh/puffin/assets/6826232/38497b89-7214-453b-9456-c9d9cbf7d2d5)
2024-01-08 16:20:38 +01:00