Python
/
uv
mirror of https://github.com/astral-sh/uv
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
uv/crates
History
Charlie Marsh 1181288078
Download, build, and install in a single pipeline phase (#605) 
## Summary

At present, we have two separate phases within the installation pipeline
related to populating wheels into the cache. The first phase downloads
the distribution, and then builds any source distributions into wheels;
the second phase unzips all the built wheels into the cache.

This PR merges those two phases into one, such that we seamlessly
download, build, and unzip wheels in one pass. This is more efficient,
since we can start unzipping while we build. It also ensures that if the
install _fails_ partway through, we don't end up with a bunch of
downloaded wheels that we never had a chance to unzip. The code is also
much simpler.

The main downside is that the user-facing feedback isn't as granular,
since we only have one phase and one progress bar for what was
originally three distinct phases.

Closes https://github.com/astral-sh/puffin/issues/571.

## Test Plan

I ran the benchmark script on two separate requirements files, and saw a
7% and 31% speedup respectively:

```text
+ TARGET=./scripts/benchmarks/requirements.txt
+ hyperfine --runs 100 --warmup 10 --prepare 'virtualenv --clear .venv' './target/release/main pip-sync ./scripts/benchmarks/requirements.txt --no-cache' --prepare 'virtualenv --clear .venv' './target/release/puffin pip-sync ./scripts/benchmarks/requirements.txt --no-cache'
Benchmark 1: ./target/release/main pip-sync ./scripts/benchmarks/requirements.txt --no-cache
  Time (mean ± σ):     269.4 ms ±  33.0 ms    [User: 42.4 ms, System: 117.5 ms]
  Range (min … max):   221.7 ms … 446.7 ms    100 runs

Benchmark 2: ./target/release/puffin pip-sync ./scripts/benchmarks/requirements.txt --no-cache
  Time (mean ± σ):     250.6 ms ±  28.3 ms    [User: 41.5 ms, System: 127.4 ms]
  Range (min … max):   207.6 ms … 336.4 ms    100 runs

Summary
  './target/release/puffin pip-sync ./scripts/benchmarks/requirements.txt --no-cache' ran
    1.07 ± 0.18 times faster than './target/release/main pip-sync ./scripts/benchmarks/requirements.txt --no-cache'
```

```text
+ TARGET=./scripts/benchmarks/requirements-large.txt
+ hyperfine --runs 100 --warmup 10 --prepare 'virtualenv --clear .venv' './target/release/main pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache' --prepare 'virtualenv --clear .venv' './target/release/puffin pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache'
Benchmark 1: ./target/release/main pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache
  Time (mean ± σ):      5.053 s ±  0.354 s    [User: 1.413 s, System: 6.710 s]
  Range (min … max):    4.584 s …  6.333 s    100 runs

Benchmark 2: ./target/release/puffin pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache
  Time (mean ± σ):      3.845 s ±  0.225 s    [User: 1.364 s, System: 6.970 s]
  Range (min … max):    3.482 s …  4.715 s    100 runs

Summary
  './target/release/puffin pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache' ran
```
 		2023-12-11 15:42:29 +00:00
..
bench Use Clippy lint table over Cargo config (#490) 2023-11-22 15:10:27 +00:00
distribution-filename Impl Version debug as display (#606) 2023-12-11 16:38:14 +01:00
distribution-types Restructure Git cache to include package name (#588) 2023-12-07 20:17:41 -05:00
gourgeist Use atomic writes for the cache consistently (#546) 2023-12-04 12:02:01 -05:00
install-wheel-rs Use `rustc-hash` instead of `fxhash` crate (#594) 2023-12-08 20:27:49 +00:00
pep440-rs Impl Version debug as display (#606) 2023-12-11 16:38:14 +01:00
pep508-rs Consistent Cargo.toml syntax (#483) 2023-11-22 08:34:08 +00:00
platform-host Error when `ldd` is not in path (#506) 2023-11-28 05:55:04 +00:00
platform-tags Use `rustc-hash` instead of `fxhash` crate (#594) 2023-12-08 20:27:49 +00:00
puffin-build Use async `Command` for wheel build operations (#601) 2023-12-09 16:20:52 +00:00
puffin-cache Download, build, and install in a single pipeline phase (#605) 2023-12-11 15:42:29 +00:00
puffin-cli Download, build, and install in a single pipeline phase (#605) 2023-12-11 15:42:29 +00:00
puffin-client Fix incorrect unknown size marker in traces (#600) 2023-12-09 04:46:01 +00:00
puffin-dev Consolidate wheel caches (#524) 2023-12-01 20:16:33 +00:00
puffin-dispatch Download, build, and install in a single pipeline phase (#605) 2023-12-11 15:42:29 +00:00
puffin-distribution Download, build, and install in a single pipeline phase (#605) 2023-12-11 15:42:29 +00:00
puffin-fs Enable selective cache purging in `puffin clean` (#589) 2023-12-08 19:51:32 +00:00
puffin-git Restructure Git cache to include package name (#588) 2023-12-07 20:17:41 -05:00
puffin-installer Download, build, and install in a single pipeline phase (#605) 2023-12-11 15:42:29 +00:00
puffin-interpreter Improve path source dist caching (#578) 2023-12-06 11:47:01 -05:00
puffin-macros Use `rustc-hash` instead of `fxhash` crate (#594) 2023-12-08 20:27:49 +00:00
puffin-normalize Shard the registry cache by package (#583) 2023-12-07 05:02:46 +00:00
puffin-resolver Download, build, and install in a single pipeline phase (#605) 2023-12-11 15:42:29 +00:00
puffin-traits Use `rustc-hash` instead of `fxhash` crate (#594) 2023-12-08 20:27:49 +00:00
puffin-workspace Use Clippy lint table over Cargo config (#490) 2023-11-22 15:10:27 +00:00
pypi-types Consolidate wheel caches (#524) 2023-12-01 20:16:33 +00:00
requirements-txt Impl Version debug as display (#606) 2023-12-11 16:38:14 +01:00
README.md Remove existing built wheels when building source distributions (#559) 2023-12-05 12:45:24 -05:00

README.md

Crates

bench

Functionality for benchmarking Puffin.

distribution-filename

Parse built distribution (wheel) and source distribution (sdist) filenames to extract structured metadata.

distribution-types

Abstractions for representing built distributions (wheels) and source distributions (sdists), and the sources from which they can be downloaded.

gourgeist

A venv replacement to create virtual environments in Rust.

install-wheel-rs

Install built distributions (wheels) into a virtual environment.]

pep440-rs

Utilities for interacting with Python version numbers and specifiers.

pep508-rs

Utilities for interacting with PEP 508 dependency specifiers.

platform-host

Functionality for detecting the current platform (operating system, architecture, etc.).

platform-tags

Functionality for parsing and inferring Python platform tags as per PEP 425.

puffin-build

A PEP 517-compatible build frontend for Puffin.

puffin-cache

Functionality for caching Python packages and associated metadata.

puffin-cli

Command-line interface for the Puffin package manager.

puffin-client

Client for interacting with PyPI-compatible HTTP APIs.

puffin-dev

Development utilities for Puffin.

puffin-dispatch

A centralized struct for resolving and building source distributions in isolated environments. Implements the traits defined in puffin-traits.

puffin-distribution

Client for interacting with built distributions (wheels) and source distributions (sdists). Capable of fetching metadata, distribution contents, etc.

puffin-fs

Utilities for interacting with the filesystem.

puffin-git

Functionality for interacting with Git repositories.

puffin-installer

Functionality for installing Python packages into a virtual environment.

puffin-interpreter

Functionality for detecting and leveraging the current Python interpreter.

puffin-macros

Reusable procedural macros for Puffin.

puffin-normalize

Normalize package and extra names as per Python specifications.

puffin-package

Types and functionality for working with Python packages, e.g., parsing wheel files.

puffin-resolver

Functionality for resolving Python packages and their dependencies.

puffin-traits

Shared traits for Puffin, to avoid circular dependencies.

pypi-types

General-purpose type definitions for types used in PyPI-compatible APIs.

requirements-txt

Functionality for parsing requirements.txt files.