2652caa3e3
## Summary This PR adds support for resolving and installing dependencies via direct URLs, like: ``` werkzeug @960bb4017c/Werkzeug-2.0.0-py3-none-any.whl``` These are fairly common (e.g., with `torch`), but you most often see them as Git dependencies. Broadly, structs like `RemoteDistribution` and friends are now enums that can represent either registry-based dependencies or URL-based dependencies: ```rust /// A built distribution (wheel) that exists as a remote file (e.g., on `PyPI`). #[derive(Debug, Clone)] #[allow(clippy::large_enum_variant)] pub enum RemoteDistribution { /// The distribution exists in a registry, like `PyPI`. Registry(PackageName, Version, File), /// The distribution exists at an arbitrary URL. Url(PackageName, Url), } ``` In the resolver, we now allow packages to take on an extra, optional `Url` field: ```rust #[derive(Debug, Clone, Eq, Derivative)] #[derivative(PartialEq, Hash)] pub enum PubGrubPackage { Root, Package( PackageName, Option<DistInfoName>, #[derivative(PartialEq = "ignore")] #[derivative(PartialOrd = "ignore")] #[derivative(Hash = "ignore")] Option<Url>, ), } ``` However, for the purpose of version satisfaction, we ignore the URL. This allows for the URL dependency to satisfy the transitive request in cases like: ``` flask==3.0.0 werkzeug @254c3e9b5f/werkzeug-3.0.1-py3-none-any.whl``` There are a couple limitations in the current approach: - The caching for remote URLs is done separately in the resolver vs. the installer. I decided not to sweat this too much... We need to figure out caching holistically. - We don't support any sort of time-based cache for remote URLs -- they just exist forever. This will be a problem for URL dependencies, where we need some way to evict and refresh them. But I've deferred it for now. - I think I need to redo how this is modeled in the resolver, because right now, we don't detect a variety of invalid cases, e.g., providing two different URLs for a dependency, asking for a URL dependency and a _different version_ of the same dependency in the list of first-party dependencies, etc. - (We don't yet support VCS dependencies.)
puffin
An experimental Python packaging tool.
Motivation
Puffin is an extremely fast (experimental) Python package resolver and installer, intended to
replace pip and pip-tools (pip-compile and pip-sync).
Puffin itself is not a complete "package manager", but rather a tool for locking dependencies
(similar to pip-compile) and installing them (similar to pip-sync). Puffin can be used to
generate a set of locked dependencies from a requirements.txt file, and then install those
locked dependencies into a virtual environment.
Puffin represents an intermediary goal in our pursuit of building a "Cargo for Python": a Python
package manager that is extremely fast, reliable, and easy to use -- capable of replacing not only
pip, but also pipx, pip-tools, virtualenv, tox, setuptools, and even pyenv, by way of
managing the Python installation itself.
Puffin's limited scope allows us to solve many of the low-level problems that are required to
build such a package manager (like package installation) while shipping an immediately useful tool
with a minimal barrier to adoption. Try it today in lieu of pip and pip-tools.
Features
- Extremely fast dependency resolution and installation: install dependencies in sub-second time.
- Disk-space efficient: Puffin uses a global cache to deduplicate dependencies, and uses Copy-on-Write on supported filesystems to reduce disk usage.
Limitations
Puffin does not yet support:
- Source distributions
- VCS dependencies
- URL dependencies
- Windows
- ...
Like pip-compile, Puffin generates a platform-specific requirements.txt file (unlike, e.g.,
poetry, which generates a platform-agnostic poetry.lock file). As such, Puffin's
requirements.txt files are not portable across platforms and Python versions.
Usage
To resolve a requirements.in file:
cargo run -p puffin-cli -- pip-compile requirements.in
To install from a resolved requirements.txt file:
cargo run -p puffin-cli -- pip-sync requirements.txt
For more, see cargo run -p puffin-cli -- --help:
Usage: puffin-cli <COMMAND>
Commands:
compile Compile a `requirements.in` file to a `requirements.txt` file
sync Sync dependencies from a `requirements.txt` file
clean Clear the cache
freeze Enumerate the installed packages in the current environment
help Print this message or the help of the given subcommand(s)
Options:
-h, --help Print help
-V, --version Print version
License
Puffin is licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in Puffin by you, as defined in the Apache-2.0 license, shall be dually licensed as above, without any additional terms or conditions.
