diff --git a/README.md b/README.md index 2c5a6eaa3..835728afd 100644 --- a/README.md +++ b/README.md @@ -1,85 +1,241 @@ -# `puffin` +# Puffin -An experimental Python packaging tool. +An extremely fast Python package installer and resolver, written in Rust. Designed as a drop-in replacement for `pip` and `pip-compile`. -## Motivation +Puffin is backed by [Astral](https://astral.sh), the creators of [Ruff](https://github.com/astral-sh/ruff). -Puffin is an extremely fast (experimental) Python package resolver and installer, intended to -replace `pip` and `pip-tools` (`pip-compile` and `pip-sync`). +## Highlights -Puffin itself is not a complete "package manager", but rather a tool for locking dependencies -(similar to `pip-compile`) and installing them (similar to `pip-sync`). Puffin can be used to -generate a set of locked dependencies from a `requirements.txt` file, and then install those -locked dependencies into a virtual environment. +- โšก๏ธ 10-100x faster than `pip` and `pip-tools` (`pip-compile` and `pip-sync`). +- ๐Ÿ’พ Disk-space efficient, with a global cache for dependency duplication and Copy-on-Write + installation on supported platforms. +- โš–๏ธ Drop-in replacement for common `pip`, `pip-tools`, and `virtualenv` commands. +- ๐Ÿค Support for a wide range of familiar `pip` features, including: editable installs, Git + dependencies, direct URL dependencies, local dependencies, constraints, source distributions, HTML + and JSON indexes, and more. +- ๐Ÿ Installable via `pip`, `pipx`, `brew` etc. Puffin is a single static binary that can be + installed without Rust or even a Python environment. +- ๐Ÿงช Tested at-scale against the top 10,000 PyPI packages. -Puffin represents an intermediary goal in our pursuit of building a "Cargo for Python": a Python -package manager that is extremely fast, reliable, and easy to use -- capable of replacing not only -`pip`, but also `pipx`, `pip-tools`, `virtualenv`, `tox`, `setuptools`, and even `pyenv`, by way of -managing the Python installation itself. +## Getting Started -Puffin's limited scope allows us to solve many of the low-level problems that are required to -build such a package manager (like package installation) while shipping an immediately useful tool -with a minimal barrier to adoption. Try it today in lieu of `pip` and `pip-tools`. +Puffin is available as [`puffin`](https://pypi.org/project/puffin/) on PyPI: -## Features +```shell +pipx install puffin +``` -- Extremely fast dependency resolution and installation: install dependencies in sub-second time. -- Disk-space efficient: Puffin uses a global cache to deduplicate dependencies, and uses - Copy-on-Write on supported filesystems to reduce disk usage. +To create a virtual environment with Puffin: + +```shell +puffin venv # Create a virtual environment at .venv. +``` + +To install a package into the virtual environment: + +```shell +puffin pip-install flask # Install Flask. +puffin pip-install -r requirements.txt # Install from a requirements.txt file. +puffin pip-install -e . # Install the current project in editable mode. +``` + +To generate a set of locked dependencies from an input file: + +```shell +puffin pip-compile pyproject.toml -o requirements.txt # Read a pyproject.toml file. +puffin pip-compile requirements.in -o requirements.txt # Read a requirements.in file. +``` + +To install a set of locked dependencies into the virtual environment: + +```shell +puffin pip-sync requirements.txt # Install from a requirements.txt file. +``` + +Puffin's `pip-install` and `pip-compile` commands supports many of the same command-line arguments +as existing tools, including `-r requirements.txt`, `-c constraints.txt`, `-e .` (for editable +installs), `--index-url`, and more. + +## Background + +Puffin is an extremely fast Python package resolver and installer, designed as a drop-in +replacement for `pip` and `pip-tools` (`pip-compile` and `pip-sync`). + +Puffin is not a complete package manager. Instead, it represents an intermediary goal in our +pursuit of a "Cargo for Python": a Python package and project manager that is extremely fast, +reliable, and easy to use โ€” a single tool capable of unifying not only `pip` and `pip-tools`, but +also `pipx`, `virtualenv`, `tox`, `setuptools`, `poetry`, `pyenv`, `rye`, and more. + +In the future, Puffin will be used as the foundation for such a tool: a single binary that +bootstraps your Python installation and gives you everything you need to be productive with Python. + +In the meantime, though, Puffin's narrower scope allows us to solve many of the low-level problems +that are required to build such a package manager (like package installation) while shipping an +immediately useful tool with a minimal barrier to adoption. Try it today in lieu of `pip` and +`pip-compile`. ## Limitations -Puffin does not yet support: +Puffin does not yet support Windows ([#73](https://github.com/astral-sh/puffin/issues/73)). -- Windows -- Editable installs (`pip install -e ...`) -- Package-less requirements (`pip install https://...`) +Puffin does not support the entire `pip` feature set. Namely, Puffin won't support the following +`pip` features: + +- `.egg` dependencies +- Editable installs for Git and direct URL dependencies +- ... + +On the other hand, Puffin plans to (but does not currently) support: + +- Hash checking - `--find-links` - ... Like `pip-compile`, Puffin generates a platform-specific `requirements.txt` file (unlike, e.g., -`poetry`, which generates a platform-agnostic `poetry.lock` file). As such, Puffin's -`requirements.txt` files are not portable across platforms and Python versions. +`poetry` and `pdm`, which generate platform-agnostic `poetry.lock` and `pdm.lock` files). As such, +Puffin's `requirements.txt` files may not be portable across platforms and Python versions. -## Usage +## Advanced Usage -To resolve a `requirements.in` file: +### Resolution strategy -```shell -cargo run -p puffin-cli -- pip-compile requirements.in -``` +By default, Puffin follows the standard Python dependency resolution strategy of preferring the +latest compatible version of each package. For example, `puffin pip-install flask` will install the +latest version of Flask (at time of writing: `3.0.0`). -To install from a resolved `requirements.txt` file: +However, Puffin's resolution strategy is parameterized, and can be configured to prefer the +lowest compatible version of each package (`--resolution=lowest`), or even the lowest compatible +version of any _direct_ dependencies (`--resolution=lowest-direct`), both of which can be useful +for library authors looking to test their packages against the oldest supported versions of their +dependencies. -```shell -cargo run -p puffin-cli -- pip-sync requirements.txt -``` - -For more, see `cargo run -p puffin-cli -- --help`: +For example, given the following `requirements.in` file: ```text -Usage: puffin [OPTIONS] - -Commands: - pip-compile Compile a `requirements.in` file to a `requirements.txt` file - pip-sync Sync dependencies from a `requirements.txt` file - pip-uninstall Uninstall packages from the current environment - clean Clear the cache - freeze Enumerate the installed packages in the current environment - venv Create a virtual environment - add Add a dependency to the workspace - remove Remove a dependency from the workspace - help Print this message or the help of the given subcommand(s) - -Options: - -q, --quiet Do not print any output - -v, --verbose Use verbose output - -n, --no-cache Avoid reading from or writing to the cache - --cache-dir Path to the cache directory [env: PUFFIN_CACHE_DIR=] - -h, --help Print help - -V, --version Print version +flask>=2.0.0 ``` +Then `puffin pip-compile requirements.in` would produce the following `requirements.txt` file: + +```text +# This file was autogenerated by Puffin v0.0.1 via the following command: +# puffin pip-compile requirements.in +blinker==1.7.0 + # via flask +click==8.1.7 + # via flask +flask==3.0.0 +itsdangerous==2.1.2 + # via flask +jinja2==3.1.2 + # via flask +markupsafe==2.1.3 + # via + # jinja2 + # werkzeug +werkzeug==3.0.1 + # via flask +``` + +However, `puffin pip-compile --resolution=lowest requirements.in` would produce: + +```text +# This file was autogenerated by Puffin v0.0.1 via the following command: +# puffin pip-compile requirements.in --resolution=lowest +click==7.1.2 + # via flask +flask==2.0.0 +itsdangerous==2.0.0 + # via flask +jinja2==3.0.0 + # via flask +markupsafe==2.0.0 + # via jinja2 +werkzeug==2.0.0 + # via flask +``` + +### Dependency caching + +Puffin uses aggressive caching to avoid re-downloading (and re-building dependencies) that have +already been accessed. + +For dependencies that are downloaded via a registry (like PyPI) or a direct URL, Puffin respects +HTTP caching headers. + +For Git dependencies, Puffin caches based on the fully-resolved Git commit hash (as such: +`puffin pip-compile` will pin Git dependencies to a specific commit hash when outputting the +resolved dependency set). + +For local path dependencies, Puffin caches based on the last-modified time of the `setup.py` or +`pyproject.toml` file. + +To force Puffin to ignore cached data for all dependencies, run `puffin pip-install --reinstall ...`. +To force Puffin to ignore cached data for a specific dependency, run, e.g., `puffin pip-install --reinstall-package flask ...`. +To clear the global cache entirely, run `puffin clean`. + +### Pre-release handling + +By default, Puffin will accept pre-release versions during dependency resolution in two cases: + +1. If the package is a direct dependency, and its version markers include a pre-release specifier + (e.g., `flask>=2.0.0rc1`). +1. If _all_ published versions of a package are pre-releases. + +If dependency resolution fails due to a transitive pre-release, Puffin will prompt the user to +re-run with `--prerelease=allow`, to allow pre-releases for all dependencies. Alternatively, add +the transitive dependency to your `requirements.in` file with a pre-release specifier (e.g., +`flask>=2.0.0rc1`) to allow pre-releases for that specific dependency. + +Pre-releases are notoriously difficult to model, and are a frequent source of bugs in other +packaging tools. Puffin's pre-release handling is _intentionally_ limited and _intentionally_ +requires user intervention to opt in to pre-releases to ensure correctness, though pre-release +handling will be revisited in future releases. + +### Dependency overrides + +Historically, `pip` has supported "constraints" (`-c constraints.txt`), which allows users to +narrow the set of acceptable versions for a given package. + +Puffin supports constraints, but also takes this concept further by allowing users to _override_ the +acceptable versions of a package across the dependency tree via overrides (`-o overrides.txt`). + +In short, overrides allow the user to lie to the resolver by overriding the declared dependencies +of a package. Overrides are a useful last resort for cases in which the user knows that a +dependency is compatible with a newer version of a package than the package declares, but the +package has not yet been updated to declare that compatibility. + +For example, if a transitive dependency declares `pydantic>=1.0,<2.0`, but the user knows that +the package is compatible with `pydantic>=2.0`, the user can override the declared dependency +with `pydantic>=2.0,<3` to allow the resolver to continue. + +While constraints are purely _additive_, and thus cannot _expand_ the set of acceptable versions for +a package, overrides _can_ expand the set of acceptable versions for a package, providing an escape +hatch for erroneous upper version bounds. + +### Multi-version resolution + +Puffin's `pip-compile` command produces a resolution that's known to be compatible with the +current platform and Python version. Unlike Poetry, PDM, and other package managers, Puffin does +not yet produce a machine-agnostic lockfile. + +However, Puffin _does_ support resolving for alternate Python versions via the `--python-version` +flag. For example, if you're running Puffin on Python 3.9, but want to resolve for Python 3.8, +you can run `puffin pip-compile --python-version=3.8 requirements.in` to produce a +Python 3.8-compatible resolution. + +## Acknowledgements + +Puffin's dependency resolver uses [PubGrub](https://github.com/pubgrub-rs/pubgrub) under the hood. +We're grateful to the PubGrub maintainers, especially [Jacob Finkelman](https://github.com/Eh2406), +for their support. + +Puffin's Git implementation draws on details from [Cargo](https://github.com/rust-lang/cargo). + +Some of Puffin's optimizations are inspired by the great work we've seen in +[Orogene](https://github.com/orogene/orogene) and [Bun](https://github.com/oven-sh/bun). We've also +learned a lot from [Posy](https://github.com/njsmith/posy). + ## License Puffin is licensed under either of