81 Commits

Author	SHA1	Message	Date
konsti	e41ec12239	Option to resolve at a fixed timestamp with `pip-compile --exclude-newer YYYY-MM-DD` (#434) ... This works by filtering out files with a more recent upload time, so if the index you use does not provide upload times, the results might be inaccurate. pypi provides upload times for all files. This is, the field is non-nullable in the warehouse schema, but the simple API PEP does not know this field. If you have only pypi dependencies, this means deterministic, reproducible(!) resolution. We could try doing the same for git repos but it doesn't seem worth the effort, i'd recommend pinning commits since git histories are arbitrarily malleable and also if you care about reproducibility and such you such not use git dependencies but a custom index. Timestamps are given either as RFC 3339 timestamps such as `2006-12-02T02:07:43Z` or as UTC dates in the same format such as `2006-12-02`. Dates are interpreted as including this day, i.e. until midnight UTC that day. Date only is required to make this ergonomic and midnight seems like an ergonomic choice. In action for `pandas`: ```console $ target/debug/puffin pip-compile --exclude-newer 2023-11-16 target/pandas.in Resolved 6 packages in 679ms # This file was autogenerated by Puffin v0.0.1 via the following command: # target/debug/puffin pip-compile --exclude-newer 2023-11-16 target/pandas.in numpy==1.26.2 # via pandas pandas==2.1.3 python-dateutil==2.8.2 # via pandas pytz==2023.3.post1 # via pandas six==1.16.0 # via python-dateutil tzdata==2023.3 # via pandas $ target/debug/puffin pip-compile --exclude-newer 2022-11-16 target/pandas.in Resolved 5 packages in 655ms # This file was autogenerated by Puffin v0.0.1 via the following command: # target/debug/puffin pip-compile --exclude-newer 2022-11-16 target/pandas.in numpy==1.23.4 # via pandas pandas==1.5.1 python-dateutil==2.8.2 # via pandas pytz==2022.6 # via pandas six==1.16.0 # via python-dateutil $ target/debug/puffin pip-compile --exclude-newer 2021-11-16 target/pandas.in Resolved 5 packages in 594ms # This file was autogenerated by Puffin v0.0.1 via the following command: # target/debug/puffin pip-compile --exclude-newer 2021-11-16 target/pandas.in numpy==1.21.4 # via pandas pandas==1.3.4 python-dateutil==2.8.2 # via pandas pytz==2021.3 # via pandas six==1.16.0 # via python-dateutil ```	2023-11-16 19:46:17 +00:00
konsti	76a41066ac	Filter out incompatible dists (#398) ... Filter out source dists and wheels whose `requires-python` from the simple api is incompatible with the current python version. This change showed an important problem: When we use a fake python version for resolving, building source distributions breaks down because we can only build with versions we actually have. This change became surprisingly big. The tests now require python 3.7 to be installed, but changing that would mean an even bigger change. Fixes #388	2023-11-13 17:14:07 +01:00
konsti	d8408b1783	Add source to failing metadata parsing (#387) ... Before: ``` cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]" puffin-dev failed Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision] Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters ``` After: ``` cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]" puffin-dev failed Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision] Caused by: Couldn't parse metadata in fastapi-0.10.1-py3-none-any.whl (97ac91cb7cd2baab1a50b0c7a17d83/fastapi-0.10.1-py3-none-any.whl) Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters ```	2023-11-10 18:33:49 +00:00
Charlie Marsh	6a15950cb5	Rename `Distribution` to `Dist` in all structs and traits (#384) ... We tend to avoid abbreviations, but this one is just so long and absolutely ubiquitous.	2023-11-10 14:55:11 +00:00
Charlie Marsh	a148f9d0be	Refactor distribution types to adhere to a clear hierarchy (#369) ... ## Summary This PR refactors our `RemoteDistribution` type such that it now follows a clear hierarchy that matches the actual variants, and encodes the differences between source and built distributions: ```rust pub enum Distribution { Built(BuiltDistribution), Source(SourceDistribution), } pub enum BuiltDistribution { Registry(RegistryBuiltDistribution), DirectUrl(DirectUrlBuiltDistribution), } pub enum SourceDistribution { Registry(RegistrySourceDistribution), DirectUrl(DirectUrlSourceDistribution), Git(GitSourceDistribution), } /// A built distribution (wheel) that exists in a registry, like `PyPI`. pub struct RegistryBuiltDistribution { pub name: PackageName, pub version: Version, pub file: File, } /// A built distribution (wheel) that exists at an arbitrary URL. pub struct DirectUrlBuiltDistribution { pub name: PackageName, pub url: Url, } /// A source distribution that exists in a registry, like `PyPI`. pub struct RegistrySourceDistribution { pub name: PackageName, pub version: Version, pub file: File, } /// A source distribution that exists at an arbitrary URL. pub struct DirectUrlSourceDistribution { pub name: PackageName, pub url: Url, } /// A source distribution that exists in a Git repository. pub struct GitSourceDistribution { pub name: PackageName, pub url: Url, } ``` Most of the PR just stems downstream from this change. There are no behavioral changes, so I'm largely relying on lint, tests, and the compiler for correctness.	2023-11-10 02:45:41 +00:00
konsti	d407bbbee6	Special case missing header build errors (on linux) (#354) ... One of the most common errors i observed are build failures due to missing header files. On ubuntu, this generally means that you need to install some `<...>-dev` package that the documentation tells you about, e.g. [mysqlclient](https://github.com/PyMySQL/mysqlclient#linux) needs `default-libmysqlclient-dev`, [some psycopg versions](https://www.psycopg.org/psycopg3/docs/basic/install.html#local-installation) (i remember that this was always required at some earlier point) require `libpq-dev` and pygraphviz wants `graphviz-dev`. This is quite common for many scientific packages (where conda has an advantage because they can provide those package as a dependency). The error message can be completely inscrutable if you're just a python programmer (or user) and not a c programmer (example: pygraphviz): ``` warning: no files found matching '*.png' under directory 'doc' warning: no files found matching '*.txt' under directory 'doc' warning: no files found matching '*.css' under directory 'doc' warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '.svn' found anywhere in distribution no previously-included directories found matching 'doc/build' pygraphviz/graphviz_wrap.c:3020:10: fatal error: graphviz/cgraph.h: No such file or directory 3020 | #include "graphviz/cgraph.h" | ^~~~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/gcc' failed with exit code 1 ``` The only relevant part is `Fatal error: graphviz/cgraph.h: No such file or directory`. Why is this file not there and how do i get it to be there? This is even harder to spot in pip's output, where it's 11 lines above the last line:  I've special cased missing headers and made sure that the last line tells you the important information: We're missing some header, please check the documentation of {package} {version} for what to install:  Scrolling up:  The difference gets even clearer with a default ubuntu terminal with its 80 columns:  --- Note that the situation is better for a missing compiler, there i get: ``` [...] warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyc' found anywhere in distribution warning: no previously-included files matching '.svn' found anywhere in distribution no previously-included directories found matching 'doc/build' error: command 'gcc' failed: No such file or directory --- ``` Putting the last line into google, the first two results tell me to `sudo apt-get install gcc`, the third even tells me about `sudo apt install build-essential`	2023-11-08 15:26:39 +00:00
konsti	2ebe40b986	Add `--no-build` (#358) ... By default, we will build source distributions for both resolving and installing, running arbitrary code. `--no-build` adds an option to ban this and only install from wheels, no source distributions or git builds allowed. We also don't fetch these and instead report immediately. I've heard from users for whom this is a requirement, i'm implementing it now because it's helpful for testing. I'm thinking about adding a shared `PuffinSharedArgs` struct so we don't have to repeat each option everywhere.	2023-11-08 10:05:15 -05:00
Charlie Marsh	4fe583257e	Use a custom PubGrub error type to always show resolution report (#365) ... Closes https://github.com/astral-sh/puffin/issues/356. The example from the issue now renders as: ``` ❯ cargo run --bin puffin-dev -q -- resolve-cli tensorflow-cpu-aws puffin-dev failed Caused by: No solution found when resolving build dependencies for source distribution: Caused by: Because there is no available version for tensorflow-cpu-aws and root depends on tensorflow-cpu-aws, version solving failed. ```	2023-11-08 09:57:26 -05:00
konsti	fbe28d3b7c	Fix mastodon-py dist-info handling (#336) ... mastodon-py 1.5.1 uses a dot in its dist-info dir name, which we previously didn't handle, causing home-assistant to fail. The new implementation is based on 2f83540272/src/packaging/utils.py (L146-L172). Part of #199 ``` unzip -l Mastodon.py-1.5.1-py2.py3-none-any.whl Archive: Mastodon.py-1.5.1-py2.py3-none-any.whl Length Date Time Name --------- ---------- ----- ---- 153929 2020-02-29 17:39 mastodon/Mastodon.py 1029 2019-10-11 19:15 mastodon/__init__.py 7357 2019-10-11 20:24 mastodon/streaming.py 10 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/DESCRIPTION.rst 1398 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/metadata.json 9 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/top_level.txt 110 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/WHEEL 1543 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/METADATA 753 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/RECORD --------- ------- 166138 9 files ```	2023-11-07 12:36:11 +01:00
konsti	aac8ae997f	Rename source distribution build to source build (#334) ... This is less verbose and better reflects that we're building both source distributions and source trees passed into the function.	2023-11-07 03:55:23 +00:00
Charlie Marsh	24e30e6557	Split `puffin-package` into requirements.txt parser and `pypi-types` (#341) ... There are only two things left in this crate and they don't really have anything to do with one another.	2023-11-06 18:19:49 +00:00
Charlie Marsh	1f447892f3	Rename `PartitionedRequirements` to `InstallPlan` (#340) ... @konstin named this file at some point and I like it, it feels appropriate for the struct itself too.	2023-11-06 12:44:35 -05:00
konsti	d99ca3159b	Cache the setup.py resolution (#327) ... Cache the resolution for the setup.py requirements (`pip`, `setuptools`, `wheels`) across builds.	2023-11-06 14:14:24 +00:00
Charlie Marsh	6d672b8951	Add source distribution support to `pip-compile` (#323) ... ## Summary This is a first-pass at adding source distribution support to the installer. The previous installation flow was: 1. Come up with a plan. 1. Find a distribution (specific file) for every package that we'll need to download. 1. Download those distributions. 1. Unzip them (since we assumed they were all wheels). 1. Install them into the virtual environment. Now, Step (3) downloads both wheels and source distributions, and we insert a step between Steps (3) and (4) to build any source distributions into zipped wheels. There are a bunch of TODOs, the most important (IMO) is that we basically have two implementations of downloading and building, between the stuff in `puffin_installer` and `puffin_resolver` (namely in `crates/puffin-resolver/src/distribution`). I didn't attempt to clean that up here -- it's already a problem, and it's related to the overall problem we need to solve around unified caching and resource management. Closes #243.	2023-11-06 08:22:36 -05:00
Charlie Marsh	1637f1c216	Add source distribution support to the `DistributionFinder` (#322) ... ## Summary This just enables the `DistributionFinder` (previously known as the `WheelFinder`) to select source distributions when there are no matching wheels for a given platform. As a reminder, the `DistributionFinder` is a simple resolver that doesn't look at any dependencies: it just takes a set of pinned packages, and finds a distribution to install to satisfy each requirement.	2023-11-06 00:16:04 -05:00
Charlie Marsh	edce4ccb24	Add support for subdirectories in URL dependencies (#312) ... Closes https://github.com/astral-sh/puffin/issues/307.	2023-11-03 15:28:38 -04:00
Zanie Blue	e1382cc747	Report project name instead of `root` when using `pyproject.toml` files (#295) ... Part of https://github.com/astral-sh/puffin/issues/214 Adds a `project: Option<PackageName>` to the `Manifest`, `Resolver`, and `RequirementsSpecification`. To populate an optional `name` for `PubGubPackage::Root`. I'll work on removing the version number next. Should we consider using the parent directory name when a `pyproject.toml` file is not present?	2023-11-03 10:22:10 -05:00
Charlie Marsh	a4002fe132	Make cache non-optional in most crates (#293) ... This PR makes the cache non-optional in most of Puffin, which simplifies the code, allows us to reuse the cache within a single command (even with `--no-cache`), and also allows us to use the cache for disk storage across an invocation. I left the cache as optional for the `Virtualenv` and `InterpreterInfo` abstractions, since those are generic enough that it seems nice to have a non-cached version, but it's kind of arbitrary.	2023-11-02 13:40:20 -04:00
Charlie Marsh	89dad0c9ad	Move distribution abstraction in shared crate (#258) ... This also allows us to get rid of `PinnedPackage` _and_ to remove some `Result<...>` types due to needless conversions between otherwise-identical types.	2023-10-31 15:30:06 -04:00
Charlie Marsh	2f38701008	Remove unused wheel cache argument from downloader (#248)	2023-10-31 02:23:50 +00:00
Charlie Marsh	ae203f998a	Rename `Unzipper#download` to `Unzipper#unzip` (#247)	2023-10-31 01:19:27 +00:00
Charlie Marsh	7e7e9f8a0c	Add support for pre-release versions (#216) ... We now accept a pre-release if (1) all versions are pre-releases, or (2) there was a pre-release marker in the dependency specifiers for a direct dependency. The code is written such that we can support a variety of pre-release strategies. Closes https://github.com/astral-sh/puffin/issues/191.	2023-10-29 14:31:55 -04:00
Charlie Marsh	8b83385763	Support constraints in `requirements.in` files (#212) ... Closes #172.	2023-10-27 00:41:02 +00:00
Charlie Marsh	58011f98b6	Revert "Add TODO around preferring local wheels" (#211) ... Reverts astral-sh/puffin#208. Unclear if we actually want to do this.	2023-10-26 23:03:00 +00:00
Charlie Marsh	d5c3ff789a	Sort wheels by size when downloading and zipping (#210) ... I just learned about this from PackagingCon, and locally, it shows a nice speedup: ``` ❯ hyperfine --warmup 3 --prepare "rm -rf .venv && ./target/release/puffin venv .venv" "./target/release/puffin pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache" "./target/release/main pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache" Benchmark 1: ./target/release/puffin pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache Time (mean ± σ): 3.958 s ± 0.250 s [User: 1.323 s, System: 5.840 s] Range (min … max): 3.652 s … 4.402 s 10 runs Benchmark 2: ./target/release/main pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache Time (mean ± σ): 4.214 s ± 0.451 s [User: 1.322 s, System: 5.976 s] Range (min … max): 3.708 s … 5.268 s 10 runs Summary './target/release/puffin pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache' ran 1.06 ± 0.13 times faster than './target/release/main pip-sync ./scripts/benchmarks/requirements-large.txt --no-cache' ```	2023-10-26 20:50:56 +00:00
Charlie Marsh	12e6b46ae8	Add TODO around preferring local wheels (#208)	2023-10-26 19:09:03 +00:00
Charlie Marsh	7bce41498e	Improve debug logging in dispatcher (#206) ... Also makes the order of operations more similar to that of the `pip-compile` command.	2023-10-26 18:54:47 +00:00
konsti	5ad58474ca	Add script to check the top 8k pypi packages (#198) ... To check to top 1k (current state): ```bash scripts/resolve/get_pypi_top_8k.sh cargo run --bin puffin-dev -- resolve-many scripts/resolve/pypi_top_8k_flat.txt --limit 1000 ``` Results: ``` Errors: pywin32, geoip2, maxminddb, pypika, dirac Success: 995, Error: 5 ``` pywin32 has no solution for the build environment, 3 have no `[build-system]` entry in pyproject.toml, `dirac` is missing cmake	2023-10-26 12:03:59 +00:00
Charlie Marsh	6faaf4bc24	Respect existing versions in "lockfile" (#187) ... Like `pip-compile`, we now respect existing versions from the `requirements.txt` provided via `--output-file`, unless you pass a `--upgrade` flag. Closes #166.	2023-10-26 04:28:58 +00:00
konsti	889f6173cc	Unify python interpreter abstractions (#178) ... Previously, we had two python interpreter metadata structs, one in gourgeist and one in puffin. Both would spawn a subprocess to query overlapping metadata and both would appear in the cli crate, if you weren't careful you could even have to different base interpreters at once. This change unifies this to one set of metadata, queried and cached once. Another effect of this crate is proper separation of python interpreter and venv. A base interpreter (such as `/usr/bin/python/`, but also pyenv and conda installed python) has a set of metadata. A venv has a root and inherits the base python metadata except for `sys.prefix`, which unlike `sys.base_prefix`, gets set to the venv root. From the root and the interpreter info we can compute the paths inside the venv. We can reuse the interpreter info of the base interpreter when creating a venv without having to query the newly created `python`.	2023-10-25 20:11:36 +00:00
konsti	1fbe328257	Build source distributions in the resolver (#138) ... This is isn't ready, but it can resolve `meine_stadt_transparent==0.2.14`. The source distributions are currently being built serially one after the other, i don't know if that is incidentally due to the resolution order, because sdist building is blocking or because of something in the resolver that could be improved. It's a bit annoying that the thing that was supposed to do http requests now suddenly also has to a whole download/unpack/resolve/install/build routine, it messes up the type hierarchy. The much bigger problem though is avoid recursive crate dependencies, it's the reason for the callback and for splitting the builder into two crates (badly named atm)	2023-10-25 20:05:13 +00:00