100 Commits

Author	SHA1	Message	Date
Charlie Marsh	6f055ecf3b	Remove existing built wheels when building source distributions (#559) ... This PR modifies the source distribution building to replace any existing targets after building the new wheel. In some cases, the existence of an existing target may be indicative of a bug, so we warn. It's partially a workaround for some (but not all) of the errors in https://github.com/astral-sh/puffin/issues/554.	2023-12-05 12:45:24 -05:00
Charlie Marsh	5fddcc362e	Improve error messages for 'file not found' case (#550) ... Right now, if you specify a wheel that doesn't exist, you get: `no such file or directory` with no additional context. Oops!	2023-12-04 22:01:51 +00:00
konsti	d5abd33813	Use atomic writes for the cache consistently (#546) ... Ensure we're using atomic writes everywhere in our cache to avoid broken cache records and error with parallel puffin actions (https://github.com/astral-sh/puffin/pull/544#issuecomment-1838841581). All json files that are written to the cache are written atomically and the build wheels are written to temp dir and then moved atomically. I didn't touch venv creation though, i don't think that's worth it since python does not support atomic package installation through its design.	2023-12-04 12:02:01 -05:00
konsti	9806901a16	Consolidate wheel caches (#524) ... After this change, two wheel caches remain: `built-wheels-v0` and `wheels-v0`, docs screenshots below. Each contains both the wheel metadata, cache policy and zip or unzipped wheels under the same name. The zipped/unzipped strategy is as follows: In `pip-compile`, when we build a wheel, we store it zipped. When `pip-sync` or a source dist build in `pip-compile` need to install the wheel, we unzip it, remove the file and replace it with the unzipped wheel. This removes `WheelCache` and `UrlIndex` in favor of `Cache` plus `WheelCache`. The non-built wheel cache now considers index urls and the url for url wheels. I'm unsure if we need the `Unzipper` type, this could just be a function. I move `no_index` into `IndexUrls` and started using `IndexUrl` up to the clap level. I left a number of TODOs in the code, namely performing the actual invalidation of unzipped wheels and making the `InstallPlan` understand cache invalidation (i.e. uninstall wheels when their remote changed). 	2023-12-01 20:16:33 +00:00
konsti	d89fbeb642	Migrate interpreter query to custom caching (#508) ... This removes the last usage of cacache by replacing it with a custom, flat json caching keyed by the digest of the executable path.  A step towards #478. I've made `CachedByTimestamp<T>` generic over `T` but intentionally not moved it to `puffin-cache` yet.	2023-11-28 17:14:59 +00:00
konsti	5435d44756	Introduce `Cache`, `CacheBucket` and `CacheEntry` (#507) ... This is mostly a mechanical refactor that moves 80% of our code to the same cache abstraction. It introduces cache `Cache`, which abstracts away the path of the cache and the temp dir drop and is passed throughout the codebase. To get a specific cache bucket, you need to requests your `CacheBucket` from `Cache`. `CacheBucket` is the centralizes the names of all cache buckets, moving them away from the string constants spread throughout the crates. Specifically for working with the `CachedClient`, there is a `CacheEntry`. I'm not sure yet if that is a strict improvement over `cache_dir: PathBuf, cache_file: String`, i may have to rotate that later. The interpreter cache moved into `interpreter-v0`. We can use the `CacheBucket` page to document the cache structure in each bucket: 	2023-11-28 17:11:14 +00:00
konsti	8855f44b5f	Move simple index queries to `CachedClient` (#504) ... Replaces the usage of `http-cache-reqwest` for simple index queries with our custom cached client, removing `http-cache-reqwest` altogether. The new cache paths are `<cache>/simple-v0/<index>/<package_name>.json`. I could not test with a non-pypi index since i'm not aware of any other json indices (jax and torch are both html indices). In a future step, we can transform the response to be a `HashMap<Version, {source_dists: Vec<(SourceDistFilename, File)>, wheels: Vec<(WheeFilename, File)>}` (independent of python version, this cache is used by all environments together). This should speed up cache deserialization a bit, since we don't need to try source dist and wheel anymore and drop incompatible dists, and it should make building the `VersionMap` simpler. We can speed this up even further by splitting into a version lists and the info for each version. I'm mentioning this because deserialization was a major bottleneck in the rust part of the old python prototype. Fixes #481	2023-11-28 00:11:03 +00:00
Charlie Marsh	afda835544	Avoid clone for `WheelMetadataCache` (#500) ... This doesn't need to own the underlying data which allows us to remove a number of clones.	2023-11-25 23:33:59 +00:00
konsti	d54e780843	Source dist metadata refactor (#468) ... ## Summary and motivation For a given source dist, we store the metadata of each wheel built through it in `built-wheel-metadata-v0/pypi/<source dist filename>/metadata.json`. During resolution, we check the cache status of the source dist. If it is fresh, we check `metadata.json` for a matching wheel. If there is one we use that metadata, if there isn't, we build one. If the source is stale, we build a wheel and override `metadata.json` with that single wheel. This PR thereby ties the local built wheel metadata cache to the freshness of the remote source dist. This functionality is available through `SourceDistCachedBuilder`. `puffin_installer::Builder`, `puffin_installer::Downloader` and `Fetcher` are removed, instead there are now `FetchAndBuild` which calls into the also new `SourceDistCachedBuilder`. `FetchAndBuild` is the new main high-level abstraction: It spawns parallel fetching/building, for wheel metadata it calls into the registry client, for wheel files it fetches them, for source dists it calls `SourceDistCachedBuilder`. It handles locks around builds, and newly added also inter-process file locking for git operations. Fetching and building source distributions now happens in parallel in `pip-sync`, i.e. we don't have to wait for the largest wheel to be downloaded to start building source distributions. In a follow-up PR, I'll also clear built wheels when they've become stale. Another effect is that in a fully cached resolution, we need neither zip reading nor email parsing. Closes #473 ## Source dist cache structure Entries by supported sources: * `<build wheel metadata cache>/pypi/foo-1.0.0.zip/metadata.json` * `<build wheel metadata cache>/<sha256(index-url)>/foo-1.0.0.zip/metadata.json` * `<build wheel metadata cache>/url/<sha256(url)>/foo-1.0.0.zip/metadata.json` But the url filename does not need to be a valid source dist filename (<https://github.com/search?q=path%3A**%2Frequirements.txt+master.zip&type=code>), so it could also be the following and we have to take any string as filename: * `<build wheel metadata cache>/url/<sha256(url)>/master.zip/metadata.json` Example: ```text # git source dist pydantic-extra-types @ git+https://github.com/pydantic/pydantic-extra-types.git # pypi source dist django_allauth==0.51.0 # url source dist werkzeug @ ff1904eb5e2853bf83db817a7dd53d/werkzeug-3.0.1.tar.gz ``` will be stored as ```text built-wheel-metadata-v0 ├── git │ └── 5c56bc1c58c34c11 │ └── 843b753e9e8cb74e83cac55598719b39a4d5ef1f │ └── metadata.json ├── pypi │ └── django-allauth-0.51.0.tar.gz │ └── metadata.json └── url └── 6781bd6440ae72c2 └── werkzeug-3.0.1.tar.gz └── metadata.json ``` The inside of a `metadata.json`: ```json { "data": { "django_allauth-0.51.0-py3-none-any.whl": { "metadata-version": "2.1", "name": "django-allauth", "version": "0.51.0", ... } } } ```	2023-11-24 17:47:58 +00:00
Charlie Marsh	443a0a9df2	Use a sparse Metadata 2.1 representation (#488) ... This is an optimization to avoid parsing the entire Metadata 2.1 when we only need a small subset of the fields. Closes #175.	2023-11-22 13:25:35 +00:00
konsti	934e32ea98	Remove outdated todos (#476)	2023-11-21 13:57:40 +00:00
Charlie Marsh	f1aa70d9d3	Refactor distribution types to return `Result` (#470) ... ## Summary A variety of small refactors to the distribution types crate to (1) return `Result` if we find an invalid wheel, rather than treating it as a source distribution with a `.whl` suffix, and (2) DRY up some repeated code around URLs.	2023-11-20 23:08:54 +00:00
konsti	f0841cdb6e	Wheel metadata refactor (#462) ... A consistent cache structure for remote wheel metadata: * `<wheel metadata cache>/pypi/foo-1.0.0-py3-none-any.json` * `<wheel metadata cache>/<digest(index-url)>/foo-1.0.0-py3-none-any.json` * `<wheel metadata cache>/url/<digest(url)>/foo-1.0.0-py3-none-any.json` The source dist caching will use a similar structure (#468).	2023-11-20 17:26:36 +01:00
Charlie Marsh	8decb29bad	Use a dedicated error type for `puffin-distribution` (#466)	2023-11-20 11:38:27 +00:00
konsti	46bb18f06e	Track file index (#452) ... Track the index (or at least its url) where we got a file from across the source code. Fixes #448	2023-11-20 08:48:16 +00:00
konsti	751f7fa9c6	Improve PEP 691 compatibility (#428) ... [PEP 691](https://peps.python.org/pep-0691/#project-detail) has slightly different, more relaxed rules around file metadata. These changes are now reflected in the `File` struct. This will make it easier to support alternative indices. I had expected that i need to introduce a separate type for that, so i'm happy it's two `Option`s more and an alias. Part of #412	2023-11-16 19:03:44 +01:00
konsti	81c9cd0d4a	Print url for bad json error (#409) ... Split out from #382	2023-11-13 11:41:20 +00:00
konsti	d8408b1783	Add source to failing metadata parsing (#387) ... Before: ``` cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]" puffin-dev failed Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision] Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters ``` After: ``` cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]" puffin-dev failed Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision] Caused by: Couldn't parse metadata in fastapi-0.10.1-py3-none-any.whl (97ac91cb7cd2baab1a50b0c7a17d83/fastapi-0.10.1-py3-none-any.whl) Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters ```	2023-11-10 18:33:49 +00:00
konsti	5cef40d87a	Add proper caching for pypi metadata fetching kinds (#368) ... I intend this to become the main form of caching for puffin: You can make http requests, you tranform the data to what you really need, you have control over the cache key, and the cache is always json (or anything else much faster we want to replace it with as long as it's serde!)	2023-11-10 11:03:40 +00:00
Charlie Marsh	a148f9d0be	Refactor distribution types to adhere to a clear hierarchy (#369) ... ## Summary This PR refactors our `RemoteDistribution` type such that it now follows a clear hierarchy that matches the actual variants, and encodes the differences between source and built distributions: ```rust pub enum Distribution { Built(BuiltDistribution), Source(SourceDistribution), } pub enum BuiltDistribution { Registry(RegistryBuiltDistribution), DirectUrl(DirectUrlBuiltDistribution), } pub enum SourceDistribution { Registry(RegistrySourceDistribution), DirectUrl(DirectUrlSourceDistribution), Git(GitSourceDistribution), } /// A built distribution (wheel) that exists in a registry, like `PyPI`. pub struct RegistryBuiltDistribution { pub name: PackageName, pub version: Version, pub file: File, } /// A built distribution (wheel) that exists at an arbitrary URL. pub struct DirectUrlBuiltDistribution { pub name: PackageName, pub url: Url, } /// A source distribution that exists in a registry, like `PyPI`. pub struct RegistrySourceDistribution { pub name: PackageName, pub version: Version, pub file: File, } /// A source distribution that exists at an arbitrary URL. pub struct DirectUrlSourceDistribution { pub name: PackageName, pub url: Url, } /// A source distribution that exists in a Git repository. pub struct GitSourceDistribution { pub name: PackageName, pub url: Url, } ``` Most of the PR just stems downstream from this change. There are no behavioral changes, so I'm largely relying on lint, tests, and the compiler for correctness.	2023-11-10 02:45:41 +00:00
konsti	fbe28d3b7c	Fix mastodon-py dist-info handling (#336) ... mastodon-py 1.5.1 uses a dot in its dist-info dir name, which we previously didn't handle, causing home-assistant to fail. The new implementation is based on 2f83540272/src/packaging/utils.py (L146-L172). Part of #199 ``` unzip -l Mastodon.py-1.5.1-py2.py3-none-any.whl Archive: Mastodon.py-1.5.1-py2.py3-none-any.whl Length Date Time Name --------- ---------- ----- ---- 153929 2020-02-29 17:39 mastodon/Mastodon.py 1029 2019-10-11 19:15 mastodon/__init__.py 7357 2019-10-11 20:24 mastodon/streaming.py 10 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/DESCRIPTION.rst 1398 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/metadata.json 9 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/top_level.txt 110 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/WHEEL 1543 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/METADATA 753 2020-03-14 18:14 Mastodon.py-1.5.1.dist-info/RECORD --------- ------- 166138 9 files ```	2023-11-07 12:36:11 +01:00
Charlie Marsh	24e30e6557	Split `puffin-package` into requirements.txt parser and `pypi-types` (#341) ... There are only two things left in this crate and they don't really have anything to do with one another.	2023-11-06 18:19:49 +00:00
konsti	b2439b24a1	Fetch wheel metadata by async range requests on the remote wheel (#301) ... Use range requests and async zip to extract the METADATA file from a remote wheel. We currently only cache when the remote says the remote declares the resource as immutable, see https://github.com/06chaynes/http-cache/issues/57 and https://github.com/baszalmstra/async_http_range_reader/pull/1 . The cache is stored as json with the description omitted, this improve cache deserialization performance.	2023-11-06 15:06:49 +01:00
konsti	4adaa9a700	Wheel filename distribution package name (#278) ... The normalized name abstractions were not consistently, this PR uses them where they were previously missing: * `WheelFilename::distribution` * `Requirement::name` * `Requirement::extras` * `Metadata21::name` * `Metadata21::provides_dist` With `puffin-package` depending on `pep508_rs` this would be cyclical crate dependency, so `puffin-normalize` gets split out from `puffin-package`. `DistInfoName` has the same task and semantics as `PackageName`, so it's merged into the latter. `PackageName` and `ExtraName` documentation is moved onto the type and their constructors are called `new` instead of `normalize`. We now use these constructors rarely enough the implicit allocation by `to_string()` shouldn't matter anymore, while more actual cloning becomes visible.	2023-11-02 11:15:27 +00:00
konsti	d1af90163b	Improve client reqwest errors (#276) ... I had to debug a failure involving these errors and had to improve their output.	2023-11-01 16:52:58 +01:00
Charlie Marsh	16aac834ee	Move PyPI-oriented types out of `puffin-client` crate (#255) ... Just an internal change to avoid a dependency on `puffin-client` for those crates that need access to PyPI-metadata types.	2023-10-31 17:10:23 +00:00
konsti	1fbe328257	Build source distributions in the resolver (#138) ... This is isn't ready, but it can resolve `meine_stadt_transparent==0.2.14`. The source distributions are currently being built serially one after the other, i don't know if that is incidentally due to the resolution order, because sdist building is blocking or because of something in the resolver that could be improved. It's a bit annoying that the thing that was supposed to do http requests now suddenly also has to a whole download/unpack/resolve/install/build routine, it messes up the type hierarchy. The much bigger problem though is avoid recursive crate dependencies, it's the reason for the callback and for splitting the builder into two crates (badly named atm)	2023-10-25 20:05:13 +00:00
Charlie Marsh	0e097874f8	Add support for alternate index URLs (#169) ... As elsewhere, we just use the `pip` and `pip-compile` APIs. So we support `--index-url` to override PyPI, then `--extra-index-url` to add _additional_ indexes, and `--no-index` to avoid hitting the index at all. Closes #156.	2023-10-23 03:18:30 +00:00
konsti	8cc4fe0d44	Install source distribution requirements with puffin itself instead of pip (#122) ... This is also a lot faster. Unfortunately it copies a lot of code from the sync cli since the `Printer` is private. The first commit are some refactorings i made when i thought about how i could reuse the existing code.	2023-10-18 19:11:17 +00:00
Charlie Marsh	471a1d657d	Migrate resolver proof-of-concept to PubGrub (#97) ... ## Summary This PR enables the proof-of-concept resolver to backtrack by way of using the `pubgrub-rs` crate. Rather than using PubGrub as a _framework_ (implementing the `DependencyProvider` trait, letting PubGrub call us), I've instead copied over PubGrub's primary solver hook (which is only ~100 lines or so) and modified it for our purposes (e.g., made it async). There's a lot to improve here, but it's a start that will let us understand PubGrub's appropriateness for this problem space. A few observations: - In simple cases, the resolver is slower than our current (naive) resolver. I think it's just that the pipelining isn't as efficient as in the naive case, where we can just stream package and version fetches concurrently without any bottlenecks. - A lot of the code here relates to bridging PubGrub with our own abstractions -- so we need a `PubGrubPackage`, a `PubGrubVersion`, etc.	2023-10-15 22:05:44 -04:00
Charlie Marsh	ed68d31e03	Add a basic test for the resolver (#86) ... Mocks out the PyPI client using some checked-in fixtures. The test is very basic, and I'm not very happy with all the ceremony around the mocks and such, but it's an interesting experiment at least.	2023-10-11 03:30:53 +00:00
Charlie Marsh	ba2b200fce	Enable release builds via `cargo-dist` (#79)	2023-10-09 20:48:55 +00:00
Charlie Marsh	d1ed41170b	Cache environment marker lookups (#55) ... Closes https://github.com/astral-sh/puffin/issues/53.	2023-10-08 05:31:19 +00:00
Charlie Marsh	92160e37df	Surface error when unable to find package (#45)	2023-10-07 19:43:12 +00:00
Charlie Marsh	162952bf64	Add a content-addressed cache for wheels (#38) ... Closes https://github.com/astral-sh/puffin/issues/4.	2023-10-07 14:24:52 +00:00
Charlie Marsh	6c31631913	Fetch from `data-dist-info-metadata` when available (#37) ... As specified in https://peps.python.org/pep-0658/#specification.	2023-10-07 13:05:29 +00:00
Charlie Marsh	ca6aa207ff	Move to workspace dependencies (#25)	2023-10-06 19:49:41 +00:00
Charlie Marsh	28721cf5fc	Avoid caching wheel fetches	2023-10-06 00:50:30 -04:00
Charlie Marsh	a43328d914	Support wheel installation (#19) ... Closes https://github.com/astral-sh/puffin/issues/8.	2023-10-06 00:47:45 -04:00
Charlie Marsh	8b151a64d5	Rename `puffin-requirements` to `puffin-package` (#16) ... Closes https://github.com/astral-sh/puffin/issues/7.	2023-10-05 21:03:20 -04:00
Charlie Marsh	2d6266b167	Add an HTTP cache (and `--no-cache` argument) (#14) ... Closes https://github.com/astral-sh/puffin/issues/3.	2023-10-05 19:14:05 -04:00
Charlie Marsh	1063d8c150	Add Python interpreter detection (#11) ... Closes https://github.com/astral-sh/puffin/issues/2.	2023-10-05 15:09:22 -04:00
Charlie Marsh	b4828fb3f2	Remove progress bar	2023-10-05 12:45:38 -04:00
Charlie Marsh	7f497fa43f	Add progress bar	2023-10-05 12:45:38 -04:00
Charlie Marsh	8032d4606e	Misc. changes	2023-10-05 12:45:38 -04:00
Charlie Marsh	f51432382a	Do basic resolution	2023-10-05 12:45:38 -04:00
Charlie Marsh	0f10595ac3	Add version selection	2023-10-05 12:45:38 -04:00
Charlie Marsh	b08e8c78b5	Remove normalized representation of SimpleJson	2023-10-05 12:45:38 -04:00
Charlie Marsh	610fd9994f	Add client networking stack	2023-10-05 12:45:38 -04:00
Charlie Marsh	1a2f35801b	Add client networking stack	2023-10-05 12:45:38 -04:00