Python
/
uv
mirror of https://github.com/astral-sh/uv
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
725 Commits 563 Branches 239 Tags 205 MiB
Commit Graph

94 Commits

Author SHA1 Message Date
konsti 8855f44b5f
Move simple index queries to `CachedClient` (#504) 
Replaces the usage of `http-cache-reqwest` for simple index queries with
our custom cached client, removing `http-cache-reqwest` altogether.

The new cache paths are `<cache>/simple-v0/<index>/<package_name>.json`.
I could not test with a non-pypi index since i'm not aware of any other
json indices (jax and torch are both html indices).

In a future step, we can transform the response to be a
`HashMap<Version, {source_dists: Vec<(SourceDistFilename, File)>,
wheels: Vec<(WheeFilename, File)>}` (independent of python version, this
cache is used by all environments together). This should speed up cache
deserialization a bit, since we don't need to try source dist and wheel
anymore and drop incompatible dists, and it should make building the
`VersionMap` simpler. We can speed this up even further by splitting
into a version lists and the info for each version. I'm mentioning this
because deserialization was a major bottleneck in the rust part of the
old python prototype.

Fixes #481
 2023-11-28 00:11:03 +00:00
Charlie Marsh afda835544
Avoid clone for `WheelMetadataCache` (#500) 
This doesn't need to own the underlying data which allows us to remove a
number of clones.
 2023-11-25 23:33:59 +00:00
konsti d54e780843
Source dist metadata refactor (#468) 
## Summary and motivation

For a given source dist, we store the metadata of each wheel built
through it in `built-wheel-metadata-v0/pypi/<source dist
filename>/metadata.json`. During resolution, we check the cache status
of the source dist. If it is fresh, we check `metadata.json` for a
matching wheel. If there is one we use that metadata, if there isn't, we
build one. If the source is stale, we build a wheel and override
`metadata.json` with that single wheel. This PR thereby ties the local
built wheel metadata cache to the freshness of the remote source dist.
This functionality is available through `SourceDistCachedBuilder`.

`puffin_installer::Builder`, `puffin_installer::Downloader` and
`Fetcher` are removed, instead there are now `FetchAndBuild` which calls
into the also new `SourceDistCachedBuilder`. `FetchAndBuild` is the new
main high-level abstraction: It spawns parallel fetching/building, for
wheel metadata it calls into the registry client, for wheel files it
fetches them, for source dists it calls `SourceDistCachedBuilder`. It
handles locks around builds, and newly added also inter-process file
locking for git operations.

Fetching and building source distributions now happens in parallel in
`pip-sync`, i.e. we don't have to wait for the largest wheel to be
downloaded to start building source distributions.

In a follow-up PR, I'll also clear built wheels when they've become
stale.

Another effect is that in a fully cached resolution, we need neither zip
reading nor email parsing.

Closes #473

## Source dist cache structure 

Entries by supported sources:
 * `<build wheel metadata cache>/pypi/foo-1.0.0.zip/metadata.json`
* `<build wheel metadata
cache>/<sha256(index-url)>/foo-1.0.0.zip/metadata.json`
* `<build wheel metadata
cache>/url/<sha256(url)>/foo-1.0.0.zip/metadata.json`
But the url filename does not need to be a valid source dist filename

(<https://github.com/search?q=path%3A**%2Frequirements.txt+master.zip&type=code>),
so it could also be the following and we have to take any string as
filename:
* `<build wheel metadata
cache>/url/<sha256(url)>/master.zip/metadata.json`

Example:
```text
# git source dist
pydantic-extra-types @ git+https://github.com/pydantic/pydantic-extra-types.git
# pypi source dist
django_allauth==0.51.0
# url source dist
werkzeug @ ff1904eb5e2853bf83db817a7dd53d/werkzeug-3.0.1.tar.gz
```
will be stored as
```text
built-wheel-metadata-v0
├── git
│   └── 5c56bc1c58c34c11
│       └── 843b753e9e8cb74e83cac55598719b39a4d5ef1f
│           └── metadata.json
├── pypi
│   └── django-allauth-0.51.0.tar.gz
│       └── metadata.json
└── url
    └── 6781bd6440ae72c2
        └── werkzeug-3.0.1.tar.gz
            └── metadata.json
```

The inside of a `metadata.json`:
```json
{
  "data": {
    "django_allauth-0.51.0-py3-none-any.whl": {
      "metadata-version": "2.1",
      "name": "django-allauth",
      "version": "0.51.0",
      ...
    }
  }
}
```
 2023-11-24 17:47:58 +00:00
Charlie Marsh 443a0a9df2
Use a sparse Metadata 2.1 representation (#488) 
This is an optimization to avoid parsing the entire Metadata 2.1 when we
only need a small subset of the fields.

Closes #175.
 2023-11-22 13:25:35 +00:00
konsti 934e32ea98
Remove outdated todos (#476) 2023-11-21 13:57:40 +00:00
Charlie Marsh f1aa70d9d3
Refactor distribution types to return `Result` (#470) 
## Summary

A variety of small refactors to the distribution types crate to (1)
return `Result` if we find an invalid wheel, rather than treating it as
a source distribution with a `.whl` suffix, and (2) DRY up some repeated
code around URLs.
 2023-11-20 23:08:54 +00:00
konsti f0841cdb6e
Wheel metadata refactor (#462) 
A consistent cache structure for remote wheel metadata:

 * `<wheel metadata cache>/pypi/foo-1.0.0-py3-none-any.json`
* `<wheel metadata
cache>/<digest(index-url)>/foo-1.0.0-py3-none-any.json`
* `<wheel metadata cache>/url/<digest(url)>/foo-1.0.0-py3-none-any.json`

The source dist caching will use a similar structure (#468).
 2023-11-20 17:26:36 +01:00
Charlie Marsh 8decb29bad
Use a dedicated error type for `puffin-distribution` (#466) 2023-11-20 11:38:27 +00:00
konsti 46bb18f06e
Track file index (#452) 
Track the index (or at least its url) where we got a file from across
the source code.

Fixes #448
 2023-11-20 08:48:16 +00:00
konsti 751f7fa9c6
Improve PEP 691 compatibility (#428) 
[PEP 691](https://peps.python.org/pep-0691/#project-detail) has slightly
different, more relaxed rules around file metadata. These changes are
now reflected in the `File` struct. This will make it easier to support
alternative indices.

I had expected that i need to introduce a separate type for that, so i'm
happy it's two `Option`s more and an alias.

Part of #412
 2023-11-16 19:03:44 +01:00
konsti 81c9cd0d4a
Print url for bad json error (#409) 
Split out from #382
 2023-11-13 11:41:20 +00:00
konsti d8408b1783
Add source to failing metadata parsing (#387) 
Before:
```
cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]"
puffin-dev failed
  Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision]
  Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters
```
After:
```
cargo run --bin puffin-dev -q -- resolve-cli "transformers[accelerate, agents, all, audio, codecarbon, deepspeed, deepspeed-testing, dev, dev-tensorflow, dev-torch, docs, docs_specific, flax, flax-speech, ftfy, integrations, ja, modelcreation, onnx, onnxruntime, optuna, quality, ray, retrieval, sagemaker, sentencepiece, serving, sigopt, sklearn, speech, testing, tf, tf-cpu, tf-speech, timm, tokenizers, torch, torch-speech, torch-vision, torchhub, video, vision]"
puffin-dev failed
  Caused by: No solution found when resolving: transformers[accelerate,agents,all,audio,codecarbon,deepspeed,deepspeed-testing,dev,dev-tensorflow,dev-torch,docs,docs-specific,flax,flax-speech,ftfy,integrations,ja,modelcreation,onnx,onnxruntime,optuna,quality,ray,retrieval,sagemaker,sentencepiece,serving,sigopt,sklearn,speech,testing,tf,tf-cpu,tf-speech,timm,tokenizers,torch,torch-speech,torch-vision,torchhub,video,vision]
  Caused by: Couldn't parse metadata in fastapi-0.10.1-py3-none-any.whl (97ac91cb7cd2baab1a50b0c7a17d83/fastapi-0.10.1-py3-none-any.whl)
  Caused by: Not a valid package or extra name: ".none". Names must start and end with a letter or digit and may only contain -, _, ., and alphanumeric characters
```
 2023-11-10 18:33:49 +00:00
konsti 5cef40d87a
Add proper caching for pypi metadata fetching kinds (#368) 
I intend this to become the main form of caching for puffin: You can
make http requests, you tranform the data to what you really need, you
have control over the cache key, and the cache is always json (or
anything else much faster we want to replace it with as long as it's
serde!)
 2023-11-10 11:03:40 +00:00
Charlie Marsh a148f9d0be
Refactor distribution types to adhere to a clear hierarchy (#369) 
## Summary

This PR refactors our `RemoteDistribution` type such that it now follows
a clear hierarchy that matches the actual variants, and encodes the
differences between source and built distributions:

```rust
pub enum Distribution {
    Built(BuiltDistribution),
    Source(SourceDistribution),
}

pub enum BuiltDistribution {
    Registry(RegistryBuiltDistribution),
    DirectUrl(DirectUrlBuiltDistribution),
}

pub enum SourceDistribution {
    Registry(RegistrySourceDistribution),
    DirectUrl(DirectUrlSourceDistribution),
    Git(GitSourceDistribution),
}

/// A built distribution (wheel) that exists in a registry, like `PyPI`.
pub struct RegistryBuiltDistribution {
    pub name: PackageName,
    pub version: Version,
    pub file: File,
}

/// A built distribution (wheel) that exists at an arbitrary URL.
pub struct DirectUrlBuiltDistribution {
    pub name: PackageName,
    pub url: Url,
}

/// A source distribution that exists in a registry, like `PyPI`.
pub struct RegistrySourceDistribution {
    pub name: PackageName,
    pub version: Version,
    pub file: File,
}

/// A source distribution that exists at an arbitrary URL.
pub struct DirectUrlSourceDistribution {
    pub name: PackageName,
    pub url: Url,
}

/// A source distribution that exists in a Git repository.
pub struct GitSourceDistribution {
    pub name: PackageName,
    pub url: Url,
}
```

Most of the PR just stems downstream from this change. There are no
behavioral changes, so I'm largely relying on lint, tests, and the
compiler for correctness.
 2023-11-10 02:45:41 +00:00
konsti fbe28d3b7c
Fix mastodon-py dist-info handling (#336) 
mastodon-py 1.5.1 uses a dot in its dist-info dir name, which we
previously didn't handle, causing home-assistant to fail. The new
implementation is based on
2f83540272/src/packaging/utils.py (L146-L172).

Part of #199

```
unzip -l  Mastodon.py-1.5.1-py2.py3-none-any.whl
Archive:  Mastodon.py-1.5.1-py2.py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
   153929  2020-02-29 17:39   mastodon/Mastodon.py
     1029  2019-10-11 19:15   mastodon/__init__.py
     7357  2019-10-11 20:24   mastodon/streaming.py
       10  2020-03-14 18:14   Mastodon.py-1.5.1.dist-info/DESCRIPTION.rst
     1398  2020-03-14 18:14   Mastodon.py-1.5.1.dist-info/metadata.json
        9  2020-03-14 18:14   Mastodon.py-1.5.1.dist-info/top_level.txt
      110  2020-03-14 18:14   Mastodon.py-1.5.1.dist-info/WHEEL
     1543  2020-03-14 18:14   Mastodon.py-1.5.1.dist-info/METADATA
      753  2020-03-14 18:14   Mastodon.py-1.5.1.dist-info/RECORD
---------                     -------
   166138                     9 files
```
 2023-11-07 12:36:11 +01:00
Charlie Marsh 24e30e6557
Split `puffin-package` into requirements.txt parser and `pypi-types` (#341) 
There are only two things left in this crate and they don't really have
anything to do with one another.
 2023-11-06 18:19:49 +00:00
konsti b2439b24a1
Fetch wheel metadata by async range requests on the remote wheel (#301) 
Use range requests and async zip to extract the METADATA file from a
remote wheel.

We currently only cache when the remote says the remote declares the
resource as immutable, see
https://github.com/06chaynes/http-cache/issues/57 and
https://github.com/baszalmstra/async_http_range_reader/pull/1 . The
cache is stored as json with the description omitted, this improve cache
deserialization performance.
 2023-11-06 15:06:49 +01:00
konsti 4adaa9a700
Wheel filename distribution package name (#278) 
The normalized name abstractions were not consistently, this PR uses
them where they were previously missing:
* `WheelFilename::distribution`
* `Requirement::name`
* `Requirement::extras`
* `Metadata21::name`
* `Metadata21::provides_dist`

With `puffin-package` depending on `pep508_rs` this would be cyclical
crate dependency, so `puffin-normalize` gets split out from
`puffin-package`.

`DistInfoName` has the same task and semantics as `PackageName`, so it's
merged into the latter.

`PackageName` and `ExtraName` documentation is moved onto the type and
their constructors are called `new` instead of `normalize`. We now use
these constructors rarely enough the implicit allocation by
`to_string()` shouldn't matter anymore, while more actual cloning
becomes visible.
 2023-11-02 11:15:27 +00:00
konsti d1af90163b
Improve client reqwest errors (#276) 
I had to debug a failure involving these errors and had to improve their
output.
 2023-11-01 16:52:58 +01:00
Charlie Marsh 16aac834ee
Move PyPI-oriented types out of `puffin-client` crate (#255) 
Just an internal change to avoid a dependency on `puffin-client` for
those crates that need access to PyPI-metadata types.
 2023-10-31 17:10:23 +00:00
konsti 1fbe328257
Build source distributions in the resolver (#138) 
This is isn't ready, but it can resolve
`meine_stadt_transparent==0.2.14`.

The source distributions are currently being built serially one after
the other, i don't know if that is incidentally due to the resolution
order, because sdist building is blocking or because of something in the
resolver that could be improved.

It's a bit annoying that the thing that was supposed to do http requests
now suddenly also has to a whole download/unpack/resolve/install/build
routine, it messes up the type hierarchy. The much bigger problem though
is avoid recursive crate dependencies, it's the reason for the callback
and for splitting the builder into two crates (badly named atm)
 2023-10-25 20:05:13 +00:00
Charlie Marsh 0e097874f8
Add support for alternate index URLs (#169) 
As elsewhere, we just use the `pip` and `pip-compile` APIs. So we
support `--index-url` to override PyPI, then `--extra-index-url` to add
_additional_ indexes, and `--no-index` to avoid hitting the index at
all.

Closes #156.
 2023-10-23 03:18:30 +00:00
konsti 8cc4fe0d44
Install source distribution requirements with puffin itself instead of pip (#122) 
This is also a lot faster. Unfortunately it copies a lot of code from
the sync cli since the `Printer` is private.

The first commit are some refactorings i made when i thought about how i
could reuse the existing code.
 2023-10-18 19:11:17 +00:00
Charlie Marsh 471a1d657d
Migrate resolver proof-of-concept to PubGrub (#97) 
## Summary

This PR enables the proof-of-concept resolver to backtrack by way of
using the `pubgrub-rs` crate.

Rather than using PubGrub as a _framework_ (implementing the
`DependencyProvider` trait, letting PubGrub call us), I've instead
copied over PubGrub's primary solver hook (which is only ~100 lines or
so) and modified it for our purposes (e.g., made it async).

There's a lot to improve here, but it's a start that will let us
understand PubGrub's appropriateness for this problem space. A few
observations:

- In simple cases, the resolver is slower than our current (naive)
resolver. I think it's just that the pipelining isn't as efficient as in
the naive case, where we can just stream package and version fetches
concurrently without any bottlenecks.
- A lot of the code here relates to bridging PubGrub with our own
abstractions -- so we need a `PubGrubPackage`, a `PubGrubVersion`, etc.
 2023-10-15 22:05:44 -04:00
Charlie Marsh ed68d31e03
Add a basic test for the resolver (#86) 
Mocks out the PyPI client using some checked-in fixtures. The test is
very basic, and I'm not very happy with all the ceremony around the
mocks and such, but it's an interesting experiment at least.
 2023-10-11 03:30:53 +00:00
Charlie Marsh ba2b200fce
Enable release builds via `cargo-dist` (#79) 2023-10-09 20:48:55 +00:00
Charlie Marsh d1ed41170b
Cache environment marker lookups (#55) 
Closes https://github.com/astral-sh/puffin/issues/53.
 2023-10-08 05:31:19 +00:00
Charlie Marsh 92160e37df
Surface error when unable to find package (#45) 2023-10-07 19:43:12 +00:00
Charlie Marsh 162952bf64
Add a content-addressed cache for wheels (#38) 
Closes https://github.com/astral-sh/puffin/issues/4.
 2023-10-07 14:24:52 +00:00
Charlie Marsh 6c31631913
Fetch from `data-dist-info-metadata` when available (#37) 
As specified in https://peps.python.org/pep-0658/#specification.
 2023-10-07 13:05:29 +00:00
Charlie Marsh ca6aa207ff
Move to workspace dependencies (#25) 2023-10-06 19:49:41 +00:00
Charlie Marsh 28721cf5fc Avoid caching wheel fetches 2023-10-06 00:50:30 -04:00
Charlie Marsh a43328d914
Support wheel installation (#19) 
Closes https://github.com/astral-sh/puffin/issues/8.
 2023-10-06 00:47:45 -04:00
Charlie Marsh 8b151a64d5
Rename `puffin-requirements` to `puffin-package` (#16) 
Closes https://github.com/astral-sh/puffin/issues/7.
 2023-10-05 21:03:20 -04:00
Charlie Marsh 2d6266b167
Add an HTTP cache (and `--no-cache` argument) (#14) 
Closes https://github.com/astral-sh/puffin/issues/3.
 2023-10-05 19:14:05 -04:00
Charlie Marsh 1063d8c150
Add Python interpreter detection (#11) 
Closes https://github.com/astral-sh/puffin/issues/2.
 2023-10-05 15:09:22 -04:00
Charlie Marsh b4828fb3f2 Remove progress bar 2023-10-05 12:45:38 -04:00
Charlie Marsh 7f497fa43f Add progress bar 2023-10-05 12:45:38 -04:00
Charlie Marsh 8032d4606e Misc. changes 2023-10-05 12:45:38 -04:00
Charlie Marsh f51432382a Do basic resolution 2023-10-05 12:45:38 -04:00
Charlie Marsh 0f10595ac3 Add version selection 2023-10-05 12:45:38 -04:00
Charlie Marsh b08e8c78b5 Remove normalized representation of SimpleJson 2023-10-05 12:45:38 -04:00
Charlie Marsh 610fd9994f Add client networking stack 2023-10-05 12:45:38 -04:00
Charlie Marsh 1a2f35801b Add client networking stack 2023-10-05 12:45:38 -04:00