Commit Graph

559 Commits

Author SHA1 Message Date
konsti 673bece595
Allow `pip-compile` without a venv (#494)
The semantics are a bit unintuitive because `--python-version` is a
preference when looking for a python version without a venv, but if we
don't find that exact version we'll take `python3` and patch the
markers. This will make more sense once we start provisioning python
builds.

We can now resolve black with both python 3.8 and 3.12, with or without
that python version being in scope. In the example below,
`PATH=$HOME/.cargo/bin:/usr/bin` removes the pyenv builds and leaves
only `python3`, which is python 3.11.

```console
$ RUST_LOG=puffin::commands=debug cargo run --bin puffin -q -- pip-compile -v scripts/benchmarks/requirements/black.in --python-version py38
    0.004108s DEBUG puffin::commands::pip_compile Using Python 3.8 at /home/konsti/.local/bin/python3.8
Resolved 8 packages in 44ms
# This file was autogenerated by Puffin v0.0.1 via the following command:
#    puffin pip-compile -v scripts/benchmarks/requirements/black.in --python-version py38
black==23.11.0
[...]
platformdirs==4.0.0
    # via black
tomli==2.0.1
    # via black
typing-extensions==4.8.0
    # via black
$ PATH=$HOME/.cargo/bin:/usr/bin RUST_LOG=puffin::commands=debug cargo run --bin puffin -q -- pip-compile -v scripts/benchmarks/requirements/black.in --python-version py38
    0.004315s DEBUG puffin::commands::pip_compile Using Python 3.11 at /usr/bin/python3
Resolved 8 packages in 43ms
# This file was autogenerated by Puffin v0.0.1 via the following command:
#    puffin pip-compile -v scripts/benchmarks/requirements/black.in --python-version py38
black==23.11.0
[...]
platformdirs==4.0.0
    # via black
tomli==2.0.1
    # via black
typing-extensions==4.8.0
    # via black
```

```console
$ RUST_LOG=puffin::commands=debug cargo run --bin puffin -q -- pip-compile -v scripts/benchmarks/requirements/black.in --python-version py312
    0.004216s DEBUG puffin::commands::pip_compile Using Python 3.12 at /home/konsti/.local/bin/python3.12
Resolved 6 packages in 37ms
# This file was autogenerated by Puffin v0.0.1 via the following command:
#    puffin pip-compile -v scripts/benchmarks/requirements/black.in --python-version py312
black==23.11.0
[...]
platformdirs==4.0.0
    # via black
$ PATH=$HOME/.cargo/bin:/usr/bin RUST_LOG=puffin::commands=debug cargo run --bin puffin -q -- pip-compile -v scripts/benchmarks/requirements/black.in --python-version py312
    0.004190s DEBUG puffin::commands::pip_compile Using Python 3.11 at /usr/bin/python3
Resolved 6 packages in 39ms
# This file was autogenerated by Puffin v0.0.1 via the following command:
#    puffin pip-compile -v scripts/benchmarks/requirements/black.in --python-version py312
black==23.11.0
[...]
platformdirs==4.0.0
    # via black
```

Fixes #235.

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2024-01-05 15:01:06 +00:00
Charlie Marsh 76064cdec2
Document Python interpreter discovery in README (#792) 2024-01-05 09:44:06 -05:00
Zanie Blue 0cd57a6cd8
Add pre-release scenarios (#790)
Scenarios added in https://github.com/zanieb/packse/pull/58
2024-01-05 03:10:43 +00:00
Zanie Blue 3d6ea7809a
Update scenario tests to include `requires-python` coverage (#769)
Includes creating a virtual env with the relevant environment python
version.

Scenarios added in https://github.com/zanieb/packse/pull/55
2024-01-04 14:15:13 -06:00
konsti 57c96df288
Explain ld errors (#773)
One of the most common ways source dists fail to build (on linux) is
when the linker fails because the shared library of a native dependency
is not installed. These errors are hard to understand when you're not a
c programmer:

```
       In file included from /usr/include/python3.10/unicodeobject.h:1046,
                        from /usr/include/python3.10/Python.h:83,
                        from Modules/3.x/readline.c:8:
       Modules/3.x/readline.c: In function ‘on_completion’:
       /usr/include/python3.10/cpython/unicodeobject.h:744:29: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
         744 | #define _PyUnicode_AsString PyUnicode_AsUTF8
             |                             ^~~~~~~~~~~~~~~~
       Modules/3.x/readline.c:842:23: note: in expansion of macro ‘_PyUnicode_AsString’
         842 |             char *s = _PyUnicode_AsString(r);
             |                       ^~~~~~~~~~~~~~~~~~~
       Modules/3.x/readline.c: In function ‘readline_until_enter_or_signal’:
       Modules/3.x/readline.c:1044:9: warning: ‘sigrelse’ is deprecated: Use the sigprocmask function instead [-Wdeprecated-declarations]
        1044 |         sigrelse(SIGINT);
             |         ^~~~~~~~
       In file included from Modules/3.x/readline.c:10:
       /usr/include/signal.h:359:12: note: declared here
         359 | extern int sigrelse (int __sig) __THROW
             |            ^~~~~~~~
       Modules/3.x/readline.c: In function ‘PyInit_readline’:
       Modules/3.x/readline.c:1179:34: warning: assignment to ‘char * (*)(FILE *, FILE *, const char *)’ from incompatible pointer type ‘char * (*)(FILE *, FILE *, char *)’ [-Wincompatible-pointer-types]
        1179 |     PyOS_ReadlineFunctionPointer = call_readline;
             |                                  ^
       In file included from /usr/include/string.h:535,
                        from /usr/include/python3.10/Python.h:30,
                        from Modules/3.x/readline.c:8:
       In function ‘strncpy’,
           inlined from ‘call_readline’ at Modules/3.x/readline.c:1124:9:
       /usr/include/x86_64-linux-gnu/bits/string_fortified.h:95:10: warning: ‘__builtin_strncpy’ output truncated before terminating nul copying as many bytes from a string as its length [-Wstringop-truncation]
          95 |   return __builtin___strncpy_chk (__dest, __src, __len,
             |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          96 |                                   __glibc_objsize (__dest));
             |                                   ~~~~~~~~~~~~~~~~~~~~~~~~~
       Modules/3.x/readline.c: In function ‘call_readline’:
       Modules/3.x/readline.c:1099:9: note: length computed here
        1099 |     n = strlen(p);
             |         ^~~~~~~~~
       /usr/bin/ld: cannot find -lncurses: No such file or directory
       collect2: error: ld returned 1 exit status
       error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
       ---
```

We parse these errors out, tell the user about the missing shared
library and even the most likely debian/ubuntu package name:

```
This error likely indicates that you need to install the library that provides a shared library for ncurses for pygraphviz-1.11 (e.g. libncurses-dev)
```
2024-01-04 20:56:38 +01:00
Zanie Blue 8ac6f9a198
Wrap scenario descriptions in docstrings (#787)
Otherwise, the lines can get kind of long.
2024-01-04 19:43:50 +00:00
Zanie Blue f89c6456e3
Explicitly pin scenarios to a packse commit (#788)
Previously, we just pulled the latest commit from `main` on every
update. This causes problems when you do not intend to update the
scenarios as in #787.

This bumps to the latest `packse` commit without new scenarios.
2024-01-04 19:38:48 +00:00
Zanie Blue 5e04a95c45
Disable line wrapping during scenario tests (#784)
Adds support for a `PUFFIN_NO_WRAP` environment variable which disables
line wrapping in `miette` output.

We set this variable in the scenario tests to improve the readability of
snapshots.

I contributed the ability to disable line wrapping upstream at
https://github.com/zkat/miette/pull/328
2024-01-04 19:07:16 +00:00
Andrew Gallant d7c9b151fb
pep440: some minor refactoring, mostly around error types (#780)
This PR does a bit of refactoring to the pep440 crate, and in
particular around the erorr types. This PR is meant to be a precursor
to another PR that does some surgery (both in parsing and in `Version`
representation) that benefits somewhat from this refactoring.

As usual, please review commit-by-commit.
2024-01-04 12:28:36 -05:00
Andrew Gallant 1cc3250e76
puffin-cli: fix botched merge (#785)
This fixes a compilation error with tests on current `main`. I didn't
track down the exact provenance, but I'd guess it's the result of a
botched merge. (i.e., Two or more PRs that pass CI independently, but
when merged cause failures.)
2024-01-04 17:03:45 +00:00
Charlie Marsh c6bdc43f37
Add missing feature to `Cargo.toml` (#777) 2024-01-04 11:39:11 -05:00
Zanie Blue e75fde7bfe
Filter prefixes from scenario snapshots to improve readability (#779)
I'm a _little_ unsure since this could be confusing but the prefixes can
be pretty long and this is much easier to read.
2024-01-04 09:57:41 -06:00
konsti 9b77a8873e
Disable color output when redirecting stderr (#742)
I'm still confused about it, but this seems to do the right thing?

`HierarchicalLayer` internally has [`let ansi =
io::stderr().is_terminal();`](fcd9eed252/src/lib.rs (L74)),
so the logging itself is already correctly uncolored, but errors in the
log weren't.

Test command, ran with network deactivated:

```shell
RUST_LOG=debug cargo run --bin puffin -- pip-compile -v ./scripts/popular_packages/pypi_8k_downloads.txt 2> log.txt
```

**Before**

```
error: Request error: error sending request for url (https://pypi.org/simple/apache-airflow-providers-dbt-cloud/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
  Caused by: error sending request for url (https://pypi.org/simple/apache-airflow-providers-dbt-cloud/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
  Caused by: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
  Caused by: dns error: failed to lookup address information: Temporary failure in name resolution
  Caused by: failed to lookup address information: Temporary failure in name resolution
  ```

  **After**

  ```
  error: Request error: error sending request for url (https://pypi.org/simple/fissix/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
    Caused by: error sending request for url (https://pypi.org/simple/fissix/): error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
    Caused by: error trying to connect: dns error: failed to lookup address information: Temporary failure in name resolution
    Caused by: dns error: failed to lookup address information: Temporary failure in name resolution
    Caused by: failed to lookup address information: Temporary failure in name resolution
```
2024-01-04 16:43:44 +01:00
konsti 92c780ec2f
Run custom insta filters before generic filters (#781)
I've noticed some non-deterministic test failures when a temp dir looks
like a timestamp
(https://github.com/astral-sh/puffin/actions/runs/7410022542/job/20161416805).
Running the custom filters for e.g. the temp dirs before the generic
time filters should fix that.
2024-01-04 16:40:28 +01:00
Charlie Marsh b2230e7f4d
Make index URLs insensitive to trailing slashes (#771)
Closes https://github.com/astral-sh/puffin/issues/770.
2024-01-04 08:45:50 -05:00
konsti 7d6e6fae25
Requirement fixup for trailing comma after trailing quote (#776)
Fixup for
7349527ceadde8fc265a33e6a4e662/boto3-1.2.0-py2.py3-none-any.whl:

```
botocore>=1.3.0,<1.4.0',
```

Note that neither the quote nor the comma are right.
2024-01-04 08:45:41 -05:00
konsti 0c5ca1cdd8
Delete unused file (#772)
This is a duplicate that's not used anymore, probably a refactoring
artifact.
2024-01-04 11:32:12 +00:00
Zanie Blue e18a6a0c03
Include permalink to scenarios used to generate test cases (#767) 2024-01-03 20:41:14 -06:00
Zanie Blue 0d5252580c
Improve scenario update script (#759)
Following #757, improves the script for generating scenario test cases
with:

- A requirements file
- Support for downloading packse scenarios from GitHub dynamically
- Running rustfmt on the generated test file
- Updating snapshots / running tests
2024-01-03 20:13:11 -06:00
Charlie Marsh bf9e9daa39
Make editable installs their own test feature flag (#766)
For whatever reason these fail for me with mold, and it's not worth it
to me to disable the linker.
2024-01-03 20:33:22 -05:00
Charlie Marsh 252d53e83a
Make environment validation a `--strict` flag (#765)
I don't necessarily want users to pay this cost every time. We could
consider making this `true` by default.

Closes https://github.com/astral-sh/puffin/issues/763.
2024-01-04 01:29:06 +00:00
Charlie Marsh ae8c7d11e3
Use `create_venv_py312` in pip-uninstall tests (#764) 2024-01-04 01:16:13 +00:00
Charlie Marsh 286145bc7f
Add a dedicated error for missing RECORD files (#762)
Related to: https://github.com/astral-sh/puffin/issues/716
2024-01-04 00:28:50 +00:00
Charlie Marsh 2d1d6ac0dd
Add context and diagnostics for missing `METADATA` (#761)
Closes https://github.com/astral-sh/puffin/issues/717.
2024-01-03 19:09:23 -05:00
Zanie Blue 1f2112191f
Unpack scenario root requirements in test cases (#757)
As mentioned in #746, instead of just installing the scenario root we
will unpack the root dependencies into the install command to allow
better coverage of direct user requests with scenarios.

I added display of the package tree provided by each scenario.

Use a mustache template for iterative replacements.
2024-01-03 17:31:29 -06:00
Charlie Marsh 02b157085e
Add INSTALLER file to install-wheel-rs (#760)
See:
https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-installer-file
2024-01-03 17:30:54 -05:00
Zanie Blue c9f43e915c
Add packse scenario tests (#746)
Adds tests using packse test scenarios! Uses `test.pypi.org` as a
backing index.

Tests are generated by a simple Python script. Requires
https://github.com/zanieb/packse/pull/49.

This opens us to a slight attack surface, as we cannot force use of
`test.pypi.org` only and someone could register these package names on
the real `pypi.org` index with malicious content. I could publish these
packages there too.
2024-01-03 15:50:06 -06:00
Charlie Marsh aa75d264cd
Clean up the Puffin CLI (#755)
- Rename to `puffin pip-freeze` for consistency.
- Add a `virtualenv` alias to `venv`.
- Hide the `add` and `remove` commands.
2024-01-03 21:22:06 +00:00
Charlie Marsh cfffcbb269
Cancel waiting tasks on resolution error (#753)
## Summary

I don't understand why this works (because I don't understand why it's
erroring) but it does. See:
https://github.com/astral-sh/puffin/pull/746#issuecomment-1875722454.

## Test Plan

```
cargo run --bin puffin pip-install requires-transitive-incompatible-with-transitive-8329cfc0 --extra-index-url https://test.pypi.org/simple -n
```
2024-01-03 20:18:27 +00:00
Charlie Marsh a8e52d2899
Split `resolver.rs` into a module (#752)
This is just getting hard to navigate. No code changes, just moving
stuff around.
2024-01-03 14:02:30 -05:00
Charlie Marsh 48c7359622
Always simplify dependency sets (#748)
`simplify_set` can itself simplify to the full range, so it seems like
we should be checking if the set is `Range::full` _after_ simplifying
rather than before.
2024-01-03 13:21:03 -05:00
Charlie Marsh 607a5bee6d
Use `register_owned` in prefetch path (#750) 2024-01-03 17:31:23 +00:00
Charlie Marsh fd556ccd44
Model Python version as a PubGrub package (#745)
## Summary

This PR modifies the resolver to treat the Python version as a package,
which allows for better error messages (since we no longer treat
incompatible packages as if they "don't exist at all").

There are a few tricky pieces here...

First, we need to track both the interpreter's Python version and the
_target_ Python version, because we support resolving for other versions
via `--python 3.7`.

Second, we allow using incompatible wheels during resolution, as long as
there's a compatible source distribution. So we still need to test for
`requires-python` compatibility when selecting distributions.

This could use more testing, but it feels like an area where `packse`
would be more productive than writing PyPI tests.

Closes https://github.com/astral-sh/puffin/issues/406.
2024-01-03 15:20:45 +00:00
Charlie Marsh 5a98add54e
Always pre-fetch distribution metadata (#744)
This PR fixes our prefetching logic to ensure that we always attempt to
prefetch the "best-guess" distribution for all dependencies. This logic
already existed, but because we only attempted to prefetch when package
metadata was available, it almost never triggered. Now, we wait for the
package metadata to become available, _then_ kick off the "best-guess"
prefetch (for every package).

In my testing, this dramatically improves performance (like 2x). I'm
wondering if this regressed at some point?

Closes #743.

Co-authored-by: konsti <konstin@mailbox.org>
2024-01-03 11:37:45 +01:00
Charlie Marsh ba23115465
Remove some package clones (#749) 2024-01-02 23:21:46 -05:00
Charlie Marsh 94076d6000
Use dependency package when simplifying dependency set (#747)
This manifested itself here:
https://github.com/astral-sh/puffin/pull/745/files#r1439912440.
2024-01-02 20:26:56 -06:00
konsti 26f597a787
Add spans to all significant tasks (#740)
I've tried to investigate puffin's performance wrt to builds and
parallelism in general, but found the previous instrumentation to
granular. I've tried to add spans to every function that either needs
noticeable io or cpu resources without creating duplication. This also
fixes some wrong tracing usage on async functions
(https://docs.rs/tracing/latest/tracing/struct.Span.html#in-asynchronous-code)
and some spans that weren't actually entered.
2024-01-02 16:17:03 +00:00
konsti a3d8b3d9ca
Don't install incompatible path and url wheels (#739)
Add early tag checking for path and url wheels.

This does not check for resolve for consistency with index wheels.
2024-01-02 15:00:50 +00:00
konsti cd43708369
Flag to force latest version in resolve-many (#741)
Also fixes color when redirecting puffin-dev to a log file.
2024-01-02 11:04:26 +00:00
konsti 35f6ea204b
Remove Box::pin usages (#738)
Rust 1.75 update follow-up, simplifies the code.
2023-12-29 15:49:12 +00:00
Charlie Marsh 2cfa4a3574
Add a dedicated error message to hint users towards enabling pre-releases (#697)
This PR adds a dedicated error message for resolutions that fail, but
might've succeeded if pre-releases were allowed. Specifically, if we see
a failed resolution, and failed to find a version for a package that
included a pre-release marker, we add a hint nudging the user to
explicitly enable all pre-releases.

We'd prefer a solution like
https://github.com/astral-sh/puffin/pull/666, but believe that it will
break some assumptions in PubGrub, so this is the lighter-weight
solution.

Closes https://github.com/astral-sh/puffin/issues/659.
2023-12-28 21:44:35 -05:00
konsti 2d4cb1ebf2
Rust 1.75 (#736)
The `async fn` and return-position `impl Trait` in traits improve
`BuildContext` ergonomics. The traits use `impl Future` over `async fn`
to make the send bound explicit
(https://blog.rust-lang.org/2023/12/21/async-fn-rpit-in-traits.html).

The remaining changes are due to clippy.
2023-12-28 16:08:35 -04:00
konsti 7bf2790a25
Remove all quotes from (lenient) version specifiers (#735)
Found in
https://pypi.org/simple/algoliasearch/?format=application/vnd.pypi.simple.v1+json
and
https://pypi.org/simple/okta/?format=application/vnd.pypi.simple.v1+json
2023-12-28 14:40:42 +00:00
konsti 0ebff943e4
Finish install-many with pypi 10k most dependents (#732)
This PR combines three small changes to finish up the install-many
testing.

* Download pypi_10k_most_dependents.txt in script I'd like to have the
setup process of the large scale checks automated.
* Some install-many dev script improvements 
* Fix mkl_fft-1.3.6-58-cp310-cp310-manylinux2014_x86_64.whl:
mkl_fft-1.3.6-58-cp310-cp310-manylinux2014_x86_64.whl has multiple
Wheel-Version entries, we have to ignore that like pip

Apart from the mkl-fft fix the only other errors i've seen showing up
are
https://github.com/astral-sh/puffin/issues/520#issuecomment-1869625642.
2023-12-27 09:42:51 -05:00
Charlie Marsh 007f52bb4e
Add support for relative URLs in simple metadata responses (#721)
## Summary

This PR adds support for relative URLs in the simple JSON responses. We
already support relative URLs for HTML responses, but the handling has
been consolidated between the two. Similar to index URLs, we now store
the base alongside the metadata, and use the base when resolving the
URL.

Closes #455.

## Test Plan

`cargo test` (to test HTML indexes). Separately, I also ran `cargo run
-p puffin-cli -- pip-compile requirements.in -n
--index-url=http://localhost:3141/packages/pypi/+simple` on the
`zb/relative` branch with `packse` running, and forced both HTML and
JSON by limiting the `accept` header.
2023-12-27 08:53:21 -05:00
Charlie Marsh ae83a74309
Review feedback for HTML indexes (#733)
See: https://github.com/astral-sh/puffin/pull/719
2023-12-26 21:57:20 +00:00
Charlie Marsh bbe0246205
Change internal representation of `CacheEntry` to avoid allocations (#730)
Removes a TODO.
2023-12-26 02:10:30 +00:00
Charlie Marsh 188ab75769
Split `File` into internal and external type (#729)
## Summary

This PR makes the `pypi_types::File` a response-only type (i.e., a type
that's only used when deserializing over the wire), and adds a separate
internal `File` type. Right now, the representations are similar, but
already, we can avoid the "lenient" deserialization on our internal
`File` type, and avoid the special-casing of the property names that's
required in the JSON. Over time, we can evolve this representation
entirely separately from the representation we receive from PyPI and
other indexes.
2023-12-25 15:42:28 -05:00
Charlie Marsh 6ff21374dc
Split `puffin-cache` into Puffin-specific and generic utilities (#728)
This crate started off as generic caching utilities, but we started
adding a lot of Puffin-specific stuff (like the cache buckets
abstraction that knows about Git vs. direct URL vs. indexes and so on).
This PR moves the generic stuff into a new `cache-key` crate.
2023-12-25 14:38:56 +00:00
Charlie Marsh 187ccef4e1
Cache `Tags` on `Interpreter` (#726) 2023-12-25 13:41:10 +00:00
Charlie Marsh 5b2e381f87
Remove `platform-tags` dependency on `puffin-interpreter` (#725)
Cuts off a large internal dependency chain from what is otherwise a very
general crate.
2023-12-24 23:06:50 +00:00
Charlie Marsh ad34bb02a9
Modify some inconsistent exports (#724) 2023-12-24 22:30:03 +00:00
Charlie Marsh 343880820b
Un-escape HTML entities when decoding (#723)
I don't have a good testing strategy here (I'm manually testing against
`devpi` via `packse`), but the HTML index uses (e.g.)
`data-requires-python="&gt;=3.8"`, so we need to decode.
2023-12-24 16:35:45 -05:00
Charlie Marsh 2d721a497e
Add a `SimpleHttp` abstraction similar to `SimpleJson` (#722)
Just an internal refactor to turn some standalone functions into
associated methods (and reduce the diff in the next PR).
2023-12-24 20:55:57 +00:00
konsti e23292641f
Add pypi 10k packages with most dependents dataset (#711)
From manual inspection, this dataset generated through the [libraries.io
API](https://libraries.io/api#project-search) seems more mainstream than
the current 8k one, which is also preserved. I've added the dataset to
the repo because the API requires an API key.
2023-12-24 18:31:52 +00:00
Charlie Marsh 5bce699ee1
Add support for HTML indexes (#719)
## Summary

This PR adds support for HTML index responses (as with
`--index-url=https://download.pytorch.org/whl`).

Closes https://github.com/astral-sh/puffin/issues/412.
2023-12-24 16:04:00 +00:00
Charlie Marsh 9e6cb706a0
Update test fixtures (#720) 2023-12-24 15:50:10 +00:00
konsti b7ad97a823
Show resource and lockfile when waiting (#715)
We lock git checkout directories and the virtualenv to avoid two puffin
instances running in parallel changing files at the same time and
leading to a broken state. When one instance is blocking another, we
need to inform the user (why is the program hanging?) and also add some
information for them to debug the situation.

The new messages will print

```
Waiting to acquire lock for /home/konsti/projects/puffin/.venv (lockfile: /home/konsti/projects/puffin/.venv/.lock)
```

or

```
Waiting to acquire lock for git+https://github.com/pydantic/pydantic-extra-types@0ce9f207a1e09a862287ab77512f0060c1625223 (lockfile: /home/konsti/projects/puffin/cache-all-kinds/git-v0/locks/f157fd329a506a34)
```

The messages aren't perfect but clear enough to see what the contention
is and in the worst case to delete the lockfile.

Fixes #714
2023-12-21 00:05:49 +01:00
konsti e60f0ec732
Update pubgrub (#713)
Easier than i expected: We simply never construct the pubgrub error
variants since we have our own main loop. The `unreachable!()`s can be
removed when never is stabilized
2023-12-20 23:56:59 +01:00
Zanie Blue e705267dac
Fix fallback download when index does not support HTTP range requests (#702)
Otherwise, when a server does not support HTTP range requests we throw
an error instead of downloading without range requests.

---------

Co-authored-by: konstin <konstin@mailbox.org>
2023-12-20 10:55:23 +00:00
Zanie Blue 665a59dae6
Fix deserialization of index response when `requires_python` field is missing (#708)
Closes https://github.com/astral-sh/puffin/issues/707
2023-12-20 11:53:48 +01:00
Zanie Blue 4e437ba7e5
Allow the default index url to be configured with `PUFFIN_INDEX_URL` (#704)
This allows the default index URL to be easily overridden with a local
index e.g. a `packse` server

```
export PUFFIN_INDEX_URL="http://localhost:3141/packages/all/+simple"
```
2023-12-20 11:52:00 +01:00
Zanie Blue ab15b08cbe
Perform 3 retries by default instead of 0 on failed index requests (#710)
As a user, I'd expect retries to occur by default.

We should also expose this via a setting probably.
2023-12-20 11:51:24 +01:00
konsti 9f8b7e7e12
Refactor `DistFinder` to allow handling errors (#709)
For the install tests, i need the ability to ignore failures in the
`DistFinder`. To avoid just copy&pasting a version that collects errors
separately, i followed
https://gendignoux.com/blog/2021/04/01/rust-async-streams-futures-part1.html
and switched the custom channel over to an async stream yielding
`Result` items.

I like the async streams mirror the normal iterator api.
2023-12-20 04:07:55 +00:00
Zanie Blue 12eedb1c12
Include `Accept` header specifying that we can only parse JSON responses (#701)
Otherwise, when an index does not support the query variable we get an
HTML response and a JSON parse error.
2023-12-19 12:22:53 -06:00
Zanie Blue 52ba65aa9c
Derive `Debug` for `CachedClientError` (#703)
Discovered while debugging https://github.com/astral-sh/puffin/pull/702
2023-12-19 12:22:39 -06:00
Andrew Gallant aa9f47bbde
improve tests for version parser (#696)
The high level goal here is to improve the tests for the version parser.
Namely, we now check not just that version strings parse successfully,
but that they parse to the expected result.

We also do a few other cleanups. Most notably, `Version` is now an
opaque type so that we can more easily change its representation going
forward.

Reviewing commit-by-commit is suggested. :-)
2023-12-19 12:25:32 -05:00
Charlie Marsh 6f90edda78
Reduce visibility of `PubGrubReportFormatter` (#699) 2023-12-19 08:53:38 -06:00
konsti 9e2bbee7f0
Name the directory whose lock we're waiting on (#700) 2023-12-19 12:19:27 +00:00
konsti 114548d945
Test that cache errors are non-fatal (#685)
The test creates a cache from multiple sources and injects faults (once
using invalid data and once by making the files unreadable on the fs
level), then resolves again.

I didn't test git because it has its own locking and correctness logic.

The main drawback is that this test is slow (2.5s for me), we could
`#[ignore]` it.
2023-12-19 12:02:49 +00:00
Charlie Marsh 878bb5c035
Remove remaining snapshot files from resolver test (#698) 2023-12-19 05:41:50 +00:00
Charlie Marsh 3660d8a08e
Introduce separate traits for ahead-of-time and installed metadata (#692)
This is a pure refactor to follow-up #690, to separate the metadata that
we know upfront about distributions (like the version, for
registry-based distributions) vs. the metadata that requires building
(like the version, for URL-based distributions).
2023-12-18 22:37:45 +00:00
Charlie Marsh 31afb39a10
Show URLs and version together for installed, URL-based dependencies (#690)
The snapshot test changes will give you a sense for the impact of the
change and the output formatting.

Closes https://github.com/astral-sh/puffin/issues/686.
2023-12-18 22:21:37 +00:00
Charlie Marsh 365c860e27
Show fully-resolved URLs in non-resolution contexts (#689)
We now show the fully-resolved URL, rather than the URL as given by the
user, _everywhere_ except for the output resolution file (which should
retain relative paths, unexpanded environment variables, etc.).

Closes https://github.com/astral-sh/puffin/issues/687.
2023-12-18 22:10:24 +00:00
konsti 43c837f7bb
Show enum defaults in `--help` output (#693)
With `Option<T>` and `.unwrap_or_default()` later, the default of `T`
isn't shown in the help output.

Old:

```
      --link-mode <LINK_MODE>
          The method to use when installing packages from the global cache

          Possible values:
          - clone:    Clone (i.e., copy-on-write) packages from the wheel into the site packages
          - copy:     Copy packages from the wheel into the site packages
          - hardlink: Hard link packages from the wheel into the site packages

      -q, --quiet
      Do not print any output

      --resolution <RESOLUTION>
          Possible values:
          - highest:       Resolve the highest compatible version of each package
          - lowest:        Resolve the lowest compatible version of each package
          - lowest-direct: Resolve the lowest compatible version of any direct dependencies, and the highest compatible version of any transitive dependencies

      --prerelease <PRERELEASE>
          Possible values:
          - disallow:                 Disallow all pre-release versions
          - allow:                    Allow all pre-release versions
          - if-necessary:             Allow pre-release versions if all versions of a package are pre-release
          - explicit:                 Allow pre-release versions for first-party packages with explicit pre-release markers in their version requirements
          - if-necessary-or-explicit: Allow pre-release versions if all versions of a package are pre-release, or if the package has an explicit pre-release marker in its version requirements
```

![Screenshot from 2023-12-18
21-04-16](https://github.com/astral-sh/puffin/assets/6826232/6b3cb47a-f224-408a-8d7a-186ebeb88ecd)

New:

```
      --link-mode <LINK_MODE>
          The method to use when installing packages from the global cache

          [default: hardlink]

          Possible values:
          - clone:    Clone (i.e., copy-on-write) packages from the wheel into the site packages
          - copy:     Copy packages from the wheel into the site packages
          - hardlink: Hard link packages from the wheel into the site packages

  -q, --quiet
          Do not print any output

      --resolution <RESOLUTION>
          [default: highest]

          Possible values:
          - highest:       Resolve the highest compatible version of each package
          - lowest:        Resolve the lowest compatible version of each package
          - lowest-direct: Resolve the lowest compatible version of any direct dependencies, and the highest compatible version of any transitive dependencies

      --prerelease <PRERELEASE>
          [default: if-necessary-or-explicit]

          Possible values:
          - disallow:                 Disallow all pre-release versions
          - allow:                    Allow all pre-release versions
          - if-necessary:             Allow pre-release versions if all versions of a package are pre-release
          - explicit:                 Allow pre-release versions for first-party packages with explicit pre-release markers in their version requirements
          - if-necessary-or-explicit: Allow pre-release versions if all versions of a package are pre-release, or if the package has an explicit pre-release marker in its version requirements
```


![image](https://github.com/astral-sh/puffin/assets/6826232/26c2c391-d959-4769-999d-481b3f179502)
2023-12-18 21:50:47 +00:00
Charlie Marsh 98fcb76015
Lock entire virtualenv during modifying commands (#695)
These commands all assume that the `site-packages` are constant
throughout.

Closes #691.
2023-12-18 16:44:45 -05:00
Charlie Marsh 207bb83a1c
Rename puffin-warnings macros to avoid tracing collision (#694)
Also more consistent with Ruff.
2023-12-18 21:33:21 +00:00
Charlie Marsh e98804141c
Re-add tf-models-nightly filter in `resolve_many.rs` (#688)
I accidentally resolved this in a prior PR.
2023-12-18 16:56:04 +00:00
Charlie Marsh dbf055fe6f
Use borrowed data in `BuildDispatch` (#679)
This PR uses borrowed data in `BuildDispatch` which makes creating a
`BuildDispatch` extremely cheap (only one allocation, for the Python
executable). I can be talked out of this, it will have no measurable
impact.
2023-12-18 16:43:03 +00:00
Charlie Marsh c400ab7d07
Add support for `file://` URLs in editable requirements (#680) 2023-12-18 14:55:37 +00:00
Charlie Marsh 74ca9128b4
Canonicalize virtualenv path once (#678)
This avoids filesystem calls when creating a `BuildDispatch`.

Co-authored-by: konsti <konstin@mailbox.org>
2023-12-18 14:42:58 +00:00
konsti 89ca0d68b9
`exclude_newer` in puffin-dev resolve-cli (#684)
Internal dev tool change.
2023-12-18 14:06:54 +00:00
konsti 7926749296
Fixup for `>=2.7,!=3.0.*,!=3.1.*,<3.4.*` (#683)
Found in
https://pypi.org/simple/wincertstore/?format=application/vnd.pypi.simple.v1+json
2023-12-18 12:56:48 +00:00
konsti f4f67ebde0
Rebase: Uninstall existing non-editable versions when installing editable requirements bug (#682)
Separate branch for rebasing #677 onto main because i don't trust the
rebase enough to force push.

Closes #677.

---

If you install `black` from PyPI, then `-e ../black`, we need to
uninstall the existing `black`. This sounds simple, but that in turn
requires that we _know_ `-e ../black` maps to the package `black`, so
that we can mark it for uninstallation in the install plan. This, in
turn, means that we need to build editable dependencies prior to the
install plan.

This is just a bunch of reorganization to fix that specific bug
(installing multiple versions of `black` if you run through the above
workflow): we now run through the list of editables upfront, mark those
that are already installed, build those that aren't, and then ensure
that `InstallPlan` correctly removes those that need to be removed, etc.

Closes #676.

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2023-12-18 09:28:14 +00:00
Charlie Marsh 0bb2c92246
Add editable install support to `pip-install` (#675)
Per the title: adds support for `-e` installs to `puffin pip-install`.
There were some challenges here around threading the editable installs
to the right places. Namely, we want to build _once_, then reuse the
editable installs from the resolution. At present, we were losing the
`editable: true` flag on the `Dist` that came back through the
resolution, so it required some changes to the resolver.

Closes https://github.com/astral-sh/puffin/issues/672.
2023-12-18 09:52:32 +01:00
konsti 8c6463d220
Allow identical `VIRTUAL_ENV` and `CONDA_PREFIX` env vars (#681)
Port of https://github.com/PyO3/maturin/pull/1879 for
https://github.com/PyO3/maturin/issues/1878
2023-12-18 08:42:31 +00:00
Charlie Marsh 77c6e6fa6c
Add support for `reinstall` to editable packages (#674)
Closes https://github.com/astral-sh/puffin/issues/673.
2023-12-17 15:41:57 +00:00
Charlie Marsh 00e1c33af4
Add an editable index to the site-packages registry (#671)
This PR modifies `SitePackages` to store all distributions in a flat
vector, and maintain two indexes (hash maps) from "per-element data for
an element in the vector" to "index of that element". This enables us to
maintain a map on both package name and editable URL.
2023-12-17 03:44:36 +00:00
Charlie Marsh 08edd173db
Add support for editable packages in `pip-uninstall` (#670) 2023-12-17 02:56:37 +00:00
konsti f059c6e6a6
Support editable in pip-sync and pip-compile (#587)
Support `-e path/do/dir` in pip-sync and and pip-compile.
2023-12-16 22:37:34 +00:00
Charlie Marsh f62458f600
Add explicit error message for URLs without package names (#669)
`pip` supports installing packages without names (e.g.,
`git+https://github.com/pallets/flask.git`), but it doesn't adhere to
the PEP grammar, and we don't yet support it (and may never) (#313).

This PR adds a dedicated error path for such cases, to ensure that we
can give meaningful feedback to the user:

```
error: Couldn't parse requirement in requirements.in position 0 to 18
  Caused by: URL requirement is missing a package name; expected: `package_name @ https://google.com`
https://google.com
^^^^^^^^^^^^^^^^^^
```

Closes https://github.com/astral-sh/puffin/issues/650.
2023-12-16 21:14:34 +00:00
konsti 71964ec7a8
Switch to msgpack in the cached client (#662)
This gives a 1.23 speedup on transformers-extras. We could change to
msgpack for the entire cache if we want. I only tried this format and
postcard so far, where postcard was much slower (like 1.6s).

I don't actually want to merge it like this, i wanted to figure out the
ballpark of improvement for switching away from json.

```
hyperfine --warmup 3 --runs 10 "target/profiling/puffin pip-compile --cache-dir cache-msgpack scripts/requirements/transformers-extras.in" "target/profiling/branch pip-compile scripts/requirements/transformers-extras.in"
Benchmark 1: target/profiling/puffin pip-compile --cache-dir cache-msgpack scripts/requirements/transformers-extras.in
  Time (mean ± σ):     179.1 ms ±   4.8 ms    [User: 157.5 ms, System: 48.1 ms]
  Range (min … max):   174.9 ms … 188.1 ms    10 runs

Benchmark 2: target/profiling/branch pip-compile scripts/requirements/transformers-extras.in
  Time (mean ± σ):     221.1 ms ±   6.7 ms    [User: 208.1 ms, System: 46.5 ms]
  Range (min … max):   213.5 ms … 235.5 ms    10 runs

Summary
  target/profiling/puffin pip-compile --cache-dir cache-msgpack scripts/requirements/transformers-extras.in ran
    1.23 ± 0.05 times faster than target/profiling/branch pip-compile scripts/requirements/transformers-extras.in
```

Disadvantage: We can't manually look into the cache anymore to debug
things

- [ ] Check more formats, i currently only tested json, msgpack and
postcard, there should be other formats, too
- [x] Switch over `CachedByTimestamp` serialization (for the interpreter
caching)
- [x] Switch over error handling and make sure puffin is still resilient
to cache failure
2023-12-16 21:01:35 +00:00
Charlie Marsh e4673a0c52
Modify PEP 508 cursor to use byte offsets (#668)
This enables us to remove a number of allocations (in particular,
`peek_while` and `take_while` no longer allocate). It also makes it
trivial to move the cursor to a new location, since you can just slice
and call `.chars()`. At present, moving to a new location would require
converting the iterator to a string, then back to a character iterator.
2023-12-15 22:05:28 +00:00
Charlie Marsh 875c9a635e
Rename `CharIter` to `Cursor` (#667)
This better aligns with the analogous struct that we have in Ruff.
2023-12-15 21:57:59 +00:00
konsti 620f73b38b
Speed up version parsing for a 1.27±0.03 speedup in transformers-extras with conservative changes (#660)
Two low-hanging fruits as optimizations for version parsing: A fast path
for release only versions and removing the regex from version specifiers
(still calling into version's parsing regex if required). This enables
optimizing the serde format since we now see the serde part instead of
only PEP 440 parsing. I intentionally didn't rewrite the full PEP 440 at
this step.

```console
$ hyperfine --warmup 5 --runs 50 "target/profiling/puffin pip-compile scripts/requirements/transformers-extras.in" "target/profiling/main pip-compile scripts/requirements/transformers-extras.in"
  Benchmark 1: target/profiling/puffin pip-compile scripts/requirements/transformers-extras.in
    Time (mean ± σ):     217.1 ms ±   3.2 ms    [User: 194.0 ms, System: 55.1 ms]
    Range (min … max):   211.0 ms … 228.1 ms    50 runs

  Benchmark 2: target/profiling/main pip-compile scripts/requirements/transformers-extras.in
    Time (mean ± σ):     276.7 ms ±   5.7 ms    [User: 252.4 ms, System: 54.6 ms]
    Range (min … max):   268.9 ms … 303.5 ms    50 runs

  Summary
    target/profiling/puffin pip-compile scripts/requirements/transformers-extras.in ran
      1.27 ± 0.03 times faster than target/profiling/main pip-compile scripts/requirements/transformers-extras.in
```

---------

Co-authored-by: Andrew Gallant <andrew@astral.sh>
2023-12-15 14:03:35 -05:00
Charlie Marsh 305b9b080a
Show resolution error once on pip-install failure (#665)
Closes https://github.com/astral-sh/puffin/issues/664.
2023-12-15 18:43:23 +00:00
Charlie Marsh 47290f784e
Add fixup for invalid double quotes (#663)
Closes https://github.com/astral-sh/puffin/issues/658.
2023-12-15 18:11:22 +00:00
Charlie Marsh 9470c20e7a
Avoid double resolution during source builds (#656)
## Summary

This PR ensures that we re-use the resolution to install the build
dependencies when building a source distribution. Currently, we only
pass along the list of requirements, and then use the `Finder` to map
each requirement to a distribution. But we already determine the correct
distribution when resolving!

Closes https://github.com/astral-sh/puffin/issues/655.
2023-12-15 17:27:16 +00:00
Charlie Marsh 1129661a22
Ignore missing manifest entries in the built wheel cache (#654)
## Summary

This is more of a hypothetical problem, but the cache manifest could in
theory get out-of-sync with the contents on disk. This PR modifies the
`BuiltWheelMetadata` lookup to warn (but not fail) if the manifest
includes a wheel that no longer exists on disk. You can mimic this by
removing a wheel from the `built-wheels-v0` cache without modifying the
manifest correspondingly.
2023-12-15 17:24:09 +00:00
Charlie Marsh 84093773ef
Store source distribution sources in the cache (#653)
## Summary

This PR modifies `source_dist.rs` to store source distributions (from
remote URLs) in the cache. The cache structure for registries now looks
like:

<img width="1053" alt="Screen Shot 2023-12-14 at 10 43 43 PM"
src="https://github.com/astral-sh/puffin/assets/1309177/3c2dbf6b-5926-41f2-b69b-74031741aba8">

(I will update the docs prior to merging, if approved.)

The benefit here is that we can reuse the source distribution (avoid
download + unzipping it) if we need to build multiple wheels. In the
future, it will be even more relevant, since we'll need to reuse the
source distribution to support
https://github.com/astral-sh/puffin/issues/599.

I also included some misc. refactors to DRY up repeated operations and
add some more abstraction to `source_dist.rs`.
2023-12-15 17:19:33 +00:00