So we can skip them when there are not code changes and still enforce
our required checks (xref #4438)
Otherwise, the names are dynamic and they are forever expected (see
#4426 for example)
It was becoming problematic that the virtual environment test context
diverged from the other one i.e. we had to implement filtering twice.
This combines the contexts and tweaks the `TestContext` API and
filtering mechanisms for Python versions. Combined with my previous
changes to the test context at #4364 and
https://github.com/astral-sh/uv/pull/4368 this finally unblocks the
snapshots for test cases in #4360 and
https://github.com/astral-sh/uv/pull/4362.
This PR re-adds the `aarch64-unknown-linux-gnu` binary, which will also
add the `manylinux_2_28` wheel for `aarch64` (in addition to the now
dual-tagged `musllinux_1_1` and `manylinux_2_217` wheel for `aarch64`).
We can consider dropping that _wheel_, but in my assessment removing a
release asset should now be treated as a breaking change -- so removing
it in a patch release was incorrect.
Closes https://github.com/astral-sh/uv/issues/4122.
Extends #3726
Moves toolchain storage out of `UV_BOOTSTRAP_DIR` (`./bin`) into the
proper user data directory as defined by #3726.
Replaces `UV_BOOTSTRAP_DIR` with `UV_TOOLCHAIN_DIR` for customization.
Installed toolchains will be discovered without opt-in, but the idea is
still that these are not yet user-facing.
## Summary
This seems to be one of the most consistent benchmark cases we have in
terms of standard deviation:
```
➜ hyperfine "target/profiling/main pip compile scripts/requirements/airflow.in" --runs 200
Benchmark 1: target/profiling/main pip compile scripts/requirements/airflow.in
Time (mean ± σ): 292.6 ms ± 6.6 ms [User: 414.1 ms, System: 194.2 ms]
Range (min … max): 282.7 ms … 320.1 ms 200 runs
```
For smaller benchmarks, scispacy and dtlssocket seem to be a bit more
consistent than our current jupyter benchmark, but it hasn't given us
any problems so I'll leave it for now.
Updates our Python interpreter discovery to conform to the rules
described in #2386, please see that issue for a full description of the
behavior. Briefly, we now will search for interpreters that satisfy a
requested version without stopping at the first Python executable.
Additionally, if retrieving information about an interpreter fails we
will continue to search for a working interpreter. We also add the
plumbing necessary to request Python implementations other than CPython,
though we do not add support for other implementations at this time.
A major internal goal of this work is to prepare for user-facing managed
toolchains i.e. fetching a requested version during `uv run`. These APIs
are not introduced, but there is some managed toolchain handling as
required for our test suite.
Some noteworthy implementation changes:
- The `uv_interpreter::find_python` module has been removed in favor of
a `uv_interpreter::discovery` module.
- There are new types to help structure interpreter requests and track
sources
- Executable discovery is implemented as a big lazy iterator and is a
central authority for source precedence
- `uv_interpreter::Error` variants were split into scoped types in each
module
- There's much more unit test coverage, but not for Windows yet
Remaining work:
- [x] Write new test cases
- [x] Determine correct behavior around executables in the current
directory
- _Future_: Combine `PythonVersion` and `VersionRequest`
- _Future_: Consider splitting `ManagedToolchain` into local and remote
variants
- _Future_: Add Windows unit test coverage
- _Future_: Explore behavior around implementation precedence (i.e.
CPython over PyPy)
Refactors split into:
- #3329
- #3330
- #3331
- #3332Closes#2386
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Followup to #3624. Drops the native manylinux wheel in favor of the
statically linked musl one with a manylinux2014 tag.
<!-- How was it tested? -->
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
## Summary
Restore API-compatibility with pre-1.1.0 versions of the `zip` crate,
and pin the dependency to the 0.6 series, due to concerns discussed in
https://github.com/astral-sh/uv/issues/3642.
## Test Plan
```
cargo run -p uv-dev -- fetch-python
cargo test
```
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Fix#3439 by retagging the statically linked musl binary for
manylinux2014. This is the most conservative option;
the regular manylinux one could be removed as well, and I expect other
archs could probably do this as well.
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
Testing on manylinux.
<!-- How was it tested? -->
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
This is needed to include uv in manylinux, which has both the
musllinux_1_1 and musllinux_1_2 images.
## Summary
We are looking into including uv in the `pypa/manylinux` docker images,
so that cibuildwheel will be able to use it. But to do so, we need all
actively supported images to be able to take it, including
musllinux_1_1.
## Test Plan
The test was updated to make sure Alpine 3.12, which musllinux was based
on, can run uv.
The binary build does need to be triggered on this PR to make sure it
works. Ah,
5b6040184d/.github/workflows/build-binaries.yml (L15-L20)
is a good idea, nice.
Will need a followup to fill out the other platforms `pypa/manylinux`
supports for musllinux.
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Separates Windows tests from the rest because it's a pain and drops the
`python-patch` feature from testing so we can use the GitHub Actions
Python versions which bootstrap in 0 seconds instead of 2 minutes.
## Summary
This PR adds system install tests to verify the behavior described in
#2798. It turns out this behavior _also_ affects Fedora and Amazon
Linux, we just didn't have the right conditions enabled (specifically,
you need to create the virtualenv with `python -m venv` to get these
symlinks), so the test suite was expanded to capture that.
The issue itself is also fixed by way of deduplicating the
`site-packages` entries.
Closes: https://github.com/astral-sh/uv/issues/2798
See https://github.com/astral-sh/uv/issues/2617
Note this also includes:
- #2918
- #2931 (pending)
A first step towards Python toolchain management in Rust.
First, we add a new crate to manage Python download metadata:
- Adds a new `uv-toolchain` crate
- Adds Rust structs for Python version download metadata
- Duplicates the script which downloads Python version metadata
- Adds a script to generate Rust code from the JSON metadata
- Adds a utility to download and extract the Python version
I explored some alternatives like a build script using things like
`serde` and `uneval` to automatically construct the code from our
structs but deemed it to heavy. Unlike Rye, I don't generate the Rust
directly from the web requests and have an intermediate JSON layer to
speed up iteration on the Rust types.
Next, we add add a `uv-dev` command `fetch-python` to download Python
versions per the bootstrapping script.
- Downloads a requested version or reads from `.python-versions`
- Extracts to `UV_BOOTSTRAP_DIR`
- Links executables for path extension
This command is not really intended to be user facing, but it's a good
PoC for the `uv-toolchain` API. Hash checking (via the sha256) isn't
implemented yet, we can do that in a follow-up.
Finally, we remove the `scripts/bootstrap` directory, update CI to use
the new command, and update the CONTRIBUTING docs.
<img width="1023" alt="Screenshot 2024-04-08 at 17 12 15"
src="https://github.com/astral-sh/uv/assets/2586601/57bd3cf1-7477-4bb8-a8e9-802a00d772cb">
Reproduced https://github.com/astral-sh/uv/issues/2941 and confirmed
fix.
We probably ought to have some ecosystem test coverage — this seems like
a good starting point we can extend to other projects in the future.
Following #2735 adds a system check that uses Homebrew. I think we were
never were actually using Homebrew's Python in the past, we were mislead
or something changed in the runners recently that broke it.
## Summary
Ensures that (e.g.) installs from conda-forge, Homebrew, and other
distributions don't expose `uv self update` at all.
We'll still show `uv self update` for `pip install uv`, but it will fail
with a good error. Removing the `uv self update` from `pip`-installed
`uv` is more complicated, since we'd need to build separately for the
installer vs. for PyPI.
Closes#2588.
Add a single job for for fast lint tools. Rustfmt for rust, ruff for
python formatting and linting, prettier avoids inconsistent formatter
changes between pycharm and vscode.
If you look at https://pypi.org/project/uv/0.1.22/#files...
- We didn't upload the ARMv6 wheel (I thought I had removed the `# Skip
for `arm`, which is not supported by PyPI.`), it must've gotten re-added
in a rebase or something.
- We lost the musllinux builds for ARM. I think this is because I built
them as manylinux.
Installing and importing numpy tests for two cases:
* The python architecture and the package architecture don't match
(https://github.com/astral-sh/uv/issues/2326)
* The libc of python and that of the package don't match on linux
(musllinux vs manylinux, picking a compatible manylinux version)
All pylint deps are py3-none-any, so they don't catch those cases.
Giving this a try... just making all of these a normal part of CI.
This is probably slightly slower than our normal CI, but not by much (it
depends how bad of a roll we get on the Windows network performance).
Includes #2309 to reduce the overhead of adding more platforms.
Alternatively, we could gate these with a label and just run on main by
default (i.e. #2308)
## Summary
Adjust maturing version to use env var for reusability in the workflow.
## Test Plan
CI `Build binaries` workflow uses correct version and passes
## Summary
`pip` uses `sysconfig` for Python 3.10 and later by default; however, it
falls back to `distutils` for earlier Python versions, and distros can
actually tell `pip` to continue falling back to `distutils` via the
`_PIP_USE_SYSCONFIG` variable.
By _always_ using `sysconfig`, we're doing the wrong then when
installing into some system Pythons, e.g., on Debian prior to Python
3.10.
This PR modifies our logic to mirror `pip` exactly, which is what's been
recommended to me as the right thing to do.
Closes https://github.com/astral-sh/uv/issues/2113.
## Test Plan
Most notably, the new Debian tests pass here (which fail on main:
https://github.com/astral-sh/uv/pull/2144).
I also added Pyston as a second stress-test.
`uv --system` is failing in GitHub Actions, because `py --list-paths`
returns all the pre-cached Pythons:
```
-V:3.12 * C:\hostedtoolcache\windows\Python\3.12.2\x64\python.exe
-V:3.12-32 C:\hostedtoolcache\windows\Python\3.12.2\x86\python.exe
-V:3.11 C:\hostedtoolcache\windows\Python\3.11.8\x64\python.exe
-V:3.11-32 C:\hostedtoolcache\windows\Python\3.11.8\x86\python.exe
-V:3.10 C:\hostedtoolcache\windows\Python\3.10.11\x64\python.exe
-V:3.10-32 C:\hostedtoolcache\windows\Python\3.10.11\x86\python.exe
-V:3.9 C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe
-V:3.9-32 C:\hostedtoolcache\windows\Python\3.9.13\x86\python.exe
-V:3.8 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
-V:3.8-32 C:\hostedtoolcache\windows\Python\3.8.10\x86\python.exe
-V:3.7 C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe
-V:3.7-32 C:\hostedtoolcache\windows\Python\3.7.9\x86\python.exe
```
So, our default selector returns the first entry here. But none of these
are actually in `PATH` except the one that the user installed via
`actions/setup-python@v5` -- that's the point of the action, that it
puts the correct versions in `PATH`.
It seems to me like we should prioritize `PATH` over `py --list-paths`.
Is there a good reason not to do this?
Closes: https://github.com/astral-sh/uv/issues/2056
## Summary
This is essentially a wrapper around something like `--python $(which
python3)`, but gives users a portable and streamlined way to solve the
common pain point of using `uv` in GitHub Actions or a Docker container.
See: https://github.com/astral-sh/uv/issues/1526.
## Summary
Closes#1943
Makes sure `build-binaries` and `publish-pypi` workflows are compatible
with `actions/{download,upload}-artifact@v4`. In nature, this PR is very
similar to the changes in https://github.com/astral-sh/ruff/pull/10105.
This PR also updates cargo-dist.
## Test Plan
I ran a small non-dry-run [smoke
test](https://github.com/samypr100/uv/actions/runs/8027864059) on my own
fork CI with only linux builds (for speed) and those jobs seem to work
at a glance.
First, replace all usages in files in-place. I used my editor for this.
If someone wants to add a one-liner that'd be fun.
Then, update directory and file names:
```
# Run twice for nested directories
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
# Update files
find . -type f -print0 | xargs -0 rename s/puffin/uv/g
```
Then add all the files again
```
# Add all the files again
git add crates
git add python/uv
# This one needs a force-add
git add -f crates/uv-trampoline
```
Run `cargo test` on windows in CI, pulling the switch on tier 1 windows
support.
These changes make the bootstrap script virtually required for running
the tests. This gives us consistency between and CI, but it also locks
our tests to python-build-standalone and an articificial `PATH`.
I've deleted the shell bootstrap script in favor of only the python one,
which also runs on windows. I've left the (sym)link creation of the
bootstrap in place, even though it is not used by the tests anymore.
I've reactivated the three tests that would previously stack overflow by
doubling their stack sizes. The stack overflows only happen in debug
mode, so this is neither a user facing problem nor an actual problem
with our code and this workaround seems better than optimizing our code
for case that the (release) compiler can optimize much better for.
The handling of patch versions will be fixed in a follow-up PR.
Closes#1160Closes#1161
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
Enables tests for macOS in CI, using the M1 runners (which are free in
public, but count against our quota in private
repos). For now, I'm just running them on `main` to save quota.
I did the math, and the M1 runners are the best value:
Closes#1053.
## Summary
We have some flags in Puffin that enable us to opt-in to certain tests.
To date, they've been opt-in, so we've run our tests with
`--all-features`. This PR makes them opt-out, and we now run tests with
default features.
The main motivation here is that I want to get tests working for macOS
on CI, but for unknown reasons, macOS can't compile the PyO3 features at
the same time as everything else due to strange linker issues. By
avoiding `--all-features` for tests, we thus avoid unnecessarily
including features that we don't actually use in Puffin.
I verified that the exact same number of tests (439) are run before and
after this change. For users, the primary difference is that you now
need to specify `--no-default-features --features pypi --features
python` to avoid (e.g.) including the Git tests.
A 1:1 port of the Bash script to Python for use on Windows.
Pulls some parts of #1068 but much more minimal. Avoids an additional
dependency on `requests`. Because we require `zstandard` to unzip the
distributions we unfortunately cannot be dependency free and cannot have
`bootstrap.sh` download the Python version needed to run this script
without it doing a non-trivial amount of work.
Retains the Bash script for now so you can bootstrap without Python
available. I may drop it in the future?
Replaces https://github.com/astral-sh/puffin/pull/1068 and #1070 which
were more complicated than I wanted.
- Introduces a `.python-versions` file which defines the Python versions
needed for development
- Adds a Bash script at `scripts/bootstrap/install` which installs the
required Python versions from `python-build-standalone` to `./bin`
- Checks in a `versions.json` file with metadata about available
versions on each platform and a `fetch-version` Python script derived
from `rye` for updating the versions
- Updates CI to use these Python builds instead of the `setup-python`
action
- Updates to the latest packse scenarios which require Python 3.8+
instead of 3.7+ since we cannot use 3.7 anymore and includes new test
coverage of patch Python version requests
- Adds a `PUFFIN_PYTHON_PATH` variable to prevent lookup of system
Python versions for isolation during development
Tested on Linux (via CI) and macOS (locally) — presumably it will be a
bit more complicated to do proper Windows support.
## Background
In virtual environments, we want to install python programs as console
commands, e.g. `black .` over `python -m black .`. They may be called
[entrypoints](https://packaging.python.org/en/latest/specifications/entry-points/)
or scripts. For entrypoints, we're given a module name and function to
call in that module.
On Unix, we generate a minimal python script launcher. Text files are
runnable on unix by adding a shebang at their top, e.g.
```python
#!/usr/bin/env python
```
will make the operating system run the file with the current python
interpreter. A venv launcher for black in `/home/ferris/colorize/.venv`
(module name: `black`, function to call: `patched_main`) would look like
this:
```python
#!/home/ferris/colorize/.venv/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from black import patched_main
if __name__ == "__main__":
sys.argv[0] = re.sub(r"(-script\.pyw|\.exe)?$", "", sys.argv[0])
sys.exit(patched_main())
```
On windows, this doesn't work, we can only rely on launching `.exe`
files.
## Summary
We use posy's rust implementation of a trampoline, which is based on
distlib's c++ implementation. We pre-build a minimal exe and append the
launcher script as stored zip archive behind it. The exe will look for
the venv python interpreter next to it and use it to execute the
appended script.
The changes in this PR make the `black` entrypoint work:
```powershell
cargo run -- venv .venv
cargo run -q -- pip install black
.\.venv\Scripts\black --version
```
Integration with our existing tests will be done in follow-up PRs.
## Implementation and Details
I've vendored the posy trampoline crate. It is a formatted, renamed and
slightly changed for embedding version of
https://github.com/njsmith/posy/pull/28.
The posy launchers are smaller than the distlib launchers, 16K vs 106K
for black. Currently only `x86_64-pc-windows-msvc` is supported. The
crate requires a nightly compiler for its no-std binary size tricks.
On windows, an application can be launched with a console or without (to
create windows instead), which needs two different launchers. The gui
launcher will subsequently use `pythonw.exe` while the console launcher
uses `python.exe`.
Missing piece for the release.
## Test Plan
Built the image locally:
```shell
❯ docker run 99956098e1f8f04e209dcfc4a0afcee67df1fe8a726c164884e67f035b1a0f42
Usage: puffin [OPTIONS] <COMMAND>
Commands:
pip Resolve and install Python packages
venv Create a virtual environment
clean Clear the cache
help Print this message or the help of the given subcommand(s)
Options:
-q, --quiet Do not print any output
-v, --verbose Use verbose output
-n, --no-cache Avoid reading from or writing to the cache
--cache-dir <CACHE_DIR> Path to the cache directory [env: PUFFIN_CACHE_DIR=]
-h, --help Print help
-V, --version Print version
```
## Summary
This PR adds a release workflow powered by `cargo-dist`. It's similar to
the version that's PR'd in Ruff
(https://github.com/astral-sh/ruff/pull/9559), with the exception that
it doesn't include the Docker build or the "update dependents" step for
pre-commit.
Alternative to #875. Instead of partitioning tests across multiple
runners via nextest, we use a larger GitHub Actions runner.
Additionally, we explore using nextest to take advantage of the
increased number of cores.
On the 8-core machine, nextest is 22% faster than insta. In combination
with the vastly more readable output, I think this means we should
switch over. As noted in #875 we lose the ability to detect unreferenced
snapshot files but since we inline all of our snapshots this shouldn't
matter.
### Benchmarks
The following are the runtime of _just_ the test portion of the test job
in GitHub Actions except the partitioned case from #875 which requires a
separate build step making runner overhead relevant.
The compile times are noted as a reference as a possible lower bound of
test times. The compile time **is included** in all of the test times
shown.
Where the nextest thread count is not noted, it is inferred from the CPU
count.
```
test time diff
------------------------------------------------------
2-core (main) 4m 53s
2-core-nextest-partioned (#875) 3m 56s -19%
4-core-compilation 32s
4-core-insta 1m 47s -63%
4-core-nextest 1m 40s -66%
8-core-compilation 18s
8-core-insta 1m 9s -76%
8-core-nextest 1m 5s -78%
8-core-nextest-12-threads 54s -82%
8-core-nextest-16-threads 55s -82%
```
### Cost
We must pay per-minute costs for these runners:
> Larger runners are not eligible for the use of included minutes on
private repositories. For both private and public repositories, when
larger runners are in use, they will always be billed at the per-minute
rate.
>
> Compared to standard GitHub-hosted runners, larger runners are billed
differently. Larger runners are only billed at the per-minute rate for
the amount of time workflows are executed on them.
[[source]](https://docs.github.com/en/actions/using-github-hosted-runners/about-larger-runners/about-larger-runners#understanding-billing)
The per-minute rates are as follows:
> Linux 2 $0.008 (main)
> Linux 4 $0.016
> Linux 8 $0.032 (pull request)
[[source]](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#per-minute-rates)
The per-minute cost increases by 4x but the workflow is 5.2x faster
since we are making use of the extra compute. We will not get any free
minutes executing these runners once the repository is public.
Additionally, we will not make use of our [3,000 minutes /
month](https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions#included-storage-and-minutes)
of included minutes. Using the 8-core machines, the included 3,000
minutes should account for approximately ~$100.
Here's a brief analysis of costs from the last few
```
Minutes used
------------
November 1090 + 3000 = 4090
December 1357 + 3000 = 4357
January 2655 in 7 days
~3x more expected
= 11000 estimated
Costs
-----
November 1090 * 0.008 = $ 8.72
December 1357 * 0.008 = $10.86
January 8000 * 0.008 = $64 projected using 3000 included minutes and 2-core machines
(11000 - (0.82 * 11000)) * 0.032
= $63 projected without included minutes and 4-core machines with perf improvement
(11000 - (0.70 * 11000)) * 0.032
= $100 projected with a more conservative 70% reduction in total runtime
```
We can reduce costs (once public) by disabling larger runners for
non-organization users e.g.
https://github.com/PrefectHQ/prefect/pull/9519
Derived from https://github.com/astral-sh/puffin/pull/875
This gets us a significant speedup.
I would not read the commits individually. I can squash them but they
were used for testing various scenarios.
### Test compile times
Ranges are the lowest and highest I've seen. Huge variability in GitHub
Actions runners.
**Before:**
7m 21s - 8m 22s (cold cache)
110s - 120s (warm cache)
**After:**
6m 15s - 7m 05s (cold cache)
57s - 70s (warm cache)
**Improvement:**
4% - 25% (cold cache)
36% - 52% (warm cache)
Each cache entry is ~1 GB of our allotted 10 GB for the repository which
is quite a bit. We're probably losing cache entries all the time since
we add an entry per commit per pull request.
Saving the cache takes ~3 minutes
([example](https://github.com/astral-sh/puffin/actions/runs/7479909295/job/20358124969)),
it's probably just slowing down CI. It's ~25% of our test runtime and
~50% of our clippy runtime.
Filter out source dists and wheels whose `requires-python` from the
simple api is incompatible with the current python version.
This change showed an important problem: When we use a fake python
version for resolving, building source distributions breaks down because
we can only build with versions we actually have.
This change became surprisingly big. The tests now require python 3.7 to
be installed, but changing that would mean an even bigger change.
Fixes#388
Zanie Blue
renovate[bot]
Charlie Marsh
konsti
Ibraheem Ahmed
Henry Schreiner
Ben Beasley
samypr100
Alex Waygood
Zander
Micha Reiser
Jacob Coffee