## Summary
I believe `zlib-rs` is now a better choice on ARM and x86, so I'm just
going to assume it's a better choice everywhere. It's much easier to
build (removes our CMake dependency), and in my benchmarking, it's
substantially faster on ARM and faster or ~exactly even on my x86
Windows machine.
We migrated to `zlib-rs` once before (#9184); however, I later reverted
it as I learned that they were only doing compile-time feature
detection, and so `zlib-rs` was meaningfully slower on x86. They now
perform runtime feature detection:
https://trifectatech.org/blog/zlib-rs-is-faster-than-c/.
To benchmark, I wrote a script to create a local Simple API-compliant
registry (see the commit history) for a single package. Then I ran the
`install-cold` benchmark against that registry to install NumPy.
On ARM:
```
❯ uv run resolver --uv-pip-path ../../zlib-ng --uv-pip-path ../../zlib-rs \
--benchmark install-cold \
req.txt --warmup 10 --min-runs 30
Benchmark 1: ../../zlib-ng (install-cold)
Time (mean ± σ): 165.7 ms ± 34.7 ms [User: 64.4 ms, System: 93.2 ms]
Range (min … max): 141.8 ms … 293.2 ms 30 runs
Benchmark 2: ../../zlib-rs (install-cold)
Time (mean ± σ): 150.9 ms ± 16.2 ms [User: 57.4 ms, System: 86.4 ms]
Range (min … max): 135.3 ms … 202.4 ms 30 runs
Summary
../../zlib-rs (install-cold) ran
1.10 ± 0.26 times faster than ../../zlib-ng (install-cold)
```
I benchmarked this about 100 times on my Windows machine and found it
difficult to conclude anything beyond "They're nearly the same". Here's
an example:
```
PS C:\Users\crmar\workspace\puffin> hyperfine --prepare "uv venv" "zlib-rs.exe pip sync ./scripts/benchmark/req.txt" "zlib-ng.exe pip sync ./scripts/benchmark/req.txt" "zlib-rs.exe pip sync ./scripts/benchmark/req.txt" "zlib-ng.exe pip sync ./scripts/benchmark/req.txt" --runs 10 --warmup 5
Benchmark 1: zlib-rs.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 240.6 ms ± 10.8 ms [User: 6.1 ms, System: 92.2 ms]
Range (min … max): 229.4 ms … 267.9 ms 10 runs
Benchmark 2: zlib-ng.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 241.3 ms ± 6.2 ms [User: 7.7 ms, System: 90.6 ms]
Range (min … max): 233.9 ms … 252.1 ms 10 runs
Benchmark 3: zlib-rs.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 242.8 ms ± 7.7 ms [User: 6.2 ms, System: 23.4 ms]
Range (min … max): 236.1 ms … 262.8 ms 10 runs
Benchmark 4: zlib-ng.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 245.9 ms ± 5.7 ms [User: 1.5 ms, System: 59.4 ms]
Range (min … max): 240.9 ms … 257.3 ms 10 runs
Summary
zlib-rs.exe pip sync ./scripts/benchmark/req.txt ran
1.00 ± 0.05 times faster than zlib-ng.exe pip sync ./scripts/benchmark/req.txt
1.01 ± 0.06 times faster than zlib-rs.exe pip sync ./scripts/benchmark/req.txt
1.02 ± 0.05 times faster than zlib-ng.exe pip sync ./scripts/benchmark/req.txt
```
Closes#11885.
## Summary
Testing with `UV_PYTHON_INSTALL_DIR` environment variable has some
problems. This PR fix them.
- `UV_PYTHON_INSTALL_DIR` must be an absolute path.
- Cargo tries to find Python executables from each crates in test. If it
is relative path, cargo searches in different directories for each
tests.
- Skip the test asserting help messages.
- Clap shows the current value of the environment variables. If
`UV_PYTHON_INSTALL_DIR` is set, the test fails.
## Test Plan
<!-- How was it tested? -->
All tests pass with
`UV_PYTHON_INSTALL_DIR=/path/to/my/home/uv/target/testpython`.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
<!-- How was it tested? -->
To enforce the 100 character line limit in markdown files introduced in
https://github.com/astral-sh/uv/pull/5635, and to automate the
formatting of markdown files, i've added prettier and formatted our
markdown files with it.
I've excluded the changelog and the generated references documentation
from this for having too many changes, but we can also include them.
I'm not particular on which style we use. My main motivations are
(major) not having to reflow markdown files myself anymore and (minor)
consistence between all markdown files. I've chosen prettier for similar
reason as we chose black, it's a single good style that's automated and
shared in the community. I do prefer prettier's style of not breaking
inside of a link name though.
This PR is in two parts, the first adds prettier to CI and documents
using it, while the second actually formats the docs. When merge
conflicts arise, we can drop the last commit and regenerate it with `npx
prettier --prose-wrap always --write BENCHMARKS.md CONTRIBUTING.md
README.md STYLE.md docs/*.md docs/concepts/**/*.md docs/guides/**/*.md
docs/pip/**/*.md`.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
Removes the legacy `benchmark` directory (we'll always have it in Git)
and renames `bench` to `benchmark` for clarity. Fixes a variety of
commands and references.
## Summary
Spotted some issues in the settings documentation, and room for small
improvements by linking to PEPs/RFCs.
Also updating contribution documentation to mention that it's necessary
to enable the virtual environment before running `mkdocs serve`.
## Test Plan
Running documentation locally.
## Summary
I noticed the command to install multiple Python versions was wrong as
it was failing cause `toolchain` is not a known command.
I looked in the `ci.yml` workflow to see which command is used there and
updated the instructions accordingly.
## Test Plan
I just ran the command locally. :)
Whew this is a lot.
The user-facing changes are:
- `uv toolchain` to `uv python` e.g. `uv python find`, `uv python
install`, ...
- `UV_TOOLCHAIN_DIR` to` UV_PYTHON_INSTALL_DIR`
- `<UV_STATE_DIR>/toolchains` to `<UV_STATE_DIR>/python` (with
[automatic
migration](https://github.com/astral-sh/uv/pull/4735/files#r1663029330))
- User-facing messages no longer refer to toolchains, instead using
"Python", "Python versions" or "Python installations"
The internal changes are:
- `uv-toolchain` crate to `uv-python`
- `Toolchain` no longer referenced in type names
- Dropped unused `SystemPython` type (previously replaced)
- Clarified the type names for "managed Python installations"
- (more little things)
## Summary
We currently rely on libgit2 for most git-related functionality.
However, libgit2 has long-standing performance issues, as well as lags
significantly behind git in terms of new features. For these reasons we
now use the git CLI by default for fetching repositories
(https://github.com/astral-sh/uv/pull/1781). This PR completely drops
libgit2 in favor of the git CLI for all git-related functionality, which
should allow us to use features such as partial clones and sparse
checkouts in the future for performance.
There is also a lot of technical debt in the current git code as it's
mostly taken from Cargo. Switching to the git CLI *vastly* simplifies
the `uv-git` codebase.
Eventually we might want to look into switching to
[`gitoxide`](https://github.com/Byron/gitoxide), but it's currently too
immature for our use case.
Extends #3726
Moves toolchain storage out of `UV_BOOTSTRAP_DIR` (`./bin`) into the
proper user data directory as defined by #3726.
Replaces `UV_BOOTSTRAP_DIR` with `UV_TOOLCHAIN_DIR` for customization.
Installed toolchains will be discovered without opt-in, but the idea is
still that these are not yet user-facing.
See https://github.com/astral-sh/uv/issues/2617
Note this also includes:
- #2918
- #2931 (pending)
A first step towards Python toolchain management in Rust.
First, we add a new crate to manage Python download metadata:
- Adds a new `uv-toolchain` crate
- Adds Rust structs for Python version download metadata
- Duplicates the script which downloads Python version metadata
- Adds a script to generate Rust code from the JSON metadata
- Adds a utility to download and extract the Python version
I explored some alternatives like a build script using things like
`serde` and `uneval` to automatically construct the code from our
structs but deemed it to heavy. Unlike Rye, I don't generate the Rust
directly from the web requests and have an intermediate JSON layer to
speed up iteration on the Rust types.
Next, we add add a `uv-dev` command `fetch-python` to download Python
versions per the bootstrapping script.
- Downloads a requested version or reads from `.python-versions`
- Extracts to `UV_BOOTSTRAP_DIR`
- Links executables for path extension
This command is not really intended to be user facing, but it's a good
PoC for the `uv-toolchain` API. Hash checking (via the sha256) isn't
implemented yet, we can do that in a follow-up.
Finally, we remove the `scripts/bootstrap` directory, update CI to use
the new command, and update the CONTRIBUTING docs.
<img width="1023" alt="Screenshot 2024-04-08 at 17 12 15"
src="https://github.com/astral-sh/uv/assets/2586601/57bd3cf1-7477-4bb8-a8e9-802a00d772cb">
* Document good first issues
* Document `scripts` directory, as far as useful for contributors
* Remove compare with pip script, we don't need it anymore
I think this closes#817.
---------
Co-authored-by: Jo <10510431+j178@users.noreply.github.com>
Add a `UV_BOOTSTRAP_DIR` option to configure the python bootstrap
directory. This is helpful when working across multiple platforms in a
single IDE session.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Provides an example of local testing usage, to make it easier for new
developers who are less familiar with `cargo` / `rust` development best
practices.
## Test Plan
I followed the setup instructions earlier in the contributing guide,
then ran these 2 commands.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
This improves Contributing documentation to specifically mention `trace`
level logging can be obtained via `RUST_LOG=trace uv …` as mentioned
here:
https://github.com/astral-sh/uv/issues/1569#issuecomment-1951489407
## Test Plan
Compare the output of
```
uv pip install --verbose requests
```
and
```
RUST_LOG=trace uv pip install --verbose requests
```
## What
Fixes a missing link in `contributing.md` to the Python installation
section by rearranging the section and adding a 3rd-level heading for
#Python
## Closes
Closes#1431
First, replace all usages in files in-place. I used my editor for this.
If someone wants to add a one-liner that'd be fun.
Then, update directory and file names:
```
# Run twice for nested directories
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
find . -type d -print0 | xargs -0 rename s/puffin/uv/g
# Update files
find . -type f -print0 | xargs -0 rename s/puffin/uv/g
```
Then add all the files again
```
# Add all the files again
git add crates
git add python/uv
# This one needs a force-add
git add -f crates/uv-trampoline
```
Run `cargo test` on windows in CI, pulling the switch on tier 1 windows
support.
These changes make the bootstrap script virtually required for running
the tests. This gives us consistency between and CI, but it also locks
our tests to python-build-standalone and an articificial `PATH`.
I've deleted the shell bootstrap script in favor of only the python one,
which also runs on windows. I've left the (sym)link creation of the
bootstrap in place, even though it is not used by the tests anymore.
I've reactivated the three tests that would previously stack overflow by
doubling their stack sizes. The stack overflows only happen in debug
mode, so this is neither a user facing problem nor an actual problem
with our code and this workaround seems better than optimizing our code
for case that the (release) compiler can optimize much better for.
The handling of patch versions will be fixed in a follow-up PR.
Closes#1160Closes#1161
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
Profiling is already extensively documented in ruff, so we can just link
to that guide. I added `tracing-durations-export` to the guide because i
found it a useful tool for optimizing.
Replaces https://github.com/astral-sh/puffin/pull/1068 and #1070 which
were more complicated than I wanted.
- Introduces a `.python-versions` file which defines the Python versions
needed for development
- Adds a Bash script at `scripts/bootstrap/install` which installs the
required Python versions from `python-build-standalone` to `./bin`
- Checks in a `versions.json` file with metadata about available
versions on each platform and a `fetch-version` Python script derived
from `rye` for updating the versions
- Updates CI to use these Python builds instead of the `setup-python`
action
- Updates to the latest packse scenarios which require Python 3.8+
instead of 3.7+ since we cannot use 3.7 anymore and includes new test
coverage of patch Python version requests
- Adds a `PUFFIN_PYTHON_PATH` variable to prevent lookup of system
Python versions for isolation during development
Tested on Linux (via CI) and macOS (locally) — presumably it will be a
bit more complicated to do proper Windows support.
You need to install cmake on windows, so i added a hint about using
`pipx install cmake`, and some more general notes on building and
testing puffin.
See #817
From manual inspection, this dataset generated through the [libraries.io
API](https://libraries.io/api#project-search) seems more mainstream than
the current 8k one, which is also preserved. I've added the dataset to
the repo because the API requires an API key.
This docker container provides isolation of source distribution builds,
whether [intended to be
helpful](https://pypi.org/project/nvidia-pyindex/) or other more or less
malicious forms of host system modification.
Fixes#194
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
