## Summary
This PR enables use of the Windows Store Pythons even with `py` is not
installed. Specifically, we need to ensure that the `python.exe` and
`python3.exe` executables installed into the
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApp` directory _are_ used
when they're not "App execution aliases" (which merely open the Windows
Store, to help you install Python).
When `py` is installed, this isn't strictly necessary, since the
"resolved" executables are discovered via `py`. These look like
`C:\Users\crmar\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbs5n2kfra8p0\python.exe`.
Closes https://github.com/astral-sh/uv/issues/2264.
## Test Plan
- Removed all Python installations from my Windows machine.
- Uninstalled `py`.
- Enabled "App execution aliases".
- Verified that for both `cargo run venv --python python.exe` and `cargo
run venv --python python3.exe`, `uv` exited with a failure that no
Python could be found.
- Installed Python 3.10 via the Windows Store.
- Verified that the above commands succeeded without error.
- Verified that `cargo run venv --python python3.10.exe` _also_
succeeded.
- Now that `packse` is being published to PyPI we can install it from
there.
- Tweaks the tooling around scenario updates to manage a temporary
virtual environment for you.
- Makes use of a new index URL
- Includes local version segment scenarios (supersedes
https://github.com/astral-sh/uv/pull/2022)
Sometimes, the first time we read from the stdout of the bytecode
compiler python subprocess, we get an empty string back (no newline). If
we try to write to stdin, it will often be a broken pipe (#2245). After
we got an empty string the first time, we will get the same empty string
if we read a line again.
The details of this behavior are mysterious to me, but it seems that it
can be identified by the first empty string. We check by inserting
starting with a `Ready` message on the Python side. When we encounter
the broken state, we discard the interpreter and try again.
We have to introduce a third timeout check for the interpreter launch
itself.
Minimized test script:
```bash
#!/usr/bin/env bash
set -euo pipefail
while true; do
date --iso-8601=seconds # Progress indicator
rm -rf testenv
target/profiling/uv venv testenv -q --python 3.12
VIRTUAL_ENV=$PWD/testenv target/profiling/uv pip install -q --compile wheel==0.42.0
done
```
Run as
```
cargo build --profile profiling && bash compile_bug.sh
```
Fixes#2245
## Summary
In #2102, I did some refactor, and changed a method to return the Python
executable path rather than the parent directory path. But I missed this
one codepath for Conda on Windows.
Closes https://github.com/astral-sh/uv/issues/2269.
## Test Plan
- Installed micromamba on my Windows machine.
- Reproduced the failure in the linked issue.
- Verified that `python.exe` exists at `${CONDA_PREFIX}\python.exe`.
- Ran with this change; installed successfully.
## Summary
In the list of tags produced by `Tags::from_env`, higher-priority tags
are expected to come earlier in the list. Right now, though, we push
tags like `py38` before `py312`. So if you run `cargo run pip install
multiprocess -n --reinstall --verbose` on Python 3.12, you get the
`py38` wheel rather than the `py32` wheel.
Closes https://github.com/astral-sh/uv/issues/2261.
## Summary
This PR adds support for pip's `--no-build-isolation`. When enabled,
build requirements won't be installed during PEP 517-style builds, but
the source environment _will_ be used when executing the build steps
themselves.
Closes https://github.com/astral-sh/uv/issues/1715.
Fix parsing `pytest;'4.0'>=python_version`, where previously the
operator and the variable were incorrectly tokenized as one invalid
operator.
Fixes#2247
Running the pep508_rs tests was failing due to uv-fs depending on
`fs_err::tokio` even when not selected. But the function that used it is
unused anyway, so i removed it.
## Summary
It turns out that by keeping the `RECORD` file open, older versions of
Windows mark it for deletion, but don't allow it to be deleted until
it's closed. As such, we end up leaving the `.dist-info` directory
around, since it appears non-empty; but once the program terminates, we
_do_ delete `RECORD`, leaving it empty. This then creates the impression
that a package exists where it does not.
Closes https://github.com/astral-sh/uv/issues/2074.
## Summary
If we fallback to streaming the wheel (because the registry doesn't
support range requests), we currently don't cache the metadata at all.
This PR fixes that, ensuring that we cache based on the same HTTP
policies, etc.
## Summary
We're seeing reports that Sonatype Nexus isn't working with cached data.
Users are reporting 304 responses that show "Found modified response..."
path in the logs. I can't reproduce this on latest Sonatype Nexus, but
my best guess is that there's a 304 response that is failing our
validators, and we try to use that as if it's a "complete" response?
Closes https://github.com/astral-sh/uv/issues/1754.
## Summary
Add netrc support to the uv-client.
closes#1405
## Test Plan
I've added a corresponding test case to validate the correct header.
Furthermore a tested it against a real world private repository.
## Summary
Per the
[`EXTERNALLY-MANAGED`](https://packaging.python.org/en/latest/specifications/externally-managed-environments/)
spec, installers SHOULD add a `--break-system-packages` flag to allow
users to override the package manager warnings raised by
`EXTERNALLY-MANAGED`. This PR adds the flag to comply with the spec, and
enable system Python installs on newer versions of certain
distributions.
While this flag feels kind of bad, it's not necessarily a change in
behavior. We _already_ allow installing into these system distributions
-- it's just that `EXTERNALLY-MANAGED` doesn't exist for distributions
that were packaged prior to the spec, so we don't run into this problem.
Closes https://github.com/astral-sh/uv/issues/2234.
## Summary
This is no longer necessary as `AsyncHttpRangeReader` now accepts
`ClientWithMiddleware` -- which is good, because it means all relevant
middleware will be enforced (like offline, or `.netrc` in the future).
## Summary
Fix computing replacements pattern for pip_list tests to count
characters in the original directory string rather than the
regex::escape'd string. The latter yields incorrect results if the
workspace path contains characters such as `-` or `.`.
Fixes#2232
## Test Plan
`cargo test --test pip_list` in a directory named `uv-test` to provoke
the bug.
## Summary
Adjust maturing version to use env var for reusability in the workflow.
## Test Plan
CI `Build binaries` workflow uses correct version and passes
## Summary
Allow using http(s) urls for constraints and requirements files handed
to the CLI, by handling paths starting with `http://` or `https://`
differently. This allows commands for such as: `uv pip install -c
https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.8.txt
requests`.
closes#1332
## Test Plan
Testing install using a `constraints.txt` file hosted on github in the
airflow repository:
fbdc2eba8e/crates/uv/tests/pip_install.rs (L1440-L1484)
## Advice Needed
- filesystem/http dispatch is implemented at a relatively low level (at
`crates/uv-fs/src/lib.rs#read_to_string`). Should I change some naming
here so it is obvious that the function is able to dispatch?
- I kept the CLI argument for -c and -r as a PathBuf, even though now it
is technically either a path or a url. We could either keep this as is
for now, or implement a new enum for this case? The enum could then
handle dispatch to files/http.
- Using another abstraction layer like
https://docs.rs/object_store/latest/object_store/ for the
files/urls/[s3] could work as well, though I ran into a bug during
testing which I couldn't debug
## Summary
Follow up from discussion in https://github.com/astral-sh/uv/pull/2223
Detect CMD.exe by checking if `PROMPT` env var is set on windows,
otherwise assume it's PowerShell.
Note, this will not work if user modifies their system env vars to
include `PROMPT` by default or if they launch nested PowerShell from
Command Prompt (e.g. `Developer PowerShell for VS 2022`).
## Test Plan
Only tested locally, although we try to add some CI tests that
specifically use CMD.exe
Command Prompt
```
Microsoft Windows [Version 10.0.19044.3086]
(c) Microsoft Corporation. All rights reserved.
Z:\Users\samypr100\dev\uv>Z:\Users\samypr100\.cargo\bin\cargo.exe +stable run --color=always -- venv "Foo Bar"
Finished dev [unoptimized + debuginfo] target(s) in 0.69s
Running `target\debug\uv.exe venv "Foo Bar"`
Using Python 3.12.2 interpreter at: Z:\Users\samypr100\AppData\Local\Programs\Python\Python312\python.exe
Creating virtualenv at: Foo Bar
Activate with: "Foo Bar\Scripts\activate"
```
Power Shell
```
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
Try the new cross-platform PowerShell https://aka.ms/pscore6
PS Z:\Users\samypr100\dev\uv>Z:\Users\samypr100\.cargo\bin\cargo.exe +stable run --color=always -- venv "Foo Bar"
Finished dev [unoptimized + debuginfo] target(s) in 0.63s
Running `target\debug\uv.exe venv "Foo Bar"`
Using Python 3.12.2 interpreter at: Z:\Users\samypr100\AppData\Local\Programs\Python\Python312\python.exe
Creating virtualenv at: Foo Bar
Activate with: & "Foo Bar\Scripts\activate"
```
## Summary
Implementation for https://github.com/astral-sh/uv/issues/1594
The output will contain only the name, version and location of the
packages for now but it should be extendable to include other
information in the future.
Quite inexperienced with Rust, so please forgive me if there are things
that obviously don't make sense 😭
## Test Plan
Added a bunch of unit tests. The exit code behavior matches `pip`'s
behavior:
- When the package is found -> exit code 0
- When the package isn't found -> exit code 1
- When one package is found but another isn't -> exit code 0
Implements `pip list --format=freeze` and `pip list --format=json`
Closes https://github.com/astral-sh/uv/issues/1970
## Test Plan
Extended existing `pip list` tests to match output.
Need to look at escaping in the Windows test 🪟
## Summary
`pip` uses `sysconfig` for Python 3.10 and later by default; however, it
falls back to `distutils` for earlier Python versions, and distros can
actually tell `pip` to continue falling back to `distutils` via the
`_PIP_USE_SYSCONFIG` variable.
By _always_ using `sysconfig`, we're doing the wrong then when
installing into some system Pythons, e.g., on Debian prior to Python
3.10.
This PR modifies our logic to mirror `pip` exactly, which is what's been
recommended to me as the right thing to do.
Closes https://github.com/astral-sh/uv/issues/2113.
## Test Plan
Most notably, the new Debian tests pass here (which fail on main:
https://github.com/astral-sh/uv/pull/2144).
I also added Pyston as a second stress-test.
## Summary
This PR migrates our virtualenv creation from a setup that assumes prior
knowledge of the correct paths, to a technique borrowed from
`virtualenv` whereby we use `sysconfig` and `distutils` to determine the
paths. The general trick is to grab the expected paths with `sysconfig`,
then make them all relative, then make them absolute for a given
directory.
Closes#2095.
Closes#2153.
## Summary
This makes `--python python3` and `--python 3.10` more consistent on
Windows.
Closes https://github.com/astral-sh/uv/issues/2213.
## Test Plan
Ran `cargo run venv --python python3.12` with the Windows Store Python.
## Summary
We have logic in `python_query.rs` to filter out Windows Store shims
when you use invocations like `-p 3.10`, but not `--python python3`,
which is uncommon but allowed on Windows.
Closes#2211.
## Summary
Our Windows shim detection wasn't catching shims like `python3.12.exe`.
Closes#2208.
## Test Plan
Installed Python 3.12 via the Windows Store; verified that `cargo run
venv --python 3.12` failed before but passes after this change.
## Summary
When determining "direct" dependencies, we need to ensure that we
respect markers. In the linked issue, the user had an optional
dependency like:
```toml
[project.optional-dependencies]
dev = [
"setuptools>=64",
"setuptools_scm>=8"
]
```
By not respecting markers, we tried to resolve `setuptools` to the
lowest-available version. However, since `setuptools>=64` _isn't_
enabled (since it's optional), we won't respect _that_ constraint.
To be consistent, we need to omit optional dependencies just as we will
at resolution time.
Closes https://github.com/astral-sh/uv/issues/2203.
## Test Plan
`cargo test`
Follow-up to #2086: Don't use timeouts for the entire workers, but only
for the section that's about communicating with the (potentially broken)
`python` subprocess. I've also raised the timeout to 60s.
Add a `--compile` option to `pip install` and `pip sync`.
I chose to implement this as a separate pass over the entire venv. If we
wanted to compile during installation, we'd have to make sure that
writing is exclusive, to avoid concurrent processes writing broken
`.pyc` files. Additionally, this ensures that the entire site-packages
are bytecode compiled, even if there are packages that aren't from this
`uv` invocation. The disadvantage is that we do not update RECORD and
rely on this comment from [PEP 491](https://peps.python.org/pep-0491/):
> Uninstallers should be smart enough to remove .pyc even if it is not
mentioned in RECORD.
If this is a problem we can change it to run during installation and
write RECORD entries.
Internally, this is implemented as an async work-stealing subprocess
worker pool. The producer is a directory traversal over site-packages,
sending each `.py` file to a bounded async FIFO queue/channel. Each
worker has a long-running python process. It pops the queue to get a
single path (or exists if the channel is closed), then sends it to
stdin, waits until it's informed that the compilation is done through a
line on stdout, and repeat. This is fast, e.g. installing `jupyter
plotly` on Python 3.12 it processes 15876 files in 319ms with 32 threads
(vs. 3.8s with a single core). The python processes internally calls
`compileall.compile_file`, the same as pip.
Like pip, we ignore and silence all compilation errors
(https://github.com/astral-sh/uv/issues/1559). There is a 10s timeout to
handle the case when the workers got stuck. For the reviewers, please
check if i missed any spots where we could deadlock, this is the hardest
part of this PR.
I've added `uv-dev compile <dir>` and `uv-dev clear-compile <dir>`
commands, mainly for my own benchmarking. I don't want to expose them in
`uv`, they almost certainly not the correct workflow and we don't want
to support them.
Fixes#1788Closes#1559Closes#1928
