## Summary
A revival of an old idea (#9344) that I have slightly more confidence in
now. I abandoned this idea because (1) it couldn't capture that, e.g.,
`platform_system == 'Windows' and sys_platform == 'foo'` (or some other
unknown value) are disjoint, and (2) I thought that Android returned
`"android"` for one of `sys_platform` or `platform_system`, which
would've made this logic incorrect.
However, it looks like Android... doesn't do that? And the values here
are almost always in a small, known set. So in the end, the tradeoffs
here actually seem pretty good.
Vis-a-vis our current solution, this can (e.g.) _simplify out_
expressions like `sys_platform == 'win32' or platform_system ==
'Windows'`.
Make the local packse workflow work again:
```
# In packse:
uv run --extra index --extra serve packse serve --no-hash scenarios &
# In uv:
UV_TEST_INDEX_URL="http://localhost:3141/simple/" ./scripts/scenarios/generate.py
```
Bugs fixed:
* The default scenario pattern didn't match anything.
* The snapshot update test command was wrong since the test
centralization
* Snapshot update failures would not be reported
When publishing, we currently ask the user to set `--publish-url` to the
upload URL and `--check-url` to the simple index URL, or the equivalent
configuration keys. But that's redundant with the `[[tool.uv.index]]`
declaration. Instead, we extend `[[tool.uv.index]]` with a `publish-url`
entry and allow passing `uv publish --index <name>`.
`uv publish --index <name>` requires the `pyproject.toml` to be present
when publishing, unlike using `--publish-url ... --check-url ...` which
can be used e.g. in CI without a checkout step. `--index` also always
uses the check URL feature to aid upload consistency.
The documentation tries to explain both approaches together, which
overlap for the check URL feature.
Fixes#8864
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
I added `crates/uv-python/create-mirror.py` to make it easy to download
all the needed files to create a mirror for Python distributions in an
offline environment.
the script also has an option to iterate over the git history of the
`download-metadata.json` to make sure we have all the files needed for
all the uv versions
## Test Plan
```
uv run create-mirror.py --from-all-history --os linux --arch x86_64 --name cpython
2024-10-25 01:31:12,973 - INFO - Starting download of 466 files.
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 466/466 [06:11<00:00, 1.26file/s]
Successfully downloaded: 466
now you can run UV_PYTHON_INSTALL_MIRROR='file:///home/meitar/dev/uv/crates/uv-python/mirror' uv python install
```
then checked (the `unshare` command make sure that the process don't
have any netwok)
```
UV_PYTHON_INSTALL_MIRROR=file:///home/meitar/dev/uv/crates/uv-python/mirror sudo -E unshare -n /home/meitar/.local/bin/uv python install 3.13
Searching for Python versions matching: Python 3.13
Installed Python 3.13.0 in 2.91s
+ cpython-3.13.0-linux-x86_64-gnu
```
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
When adding excludes, we usually don't want to include python cache
files. On the contrary, I haven't seen any project in my ecosystem
research that would want any of `__pycache__`, `*.pyc`, `*.pyo` to be
included. By moving them behind a `default-excludes` toggle, they are
always active even when defining custom excludes, but can be deactivated
if the user so chooses.
With includes and excludes being this small again, we can roll back the
include-exclude anchored difference to always using anchored globs (i.e.
you would need to use `**/build-*.h` below).
A pyproject.toml with custom settings with the change applied:
```toml
[project]
name = "foo"
version = "0.1.0"
readme = "README.md"
license-files = ["LICENSE*", "third-party-licenses/*"]
[tool.uv.build-backend]
# A file we need for the source dist -> wheel step, but not in the wheel itself (currently unused)
source-include = ["data/build-script.py"]
# A temporary or generated file we want to ignore
source-exclude = ["/src/foo/not-packaged.txt"]
# Headers are build-only
wheel-exclude = ["build-*.h"]
[tool.uv.build-backend.data]
scripts = "scripts"
data = "assets"
headers = "header"
[build-system]
requires = ["uv>=0.5.5,<0.6"]
build-backend = "uv"
```
When building the source distribution, we always need to include
`pyproject.toml` and the module, when building the wheel, we always
include the module but nothing else at top level. Since we only allow a
single module per wheel, that means that there are no specific wheel
includes. This means we have source includes, source excludes, wheel
excludes, but no wheel includes: This is defined by the module root,
plus the metadata files and data directories separately.
Extra source dist includes are currently unused (they can't end up in
the wheel currently), but it makes sense to model them here, they will
be needed for any sort of procedural build step.
This results in the following fields being relevant for inclusions and
exclusion:
* `pyproject.toml` (always included in the source dist)
* project.readme: PEP 621
* project.license-files: PEP 639
* module_root: `Path`
* source_include: `Vec<Glob>`
* source_exclude: `Vec<Glob>`
* wheel_exclude: `Vec<Glob>`
* data: `Map<KnownDataName, Path>`
An opinionated choice is that that wheel excludes always contain the
source excludes: Otherwise you could have a path A in the source tree
that gets included when building the wheel directly from the source
tree, but not when going through the source dist as intermediary,
because A is in source excludes, but not in the wheel excludes. This has
been a source of errors previously.
In the process, I fixed a bug where we would skip directories and only
include the files and were missing license due to absolute globs.
When trying to upload without a password but with the keyring, check
that the keyring has a password for the upload URL and username and warn
if it doesn't.
Fixes#8781
We were previously not uploading all metadata in the formdata of an
upload request in the legacy api. Notably, we were missing the PEP 639
license-files field.
I had to switch to pdm due to https://github.com/pypa/hatch/issues/1828
Allow including data files in wheels, configured through
`pyproject.toml`. This configuration is currently only read in the build
backend. We'd only start using it in the frontend when we're adding a
fast path.
Each data entry is a directory, whose contents are copied to the
matching directory in the wheel in
`<name>-<version>.data/(purelib|platlib|headers|scripts|data)`. Upon
installation, this data is moved to its target location, as defined by
<https://docs.python.org/3.12/library/sysconfig.html#installation-paths>:
- `data`: Installed over the virtualenv environment root. Warning: This
may override existing files!
- `scripts`: Installed to the directory for executables, `<venv>/bin` on
Unix or `<venv>\Scripts` on Windows. This directory is added to PATH
when the virtual environment is activated or when using `uv run`, so
this data type can be used to install additional binaries. Consider
using `project.scripts` instead for starting Python code.
- `headers`: Installed to the include directory, where compilers
building Python packages with this package as built requirement will
search for header files.
- `purelib` and `platlib`: Installed to the `site-packages` directory.
It is not recommended to uses these two options.
For simplicity, for now we're just defining a directory to be copied for
each data directory, while using the glob based include mechanism in the
background. We thereby introduce a third mechanism next to the main
includes and the PEP 639 mechanism, which is not what we should finalize
on.
When building source distributions, we need to include the readme, so it
can become part the METADATA body when building the wheel. We also need
to support the license files from PEP 639. When building the source
distribution, we copy those file relative to their origin, when building
the wheel, we copy them to `.dist-info/licenses`.
The test for idempotence in wheel building is merged into the file
listing test, which also covers that source tree -> source dist -> wheel
is equivalent to source tree -> wheel, though we do need to check for
file inclusion stronger here.
Best reviewed commit-by-commit
A first milestone: source tree -> source dist -> wheel -> install works.
This PR adds a test for this.
There's obviously a lot still missing, including basics such as the
Readme inclusion.
Implement a full working version of local version semantics. The (AFAIA)
major move towards this was implemented in #2430. This added support
such that the version specifier `torch==2.1.0+cpu` would install
`torch@2.1.0+cpu` and consider `torch@2.1.0+cpu` a valid way to satisfy
the requirement `torch==2.1.0` in further dependency resolution.
In this feature, we more fully support local version semantics. Namely,
we now allow `torch==2.1.0` to install `torch@2.1.0+cpu` regardless of
whether `torch@2.1.0` (no local tag) actually exists.
We do this by adding an internal-only `Max` value to local versions that
compare greater to all other local versions. Then we can translate
`torch==2.1.0` into bounds: greater than 2.1.0 with no local tag and
less than 2.1.0 with the `Max` local tag.
Depends on https://github.com/astral-sh/packse/pull/227.
## Summary
This PR declares and documents all environment variables that are used
in one way or another in `uv`, either internally, or externally, or
transitively under a common struct.
I think over time as uv has grown there's been many environment
variables introduced. Its harder to know which ones exists, which ones
are missing, what they're used for, or where are they used across the
code. The docs only documents a handful of them, for others you'd have
to dive into the code and inspect across crates to know which crates
they're used on or where they're relevant.
This PR is a starting attempt to unify them, make it easier to discover
which ones we have, and maybe unlock future posibilities in automating
generating documentation for them.
I think we can split out into multiple structs later to better organize,
but given the high influx of PR's and possibly new environment variables
introduced/re-used, it would be hard to try to organize them all now
into their proper namespaced struct while this is all happening given
merge conflicts and/or keeping up to date.
I don't think this has any impact on performance as they all should
still be inlined, although it may affect local build times on changes to
the environment vars as more crates would likely need a rebuild. Lastly,
some of them are declared but not used in the code, for example those in
`build.rs`. I left them declared because I still think it's useful to at
least have a reference.
Did I miss any? Are their initial docs cohesive?
Note, `uv-static` is a terrible name for a new crate, thoughts? Others
considered `uv-vars`, `uv-consts`.
## Test Plan
Existing tests
As per
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html
Before that, there were 91 separate integration tests binary.
(As discussed on Discord — I've done the `uv` crate, there's still a few
more commits coming before this is mergeable, and I want to see how it
performs in CI and locally).
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
I started learning `uv` by inspecting the source code.
I've noticed that your shell scripts are very good! Which is rare!
## Test Plan
I propose to add `shellcheck` to the CI.
It is a great tool to help finding bugs and style issues in shell code.
Techincal details:
- This CI job will only run when any `.sh` files are changed (or the job
definition file)
- It takes just several seconds even on local machine:
```
» time shellcheck -S style **/*.sh
shellcheck -S style **/*.sh 0.02s user 0.05s system 61% cpu 0.123 total
```
- It is easy to use, for example: I just fixed the single problem you
had in your code with `# shellcheck disable=SC1091`
- I am using this tool for around 8 years now and didn't have any
issues. Examples:
ca899f3b69/.github/workflows/test.yml (L22-L27)
and
https://github.com/wemake-services/wemake-django-template/blob/master/.github/workflows/shellcheck.yml
But, I understand that build / lint tools are very subjective. So, feel
free to close :)
Merge before #6375 (see
https://github.com/astral-sh/uv/pull/6375#discussion_r1776975298)
This is testing a couple of things at once, and I'm happy to strip back
to something more simple (i.e., "just" printing).
However, I figured it would be useful to ensure that the PEP 723
metadata correctly works as well as that
additional arguments are correctly forwarded to the Python program.
cc: @burntsushi
Previously, we had dropped extras from the requirements when converting
to poetry, skewing the results towards poetry for the
`apache-airflow[all]` benchmark.
## Summary
This PR changes the definition of `--locked` from:
> Produces the same `Lock`
To:
> Passes `Lock::satisfies`
This is a subtle but important difference. Previous, if
`Lock::satisfies` failed, we would run a resolution, then do
`existing_lock == lock`. If the two weren't equal, and `--locked` was
specified, we'd throw an error.
The equality check is hard to get right. For example, it means that we
can't ship #6076 without changing our marker representation, since the
deserialized lockfile "loses" some of the internal marker state that
gets accumulated during resolution.
The downside of this change is that there could be scenarios in which
`uv lock --locked` fails even though the lockfile would actually work
and the exact TOML would be unchanged. But... I think it's ok if
`--locked` fails after the user modifies something?
## Summary
Historically, in order to "resolve from a lockfile", we've taken the
lockfile, used it to pre-populate the in-memory metadata index, then run
a resolution. If the resolution didn't match our existing resolution, we
re-resolved from scratch.
This was an appealing approach because (in theory) it didn't require any
dedicated logic beyond pre-populating the index. However, it's proven to
be _really_ hard to get right, because it's a stricter requirement than
we need. We just need the current lockfile to _satisfy_ the requirements
provided by the user. We don't actually need a second resolution to
produce the exact same result. And it's not uncommon that this second
resolution differs, because we seed it with preferences, which
fundamentally changes its course. We've worked hard to minimize those
"instabilities", but they're still present.
The approach here is intended to be much simpler. Instead of resolving
from the lockfile, we just check if the current resolution satisfies the
state of the workspace. Specifically, we check if the lockfile (1)
contains all the relevant members, and (2) matches the metadata for all
dependencies, recursively. (We skip registry dependencies, assuming that
they're immutable.)
This may actually be too conservative, since we can have resolutions
that satisfy the requirements, even if the requirements have changed
slightly. But we want to bias towards correctness for now.
My hope is that this scheme will be more performant, simpler, and more
robust.
Closes https://github.com/astral-sh/uv/issues/6063.
## Summary
Now, if you resolve against a registry, then swap it out for another, we
won't reuse the lockfile. (If you don't provide any registry
configuration, then we won't enforce this, so that `uv lock --index-url
foo` and `uv lock` is stable.)
Closes https://github.com/astral-sh/uv/issues/5920.
Currently, the entry for a package+version+source table is called
`distribution`. That is incorrect, the `sdist` and `wheel` fields inside
of that table are distributions, the table itself is for a package. We
also align ourselves closer with PEP 751.
I went through `lock.rs` and renamed all occurrences of "distribution"
that actually referred to a "package".
This change invalidates all existing lockfiles.
Bikeshedding: Do we call it `package` or `packages`? See also
https://github.com/python/peps/pull/3877
`package` is nice because it looks like a header:
```toml
[[package]]
name = "anyio"
version = "4.3.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "idna" },
{ name = "sniffio" },
]
sdist = { url = "3970183622d484d08e3285104333d3/anyio-4.3.0.tar.gz", hash = "sha256:f75253795a87df48568485fd18cdd2a3fa5c4f7c5be8e5e36637733fce06fed6", size = 159642 }
wheels = [
{ url = "2f20c40b45242c0b33774da0e2e34f/anyio-4.3.0-py3-none-any.whl", hash = "sha256:048e05d0f6caeed70d731f3db756d35dcc1f35747c8c403364a8332c630441b8", size = 85584 },
]
```
`packages` is nice because the field is not a single entry, but a list.
2/3 for https://github.com/astral-sh/uv/issues/4893
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
To enforce the 100 character line limit in markdown files introduced in
https://github.com/astral-sh/uv/pull/5635, and to automate the
formatting of markdown files, i've added prettier and formatted our
markdown files with it.
I've excluded the changelog and the generated references documentation
from this for having too many changes, but we can also include them.
I'm not particular on which style we use. My main motivations are
(major) not having to reflow markdown files myself anymore and (minor)
consistence between all markdown files. I've chosen prettier for similar
reason as we chose black, it's a single good style that's automated and
shared in the community. I do prefer prettier's style of not breaking
inside of a link name though.
This PR is in two parts, the first adds prettier to CI and documents
using it, while the second actually formats the docs. When merge
conflicts arise, we can drop the last commit and regenerate it with `npx
prettier --prose-wrap always --write BENCHMARKS.md CONTRIBUTING.md
README.md STYLE.md docs/*.md docs/concepts/**/*.md docs/guides/**/*.md
docs/pip/**/*.md`.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
Gives you a nice error message if you attempt to sync with, e.g., `-p
3.8` when that version is supported by at least one workspace member,
but your project's minimum requirement is `>=3.12`
Closes https://github.com/astral-sh/uv/issues/5662.
This `pyproject.toml`, translated from
0e4a1c3401/setup.py,
has found a number of bugs in blue jay, so i'm adding it to uv as a
development test case. (The file is too large to make it a test by
itself.)
## Summary
This PR deprecates the `--isolated` flag. The treatment varies across
the APIs:
- For non-preview APIs, we warn but treat it as equivalent to
`--no-config`.
- For preview APIs, we warn and ignore it, with two exceptions...
- For `tool run` and `run` specifically, we don't even warn, because we
can't differentiate the command-specific `--isolated` from the global
`--isolated`.
Every packse version update is currently causing a huge diff (the size
of the `lock_scenarios.rs` diff in this PR). By redacting the version
from the snapshots, we will only have the actual change in the diff and
not the redundant version change noise.
The second commit moves all remaining packse url arg values to
`common/mod.rs`, which acts as a single source of truth for the packse
version.
## Summary
Removes the legacy `benchmark` directory (we'll always have it in Git)
and renames `bench` to `benchmark` for clarity. Fixes a variety of
commands and references.
## Summary
This PR adds support for `uv lock` and `uv sync` in the standardized
benchmarks script.
Part of: https://github.com/astral-sh/uv/issues/5263.
## Test Plan
For example:
```sh
python scripts/bench/__main__.py --uv-project --benchmark resolve-cold ./scripts/requirements/trio.in --verbose
```
## Summary
You can now add `managed = false` under `[tool.uv]` in a
`pyproject.toml` to explicitly opt out of the project and workspace
APIs.
If a project sets `managed = false`, we will (1) _not_ discover it as a
workspace root, and (2) _not_ discover it as a workspace member (similar
to using `exclude` in the workspace parent).
Closes https://github.com/astral-sh/uv/issues/4551.
This PR refactors the command creation in the test suite to remove the
duplication.
**1)** We add the same set of test stubbing args to almost any uv
invocation in the tests:
```rust
command
.arg("--cache-dir")
.arg(self.cache_dir.path())
.env("VIRTUAL_ENV", self.venv.as_os_str())
.env("UV_NO_WRAP", "1")
.env("HOME", self.home_dir.as_os_str())
.env("UV_TOOLCHAIN_DIR", "")
.env("UV_TEST_PYTHON_PATH", &self.python_path())
.current_dir(self.temp_dir.path());
if cfg!(all(windows, debug_assertions)) {
// TODO(konstin): Reduce stack usage in debug mode enough that the tests pass with the
// default windows stack of 1MB
command.env("UV_STACK_SIZE", (8 * 1024 * 1024).to_string());
}
```
Centralizing these into a `TestContext::add_shared_args` method removes
them from everywhere.
**2)** Prefix all `TextContext` methods of the pip interface with
`pip_`. This is now necessary due to `uv sync` vs. `uv pip sync`.
**3)** Move command creation in the various test files into dedicated
functions or methods to avoid repeating the arguments. Except for error
message tests, there should be at most one `Command::new(get_bin())`
call per test file. `EXCLUDE_NEWER` is exclusively used in
`TestContext`.
---
I'm considering adding a `TestCommand` on top of these changes (in
another PR) that holds a reference to the `TextContext`, has
`add_shared_args` as a method and uses `Fn(Self) -> Self` instead of
`Fn(&mut Self) -> Self` for methods to improved chaining.
Make use of GitHub's markdown rendering
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
<!-- How was it tested? -->
packse has the ability to specify a project wide Requires-Python
constraint, but our lock template wasn't forwarding this to the
corresponding pyproject.toml. This update makes that happen.
I tweaked rooster to allow sections to be overridden from the CLI so we
can generate a separate preview changelog
See https://github.com/zanieb/rooster/pull/43 for the rooster changes
needed
I tested `./scripts/release.sh` as well.
The basic idea here is to make it so forking can only ever result in a
resolution that, for a particular marker environment, will only install
at most one version of a package. We can guarantee this by ensuring we
only fork on conflicting dependency specifications only when their
corresponding markers are completely disjoint. If they aren't, then
resolution _must_ find a single version of the package in the
intersection of the two dependency specifications.
A test for this case has been added to packse here:
https://github.com/astral-sh/packse/pull/182. Previously, that test
would result in a resolution with two different unconditional versions
of the same package. With this change, resolution fails (as it should).
A commit-by-commit review should be helpful here, since the first commit
is a refactor to make the second commit a bit more digestible.
## Summary
If `Requires-Python` is omitted in `uv lock` or `uv run`, we now warn
and default to `>=` the current minor version.
Closes https://github.com/astral-sh/uv/issues/4050.
This commit adds a template and does some light surgery on `generate.py`
to make use of that template. In particular, the universal tests require
using the "workspace"-aware version of `uv`, so we can't use the
existing `uv pip {compile,install}` tests.
This is just the result of running
./scripts/sync_scenarios.sh
From the root of the `uv` repository.
When I initially ran this, it produced some tests with snapshots that
weren't being updated. It turned out this was because the tests weren't
running, as they were gated behind the `python-patch` feature. In this
commit, we add `python-patch` to our `cargo insta` command, which should
update all relevant snapshots.
There are still some superfluous updates as a result of a spell checker
being run on generated files, but
The workspace test directories can be used both in tests and directly
for developing/debugging. In the latter, we shouldn't copy the venv and
the lockfile when running tests. Using the ignore crate over manual
recursion we exclude those files.
## Summary
There are a few behavior changes in here:
- We now enforce `--require-hashes` for editables, like pip. So if you
use `--require-hashes` with an editable requirement, we'll reject it. I
could change this if it seems off.
- We now treat source tree requirements, editable or not (e.g., both `-e
./black` and `./black`) as if `--refresh` is always enabled. This
doesn't mean that we _always_ rebuild them; but if you pass
`--reinstall`, then yes, we always rebuild them. I think this is an
improvement and is close to how editables work today.
Closes#3844.
Closes#2695.
Add workspace support when using `-r <path>/pyproject.toml` or `-e
<path>` in the pip interface. It is limited to all-editable
static-metadata workspaces, and tests only include a single main
workspace, ignoring path dependencies in another workspace. This can be
considered the MVP for workspace support: You can create a workspace,
you can install from it, but some options and conveniences are still
missing. I'll file follow-up tickets (support in lockfiles, support path
deps in other workspace, #3625)
There is also support in `uv run`, but we need
https://github.com/astral-sh/uv/issues/3700 first to properly support
using different current projects in the bluejay interface, currently the
resolution and therefore the lockfile depends on the current project.
I'd do this change first (it's big enough already), then #3700, and then
add workspace support properly to bluejay.
Fixes#3404
Charlie Marsh
Zanie Blue
konsti
Meitar Reihan
Eric Mark Martin
samypr100
Amos Wenger
sobolevn
Trevor Manz
Ahmed Ilyas
Andrew Gallant
Markus Hofbauer