## Summary
Follow up to https://github.com/astral-sh/uv/pull/15563
Closes https://github.com/astral-sh/uv/issues/13485
This is a first-pass at adding support for conditional support for Git
LFS between git sources, initial feedback welcome.
e.g.
```
[tool.uv.sources]
test-lfs-repo = { git = "https://github.com/zanieb/test-lfs-repo.git", lfs = true }
```
For context previously a user had to set `UV_GIT_LFS` to have uv fetch
lfs objects on git sources. This env var was all or nothing, meaning you
must always have it set to get consistent behavior and it applied to all
git sources. If you fetched lfs objects at a revision and then turned
off lfs (or vice versa), the git db, corresponding checkout lfs
artifacts would not be updated properly. Similarly, when git source
distributions were built, there would be no distinction between sources
with lfs and without lfs. Hence, it could corrupt the git, sdist, and
archive caches.
In order to support some sources being LFS enabled and other not, this
PR adds a stateful layer roughly similar to how `subdirectory` works but
for `lfs` since the git database, the checkouts and the corresponding
caching layers needed to be LFS aware (requested vs installed). The
caches also had to isolated and treated entirely separate when handling
LFS sources.
Summary
* Adds `lfs = true` or `lfs = false` to git sources in pyproject.toml
* Added `lfs=true` query param / fragments to most relevant url structs
(not parsed as user input)
* In the case of uv add / uv tool, `--lfs` is supported instead
* `UV_GIT_LFS` environment variable support is still functional for
non-project entrypoints (e.g. uv pip)
* `direct-url.json` now has an custom `git_lfs` entry under VcsInfo
(note, this is not in the spec currently -- see caveats).
* git database and checkouts have an different cache key as the sources
should be treated effectively different for the same rev.
* sdists cache also differ in the cache key of a built distribution if
it was built using LFS enabled revisions to distinguish between non-LFS
same revisions. This ensures the strong assumption for archive-v0 that
an unpacked revision "doesn't change sources" stays valid.
Caveats
* `pylock.toml` import support has not been added via git_lfs=true,
going through the spec it wasn't clear to me it's something we'd support
outside of the env var (for now).
* direct-url struct was modified by adding a non-standard `git_lfs`
field under VcsInfo which may be undersirable although the PEP 610 does
say `Additional fields that would be necessary to support such VCS
SHOULD be prefixed with the VCS command name` which could be interpret
this change as ok.
* There will be a slight lockfile and cache churn for users that use
`UV_GIT_LFS` as all git lockfile entries will get a `lfs=true` fragment.
The cache version does not need an update, but LFS sources will get
their own namespace under git-v0 and sdist-v9/git hence a cache-miss
will occur once but this can be sufficient to label this as breaking for
workflows always setting `UV_GIT_LFS`.
## Test Plan
Some initial tests were added. More tests likely to follow as we reach
consensus on a final approach.
For IT test, we may want to move to use a repo under astral namespace in
order to test lfs functionality.
Manual testing was done for common pathological cases like killing LFS
fetch mid-way, uninstalling LFS after installing an sdist with it and
reinstalling, fetching LFS artifacts in different commits, etc.
PSA: Please ignore the docker build failures as its related to depot
OIDC issues.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Co-authored-by: konstin <konstin@mailbox.org>
Part of https://github.com/astral-sh/uv/issues/4392
We shouldn't link to PyPI, and dropping the workspace-level
documentation link should mean that we get the auto-generated `docs.rs`
links.
## Summary
I spent time trying to figure out how to support this but came up empty.
It _seems_ like maybe the `DefaultFields` implementation in
`tracing-subscriber` uses debug formatting for fields...? So if you have
a string with ANSI codes, they end up printing as unformatted values? I
even reverted all our custom formatting in `logging.rs` and saw the same
thing.
Closes https://github.com/astral-sh/uv/issues/15840.
## Summary
Fixes issue #15190 where `uv sync --no-sources` fails to switch from
editable to registry package installations. The problem occurred because
the installer's satisfaction check didn't consider the `--no-sources`
flag when determining if an existing editable installation was
compatible with a registry requirement.
## Solution
Modified `RequirementSatisfaction::check()` to reject non-registry
installations when `SourceStrategy::Disabled` and the requirement is
from registry. Added `SourceStrategy` parameter threading through the
entire call chain from commands to the satisfaction check to ensure
consistent behavior between `uv sync --no-sources` and `uv pip install
--no-sources`.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
We've received several requests to validate that installed wheels match
the current Python platform. This isn't _super_ common, since it
requires that your platform changes in some meaningful way (e.g., you
switch from x86 to ARM), though in practice, it sounds like it _can_
happen in HPC environments. This seems like a good thing to do
regardless, so we now validate that the tags (as recoded in `WHEEL`) are
consistent with the current platform during installs.
Closes https://github.com/astral-sh/uv/issues/15035.
## Summary
This is causing some cyclic dependencies issues for me, because these
can be used in virtually _any_ crate (like `uv-install-wheel`), which
then means that all of `uv-configuration` becomes a dependency, etc. I
think this should be a leaf crate so that we can safely depend on it
anywhere.
## Summary
I've written a reasonably-long comment to explain what's going on here.
We should fix this, but it's better to continue using a
potentially-stale distribution than to panic.
Closes https://github.com/astral-sh/uv/issues/15386.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
This PR productionizes an idea I saw in
https://github.com/astral-sh/uv/issues/15248, which was added in Pixi:
https://github.com/prefix-dev/pixi/pull/4247. The core of the idea is
that if we install all build isolation-enabled packages first, and the
build isolation-disabled packages in a second phase, the sync is more
likely to "just work", because if all the build dependencies of the
build isolation-disabled packages are included as dependencies (as is
the case for `flash-attn`, at least), they'll be present.
This isn't really a silver bullet, because it requires that all the
build dependencies are included as first-party dependencies, and if you
have packages that want build isolation to be disabled but rely on other
packages that also require build isolation disabled, that won't work
either. I think `extra-build-dependencies` will be more robust and have
much better caching behavior, but this will get more cases right than
our current behavior, and I don't see any downsides.
Closes https://github.com/astral-sh/uv/issues/15301.
## Summary
With this PR, we track the settings that were used to build a wheel
(`--config-settings`, plus any `extra-build-dependencies` or
`extra-build-variables`) and write those to the `.dist-info` directory
upon install. This then allows us to "reject" already-installed wheels,
if the user changes the build dependencies or `--config-settings` (or,
crucially, if they use `match-runtime = true` and the resolution
changes).
Closes https://github.com/astral-sh/uv/issues/15218.
## Summary
This breaks up a cycle I'm running into in incorporating the build
configuration into our cache keys. This is actually a type that ends up
in the frontend build system, etc., so I think it makes more sense here
anyway (as opposed to `uv-configuration` which tend to be our own
user-facing types).
## Summary
Split the cleanup fixes from https://github.com/astral-sh/uv/pull/15196
into a separate PR for easier review.
This cleans up some minor env var usage / references throughout tests
and runtime code.
## Test Plan
Existing Tests. No functional changes.
## Summary
I should've noticed this during review -- my bad -- but it looks like
after lowering, we're converting back to `uv_pep508::Requirement`. This
is mostly okay, but it's lossy for some lowerings. For example, we lose
index pinning. With this PR, we now preserve the lowered types
(`Requirement`).
Closes https://github.com/astral-sh/uv/issues/15037.
## Summary
We weren't including these in the cache key when constructing the
install plan. We likely still read them from the cache later, but we may
have reported the wrong number of prepares, etc.
If a user specifies `-e /path/to/dir` and `/path/to/dir` in a `uv pip
install` command, we want the editable to "win" (rather than erroring
due to conflicting URLs). Unfortunately, this behavior meant that when
you requested a package as editable and non-editable in conflicting
groups, the editable version was _always_ used. This PR modifies the
requisite types to use `Option<bool>` rather than `bool` for the
`editable` field, so we can determine whether a requirement was
explicitly requested as editable, explicitly requested as non-editable,
or not specified (as in the case of `/path/to/dir` in a
`requirements.txt` file). In the latter case, we allow editables to
override the "unspecified" requirement.
If a project includes a path dependency twice, once with `editable =
true` and once without any `editable` annotation, those are now
considered conflicting URLs, and lead to an error, so I've marked this
change as breaking.
Closes https://github.com/astral-sh/uv/issues/14139.
## Summary
When refactoring the addition PR I accidentally introduced a bug where
the debug message would not be output if the default value is used.
cc @zanieb
## Summary
When installing packages on _very_ slow/overloaded systems it'spossible
to trigger bytecode compilation timeouts, which tends to happen in
environments such as Qemu (especially without KVM/virtio), but also on
systems that are simply overloaded. I've seen this in my Nix builds if I
for example am compiling a Linux kernel at the same time as a few other
concurrent builds.
By making the bytecode compilation timeout adjustable you can work
around such issues. I plan to set `UV_COMPILE_BYTECODE_TIMEOUT=0` in the
[pyproject.nix
builders](https://pyproject-nix.github.io/pyproject.nix/build.html) to
make them more reliable.
- Related issues
* https://github.com/astral-sh/uv/issues/6105
## Test Plan
Only manual testing was applied in this instance. There is no existing
automated tests for bytecode compilation timeout afaict.
Prior to this PR, there were numerous places where uv would leak
credentials in logs. We had a way to mask credentials by calling methods
or a recently-added `redact_url` function, but this was not secure by
default. There were a number of other types (like `GitUrl`) that would
leak credentials on display.
This PR adds a `DisplaySafeUrl` newtype to prevent leaking credentials
when logging by default. It takes a maximalist approach, replacing the
use of `Url` almost everywhere. This includes when first parsing config
files, when storing URLs in types like `GitUrl`, and also when storing
URLs in types that in practice will never contain credentials (like
`DirectorySourceUrl`). The idea is to make it easy for developers to do
the right thing and for the compiler to support this (and to minimize
ever having to manually convert back and forth). Displaying credentials
now requires an active step. Note that despite this maximalist approach,
the use of the newtype should be zero cost.
One conspicuous place this PR does not use `DisplaySafeUrl` is in the
`uv-auth` crate. That would require new clones since there are calls to
`request.url()` that return a `&Url`. One option would have been to make
`DisplaySafeUrl` wrap a `Cow`, but this would lead to lifetime
annotations all over the codebase. I've created a separate PR based on
this one (#13576) that updates `uv-auth` to use `DisplaySafeUrl` with
one new clone. We can discuss the tradeoffs there.
Most of this PR just replaces `Url` with `DisplaySafeUrl`. The core is
`uv_redacted/lib.rs`, where the newtype is implemented. To make it
easier to review the rest, here are some points of note:
* `DisplaySafeUrl` has a `Display` implementation that masks
credentials. Currently, it will still display the username when there is
both a username and password. If we think is the wrong choice, it can
now be changed in one place.
* `DisplaySafeUrl` has a `remove_credentials()` method and also a
`.to_string_with_credentials()` method. This allows us to use it in a
variety of scenarios.
* `IndexUrl::redacted()` was renamed to
`IndexUrl::removed_credentials()` to make it clearer that we are not
masking.
* We convert from a `DisplaySafeUrl` to a `Url` when calling `reqwest`
methods like `.get()` and `.head()`.
* We convert from a `DisplaySafeUrl` to a `Url` when creating a
`uv_auth::Index`. That is because, as mentioned above, I will be
updating the `uv_auth` crate to use this newtype in a separate PR.
* A number of tests (e.g., in `pip_install.rs`) that formerly used
filters to mask tokens in the test output no longer need those filters
since tokens in URLs are now masked automatically.
* The one place we are still knowingly writing credentials to
`pyproject.toml` is when a URL with credentials is passed to `uv add`
with `--raw`. Since displaying credentials is no longer automatic, I
have added a `to_string_with_credentials()` method to the `Pep508Url`
trait. This is used when `--raw` is passed. Adding it to that trait is a
bit weird, but it's the simplest way to achieve the goal. I'm open to
suggestions on how to improve this, but note that because of the way
we're using generic bounds, it's not as simple as just creating a
separate trait for that method.
Rustfmt introduces a lot of formatting changes in the 2024 edition. To
not break everything all at once, we split out the set of formatting
changes compatible with both the 2021 and 2024 edition by first
formatting with the 2024 style, and then again with the currently used
2021 style.
Notable changes are the formatting of derive macro attributes and lines
with overly long strings and adding trailing semicolons after statements
consistently.
## Summary
I don't know if I actually want to commit this, but I did it on the
plane last time and just polished it off (got it to compile) while
waiting to board.
## Summary
This crate is for standards-compliant types, but this is explicitly a
type that's custom to uv. It's also strange because we kind of want to
reference `IndexUrl` on the registry type, but that's in a crate that
_depends_ on `uv-pypi-types`, which to me is a sign that this is off.
konsti
Zanie Blue
samypr100
Zsolt Dollenstein
Charlie Marsh
chisato
adisbladis
John Mumm