Part of https://github.com/astral-sh/uv/issues/4392
We shouldn't link to PyPI, and dropping the workspace-level
documentation link should mean that we get the auto-generated `docs.rs`
links.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
Hey devs, great tool as always, you're doing amazing work.
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Adds the following fields to the `[package]` table of `Cargo.toml` files
where they were missing:
```toml
rust-version = { workspace = true }
homepage = { workspace = true }
documentation = { workspace = true }
repository = { workspace = true }
authors = { workspace = true }
license = { workspace = true }
```
Most crates already had these fields, this just aligns the rest for
consistency.
This also resolves the warnings from `cargo-deny` when using `uv` crates
as dependencies in Pixi.
## Test Plan
No tests needed, this only updates metadata.
## Summary
This PR enables users to mark a URL as an S3 endpoint, at which point uv
will sign requests to that URL by detecting credentials from the
standard AWS environment variables, configuration files, etc.
Signing is handled by the
[reqsign](https://docs.rs/reqsign/latest/reqsign/) crate, which we can
also use in the future to sign requests for other providers.
## Summary
This PR adds support for pyx to `uv auth login`, `uv auth logout`, and
`uv auth token`. These are generic uv commands that can be used to store
credentials for arbitrary indexes and other URLs, but we include a
fast-path for pyx that initiates the appropriate login or logout flow.
We're not sure what the best way to expose the native store to users is
yet and it's a bit weird that you can use this in the `uv auth` commands
but can't use any of the other keyring provider options. The simplest
path forward is to just not expose it to users as a keyring provider,
and instead frame it as a preview alternative to the plaintext uv
credentials store. We can revisit the best way to expose configuration
before stabilization.
Note this pull request retains the _internal_ keyring provider
implementation — we can refactor it out later but I wanted to avoid a
bunch of churn here.
Adds locking of the credentials store for concurrency safety. It's
important to hold the lock from read -> write so credentials are not
dropped during concurrent writes.
I opted not to attach the lock to the store itself. Instead, I return
the lock on read and require it on write to encourage safe use. Maybe
attaching the source path to the store struct and adding a `lock(&self)`
method would make sense? but then you can forget to take the lock at the
right time. The main problem with the interface here is to write a _new_
store you have to take the lock yourself, and you could make a mistake
by taking a lock for the wrong path or something. The fix for that would
be to introduce a new `CredentialStoreHandle` type or something, but
that seems overzealous rn. We also don't eagerly drop the lock on token
read, although we could.
Adds a default plain text storage mechanism to `uv auth`.
While we'd prefer to use the system store, the "native" keyring support
is experimental still and I don't want to ship an unusable interface.
@geofft also suggested that the story for secure credential storage is
much weaker on Linux than macOS and Windows and felt this approach would
be needed regardless.
We'll switch over to using the native keyring by default in the future.
On Linux, we can now fallback to a plaintext store the secret store is
not configured, which is a nice property.
Right now, we store credentials in a TOML file in the uv state
directory. I expect to also read from the uv config directory in the
future, but we don't need it immediately.
Picks up the work from
- #14559
- https://github.com/astral-sh/uv/pull/14896
There are some high-level changes from those pull requests
1. We do not stash seen credentials in the keyring automatically
2. We use `auth login` and `auth logout` (for future consistency)
3. We add a `token` command for showing the credential that will be used
As well as many smaller changes to API, messaging, testing, etc.
---------
Co-authored-by: John Mumm <jtfmumm@gmail.com>
Prior to this PR, there were numerous places where uv would leak
credentials in logs. We had a way to mask credentials by calling methods
or a recently-added `redact_url` function, but this was not secure by
default. There were a number of other types (like `GitUrl`) that would
leak credentials on display.
This PR adds a `DisplaySafeUrl` newtype to prevent leaking credentials
when logging by default. It takes a maximalist approach, replacing the
use of `Url` almost everywhere. This includes when first parsing config
files, when storing URLs in types like `GitUrl`, and also when storing
URLs in types that in practice will never contain credentials (like
`DirectorySourceUrl`). The idea is to make it easy for developers to do
the right thing and for the compiler to support this (and to minimize
ever having to manually convert back and forth). Displaying credentials
now requires an active step. Note that despite this maximalist approach,
the use of the newtype should be zero cost.
One conspicuous place this PR does not use `DisplaySafeUrl` is in the
`uv-auth` crate. That would require new clones since there are calls to
`request.url()` that return a `&Url`. One option would have been to make
`DisplaySafeUrl` wrap a `Cow`, but this would lead to lifetime
annotations all over the codebase. I've created a separate PR based on
this one (#13576) that updates `uv-auth` to use `DisplaySafeUrl` with
one new clone. We can discuss the tradeoffs there.
Most of this PR just replaces `Url` with `DisplaySafeUrl`. The core is
`uv_redacted/lib.rs`, where the newtype is implemented. To make it
easier to review the rest, here are some points of note:
* `DisplaySafeUrl` has a `Display` implementation that masks
credentials. Currently, it will still display the username when there is
both a username and password. If we think is the wrong choice, it can
now be changed in one place.
* `DisplaySafeUrl` has a `remove_credentials()` method and also a
`.to_string_with_credentials()` method. This allows us to use it in a
variety of scenarios.
* `IndexUrl::redacted()` was renamed to
`IndexUrl::removed_credentials()` to make it clearer that we are not
masking.
* We convert from a `DisplaySafeUrl` to a `Url` when calling `reqwest`
methods like `.get()` and `.head()`.
* We convert from a `DisplaySafeUrl` to a `Url` when creating a
`uv_auth::Index`. That is because, as mentioned above, I will be
updating the `uv_auth` crate to use this newtype in a separate PR.
* A number of tests (e.g., in `pip_install.rs`) that formerly used
filters to mask tokens in the test output no longer need those filters
since tokens in URLs are now masked automatically.
* The one place we are still knowingly writing credentials to
`pyproject.toml` is when a URL with credentials is passed to `uv add`
with `--raw`. Since displaying credentials is no longer automatic, I
have added a `to_string_with_credentials()` method to the `Pep508Url`
trait. This is used when `--raw` is passed. Adding it to that trait is a
bit weird, but it's the simplest way to achieve the goal. I'm open to
suggestions on how to improve this, but note that because of the way
we're using generic bounds, it's not as simple as just creating a
separate trait for that method.
Previously, we required a username to perform a fetch from the keyring
because the `keyring` CLI only supported fetching password for a given
service and username. Unfortunately, this is different from the keyring
Python API which supported fetching a username _and_ password for a
given service. We can't (easily) use the Python API because we don't
expect `keyring` to be installed in a specific environment during
network requests. This means that we did not have parity with `pip`.
Way back in https://github.com/jaraco/keyring/pull/678 we got a `--mode
creds` flag added to `keyring`'s CLI which supports parity with the
Python API. Since `keyring` is expensive to invoke and we cannot be
certain that users are on the latest version of keyring, we've not added
support for invoking keyring with this flag. However, now that we have a
mode that says authentication is _required_ for an index (#11896), we
might as well _try_ to invoke keyring with `--mode creds` when there is
no username. This will address use-cases where the username is
non-constant and move us closer to `pip` parity.
Adds a new optional key `auth-policy` to `[tool.uv.index]` that sets the
authentication policy for the index URL.
The default is `"auto"`, which attempts to authenticate when necessary.
`"always"` always attempts to authenticate and fails if the endpoint is
unauthenticated. `"never"` never attempts to authenticate.
These policy address two kinds of cases:
* Some indexes don’t fail on unauthenticated requests; instead they just
forward to the public PyPI. This can leave the user confused as to why
their package is missing. The "always" policy prevents this.
* "never" allows users to ensure their credentials couldn't be leaked to
an unexpected index, though it will only allow for successful requests
on an index that doesn't require credentials.
Closes#11600
## Summary
This lets us drop a dependency entirely. `percent-encoding` is used by
`url` and so is already in the graph, whereas `urlencoding` isn't used
by anything else.
As per
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html
Before that, there were 91 separate integration tests binary.
(As discussed on Discord — I've done the `uv` crate, there's still a few
more commits coming before this is mergeable, and I want to see how it
performs in CI and locally).
In #2976 I made some changes that led to regressions:
- We stopped tracking URLs that we had not seen credentials for in the
cache
- This means the cache no longer returns a value to indicate we've seen
a realm before
- We stopped seeding the cache with URLs
- Combined with the above, this means we no longer had a list of
locations that we would never attempt to fetch credentials for
- We added caching of credentials found on requests
- Previously the cache was only populated from the seed or credentials
found in the netrc or keyring
- This meant that the cache was populated for locations that we
previously did not cache, i.e. GitHub artifacts(?)
Unfortunately this unveiled problems with the granularity of our cache.
We cache credentials per realm (roughly the hostname) but some realms
have mixed authentication modes i.e. different credentials per URL or
URLs that do not require credentials. Applying credentials to a URL that
does not require it can lead to a failed request, as seen in #3123 where
GitHub throws a 401 when receiving credentials.
To resolve this, the cache is expanded to supporting caching at two
levels:
- URL, cached URL must be a prefix of the request URL
- Realm, exact match required
When we don't have URL-level credentials cached, we attempt the request
without authentication first. On failure, we'll search for realm-level
credentials or fetch credentials from external services. This avoids
providing credentials to new URLs unless we know we need them.
Closes https://github.com/astral-sh/uv/issues/3123
## Summary
This PR adds basic struct definitions along with a "workspace" concept
for discovering settings. (The "workspace" terminology is used to match
Ruff; I did not invent it.)
A few notes:
- We discover any `pyproject.toml` or `uv.toml` file in any parent
directory of the current working directory. (We could adjust this to
look at the directories of the input files.)
- We don't actually do anything with the configuration yet; but those
PRs are large and I want this to be reviewed in isolation.
Closes#2566
We were storing the username e.g. `charlie@astral.sh` as a
percent-encoded string `charlie%40astral.sh` which resulted in different
headers and broke JFrog's artifactory which apparently does not decode
usernames.
Tested with a JFrog artifactory and AWS CodeArtifact although it is
worth noting that AWS does _not_ have a username with an `@` — it'd be
nice to test another artifactory with percent-encoded characters in the
username and/or password.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
Adds basic keyring auth support for `uv` commands. Adds clone of `pip`'s
`--keyring-provider subprocess` argument (using CLI `keyring` tool).
See issue: https://github.com/astral-sh/uv/issues/1520
## Test Plan
<!-- How was it tested? -->
Hard to write full-suite unit tests due to reliance on
`process::Command` for `keyring` cli
Manually tested end-to-end in a project with GCP artifact registry using
keyring password:
```bash
➜ uv pip uninstall watchdog
Uninstalled 1 package in 46ms
- watchdog==4.0.0
➜ cargo run -- pip install --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
Finished dev [unoptimized + debuginfo] target(s) in 0.18s
Running `target/debug/uv pip install --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
error: HTTP status client error (401 Unauthorized) for url (https://<redacted>/pypi-mirror/simple/watchdog/)
➜ cargo run -- pip install --keyring-provider subprocess --index-url https://<redacted>/python/simple/ --extra-index-url https://<redacted>/pypi-mirror/simple/ watchdog
Finished dev [unoptimized + debuginfo] target(s) in 0.17s
Running `target/debug/uv pip install --keyring-provider subprocess --index-url 'https://<redacted>/python/simple/' --extra-index-url 'https://<redacted>/pypi-mirror/simple/' watchdog`
Resolved 1 package in 2.34s
Installed 1 package in 27ms
+ watchdog==4.0.0
```
`requirements.txt`
```
#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
# .bin/generate-requirements
#
--index-url https://<redacted>/python/simple/
--extra-index-url https://<redacted>/pypi-mirror/simple/
...
```
```bash
➜ cargo run -- pip install --keyring-provider subprocess -r requirements.txt
Finished dev [unoptimized + debuginfo] target(s) in 0.19s
Running `target/debug/uv pip install --keyring-provider subprocess -r requirements.txt`
Resolved 205 packages in 23.52s
Built <redacted>
...
Downloaded 47 packages in 19.32s
Installed 195 packages in 276ms
+ <redacted>
...
```
---------
Co-authored-by: Thomas Gilgenast <thomas@vant.ai>
Co-authored-by: Zanie Blue <contact@zanie.dev>
Closes https://github.com/astral-sh/uv/issues/1709
Closes https://github.com/astral-sh/uv/issues/1371
Tested with the reproduction provided in #1709 which gets past the HTTP
401.
Reuses the same copying logic we introduced in
https://github.com/astral-sh/uv/pull/1874 to ensure authentication is
attached to file URLs with a realm that matches that of the index. I had
to move the authentication logic into a new crate so it could be used in
`distribution-types`.
We will want to something more robust in the future, like track all
realms with authentication in a central store and perform lookups there.
That's what `pip` does and it allows consolidation of logic like netrc
lookups. That refactor feels significant though, and I'd like to get
this fixed ASAP so this is a minimal fix.
konsti
Zanie Blue
Zsolt Dollenstein
Ruben Arts
Charlie Marsh
John Mumm
samypr100
Amos Wenger
哇呜哇呜呀咦耶
Hans Baker