From c80d5c6ffbe705a8b337f7e9df5c459f1c3b2860 Mon Sep 17 00:00:00 2001 From: Jonathon Belotti Date: Sun, 25 Feb 2024 23:28:22 -0500 Subject: [PATCH] fix 'uv pip install' handling of gzip'd response and PEP 691 (#1978) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thank you for writing `uv`! We're already using it internally on some container image builds and finding that it's noticeably faster 💯 ## Summary I was attempting to use `uv` alongside [modal](https://modal.com/)'s internal PyPi mirror and ran into some issues. The first issue was the following error: ``` error: Failed to download: nltk==3.8.1 Caused by: content-length header is missing from response ``` This error was coming from within `RegistryClient::wheel_metadata_no_pep658`. By logging requests on the client (uv) and server (internal mirror) sides I've concluded that it's occurring because `uv` is sending a header suggesting that it can accept a gzip'd response, but decompressing the gzip'd response strips the `content-length` header: https://github.com/seanmonstar/reqwest/issues/294. **Logged request, client-side:** ``` 0.981664s 0ms INFO uv_client::registry_client JONO, REQ: Request { method: HEAD, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Ipv4(172.21.0.1)), port: Some(5555), path: "/simple/joblib/joblib-1.3.2-py3-none-any.whl", query: None, fragment: None }, headers: {} } ``` No headers set explicitly by `uv`. **Logged request, server-side:** ``` 2024-02-26T03:45:08.598272Z DEBUG pypi_mirror: origin request = Request { method: HEAD, uri: /simple/joblib/joblib-1.3.2-py3-none-any.whl, version: HTTP/1.1, headers: {"accept": "*/*", "user-agent": "uv", "accept-encoding": "gzip, br", "host": "172.21.0.1:5555"}, body: Body(Empty) } ``` Server receives `"accept-encoding": "gzip, br",`. My change adding the header to the request fixed this issue. But our internal mirror is just passing through PyPI responses and PyPI responses do contain PEP 658 data, and so `wheel_metadata_no_pep658` shouldn't execute. The issue there is that the PyPi response field has _dashes_ not _underscores_ (https://peps.python.org/pep-0691/). image After changing the `alias` the PEP 658 codepath now runs correctly :) ## Test Plan I tested by installing against both our mirror and against PyPi: ``` RUST_LOG="uv=trace" UV_NO_CACHE=true UV_INDEX_URL="http://172.21.0.1:5555/simple" target/release/uv pip install -v nltk RUST_LOG="uv=trace" UV_NO_CACHE=true UV_INDEX_URL="http://localhost:5555/simple" target/release/uv pip uninstall -v nltk ``` ``` target/release/uv pip install -v nltk target/release/uv pip uninstall -v nltk ``` --- crates/pypi-types/src/simple_json.rs | 2 +- crates/uv-client/src/registry_client.rs | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/crates/pypi-types/src/simple_json.rs b/crates/pypi-types/src/simple_json.rs index 9380a9012..a0de88571 100644 --- a/crates/pypi-types/src/simple_json.rs +++ b/crates/pypi-types/src/simple_json.rs @@ -38,7 +38,7 @@ fn sorted_simple_json_files<'de, D: Deserializer<'de>>(d: D) -> Result #[serde(rename_all = "kebab-case")] pub struct File { // Non-PEP 691-compliant alias used by PyPI. - #[serde(alias = "data_dist_info_metadata")] + #[serde(alias = "data-dist-info-metadata")] pub dist_info_metadata: Option, pub filename: String, pub hashes: Hashes, diff --git a/crates/uv-client/src/registry_client.rs b/crates/uv-client/src/registry_client.rs index f0f2915da..01497c9c4 100644 --- a/crates/uv-client/src/registry_client.rs +++ b/crates/uv-client/src/registry_client.rs @@ -459,6 +459,10 @@ impl RegistryClient { .client .uncached() .head(url.clone()) + .header( + "accept-encoding", + http::HeaderValue::from_static("identity"), + ) .build() .map_err(ErrorKind::RequestError)?;