## Summary
We'll revert back to the crates.io release once it's up-to-date, but
better to get this out now that Python 3.12 is released.
## Test Plan
`cargo test`
## Summary
This PR enables `ruff format` to format Jupyter notebooks.
Most of the work is contained in a new `format_source` method that
formats a generic `SourceKind`, then returns `Some(transformed)` if the
source required formatting, or `None` otherwise.
Closes https://github.com/astral-sh/ruff/issues/7598.
## Test Plan
Ran `cat foo.py | cargo run -p ruff_cli -- format --stdin-filename
Untitled.ipynb`; verified that the console showed a reasonable error:
```console
warning: Failed to read notebook Untitled.ipynb: Expected a Jupyter Notebook, which must be internally stored as JSON, but this file isn't valid JSON: EOF while parsing a value at line 1 column 0
```
Ran `cat Untitled.ipynb | cargo run -p ruff_cli -- format
--stdin-filename Untitled.ipynb`; verified that the JSON output
contained formatted source code.
## Summary
When writing back notebooks via `stdout`, we need to write back the
entire JSON content, not _just_ the fixed source code. Otherwise,
writing the output _back_ to the file will yield an invalid notebook.
Closes https://github.com/astral-sh/ruff/issues/7747
## Test Plan
`cargo test`
## Summary
There's no way for users to fix this warning if they're intentionally
using an "invalid" PEP 593 annotation, as is the case in CPython. This
is a symptom of having warnings that aren't themselves diagnostics. If
we want this to be user-facing, we should add a diagnostic for it!
## Test Plan
Ran `cargo run -p ruff_cli -- check foo.py -n` on:
```python
from typing import Annotated
Annotated[int]
```
## Summary
If we have, e.g.:
```python
sum((
factor.dims for factor in bases
), [])
```
We generate three edits: two insertions (for the `operator` and
`functools` imports), and then one replacement (for the `sum` call
itself). We need to ensure that the insertions come before the
replacement; otherwise, the edits will appear overlapping and
out-of-order.
Closes https://github.com/astral-sh/ruff/issues/7718.
## Summary
This PR modifies the `line-too-long` and `doc-line-too-long` rules to
ignore lines that are too long due to the presence of a pragma comment
(e.g., `# type: ignore` or `# noqa`). That is, if a line only exceeds
the limit due to the pragma comment, it will no longer be flagged as
"too long". This behavior mirrors that of the formatter, thus ensuring
that we don't flag lines under E501 that the formatter would otherwise
avoid wrapping.
As a concrete example, given a line length of 88, the following would
_no longer_ be considered an E501 violation:
```python
# The string literal is 88 characters, including quotes.
"shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:sh" # type: ignore
```
This, however, would:
```python
# The string literal is 89 characters, including quotes.
"shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:shape:sha" # type: ignore
```
In addition to mirroring the formatter, this also means that adding a
pragma comment (like `# noqa`) won't _cause_ additional violations to
appear (namely, E501). It's very common for users to add a `# type:
ignore` or similar to a line, only to find that they then have to add a
suppression comment _after_ it that was required before, as in `# type:
ignore # noqa: E501`.
Closes https://github.com/astral-sh/ruff/issues/7471.
## Test Plan
`cargo test`
## Summary
This PR implements a variety of optimizations to improve performance of
the Eradicate rule, which always shows up in all-rules benchmarks and
bothers me. (These improvements are not hugely important, but it was
kind of a fun Friday thing to spent a bit of time on.)
The improvements include:
- Doing cheaper work first (checking for some explicit substrings
upfront).
- Using `aho-corasick` to speed an exact substring search.
- Merging multiple regular expressions using a `RegexSet`.
- Removing some unnecessary `\s*` and other pieces from the regular
expressions (since we already trim strings before matching on them).
## Test Plan
I benchmarked this function in a standalone crate using a variety of
cases. Criterion reports that this version is up to 80% faster, and
almost every case is at least 50% faster:
```
Eradicate/Detection/# Warn if we are installing over top of an existing installation. This can
time: [101.84 ns 102.32 ns 102.82 ns]
change: [-77.166% -77.062% -76.943%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high mild
Eradicate/Detection/#from foo import eradicate
time: [74.872 ns 75.096 ns 75.314 ns]
change: [-84.180% -84.131% -84.079%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
Eradicate/Detection/# encoding: utf8
time: [46.522 ns 46.862 ns 47.237 ns]
change: [-29.408% -28.918% -28.471%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
6 (6.00%) high mild
1 (1.00%) high severe
Eradicate/Detection/# Issue #999
time: [16.942 ns 16.994 ns 17.058 ns]
change: [-57.243% -57.064% -56.815%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
Eradicate/Detection/# type: ignore
time: [43.074 ns 43.163 ns 43.262 ns]
change: [-17.614% -17.390% -17.152%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe
Eradicate/Detection/# user_content_type, _ = TimelineEvent.objects.using(db_alias).get_or_create(
time: [209.40 ns 209.81 ns 210.23 ns]
change: [-32.806% -32.630% -32.470%] (p = 0.00 < 0.05)
Performance has improved.
Eradicate/Detection/# this is = to that :(
time: [72.659 ns 73.068 ns 73.473 ns]
change: [-68.884% -68.775% -68.655%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
7 (7.00%) high mild
2 (2.00%) high severe
Eradicate/Detection/#except Exception:
time: [92.063 ns 92.366 ns 92.691 ns]
change: [-64.204% -64.052% -63.909%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe
Eradicate/Detection/#print(1)
time: [68.359 ns 68.537 ns 68.725 ns]
change: [-72.424% -72.356% -72.278%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
Eradicate/Detection/#'key': 1 + 1,
time: [79.604 ns 79.865 ns 80.135 ns]
change: [-69.787% -69.667% -69.549%] (p = 0.00 < 0.05)
Performance has improved.
```
## Summary
The parser now uses the raw source code as global context and slices
into it to parse debug text. It turns out we were always passing in the
_old_ source code, so when code was fixed, we were making invalid
accesses. This PR modifies the call to use the _fixed_ source code,
which will always be consistent with the tokens.
Closes https://github.com/astral-sh/ruff/issues/7711.
## Test Plan
`cargo test`
## Summary
This wasn't necessary in the past, since we _only_ applied this rule to
bodies that contained two statements, one of which was a `pass`. Now
that it applies to any `pass` in a block with multiple statements, we
can run into situations in which we remove both passes, and so need to
apply the fixes in isolation.
See:
https://github.com/astral-sh/ruff/issues/7455#issuecomment-1741107573.
## Summary
The markdown documentation was present, but in the wrong place, so was
not displaying on the website. I moved it and added some references.
Related to #2646.
## Test Plan
`python scripts/check_docs_formatted.py`
## Summary
Extend `unnecessary-pass` (`PIE790`) to trigger on all unnecessary
`pass` statements by checking for `pass` statements in any class or
function body with more than one statement.
Closes#7600.
## Test Plan
`cargo test`
Part of #1646.
## Summary
Implement `S505`
([`weak_cryptographic_key`](https://bandit.readthedocs.io/en/latest/plugins/b505_weak_cryptographic_key.html))
rule from `bandit`.
For this rule, `bandit` [reports the issue
with](https://github.com/PyCQA/bandit/blob/1.7.5/bandit/plugins/weak_cryptographic_key.py#L47-L56):
- medium severity for DSA/RSA < 2048 bits and EC < 224 bits
- high severity for DSA/RSA < 1024 bits and EC < 160 bits
Since Ruff does not handle severities for `bandit`-related rules, we
could either report the issue if we have lower values than medium
severity, or lower values than high one. Two reasons led me to choose
the first option:
- a medium severity issue is still a security issue we would want to
report to the user, who can then decide to either handle the issue or
ignore it
- `bandit` [maps the EC key algorithms to their respective key lengths
in
bits](https://github.com/PyCQA/bandit/blob/1.7.5/bandit/plugins/weak_cryptographic_key.py#L112-L133),
but there is no value below 160 bits, so technically `bandit` would
never report medium severity issues for EC keys, only high ones
Another consideration is that as shared just above, for EC key
algorithms, `bandit` has a mapping to map the algorithms to their
respective key lengths. In the implementation in Ruff, I rather went
with an explicit list of EC algorithms known to be vulnerable (which
would thus be reported) rather than implementing a mapping to retrieve
the associated key length and comparing it with the minimum value.
## Test Plan
Snapshot tests from
https://github.com/PyCQA/bandit/blob/1.7.5/examples/weak_cryptographic_key_sizes.py.
## Summary
Extend the `task-tags` checking logic to ignore TODO tags (with or
without parentheses). For example,
```python
# TODO(tjkuson): Rewrite in Rust
```
is no longer flagged as commented-out code.
Closes#7031.
I also updated the documentation to inform users that the rule is prone
to false positives like this!
EDIT: Accidentally linked to the wrong issue when first opening this PR,
now corrected.
## Test Plan
`cargo test`
Closes#7434
Replaces the `PREVIEW` selector (removed in #7389) with a configuration
option `explicit-preview-rules` which requires selectors to use exact
rule codes for all preview rules. This allows users to enable preview
without opting into all preview rules at once.
## Test plan
Unit tests
## Summary
`PGH002`, which checks for use of deprecated `logging.warn` calls, did
not check for calls made on the attribute `warn` yet. Since
https://github.com/astral-sh/ruff/pull/7521 we check both cases for
similar rules wherever possible. To be consistent this PR expands PGH002
to do the same.
## Test Plan
Expanded existing fixtures with `logger.warn()` calls
## Issue links
Fixes final inconsistency mentioned in
https://github.com/astral-sh/ruff/issues/7502
## Summary
As we bind the `ast::ExprCall` in the big `match expr` in
`expression.rs`
```rust
Expr::Call(
call @ ast::ExprCall {
...
```
There is no need for additional `let/if let` checks on `ExprCall` in
downstream rules. Found a few older rules which still did this while
working on something else. This PR removes the redundant check from
these rules.
## Test Plan
`cargo test`
## Summary
This PR adds support for named expressions when analyzing `__all__`
assignments, as per https://github.com/astral-sh/ruff/issues/7672. It
also loosens the enforcement around assignments like: `__all__ =
list(some_other_expression)`. We shouldn't flag these as invalid, even
though we can't analyze the members, since we _know_ they evaluate to a
`list`.
Closes https://github.com/astral-sh/ruff/issues/7672.
## Test Plan
`cargo test`
## Summary
Fixes#7616 by ensuring that
[B006](https://docs.astral.sh/ruff/rules/mutable-argument-default/#mutable-argument-default-b006)
fixes are inserted after module imports.
I have created a new test file, `B006_5.py`. This is mainly because I
have been working on this on and off, and the merge conflicts were
easier to handle in a separate file. If needed, I can move it into
another file.
## Test Plan
`cargo test`
## Summary
Expands several rules to also check for `Expr::Name` values. As they
would previously not consider:
```python
from logging import error
error("foo")
```
as potential violations
```python
import logging
logging.error("foo")
```
as potential violations leading to inconsistent behaviour.
The rules impacted are:
- `BLE001`
- `TRY400`
- `TRY401`
- `PLE1205`
- `PLE1206`
- `LOG007`
- `G001`-`G004`
- `G101`
- `G201`
- `G202`
## Test Plan
Fixtures for all impacted rules expanded.
## Issue Link
Refers: https://github.com/astral-sh/ruff/issues/7502
## Summary
Pass around a `Settings` struct instead of individual members to
simplify function signatures and to make it easier to add new settings.
This PR was suggested in [this
comment](https://github.com/astral-sh/ruff/issues/1567#issuecomment-1734182803).
## Note on the choices
I chose which functions to modify based on which seem most likely to use
new settings, but suggestions on my choices are welcome!
**Summary** Check that `closefd` and `opener` aren't being used with
`builtin.open()` before suggesting `Path.open()` because pathlib doesn't
support these arguments.
Closes#7620
**Test Plan** New cases in the fixture.
## Summary
This is a follow-up to #7469 that attempts to achieve similar gains, but
without introducing malachite. Instead, this PR removes the `BigInt`
type altogether, instead opting for a simple enum that allows us to
store small integers directly and only allocate for values greater than
`i64`:
```rust
/// A Python integer literal. Represents both small (fits in an `i64`) and large integers.
#[derive(Clone, PartialEq, Eq, Hash)]
pub struct Int(Number);
#[derive(Debug, Clone, PartialEq, Eq, Hash)]
pub enum Number {
/// A "small" number that can be represented as an `i64`.
Small(i64),
/// A "large" number that cannot be represented as an `i64`.
Big(Box<str>),
}
impl std::fmt::Display for Number {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Number::Small(value) => write!(f, "{value}"),
Number::Big(value) => write!(f, "{value}"),
}
}
}
```
We typically don't care about numbers greater than `isize` -- our only
uses are comparisons against small constants (like `1`, `2`, `3`, etc.),
so there's no real loss of information, except in one or two rules where
we're now a little more conservative (with the worst-case being that we
don't flag, e.g., an `itertools.pairwise` that uses an extremely large
value for the slice start constant). For simplicity, a few diagnostics
now show a dedicated message when they see integers that are out of the
supported range (e.g., `outdated-version-block`).
An additional benefit here is that we get to remove a few dependencies,
especially `num-bigint`.
## Test Plan
`cargo test`
## Summary
This PR fixes the autofix behavior for `PT022` to create an additional
edit for the return type if it's present. The edit will update the
return type from `Generator[T, ...]` to `T`. As per the [official
documentation](https://docs.python.org/3/library/typing.html?highlight=typing%20generator#typing.Generator),
the first position is the yield type, so we can ignore other positions.
```python
typing.Generator[YieldType, SendType, ReturnType]
```
## Test Plan
Add new test cases, `cargo test` and review the snapshots.
fixes: #7610
## Summary
Implement
[`simplify-print`](https://github.com/dosisod/refurb/blob/master/refurb/checks/builtin/print.py)
as `print-empty-string` (`FURB105`).
Extends the original rule in that it also checks for multiple empty
string positional arguments with an empty string separator.
Related to #1348.
## Test Plan
`cargo test`
## Summary
B005 only flags `.strip()` calls for which the argument includes
duplicate characters. This is consistent with bugbear, but isn't
explained in the documentation.
## Summary
This is only used for the `level` field in relative imports (e.g., `from
..foo import bar`). It seems unnecessary to use a wrapper here, so this
PR changes to a `u32` directly.
## Summary
## Stack Summary
This stack splits `Settings` into `FormatterSettings` and `LinterSettings` and moves it into `ruff_workspace`. This change is necessary to add the `FormatterSettings` to `Settings` without adding `ruff_python_formatter` as a dependency to `ruff_linter` (and the linter should not contain the formatter settings).
A quick overview of our settings struct at play:
* `Options`: 1:1 representation of the options in the `pyproject.toml` or `ruff.toml`. Used for deserialization.
* `Configuration`: Resolved `Options`, potentially merged from multiple configurations (when using `extend`). The representation is very close if not identical to the `Options`.
* `Settings`: The resolved configuration that uses a data format optimized for reading. Optional fields are initialized with their default values. Initialized by `Configuration::into_settings` .
The goal of this stack is to split `Settings` into tool-specific resolved `Settings` that are independent of each other. This comes at the advantage that the individual crates don't need to know anything about the other tools. The downside is that information gets duplicated between `Settings`. Right now the duplication is minimal (`line-length`, `tab-width`) but we may need to come up with a solution if more expensive data needs sharing.
This stack focuses on `Settings`. Splitting `Configuration` into some smaller structs is something I'll follow up on later.
## PR Summary
This PR moves the `ResolverSettings` and `Settings` struct to `ruff_workspace`. `LinterSettings` remains in `ruff_linter` because it gets passed to lint rules, the `Checker` etc.
## Test Plan
`cargo test`
## Stack Summary
This stack splits `Settings` into `FormatterSettings` and `LinterSettings` and moves it into `ruff_workspace`. This change is necessary to add the `FormatterSettings` to `Settings` without adding `ruff_python_formatter` as a dependency to `ruff_linter` (and the linter should not contain the formatter settings).
A quick overview of our settings struct at play:
* `Options`: 1:1 representation of the options in the `pyproject.toml` or `ruff.toml`. Used for deserialization.
* `Configuration`: Resolved `Options`, potentially merged from multiple configurations (when using `extend`). The representation is very close if not identical to the `Options`.
* `Settings`: The resolved configuration that uses a data format optimized for reading. Optional fields are initialized with their default values. Initialized by `Configuration::into_settings` .
The goal of this stack is to split `Settings` into tool-specific resolved `Settings` that are independent of each other. This comes at the advantage that the individual crates don't need to know anything about the other tools. The downside is that information gets duplicated between `Settings`. Right now the duplication is minimal (`line-length`, `tab-width`) but we may need to come up with a solution if more expensive data needs sharing.
This stack focuses on `Settings`. Splitting `Configuration` into some smaller structs is something I'll follow up on later.
## PR Summary
This PR extracts the linter-specific settings into a new `LinterSettings` struct and adds it as a `linter` field to the `Settings` struct. This is in preparation for moving `Settings` from `ruff_linter` to `ruff_workspace`
## Test Plan
`cargo test`
## Stack Summary
This stack splits `Settings` into `FormatterSettings` and `LinterSettings` and moves it into `ruff_workspace`. This change is necessary to add the `FormatterSettings` to `Settings` without adding `ruff_python_formatter` as a dependency to `ruff_linter` (and the linter should not contain the formatter settings).
A quick overview of our settings struct at play:
* `Options`: 1:1 representation of the options in the `pyproject.toml` or `ruff.toml`. Used for deserialization.
* `Configuration`: Resolved `Options`, potentially merged from multiple configurations (when using `extend`). The representation is very close if not identical to the `Options`.
* `Settings`: The resolved configuration that uses a data format optimized for reading. Optional fields are initialized with their default values. Initialized by `Configuration::into_settings` .
The goal of this stack is to split `Settings` into tool-specific resolved `Settings` that are independent of each other. This comes at the advantage that the individual crates don't need to know anything about the other tools. The downside is that information gets duplicated between `Settings`. Right now the duplication is minimal (`line-length`, `tab-width`) but we may need to come up with a solution if more expensive data needs sharing.
This stack focuses on `Settings`. Splitting `Configuration` into some smaller structs is something I'll follow up on later.
## PR Summary
This PR extracts a `ResolverSettings` struct that holds all the resolver-relevant fields (uninteresting for the `Formatter` or `Linter`). This will allow us to move the `ResolverSettings` out of `ruff_linter` further up in the stack.
## Test Plan
`cargo test`
(I'll to more extensive testing at the top of this stack)
## Summary
This PR fixes the way NoQA range is inserted to the `NoqaMapping`.
Previously, the way the mapping insertion logic worked was as follows:
1. If the range which is to be inserted _touched_ the previous range, meaning
that the end of the previous range was the same as the start of the new
range, then the new range was added in addition to the previous range.
2. Else, if the new range intersected the previous range, then the previous
range was replaced with the new _intersection_ of the two ranges.
The problem with this logic is that it does not work for the following case:
```python
assert foo, \
"""multi-line
string"""
```
Now, the comments cannot be added to the same line which ends with a continuation
character. So, the `NoQA` directive has to be added to the next line. But, the
next line is also a triple-quoted string, so the `NoQA` directive for that line
needs to be added to the next line. This creates a **union** pattern instead of an
**intersection** pattern.
But, only union doesn't suffice because (1) means that for the edge case where
the range touch only at the end, the union won't take place.
### Solution
1. Replace '<=' with '<' to have a _strict_ insertion case
2. Use union instead of intersection
## Test Plan
Add a new test case. Run the test suite to ensure that nothing is broken.
### Integration
1. Make a `test.py` file with the following contents:
```python
assert foo, \
"""multi-line
string"""
```
2. Run the following command:
```console
$ cargo run --bin ruff -- check --isolated --no-cache --select=F821 test.py
/Users/dhruv/playground/ruff/fstring.py:1:8: F821 Undefined name `foo`
Found 1 error.
```
3. Use `--add-noqa`:
```console
$ cargo run --bin ruff -- check --isolated --no-cache --select=F821 --add-noqa test.py
Added 1 noqa directive.
```
4. Check that the NoQA directive was added in the correct position:
```python
assert foo, \
"""multi-line
string""" # noqa: F821
```
5. Run the `check` command to ensure that the NoQA directive is respected:
```console
$ cargo run --bin ruff -- check --isolated --no-cache --select=F821 test.py
```
fixes: #7530
Charlie Marsh
konsti
dependabot[bot]
jan Apisu
Tom Kuson
Dhruv Manilawala
Daniel Parizher
Mathieu Kniewallner
Zanie Blue
qdegraaf
Simon Høxbro Hansen
bluthej
Micha Reiser