mirror of https://github.com/astral-sh/ruff
195bb433db
## Summary This PR adds a CI job that causes GitHub to add annotations to a PR diff when mdtest assertions fail. For example: <details> <summary>Screenshot</summary>  </details> ## Motivation Debugging mdtest failures locally is currently a really nice experience: - Errors are displayed with pretty colours, which makes them much more readable - If you run the test from inside an IDE, you can CTRL-click on a path and jump directly to the line that had the failing assertion - If you use [`mdtest.py`](https://github.com/astral-sh/ruff/blob/main/crates/red_knot_python_semantic/mdtest.py), you don't even need to recompile anything after changing an assertion in an mdtest, amd the test results instantly live-update with each change to the MarkDown file Debugging mdtest failures in CI is much more unpleasant, however. Sometimes an error message is just > [static-assert-error] Argument evaluates to `False` ...which doesn't tell you very much unless you navigate to the line in question that has the failing mdtest assertion. The line in question might not even be touched by the PR, and even if it is, it can be hard to find the line if the PR touches many files. Unlike locally, you can't click on the error and jump straight to the line that contains the failing assertion. You also don't get colourised output in CI (https://github.com/astral-sh/ruff/issues/13939). GitHub PR annotations should make it really easy to debug why mdtests are failing on PRs, making PR review much easier. ## Test Plan I opened a PR to my fork [here](https://github.com/AlexWaygood/ruff/pull/11/files) with some bogus changes to an mdtest to show what it looks like when there are failures in CI and this job has been added. Scroll down to `crates/red_knot_python_semantic/resources/mdtest/type_properties/is_equivalent_to.md` on the "files changed" tab for that PR to see the annotations. |
||
|---|---|---|
| .. | ||
| red_knot | ||
| red_knot_ide | ||
| red_knot_project | ||
| red_knot_python_semantic | ||
| red_knot_server | ||
| red_knot_test | ||
| red_knot_vendored | ||
| red_knot_wasm | ||
| ruff | ||
| ruff_annotate_snippets | ||
| ruff_benchmark | ||
| ruff_cache | ||
| ruff_db | ||
| ruff_dev | ||
| ruff_diagnostics | ||
| ruff_formatter | ||
| ruff_graph | ||
| ruff_index | ||
| ruff_linter | ||
| ruff_macros | ||
| ruff_notebook | ||
| ruff_python_ast | ||
| ruff_python_ast_integration_tests | ||
| ruff_python_codegen | ||
| ruff_python_formatter | ||
| ruff_python_index | ||
| ruff_python_literal | ||
| ruff_python_parser | ||
| ruff_python_resolver | ||
| ruff_python_semantic | ||
| ruff_python_stdlib | ||
| ruff_python_trivia | ||
| ruff_python_trivia_integration_tests | ||
| ruff_server | ||
| ruff_source_file | ||
| ruff_text_size | ||
| ruff_wasm | ||
| ruff_workspace | ||