mirror of https://github.com/astral-sh/ruff
93eff7f174
## Summary Adds a markdown-based test framework for writing tests of type inference and type checking. Fixes #11664. Implements the basic required features. A markdown test file is a suite of tests, each test can contain one or more Python files, with optionally specified path/name. The test writes all files to an in-memory file system, runs red-knot, and matches the resulting diagnostics against `Type: ` and `Error: ` assertions embedded in the Python source as comments. We will want to add features like incremental tests, setting custom configuration for tests, writing non-Python files, testing syntax errors, capturing full diagnostic output, etc. There's also plenty of room for improved UX (colored output?). ## Test Plan Lots of tests! Sample of the current output when a test fails: ``` Running tests/inference.rs (target/debug/deps/inference-7c96590aa84de2a4) running 1 test test inference::path_1_resources_inference_numbers_md ... FAILED failures: ---- inference::path_1_resources_inference_numbers_md stdout ---- inference/numbers.md - Numbers - Floats /src/test.py line 2: unexpected error: [invalid-assignment] "Object of type `Literal["str"]` is not assignable to `int`" thread 'inference::path_1_resources_inference_numbers_md' panicked at crates/red_knot_test/src/lib.rs:60:5: Some tests failed. note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: inference::path_1_resources_inference_numbers_md test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.19s error: test failed, to rerun pass `-p red_knot_test --test inference` ``` --------- Co-authored-by: Micha Reiser <micha@reiser.io> Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com> |
||
|---|---|---|
| .. | ||
| red_knot | ||
| red_knot_python_semantic | ||
| red_knot_server | ||
| red_knot_test | ||
| red_knot_vendored | ||
| red_knot_wasm | ||
| red_knot_workspace | ||
| ruff | ||
| ruff_benchmark | ||
| ruff_cache | ||
| ruff_db | ||
| ruff_dev | ||
| ruff_diagnostics | ||
| ruff_formatter | ||
| ruff_graph | ||
| ruff_index | ||
| ruff_linter | ||
| ruff_macros | ||
| ruff_notebook | ||
| ruff_python_ast | ||
| ruff_python_ast_integration_tests | ||
| ruff_python_codegen | ||
| ruff_python_formatter | ||
| ruff_python_index | ||
| ruff_python_literal | ||
| ruff_python_parser | ||
| ruff_python_resolver | ||
| ruff_python_semantic | ||
| ruff_python_stdlib | ||
| ruff_python_trivia | ||
| ruff_python_trivia_integration_tests | ||
| ruff_server | ||
| ruff_source_file | ||
| ruff_text_size | ||
| ruff_wasm | ||
| ruff_workspace | ||