[ty] ecosystem-analyzer workflow (#18719)

## Summary

Adds a new ecosystem-analyzer workflow with a similar purpose to the
mypy-primer workflow. It creates a richer ecosystem diff report using
[ecosystem-analyzer](https://github.com/astral-sh/ecosystem-analyzer/)
([example
report](https://shark.fish/diff-attr-subscript-narrowing.html)). This is
still experimental and also quite a bit slower than mypy_primer, so I
chose to make this opt-in for now via a `ecosystem-analyzer` label. This
would give us a way to play with this while still evaluating if we
should further invest in this or not.

Advantages over the mypy_primer diff output:
- Interactive filtering of diagnostics
- Statistics overview which breaks down added/removed/changed
diagnostics across lint rules
- Has the concept of "changed" diagnostics, which makes it easier to
review changes where diagnostic messages have changed (along with other
changes).
- Compute diff based on old and new project-lists (`good.txt`). This
allows us to diff changes to the project list itself. This has caused
confusion in the past where we tried to add new projects to `good.txt`,
but then ran the `main`-branch version of ty on that new list (where the
bug was not yet fixed)

Disadvantages:
- The report currently needs to be downloaded from the workflow run, as
I don't know if we have a way of deploying HTML files like this
temporarily to some hosted infrastructure.
This commit is contained in:
David Peter 2025-06-20 15:23:22 +02:00 committed by GitHub
parent 7982edac90
commit 0ef324b2dd
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 94 additions and 0 deletions

View File

@ -0,0 +1,94 @@
name: ty ecosystem-analyzer
permissions: {}
on:
pull_request:
types: [labeled]
concurrency:
group: ${{ github.workflow }}-${{ github.ref_name }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true
env:
CARGO_INCREMENTAL: 0
CARGO_NET_RETRY: 10
CARGO_TERM_COLOR: always
RUSTUP_MAX_RETRIES: 10
RUST_BACKTRACE: 1
REF_NAME: ${{ github.ref_name }}
jobs:
ty-ecosystem-analyzer:
name: Compute diagnostic diff
runs-on: depot-ubuntu-22.04-32
timeout-minutes: 20
if: contains(github.event.label.name, 'ecosystem-analyzer')
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
path: ruff
fetch-depth: 0
persist-credentials: false
- name: Install the latest version of uv
uses: astral-sh/setup-uv@f0ec1fc3b38f5e7cd731bb6ce540c5af426746bb # v6.1.0
- uses: Swatinem/rust-cache@9d47c6ad4b02e050fd481d890b2ea34778fd09d6 # v2.7.8
with:
workspaces: "ruff"
- name: Install Rust toolchain
run: rustup show
- name: Compute diagnostic diff
shell: bash
run: |
cd ruff
echo "Enabling configuration overloads (see .github/mypy-primer-ty.toml)"
mkdir -p ~/.config/ty
cp .github/mypy-primer-ty.toml ~/.config/ty/ty.toml
echo "new commit"
git checkout -b new_commit "$GITHUB_SHA"
git rev-list --format=%s --max-count=1 new_commit
cp crates/ty_python_semantic/resources/primer/good.txt projects_new.txt
echo "old commit (merge base)"
MERGE_BASE="$(git merge-base "$GITHUB_SHA" "origin/$GITHUB_BASE_REF")"
git checkout -b old_commit "$MERGE_BASE"
git rev-list --format=%s --max-count=1 old_commit
cp crates/ty_python_semantic/resources/primer/good.txt projects_old.txt
cd ..
uv tool install "git+https://github.com/astral-sh/ecosystem-analyzer@9c34dc514ee9aef6735db1dfebb80f63acbc3440"
ecosystem-analyzer \
--repository ruff \
analyze \
--projects ruff/projects_old.txt \
--commit old_commit \
--output diagnostics_old.json
ecosystem-analyzer \
--repository ruff \
analyze \
--projects ruff/projects_new.txt \
--commit new_commit \
--output diagnostics_new.json
ecosystem-analyzer \
generate-diff \
diagnostics_old.json \
diagnostics_new.json \
--old-name "main (merge base)" \
--new-name "$REF_NAME" \
--output-html diff.html
- name: Upload HTML diff report
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: diff.html
path: diff.html