Skip to content

ci: Auto-accept API verifier changes in PRs#5228

Open
jamescrosswell wants to merge 5 commits into
mainfrom
ci/issue-5157-auto-verify-api
Open

ci: Auto-accept API verifier changes in PRs#5228
jamescrosswell wants to merge 5 commits into
mainfrom
ci/issue-5157-auto-verify-api

Conversation

@jamescrosswell
Copy link
Copy Markdown
Collaborator

@jamescrosswell jamescrosswell commented May 13, 2026

Summary

Adds a new verify api workflow that auto-accepts API approval test snapshot changes on PRs, similar to the existing format-code workflow.

  • Runs ApiApprovalTests on a matrix of macos-15 (covers .NET / netstandard / iOS / MacCatalyst / Android TFMs) and windows-latest (covers the net48 / .NET Framework TFM, which is the common pain point when developing on non-Windows).
  • Uploads each runner's *.received.* files as artifacts.
  • A finalizer job downloads all received files, runs scripts/accept-verifier-changes.ps1, and pushes the accepted snapshots back to the PR branch.
  • When API changes are accepted, the PR is also labelled public API
  • For PRs from forks (where the workflow can't push back to the contributor's branch), the workflow fails with a clear message instructing the contributor to accept the snapshots locally — mirroring the fork-fallback in format-code.yml.

The Alternative suggested in the issue (switching to Microsoft.CodeAnalysis.PublicApiAnalyzers) is intentionally out of scope for this PR and can be considered in a follow-up.

Closes #5157

Adds a new `verify api` workflow that runs the ApiApprovalTests on
macOS (covers .NET / netstandard / iOS / MacCatalyst / Android TFMs)
and Windows (covers the net48 / .NET Framework TFM), then runs
scripts/accept-verifier-changes.ps1 over the resulting *.received.*
files and pushes the accepted snapshots back to the PR branch. When
API changes are accepted, the PR is also labelled `public API`.

For PRs from forks (where we can't push back), the workflow fails
with a hint telling the contributor how to accept the changes
locally.

Closes #5157

Co-Authored-By: Claude <noreply@anthropic.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented May 13, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.12%. Comparing base (0f91f7d) to head (ed7001e).
⚠️ Report is 7 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5228      +/-   ##
==========================================
- Coverage   74.13%   74.12%   -0.02%     
==========================================
  Files         508      508              
  Lines       18282    18282              
  Branches     3574     3574              
==========================================
- Hits        13554    13552       -2     
- Misses       3859     3860       +1     
- Partials      869      870       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jamescrosswell jamescrosswell marked this pull request as ready for review May 20, 2026 00:37
@jamescrosswell jamescrosswell requested a review from Flash0ver as a code owner May 20, 2026 00:37
Comment thread .github/workflows/verify-api.yml
Comment thread .github/workflows/verify-api.yml
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit dc60272. Configure here.

# here and rely on the produced *.received.txt files to detect and accept the change.
- name: Run API Approval Tests
continue-on-error: true
run: dotnet test ${{ matrix.slnf }} -c Release --no-build --nologo --filter "FullyQualifiedName~ApiApprovalTests"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

API test failures ignored

Medium Severity

The API approval test step uses continue-on-error, so run-api-tests still succeeds when dotnet test fails without producing *.received.txt. No later step checks that outcome, so the workflow can pass while ApiApprovalTests actually failed for non-snapshot reasons.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit dc60272. Configure here.

Copy link
Copy Markdown
Collaborator Author

@jamescrosswell jamescrosswell May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine. If that happens we'll know about it from the build workflow that is running concurrently. This workflow isn't intended to replace the full build - it's just a convenience so devs don't have to build the solution and run it on multiple operating systems every time one of the verify API files changes.

We're not actually running the tests to see if they pass or fail here - we run them simply as a way to get the latest verify files.

Comment on lines +94 to +99
- name: Download Received API Files
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: api-verify-received-*
merge-multiple: true

Copy link
Copy Markdown

@sentry sentry Bot May 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in b011745

Bug: The accept-api-changes job may fail on PRs with no API changes because the download-artifact step errors when no API change artifacts are found to download.
Severity: MEDIUM

Suggested Fix

Add continue-on-error: true to the Download Received API Files step in the accept-api-changes job. This will prevent the workflow from failing when no API change artifacts are found. Apply the same fix to the equivalent step in the report-fork-api-changes job.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: .github/workflows/verify-api.yml#L94-L99

Potential issue: The `accept-api-changes` job includes a step to upload API verification
files, which uses `if-no-files-found: ignore`. Consequently, for pull requests without
any public API changes, no `api-verify-received-*` artifact is created. A subsequent
step attempts to download artifacts using the pattern `api-verify-received-*`. Since
`actions/download-artifact@v8.0.1` is expected to fail when a pattern matches no
artifacts and there is no `continue-on-error` configured, this job will likely fail on
every clean PR. The `report-fork-api-changes` job has the same vulnerability.

Also affects:

  • .github/workflows/verify-api.yml:148~151

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b011745

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted per #5228 (comment)

Looking at the source for the action it only throws an error if there are no files to download if either isDownloadByIds or isSingleArtifactDownload is true... so we don't need continue on error in the download step.

So the initial review comment was erroneous.

Comment thread .github/workflows/verify-api.yml Outdated
The workflow-level `contents: write` token was inherited by
`run-api-tests`, which builds and runs untrusted PR code.

Scope tokens per-job instead:
  - run-api-tests:           contents: read
  - accept-api-changes:      contents: write, pull-requests: write
  - report-fork-api-changes: contents: read

Co-Authored-By: Claude <noreply@anthropic.com>
Comment thread .github/workflows/verify-api.yml
The pattern branch of actions/download-artifact (no `name`/`artifact-ids`,
only `pattern`) tolerates zero matches without erroring — see
https://github.com/actions/download-artifact/blob/main/src/download-artifact.ts.
So `continue-on-error: true` isn't needed for the clean-PR case and would
otherwise mask genuine download failures.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: Auto-Verify API changes in Pull Requests

1 participant