Skip to content

Add CI workflow for running benchmarks#341

Merged
dfremont merged 2 commits intoBerkeleyLearnVerify:mainfrom
lola831:benchmark-ci
Apr 13, 2026
Merged

Add CI workflow for running benchmarks#341
dfremont merged 2 commits intoBerkeleyLearnVerify:mainfrom
lola831:benchmark-ci

Conversation

@lola831
Copy link
Copy Markdown
Collaborator

@lola831 lola831 commented Apr 9, 2025

Description

This PR introduces a new GitHub Actions workflow (.github/workflows/run-benchmarks.yml) that runs benchmarks via an issue comment trigger. It verifies that the commenter has sufficient permissions, checks out both the base branch and compare branch (using the merge ref if available), runs benchmarks from tools/benchmarking/ci/benchmarks.py, compares the results with pyperf, and posts the comparison back to the PR comment.

Issue Link

N/A

Checklist

  • I have tested the changes locally via pytest and/or other means
  • I have added or updated relevant documentation
  • I have autoformatted the code with black and isort
  • I have added test cases (if applicable)

Additional Notes

N/A

@lola831 lola831 requested a review from dfremont April 9, 2025 15:52
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 9, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.66%. Comparing base (b72430c) to head (a53551c).
⚠️ Report is 69 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #341      +/-   ##
==========================================
- Coverage   91.35%   89.66%   -1.69%     
==========================================
  Files          54       48       -6     
  Lines       13584    13194     -390     
==========================================
- Hits        12409    11830     -579     
- Misses       1175     1364     +189     

see 55 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread .github/workflows/run-benchmarks.yml Outdated
Comment thread .github/workflows/run-benchmarks.yml Outdated
Comment thread .github/workflows/run-benchmarks.yml Outdated
* Benchmark-running script has more informative name

* Sort benchmarks by speedup; ignore differences below 10%

Co-authored-by: Daniel Fremont <dfremont@ucsc.edu>
Copy link
Copy Markdown
Collaborator

@dfremont dfremont left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, Lola! I made a couple tiny changes. Will follow up shortly with the actual benchmarks and runner script in a new PR.

@dfremont dfremont merged commit 3c5102e into BerkeleyLearnVerify:main Apr 13, 2026
17 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants