Skip to content

Activate nightly workflow for random and stress tests#3500

Merged
kdeldycke merged 1 commit into
pallets:mainfrom
kdeldycke:activate-random-and-stress-tests
May 23, 2026
Merged

Activate nightly workflow for random and stress tests#3500
kdeldycke merged 1 commit into
pallets:mainfrom
kdeldycke:activate-random-and-stress-tests

Conversation

@kdeldycke
Copy link
Copy Markdown
Collaborator

@kdeldycke kdeldycke commented May 22, 2026

In #3139 we added stress tests and in #3151 we added randomization and parallel tests.

We decided to not activate them by default in #3151 (comment) and rely on contributors diligence to run them by hand to catch regressions. In practice nobody does. I even often forget to run them on my own during my sessions, only focused on the task at hand.

I'd like to re-assess this decision in light of regressions that could have been caught if these test were active. See the following PRs:

Also why even support parallel tests if we don't exercice them regularly? (ref: #2899)

Currently these tests adds ~15 minutes run-time. If you find this too long and invasive, I propose a compromise and run them nightly. But my preference would be to run them as part of the standard CI to incentivize contributors. And reduce surprises and and follow-up fixes.

Given the reversal in policy this change imply, I'm aiming for the next 8.5.0 minor release. I'm opening this PR as a way to discuss the best strategy forward.

@kdeldycke kdeldycke added f:test runner feature: cli test runner github_actions Pull requests that update GitHub Actions code labels May 22, 2026
@kdeldycke kdeldycke added this to the 8.5.0 milestone May 22, 2026
@kdeldycke kdeldycke marked this pull request as draft May 22, 2026 11:07
@kdeldycke kdeldycke merged commit 5b1aba8 into pallets:main May 23, 2026
24 of 25 checks passed
@Rowlando13
Copy link
Copy Markdown
Collaborator

@kdeldycke I agree with the nightly run.

@kdeldycke kdeldycke deleted the activate-random-and-stress-tests branch May 27, 2026 07:48
@kdeldycke
Copy link
Copy Markdown
Collaborator Author

I think we were overly cautious just running these tests nightly as in the end they only take 5 minutes, not 15: https://github.com/pallets/click/actions/runs/26496766042/usage . To me the cost-benefit is excellent to trigger on every PR: a 5 minute wait is nothing in comparison to the effort and synchronization time required to fix leaky issues from non-core maintainers. But I will let that decision for later.

Thanks @Rowlando13 for the merge!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

f:test runner feature: cli test runner github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants