Drop fallback release queries from 3 to 1.#3553
Conversation
This change stops checking 4 total prior releases for basis, and instead checks only 2. Based on the data I can see and observed behavior, we are not benefiting much from being this sensitive. Flakes are not at true consistent rates, if there's a problem, we will typically see it at some point in time and now have many people monitoring for problems. Data shows that less 2-4% of regressions matched beyond the prior two base releases, and this does not account for those that also would have matched against a newer one, we just found a better pass percentage by going further. (I do not have this data) Reducing to just check our base release plus one fallback will save considerable query costs, and still catch over 96% of regressions we catch today, factoring in those that would still match we're probably around 98-99%.
|
Pipeline controller notification For optional jobs, comment This repository is configured in: automatic mode |
WalkthroughThe PR parameterizes the fallback release count in the component readiness middleware. A new constant controls the default fallback limit, a wrapper function is introduced, and the underlying ChangesFallback Release Parameterization
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes 🚥 Pre-merge checks | ✅ 15 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (15 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/hold Checking what this would do to 4.22 and 5.0 today |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgoodwin The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Scheduling required tests: |
|
/hold cancel No change to regression counts for 4.22 or 5.0 on the active boards when reducing from 3 to 1. |
|
@dgoodwin: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This change stops checking 4 total prior releases for basis, and instead
checks only 2. Based on the data I can see and observed behavior, we are
not benefiting much from being this sensitive. Flakes are not at true
consistent rates, if there's a problem, we will typically see it at some
point in time and now have many people monitoring for problems.
Data shows that less 2-4% of regressions matched beyond the prior two
base releases, and this does not account for those that also would have
matched against a newer one, we just found a better pass percentage by
going further. (I do not have this data)
Reducing to just check our base release plus one fallback will save
considerable query costs, and still catch over 96% of regressions we
catch today, factoring in those that would still match we're probably
around 98-99%.
Testing is preserved for 3 so we can keep the functionality if we want it again.
Data from regression tracker on what release we matched against:
sippy_openshift=> select base_release, count(base_release) from test_regressions where release='4.21' and base_release != '4.21' group by base_release order by base_release;
base_release | count
--------------+-------
4.17 | 170
4.18 | 100
4.19 | 481
4.20 | 5764
(4 rows)
sippy_openshift=> select base_release, count(base_release) from test_regressions where release='4.22' and base_release != '4.22' group by base_release order by base_release;
base_release | count
--------------+-------
4.18 | 26
4.19 | 131
4.20 | 649
4.21 | 2923
(4 rows)
sippy_openshift=> select base_release, count(base_release) from test_regressions where release='5.0' and base_release != '5.0' group by base_release order by base_release;
base_release | count
--------------+-------
4.19 | 11
4.20 | 20
4.21 | 529
4.22 | 795
(4 rows)