[FEATURE] Added a "Never Worked" state to the tests#1071
Open
pulk17 wants to merge 1 commit intoCCExtractor:masterfrom
Open
[FEATURE] Added a "Never Worked" state to the tests#1071pulk17 wants to merge 1 commit intoCCExtractor:masterfrom
pulk17 wants to merge 1 commit intoCCExtractor:masterfrom
Conversation
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



[FEATURE] Add "Never Worked" status for regression tests
In raising this pull request, I confirm the following (please check boxes):
My familiarity with the project is as follows (check one):
What's the problem?
Right now, every regression test is either Pass or Fail. The issue is that some samples have never worked with any version of CCExtractor — they're broken by nature, not because of anything a PR did. When these tests fail in CI, they show up as regressions in the PR comment alongside tests that the PR genuinely broke. This makes it harder to see what actually needs attention, and it's unfair to contributors whose PRs get flagged for failures that existed long before their changes.
There's currently no way to communicate "yes, this test is failing, but it has always failed and this is a known pre-existing issue" — the system just treats it the same as a real regression.
What this PR does
Adds a third classification: Never Worked.
Admins can flag any regression test as "never worked" from the test edit and add pages using a simple checkbox. Once flagged, the test still runs as normal, but its failure is interpreted differently across the whole system:
extra_failed_tests), never-worked tests are excluded from that decision entirelyChanges
Database:
never_workedboolean column on theregression_testtable,NOT NULL, defaults tofalseBackend:
TestResultStatusenum (passed/failed/never_worked) inmod_test/models.py— this is derived at query time from test output and thenever_workedflag, it is not stored in the databaseget_test_results()inmod_test/controllers.pynow classifies each test into one of the three states and initializesstatusanderrorfields upfront for type safety. Also cachesget_customized_regressiontests()into a set to avoid repeated calls inside the comprehensionget_info_for_pr_comment()inmod_ci/controllers.pybuckets never-worked tests into their own list, separate fromextra_failedandcommon_failedcomment_pr()now explicitly returnsStatus.SUCCESSorStatus.FAILUREbased solely on whether the PR introduced new regressionsPrCommentInfodataclass updated with the newnever_worked_testsfieldRegressionTestmodel and constructor updated with the new fieldBooleanFieldfor the flagTemplates:
pr_comment.txt— new section for never-worked tests, rendered only when presentregression/index.html— "Never worked" badge on flagged tests in the listregression/test_add.htmlandtest_edit.html— checkbox for the flag with a short explanationregression/test_view.html— status field showing Normal or Never Workedtest/by_id.html— per-row "Never worked" label in results table, amber category header when all failures in a category are never-workedTests:
test_comments_successfuly_in_failed_pr_testto includestatusin the mock data and assert the return value isStatus.FAILUREtest_get_info_for_pr_comment_buckets_tests_correctlycovering all four cases: passed, new regression, pre-existing failure, and never-worked — verifying each ends up in the right list