OCPBUGS-65880: Reset StorageVersionMigrationDegraded condition

[jhadvig]

Per bug, we havent been reseting the StorageVersionMigrationDegraded condition when the degradation cause got fixed.
Due to that Ive created a syncStorageVersionMigration method to perform all the SVM related actions and return appropriate reason and error which will be used to set/unset the condition and flush.

/assign @TheRealJon

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-65880, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Per bug, we havent been reseting the StorageVersionMigrationDegraded condition when the degradation cause got fixed.
Due to that Ive created a syncStorageVersionMigration method to perform all the SVM related actions and return appropriate reason and error which will be used to set/unset the condition and flush.

/assign @TheRealJon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jhadvig]

/jira refresh

[openshift-ci-robot]

@jhadvig: This pull request references Jira Issue OCPBUGS-65880, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.21.0) matches configured target version for branch (4.21.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[Leo6Leo]

Review
/assign @Leo6Leo

[Leo6Leo]

The problem is the controller never calls HandleDegraded() to clear the condition, and it only calls to set the condition, therefore causing it to stuck in degraded forever. jakub's pr handles it pretty well by extracting the logic and centralizing condition management

// setting reason and err to empty string will clear the degraded status
statusHandler.AddCondition(status.HandleDegraded("StorageVersionMigration", reason, err))

and the changes looks good to me, ping QE for verification
/assign @yapei

/lgtm

Test seems like a flaky cypress test, retesting:
/retest-required

[jhadvig]

/retest

[yapei]

1. Create verification cluster (launch 4.21.0-0.nightly-2025-11-26-140450,openshift/console-operator#1071 aws)
2. Manually update consoles.operator status

$ oc get clusterversion
NAME      VERSION                                                AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.21.0-0-2025-11-27-074856-test-ci-ln-rrjynmb-latest   True        False         18m     Cluster version is 4.21.0-0-2025-11-27-074856-test-ci-ln-rrjynmb-latest


$ oc get consoles.operator.openshift.io cluster -o json | jq '.status.conditions[] | select(.type == "StorageVersionMigrationDegraded")'
{
  "lastTransitionTime": "2025-11-27T08:07:46Z",
  "message": "rpc error: code = Unavailable desc = error reading from server",
  "reason": "FailedCheckCRDStoredVersions",
  "status": "True",
  "type": "StorageVersionMigrationDegraded"
}

3. Monitor co/console status, but I didn't see the error status was cleared/reset

$ oc get co console
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-11-27-074856-test-ci-ln-rrjynmb-latest   True        False         True       29m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
$ oc get co console
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-11-27-074856-test-ci-ln-rrjynmb-latest   True        False         True       34m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
oc get co console
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-11-27-074856-test-ci-ln-rrjynmb-latest   True        False         True       42m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server

[jhadvig]

@yapei thank you for review. Added resync every 10minutes 👍

[TheRealJon]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig, Leo6Leo, TheRealJon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [TheRealJon,jhadvig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@jhadvig: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[yapei]

StorageVersionMigrationDegraded status can be re-sync and reset

$ oc edit consoles.operator.openshift.io cluster --subresource=status
console.operator.openshift.io/cluster edited
$ oc get consoles.operator.openshift.io cluster -o json | jq '.status.conditions[] | select(.type == "StorageVersionMigrationDegraded")'
{
  "lastTransitionTime": "2025-12-03T01:33:34Z",
  "message": "rpc error: code = Unavailable desc = error reading from server",
  "reason": "FailedCheckCRDStoredVersions",
  "status": "True",
  "type": "StorageVersionMigrationDegraded"
}
$ while true; do oc get co console; sleep 60; done
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       51m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       52m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       53m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       54m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       55m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       56m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       57m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       58m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         False      59m
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         False      60m

/verified by @yapei

[openshift-ci-robot]

@yapei: This PR has been marked as verified by @yapei.

Details

In response to this:

StorageVersionMigrationDegraded status can be re-sync and reset

$ oc edit consoles.operator.openshift.io cluster --subresource=status
console.operator.openshift.io/cluster edited
$ oc get consoles.operator.openshift.io cluster -o json | jq '.status.conditions[] | select(.type == "StorageVersionMigrationDegraded")'
{
 "lastTransitionTime": "2025-12-03T01:33:34Z",
 "message": "rpc error: code = Unavailable desc = error reading from server",
 "reason": "FailedCheckCRDStoredVersions",
 "status": "True",
 "type": "StorageVersionMigrationDegraded"
}
$ while true; do oc get co console; sleep 60; done
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       51m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       52m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       53m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       54m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       55m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       56m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       57m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         True       58m     StorageVersionMigrationDegraded: rpc error: code = Unavailable desc = error reading from server
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         False      59m
NAME      VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
console   4.21.0-0-2025-12-03-011123-test-ci-ln-0tmm1tb-latest   True        False         False      60m

/verified by @yapei

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@jhadvig: Jira Issue Verification Checks: Jira Issue OCPBUGS-65880
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-65880 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Per bug, we havent been reseting the StorageVersionMigrationDegraded condition when the degradation cause got fixed.
Due to that Ive created a syncStorageVersionMigration method to perform all the SVM related actions and return appropriate reason and error which will be used to set/unset the condition and flush.

/assign @TheRealJon

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[Leo6Leo]

/cherry-pick release-4.20

[openshift-cherrypick-robot]

@Leo6Leo: new pull request created: #1085

Details

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.