OCPBUGS-84169: Handle zero-worker HyperShift clusters in daemonset rollout [4.21 backport] by weliang1 · Pull Request #2972 · openshift/cluster-network-operator

weliang1 · 2026-04-24T15:48:36Z

Summary

Backport of #2897 to release-4.21

This PR fixes control-plane upgrade blocking in zero-worker HyperShift clusters by:

Adding early return in daemonSetProgressing() when DesiredNumberScheduled == 0
Treating rollout as complete when there are no nodes to schedule pods on

Problem

In zero-worker HyperShift clusters, the ovnkube-node DaemonSet has no pods to roll out. The existing logic in daemonSetProgressing() would return true indefinitely because NumberAvailable == 0, blocking control-plane upgrades waiting for a node rollout that will never complete.

Solution

Add an early check for DesiredNumberScheduled == 0 and return false (not progressing) since there is nothing to roll out.

Backport Details

Original PR: CORENET-6871: Handle zero-worker HyperShift clusters in daemonset rollout #2897
Jira Bug: https://redhat.atlassian.net/browse/OCPBUGS-84169
Original Bug: https://redhat.atlassian.net/browse/OCPBUGS-32382
Target Version: 4.21

Cherry-picked commits

4fe8ca6 - Handle zero-worker HyperShift clusters in daemonset rollout
04718f8 - Address CodeRabbit review: add docstring and assertion messages

🤖 Generated with Claude Code

Fix daemonSetProgressing() to treat a DaemonSet with DesiredNumberScheduled == 0 as not progressing, by changing the condition from: status.NumberAvailable == 0 to: status.NumberAvailable == 0 && status.DesiredNumberScheduled > 0 This aligns with the existing logic in status_manager.go and fixes control-plane upgrade blocking in zero-worker HyperShift clusters, where the ovnkube-node DaemonSet has no pods to roll out and would otherwise block indefinitely. Add unit tests for daemonSetProgressing() covering the zero-worker case and existing progressing conditions.

- Add doc comment to TestDaemonSetProgressing describing its purpose - Add descriptive failure message to Gomega assertion so test failures report the case name and expected vs actual values Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

openshift-ci-robot · 2026-04-24T15:48:42Z

@weliang1: This pull request references Jira Issue OCPBUGS-84169, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-84169 to depend on a bug targeting a version in 4.22.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Backport of #2897 to release-4.21

This PR fixes control-plane upgrade blocking in zero-worker HyperShift clusters by:

Adding early return in daemonSetProgressing() when DesiredNumberScheduled == 0

Treating rollout as complete when there are no nodes to schedule pods on

Problem

In zero-worker HyperShift clusters, the ovnkube-node DaemonSet has no pods to roll out. The existing logic in daemonSetProgressing() would return true indefinitely because NumberAvailable == 0, blocking control-plane upgrades waiting for a node rollout that will never complete.

Solution

Add an early check for DesiredNumberScheduled == 0 and return false (not progressing) since there is nothing to roll out.

Backport Details

Original PR: CORENET-6871: Handle zero-worker HyperShift clusters in daemonset rollout #2897

Jira Bug: https://redhat.atlassian.net/browse/OCPBUGS-84169

Original Bug: https://redhat.atlassian.net/browse/OCPBUGS-32382

Target Version: 4.21

Cherry-picked commits

4fe8ca6 - Handle zero-worker HyperShift clusters in daemonset rollout

04718f8 - Address CodeRabbit review: add docstring and assertion messages

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

coderabbitai · 2026-04-24T15:48:49Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 59a5f2e5-fca0-42ad-85de-af6ce306d801

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

weliang1 · 2026-04-24T19:35:43Z

/retest

weliang1 · 2026-04-27T13:17:38Z

/retest

weliang1 · 2026-04-27T19:29:28Z

/retest

weliang1 · 2026-04-29T19:26:00Z

/retest

weliang1 · 2026-04-30T17:31:25Z

/verified by @weliang1

openshift-ci-robot · 2026-04-30T17:31:38Z

@weliang1: This PR has been marked as verified by @weliang1.

Details

In response to this:

/verified by @weliang1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

weliang1 · 2026-04-30T17:32:38Z

/retest

weliang1 · 2026-05-01T13:34:35Z

/test e2e-azure-ovn-upgrade

weliang1 · 2026-05-04T18:52:55Z

/test e2e-azure-ovn-dualstack

weliang1 · 2026-05-05T16:19:49Z

/retest

weliang1 · 2026-05-05T22:32:13Z

/test e2e-aws-hypershift-ovn-kubevirt

weliang1 · 2026-05-06T16:03:15Z

/test e2e-azure-ovn-dualstack

weliang1 · 2026-05-06T16:03:17Z

/test e2e-vsphere-ovn-dualstack-primaryv6

weliang1 · 2026-05-06T16:03:25Z

/test e2e-openstack-ovn

weliang1 · 2026-05-06T16:03:27Z

/test e2e-aws-hypershift-ovn-kubevirt

weliang1 · 2026-05-06T19:46:39Z

/test e2e-aws-hypershift-ovn-kubevirt

weliang1 · 2026-05-06T19:46:41Z

/test e2e-azure-ovn-dualstack

weliang1 · 2026-05-06T19:46:45Z

/test e2e-vsphere-ovn-dualstack-primaryv6

weliang1 · 2026-05-07T14:36:05Z

Re test the failed many times, below analysis approved all failures are not related the new code itself:

Test Failure Analysis Complete                                                                                                                                             
                                                                                                                                                                             
Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-vsphere-ovn-dualstack-primaryv6                                                                      
Build ID: 2052112886200799232                 
Error: Cluster installation failed - failed to provision control-plane machines within 15m0s                                                                                                                                                                                                                                                                                                                                                                                                                             
Root Cause                                                                                                                                                                                                                                                                                                                                            
This is a vSphere CI infrastructure issue, NOT a code issue with PR #2972.                                                                                                 
The installation failed because IPv6 DHCP is disabled in the vSphere test environment, but it's required for a dualstack primaryv6 deployment.     

Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-openstack-ovn                                                                                        
Build ID: 2052056976401108992                                                                                                                                              
Error: 1 blocking test failure out of 1832 passed tests                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
Root Cause                                                                                                                                                                                                                                                                                                                                           
This is an OpenStack infrastructure issue, NOT a code issue with PR #2972.                                                                                                                                                                                                                                                                           
The test suite failed due to insufficient CPU resources and a severe metric overflow bug in the OpenStack worker node.     

Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-azure-ovn-dualstack                                                                                  
Build ID: 2052112873068433408                             
Error: Cluster installation failed during bootstrap - timed out waiting for network operator                                                                                                                                                                                                                                                                                                                                                                                                                            
Root Cause                                                                                                                                                                                                                                                                                                                                             
This is an Azure infrastructure issue, NOT a code issue with PR #2972.                                                                                                                                                                                                                                                                                 
The installation failed because Azure Load Balancer NAT rules conflicted during master node provisioning.         

Prow Job: pull-ci-openshift-cluster-network-operator-release-4.21-e2e-aws-hypershift-ovn-kubevirt                                                                          
Build ID: 2052112860506492928                                                                                                                                              
Error: Massive test suite failure - 110 blocking failures out of 1764 tests                                                                                                                                                                                                                                                                                                                                                                                                                                      
Root Cause                                                                                                                                                                                                                                                                                                                                            
This is a HyperShift KubeVirt infrastructure issue, NOT a code issue with PR #2972.                                                                                                                                                                                                                                                                  
The test suite failed due to severe API server performance degradation caused by client rate limiting and widespread timeouts.

@kyrtapz Please override failed job and approve this PR.

kyrtapz · 2026-05-08T07:18:51Z

@weliang1 why didn't you use cherry-pick automation?
/lgtm
/label backport-risk-assessed
/retest-required

retesting required jobs, infra issues likely mean that CNO didn't get a chance to prove it works fine.
We don't usually just override all failing required jobs without a bug reported.

openshift-ci · 2026-05-08T07:19:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kyrtapz, weliang1

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kyrtapz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kyrtapz · 2026-05-08T09:15:26Z

/test e2e-aws-hypershift-ovn-kubevirt

openshift-ci · 2026-05-08T14:39:34Z

@weliang1: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/security	`f63208a`	link	false	`/test security`
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6	`f63208a`	link	false	`/test e2e-vsphere-ovn-dualstack-primaryv6`
ci/prow/e2e-openstack-ovn	`f63208a`	link	false	`/test e2e-openstack-ovn`
ci/prow/e2e-azure-ovn-dualstack	`f63208a`	link	false	`/test e2e-azure-ovn-dualstack`
ci/prow/e2e-aws-hypershift-ovn-kubevirt	`f63208a`	link	false	`/test e2e-aws-hypershift-ovn-kubevirt`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

weliang1 · 2026-05-11T14:48:24Z

@weliang1 why didn't you use cherry-pick automation? /lgtm /label backport-risk-assessed /retest-required

retesting required jobs, infra issues likely mean that CNO didn't get a chance to prove it works fine. We don't usually just override all failing required jobs without a bug reported.

@kyrtapz Should I close this PR and using cherry-pick automation to create a new one?

weliang1 and others added 2 commits April 24, 2026 11:48

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Apr 24, 2026

openshift-ci Bot requested review from arkadeepsen and jcaamano April 24, 2026 15:49

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Apr 30, 2026

openshift-ci Bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label May 8, 2026

openshift-ci Bot assigned anuragthehatter, asood-rh, imatza-rh, jechen0648, mffiedler, rbbratta, weliang1 and kyrtapz May 8, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label May 8, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 8, 2026

Conversation

weliang1 commented Apr 24, 2026

Summary

Problem

Solution

Backport Details

Cherry-picked commits

Uh oh!

openshift-ci-robot commented Apr 24, 2026

Summary

Problem

Solution

Backport Details

Cherry-picked commits

Uh oh!

coderabbitai Bot commented Apr 24, 2026

Review skipped

Uh oh!

weliang1 commented Apr 24, 2026

Uh oh!

weliang1 commented Apr 27, 2026

Uh oh!

weliang1 commented Apr 27, 2026

Uh oh!

weliang1 commented Apr 29, 2026

Uh oh!

weliang1 commented Apr 30, 2026

Uh oh!

openshift-ci-robot commented Apr 30, 2026

Uh oh!

weliang1 commented Apr 30, 2026

Uh oh!

weliang1 commented May 1, 2026

Uh oh!

weliang1 commented May 4, 2026

Uh oh!

weliang1 commented May 5, 2026

Uh oh!

weliang1 commented May 5, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 6, 2026

Uh oh!

weliang1 commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kyrtapz commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented May 8, 2026

Uh oh!

kyrtapz commented May 8, 2026

Uh oh!

openshift-ci Bot commented May 8, 2026

Uh oh!

weliang1 commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

weliang1 commented May 7, 2026 •

edited

Loading

kyrtapz commented May 8, 2026 •

edited

Loading