Skip to content

OCPBUGS-77560: Disable non-azure permafailing hypershift-kubevirt jobs#79396

Open
orenc1 wants to merge 1 commit into
openshift:mainfrom
orenc1:diabled_hypershift_permafailing_jobs
Open

OCPBUGS-77560: Disable non-azure permafailing hypershift-kubevirt jobs#79396
orenc1 wants to merge 1 commit into
openshift:mainfrom
orenc1:diabled_hypershift_permafailing_jobs

Conversation

@orenc1
Copy link
Copy Markdown
Contributor

@orenc1 orenc1 commented May 18, 2026

These kubevirt CI jobs have had 0% pass rate for months, wasting thousands of hours of CI time. Set them to run once a year instead of removing them entirely.

Jobs affected:

release-4.19:

- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.20:

- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.21:

- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.22:

- periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-kubevirt-aws-ovn

Disable permafailing hypershift-kubevirt periodic CI jobs

This PR reduces CI resource waste in the OpenShift release repository by scaling back the frequency of several permafailing Hypershift KubeVirt periodic jobs. It updates CI job schedules in ci-operator config for the hypershift component so the jobs run only once per year (on May 17th) instead of daily or multiple times per day, preserving job definitions while avoiding continuous failures.

What's changing

  • Affected CI configuration: ci-operator/config/openshift/hypershift/ for releases 4.19–4.22.
  • Affected jobs (10 total):
    • Release 4.19 (3 jobs): periodic e2e-kubevirt-aws-ovn, e2e-kubevirt-metal-ovn, e2e-kubevirt-metal-ovn-disconnected
    • Release 4.20 (3 jobs): periodic e2e-kubevirt-aws-ovn, e2e-kubevirt-metal-ovn, e2e-kubevirt-metal-ovn-disconnected
    • Release 4.21 (3 jobs): periodic e2e-kubevirt-aws-ovn, e2e-kubevirt-metal-ovn, e2e-kubevirt-metal-ovn-disconnected
    • Release 4.22 (1 job): periodic e2e-kubevirt-aws-ovn

Implementation

  • For each affected job the cron expression was changed to "0 0 17 5 *" (run once per year on May 17).
  • Only the cron schedule fields were modified in the YAML files; no other job configuration (workflows, cluster profiles, env, capabilities, metadata) was changed.

Rationale

  • These jobs have shown a ~0% pass rate for months. Changing the schedule preserves the jobs (for traceability and occasional validation) while substantially reducing wasted CI runs, per the referenced Jira (OCPBUGS-77560).

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 18, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@orenc1: This pull request references Jira Issue OCPBUGS-77560, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

These kubevirt CI jobs have had 0% pass rate for months, wasting thousands of hours of CI time. Set them to run once a year (May 17th) instead of removing them entirely, per the Jira ticket request.

Jobs affected:

release-4.19:

- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.20:

- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.21:

- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.22:

- periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-kubevirt-aws-ovn

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 18, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 13004a04-f008-4034-9d0c-ec2ae19edeb6

📥 Commits

Reviewing files that changed from the base of the PR and between 9f04820 and 3e1c8ca.

⛔ Files ignored due to path filters (4)
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.19-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.20-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.21-periodics.yaml is excluded by !ci-operator/jobs/**
  • ci-operator/jobs/openshift/hypershift/openshift-hypershift-release-4.22-periodics.yaml is excluded by !ci-operator/jobs/**
📒 Files selected for processing (4)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics.yaml
✅ Files skipped from review due to trivial changes (2)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19__periodics.yaml
🚧 Files skipped from review as they are similar to previous changes (2)
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml
  • ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics.yaml

Walkthrough

This pull request consolidates the periodic test schedules for three KubeVirt OVN-related test jobs across four OpenShift HyperShift release branch configurations. All targeted tests now execute on a unified cron schedule of 0 0 17 5 * (00:00 UTC on May 17th), replacing their previous distinct daily, weekly, and multi-hour schedules.

Changes

KubeVirt Periodic Test Schedule Consolidation

Layer / File(s) Summary
Update e2e-kubevirt-metal-ovn across releases
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19__periodics.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml
e2e-kubevirt-metal-ovn cron changed to 0 0 17 5 * in 4.19, 4.20, and 4.21.
Update e2e-kubevirt-metal-ovn-disconnected across releases
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19__periodics.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml
e2e-kubevirt-metal-ovn-disconnected cron changed to 0 0 17 5 * in 4.19, 4.20, and 4.21.
Update e2e-kubevirt-aws-ovn across releases
ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.19__periodics.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.20__periodics.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.21__periodics.yaml, ci-operator/config/openshift/hypershift/openshift-hypershift-release-4.22__periodics.yaml
e2e-kubevirt-aws-ovn cron changed to 0 0 17 5 * in 4.19, 4.20, 4.21, and 4.22 (previously multi-run daily schedules in some branches).

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Suggested reviewers

  • muraee
  • enxebre
  • Nirshal
🚥 Pre-merge checks | ✅ 12
✅ Passed checks (12 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: disabling (reducing frequency of) non-azure permafailing hypershift-kubevirt jobs across multiple release versions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR modifies only YAML CI configuration files with cron schedules. No Ginkgo test source files or test names are present in the modified files. The check is not applicable.
Test Structure And Quality ✅ Passed The custom check applies to Ginkgo test code quality, but this PR only modifies CI configuration YAML files. No test code exists in this PR, so the check is not applicable.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests added. PR only modifies cron schedules in CI YAML config files for existing hypershift periodic jobs. Custom check applies only to new test code additions.
Single Node Openshift (Sno) Test Compatibility ✅ Passed This PR does not add new Ginkgo e2e tests. It only modifies YAML CI job configuration files to update cron schedules. The SNO compatibility check applies only to new test code additions.
Topology-Aware Scheduling Compatibility ✅ Passed Custom check not applicable. PR modifies only CI periodic job cron schedules in ci-operator/config, not deployment manifests, operator code, or controllers with scheduling constraints.
Ote Binary Stdout Contract ✅ Passed Check not applicable. PR modifies only YAML CI configuration files (cron schedules), not Go code, test binaries, or OTE extensions. No stdout violations possible.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This custom check applies only to new Ginkgo e2e test code additions. The PR exclusively modifies YAML CI configuration files (cron schedules) with no new test code additions.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

These kubevirt CI jobs have had 0% pass rate for months, wasting CI resources.
Set them to run once a year instead of removing them entirely.

Jobs affected:

release-4.19:
- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.20:
- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.21:
- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-aws-ovn
- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn
- periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn-disconnected

release-4.22:
- periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-kubevirt-aws-ovn

Signed-off-by: Oren Cohen <ocohen@redhat.com>
Assisted-by: Claude <noreply@anthropic.com>
@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 18, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: orenc1
Once this PR has been reviewed and has the lgtm label, please assign devguyio for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot requested review from Nirshal and sdminonne May 18, 2026 09:18
@orenc1 orenc1 force-pushed the diabled_hypershift_permafailing_jobs branch from 9f04820 to 3e1c8ca Compare May 18, 2026 09:19
@openshift-merge-bot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@orenc1: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn-disconnected N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn-disconnected N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn-disconnected N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn N/A periodic Ci-operator config changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@orenc1: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn-disconnected N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-kubevirt-metal-ovn-disconnected N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn-disconnected N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.22-periodics-e2e-kubevirt-aws-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-kubevirt-metal-ovn N/A periodic Ci-operator config changed
periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-metal-ovn N/A periodic Ci-operator config changed
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@orenc1
Copy link
Copy Markdown
Contributor Author

orenc1 commented May 18, 2026

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 18, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@orenc1: This pull request references Jira Issue OCPBUGS-77560, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (yli2@redhat.com), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 18, 2026

@orenc1: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copy link
Copy Markdown
Contributor

@jparrill jparrill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropped a couple of questions. Thanks!

capabilities:
- intranet
cron: 31 0 * * 0
cron: 0 0 17 5 *
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All 10 jobs share the same cron (0 0 17 5 *), so they'll all fire at midnight on May 17 simultaneously. That could overload the CI nodes, especially the metal jobs on equinix-ocp-hcp. Could we spread them across different hours to avoid the spike?

workflow: hypershift-kubevirt-azure-conformance
- as: e2e-kubevirt-aws-ovn
cron: 0 0,6,12,18 * * *
cron: 0 0 17 5 *
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same kubevirt jobs (e2e-kubevirt-aws-ovn, e2e-kubevirt-metal-ovn, e2e-kubevirt-metal-ovn-disconnected) are still running on active schedules in release-4.23 and 5.0. If the kubevirt infra issues affect all releases, those are still burning CI time. Were 4.23/5.0 intentionally left out?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants