Skip to content

OCPEDGE-2381: Validate no WAL corruption when both nodes shutdown gracefully#30925

Open
kasturinarra wants to merge 1 commit intoopenshift:mainfrom
kasturinarra:automate_case_86430
Open

OCPEDGE-2381: Validate no WAL corruption when both nodes shutdown gracefully#30925
kasturinarra wants to merge 1 commit intoopenshift:mainfrom
kasturinarra:automate_case_86430

Conversation

@kasturinarra
Copy link
Contributor

No description provided.

@openshift-ci-robot
Copy link

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 24, 2026
@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 24, 2026

@kasturinarra: This pull request references OCPEDGE-2381 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link

coderabbitai bot commented Mar 24, 2026

Walkthrough

A new test case is added to the etcd recovery test suite that validates cluster recovery behavior when both nodes undergo simultaneous graceful shutdown. The test schedules reboot on both nodes, waits 90 seconds, validates etcd recovery state with expected member roles, and verifies that the openshift-etcd container is running on both nodes afterward.

Changes

Cohort / File(s) Summary
Etcd Recovery Testing
test/extended/two_node/tnf_recovery.go
Adds new Ginkgo test case for simultaneous graceful shutdown of both nodes and etcd recovery validation, including verification that openshift-etcd containers are running on both nodes post-recovery. Includes strings import addition.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.11.3)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci bot requested review from jaypoulz and jeff-roche March 24, 2026 09:38
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 24, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kasturinarra
Once this PR has been reviewed and has the lgtm label, please assign jaypoulz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
test/extended/two_node/tnf_recovery.go (1)

413-435: Avoid a second copy of this suite block.

This repeats the Describe/BeforeEach above, and the copy has already drifted by dropping [OCPFeatureGate:DualReplica]. Please add the new It to the existing suite instead so markers and setup stay in one place.

As per coding guidelines, "Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/two_node/tnf_recovery.go` around lines 413 - 435, There is a
duplicated Describe/BeforeEach block for the "[sig-etcd] Two Node with Fencing
etcd recovery" suite; remove the second Describe block and instead add the new
test It into the original suite so setup and markers (including the
[OCPFeatureGate:DualReplica] tag) remain intact. Locate the duplicate
Describe(...) block that defines oc, etcdClientFactory, peerNode, targetNode and
its BeforeEach, delete that duplicated Describe/BeforeEach, and move the new It
test body into the existing Describe that already declares oc and uses
BeforeEach/etcdClientFactory so the markers and shared setup are not dropped.
Ensure the moved It references the same oc, etcdClientFactory, peerNode and
targetNode variables and keep all original markers on the suite.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/two_node/tnf_recovery.go`:
- Around line 449-463: The test currently calls
validateEtcdRecoveryState(targetNode, peerNode, ...) without first proving the
scheduled reboots actually happened; capture each node's pre-disruption boot-id
(e.g., via exutil.DebugNodeRetryWithOptionsAndChroot calling "cat
/proc/sys/kernel/random/boot_id") before issuing "shutdown -r 1", then poll both
nodes after the disruption and wait until their boot-id values have changed
(with a timeout) before calling validateEtcdRecoveryState; place this boot-id
capture and change-check around the existing shutdown/recovery logic and use the
same targetNode and peerNode identifiers so the test fails if a node never
rebooted.

---

Nitpick comments:
In `@test/extended/two_node/tnf_recovery.go`:
- Around line 413-435: There is a duplicated Describe/BeforeEach block for the
"[sig-etcd] Two Node with Fencing etcd recovery" suite; remove the second
Describe block and instead add the new test It into the original suite so setup
and markers (including the [OCPFeatureGate:DualReplica] tag) remain intact.
Locate the duplicate Describe(...) block that defines oc, etcdClientFactory,
peerNode, targetNode and its BeforeEach, delete that duplicated
Describe/BeforeEach, and move the new It test body into the existing Describe
that already declares oc and uses BeforeEach/etcdClientFactory so the markers
and shared setup are not dropped. Ensure the moved It references the same oc,
etcdClientFactory, peerNode and targetNode variables and keep all original
markers on the suite.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1c865daf-4792-4b5c-b9f9-dcd740484608

📥 Commits

Reviewing files that changed from the base of the PR and between 02f8514 and 011bd7f.

📒 Files selected for processing (1)
  • test/extended/two_node/tnf_recovery.go

@openshift-ci-robot
Copy link

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 24, 2026

@kasturinarra: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-microshift-serial 011bd7f link true /test e2e-aws-ovn-microshift-serial
ci/prow/e2e-aws-ovn-fips 011bd7f link true /test e2e-aws-ovn-fips

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants