Fix disk removal deadlock during rolling upgrade#254
Open
dobrerazvan wants to merge 3 commits into
Open
Conversation
When a broker pod is deleted during rolling upgrade and a disk removal is pending (GracefulDiskRemovalScheduled), the operator enters a deadlock: reconcileKafkaPvc blocks the entire reconcile with "Disk removal pending", preventing reconcileKafkaPod from recreating the missing pod. Meanwhile, Cruise Control cannot complete the disk removal because the broker isn't running. Fix: move runningBrokers map building before reconcileKafkaPvc and pass it in. Before returning the blocking error, check if any broker with pending disk removal has a missing pod. If so, allow the reconcile to proceed so the pod can be recreated. The disk removal check is re-evaluated on the next cycle once the broker is back up. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… add tests - Fix #1 (HIGH): Override now checks IsDiskRebalance() in addition to IsDiskRemoval(), closing the same deadlock vector for rebalance states - Fix #2 (LOW): Include mountPath in the bypass log message for consistency with other disk-removal log messages - Fix #3 (LOW): Add tests for rebalance-state deadlock bypass and for newly-marked-for-removal with missing pod Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Proposal, design, and task tracking for the fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a broker pod is deleted during rolling upgrade and a disk removal is pending (GracefulDiskRemovalScheduled), the operator enters a deadlock: reconcileKafkaPvc blocks the entire reconcile with "Disk removal pending", preventing reconcileKafkaPod from recreating the missing pod. Meanwhile, Cruise Control cannot complete the disk removal because the broker isn't running.
Fix: move runningBrokers map building before reconcileKafkaPvc and pass it in. Before returning the blocking error, check if any broker with pending disk removal has a missing pod. If so, allow the reconcile to proceed so the pod can be recreated. The disk removal check is re-evaluated on the next cycle once the broker is back up.
Description
Please provide a meaningful description of what this change will do, or is for. Bonus points for including links to
related issues, other PRs, or technical references.
Note that by not including a description, you are asking reviewers to do extra work to understand the context of this
change, which may lead to your PR taking much longer to review, or result in it not being reviewed at all.
Type of Change
Checklist