feat: batch remove_broker operations into single CruiseControl task#251
Open
dobrerazvan wants to merge 1 commit into
Open
feat: batch remove_broker operations into single CruiseControl task#251dobrerazvan wants to merge 1 commit into
dobrerazvan wants to merge 1 commit into
Conversation
When removing multiple brokers from a KafkaCluster spec, the operator previously created one CruiseControlOperation per broker. CC can only run one operation at a time, causing data to be shuffled onto brokers that will themselves be decommissioned, far more partition movements than necessary. Collect all broker IDs in GracefulDownscaleRequired state and submit them as a single remove_broker CC operation, mirroring the existing addBrokers pattern. Because the KafkaCluster reconciler sets all removed brokers to Required via a single atomic status patch, all brokers from a single manifest apply are guaranteed to land in the same batch. Includes unit test (createCCOperation multi-broker params), integration test (exactly one CCOperation created for two simultaneous removals), e2e test scaffold, and 5-broker sample manifest. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When removing multiple brokers from a KafkaCluster spec, the operator previously created one CruiseControlOperation per broker. CC can only run one operation at a time, causing data to be shuffled onto brokers that will themselves be decommissioned, far more partition movements than necessary.
Collect all broker IDs in GracefulDownscaleRequired state and submit them as a single remove_broker CC operation, mirroring the existing addBrokers pattern. Because the KafkaCluster reconciler sets all removed brokers to Required via a single atomic status patch, all brokers from a single manifest apply are guaranteed to land in the same batch.
Includes unit test (createCCOperation multi-broker params), integration test (exactly one CCOperation created for two simultaneous removals), e2e test scaffold, and 5-broker sample manifest.
Description
Please provide a meaningful description of what this change will do, or is for. Bonus points for including links to
related issues, other PRs, or technical references.
Note that by not including a description, you are asking reviewers to do extra work to understand the context of this
change, which may lead to your PR taking much longer to review, or result in it not being reviewed at all.
Type of Change
Checklist