Skip to content

Skip JMX version fetch when broker status is already current#249

Open
hvan wants to merge 1 commit into
feature/reconcile-improvementsfrom
feature/broker-version-update-guard
Open

Skip JMX version fetch when broker status is already current#249
hvan wants to merge 1 commit into
feature/reconcile-improvementsfrom
feature/broker-version-update-guard

Conversation

@hvan
Copy link
Copy Markdown
Collaborator

@hvan hvan commented May 12, 2026

The PR addresses two performance problems in the broker reconcile loop:

1. Eliminating redundant JMX calls

Previously, every reconcile cycle made a JMX HTTP call to every broker — even if nothing had changed. Since the reconciler runs continuously, this meant constant
network traffic to all brokers at all times. The new brokerNeedsVersionUpdate guard checks the cluster's stored state first: if the broker's recorded image and version
already match what's desired, the JMX call is skipped entirely. Only brokers that are new, have an empty version, or had their image changed will trigger a fetch.

2. Parallelizing the JMX calls that do happen

Previously the JMX calls that did run were sequential — broker 0 had to finish before broker 1 started, and so on. In a cluster with N brokers, total wait time was N ×
(JMX latency). The refactored updateStatusWithDockerImageAndVersion fans out all necessary JMX calls concurrently using goroutines and collects results over a
channel, so the total wait time is bounded by the slowest single broker rather than the sum of all of them.

Combined effect: In steady state (no image changes), zero JMX calls are made. During a rolling upgrade, only the brokers being updated trigger calls, and those calls
run in parallel. The 30s HTTP timeout added alongside this prevents any one stalled broker from blocking the reconciler indefinitely.


Summary

  • Skip redundant JMX fetches: Adds brokerNeedsVersionUpdate to guard JMX version extraction — brokers whose recorded image and version already match the desired image
    are skipped, reducing unnecessary network calls on every reconcile loop.

Test plan

  • TestBrokerNeedsVersionUpdate covers all guard conditions: missing state, empty version, stale image, up-to-date image, broker-level image override, and per-broker
    isolation.
  • Run go test ./pkg/resources/kafka/... ./pkg/jmxextractor/... locally.

Adds brokerNeedsVersionUpdate to guard the BrokersState write so that JMX
fetches are skipped for brokers whose recorded image and version already
match the desired image, reducing unnecessary reconcile work.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant