feat(helm): Add monitoring.scrapeAnnotations to control whether or not to add prometheus.io/* annotations to Service#825
Open
azagarelz wants to merge 1 commit into
Conversation
azagarelz
changed the title
…nnotations When monitoring.enabled is true (or auto with CRDs installed), the chart creates ServiceMonitor CRDs that instruct the Prometheus Operator to scrape CloudZero Agent metrics. In that configuration, the prometheus.io/* annotations on Services become redundant. In clusters where both annotation-based and CRD-based discovery are active simultaneously, metrics may be scraped twice, generating unnecessary load on the metrics pipeline. Adds components.monitoring.scrapeAnnotations (default: true) to address this. When set to false, prometheus.io/* annotations are omitted from the agent, aggregator, and webhook Services, eliminating the risk of double scraping while keeping ServiceMonitor-based discovery intact. Updates monitoring-infrastructure.md to document the new flag and the double scraping scenario, and adds helm unit tests covering both true and false states for all three affected Services. The default value of true preserves full backward compatibility for users relying on annotation-based Prometheus discovery without the Operator. This approach is purely additive, users without ServiceMonitors are unaffected, while Operator users gain a clean way to eliminate redundant scrape targets. A future consideration would be auto-disabling annotations when monitoring.enabled is true, though that would be a breaking change requiring a major version bump.
azagarelz
force-pushed
the
feat/monitoring-scrape-annotations-toggle
branch
from
f570231 to
20a7bf7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why?
When
components.monitoring.enabledistrue(orautowith CRDSs installed), the chart creates ServiceMonitor CRDs that instruct the Prometheus Operator on how to scrape CZ metrics. In that configuration, theprometheus.io/*annotations on Services become redundant.In clusters where both annotation-based discovery and CRD-based discovery are active simultaneously, CZ metrics may be scraped twice, generating unnecessary load.
Setting
scrapeAnnotations: falseremoves the annotations from Services, eliminating the risk of double scraping while keeping the ServiceMonitor based discovery intact. The default remainstruefor full backward compatibility.What
components.monitoring.scrapeAnnotationsvalue (default:true)false, removeprometheus.io/*annotations from the agent, aggregator, and webhook Servicesmonitoring-infrastructure.mdto document the new flag and the double scraping scenarioHow Tested
Helm unit tests added in
charts/cloudzero-agent/tests/defaults_service_test.yamlcovering:prometheus.io/scrapeannotation present on all three ServicesscrapeAnnotations: false