[feat][monitor] PIP-447 for Customizable Prometheus Labels for Topic Metrics #24991

coderzc · 2025-11-16T16:14:21Z

PIP: https://github.com/apache/pulsar/blob/master/pip/pip-447.md

Motivation

Modifications

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

Documentation

doc
doc-required
doc-not-needed
doc-complete

Matching PR in forked repository

PR in forked repository:

lhotari

One comment about the usage of PolicyName.MAX_SUBSCRIPTIONS

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java

Copilot

Pull request overview

This PR implements PIP-447 to add support for customizable Prometheus labels on topic metrics, allowing operators to attach custom metadata (like SLA tier, team ownership, etc.) to topics that will be exposed as Prometheus labels.

Key changes:

Adds customMetricLabels field to TopicPolicies with clone and validation support
Implements REST APIs, admin client methods, and CLI commands for managing custom metric labels
Integrates custom labels into Prometheus metrics generation in TopicStats
Adds configuration options to enable/disable the feature and control allowed label keys and value lengths

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/TopicPolicies.java	Adds customMetricLabels field with proper cloning and accessor methods
pulsar-common/src/main/java/org/apache/pulsar/common/policies/data/HierarchyTopicPolicies.java	Adds PolicyHierarchyValue for customMetricLabels to support policy hierarchy
pulsar-client-admin-api/src/main/java/org/apache/pulsar/client/admin/TopicPolicies.java	Adds API methods for setting, getting, and removing custom metric labels
pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/TopicPoliciesImpl.java	Implements client-side REST API calls for custom metric labels operations
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java	Adds CLI commands for managing custom metric labels in CmdTopics
pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopicPolicies.java	Adds CLI commands for managing custom metric labels in CmdTopicPolicies
pulsar-client-tools/src/test/java/org/apache/pulsar/admin/cli/CmdTopicPoliciesTest.java	Adds comprehensive tests for CLI command functionality
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java	Adds REST endpoints for custom metric labels operations
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java	Implements business logic for setting, getting, and removing custom metric labels with validation
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractTopic.java	Updates topic policy loading to include customMetricLabels
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/TopicStats.java	Modifies metrics generation to include custom labels in Prometheus output
pulsar-broker/src/main/java/org/apache/pulsar/broker/stats/prometheus/NamespaceStatsAggregator.java	Retrieves and passes custom metric labels to TopicStats for metrics generation
pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/TopicPoliciesTest.java	Adds integration tests for custom metric labels feature including validation scenarios
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java	Adds three configuration properties for the custom metric labels feature
conf/broker.conf	Documents the new configuration properties with examples

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

conf/broker.conf

...ar-client-admin/src/main/java/org/apache/pulsar/client/admin/internal/TopicPoliciesImpl.java

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java

…s and expose them in Prometheus

…tics

…tency in topic statistics

…with CLI commands

…configuration - Add namespace-level allowed_topic_properties_for_metrics field in Policies.java - Add admin API methods for set/get/remove allowedTopicPropertiesForMetrics - Add REST endpoints in Namespaces.java (v2) - Add CLI commands: set-allowed-topic-properties-for-metrics, get-allowed-topic-properties-for-metrics, remove-allowed-topic-properties-for-metrics - Integrate with Prometheus metrics generation to merge broker-level and namespace-level configurations - Add test for namespace-level allowedTopicPropertiesForMetrics

Since PIP-447 now uses topic properties instead of topic policies, this removes: - TopicPolicies.customMetricLabels field and methods - HierarchyTopicPolicies.customMetricLabels field - TopicPolicies admin API methods (set/get/removeCustomMetricLabels) - CLI commands in CmdTopicPolicies and CmdTopics - TopicPoliciesImpl implementation - REST API endpoints in PersistentTopics.java (v2) - Internal methods in PersistentTopicsBase.java - References in AbstractTopic.java - Unit tests in CmdTopicPoliciesTest.java - Renamed test method in PrometheusMetricsLabelsTest.java

…rectly and update usage in NamespacesBase

… operations

…x config validation order

…LabelValueLength and clarify allowedTopicPropertiesForMetrics documentation

…nore hidden directories

… policies

…metrics - Move logic for fetching and filtering custom metric labels to TopicStats - Simplify NamespaceStatsAggregator by delegating label handling to TopicStats - Ensure only allowed topic properties are exposed as custom metric labels

…tats to AbstractTopic

…_METRIC_LABELS for consistency

…xception with detailed error messages

coderzc added the area/metrics label Nov 16, 2025

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Nov 16, 2025

coderzc force-pushed the feat_pip_447 branch from d8179f9 to bc8eab2 Compare November 17, 2025 02:42

lhotari requested a review from Copilot January 2, 2026 14:06

Copilot started reviewing on behalf of lhotari January 2, 2026 14:06 View session

lhotari reviewed Jan 2, 2026

View reviewed changes

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/v2/PersistentTopics.java Outdated Show resolved Hide resolved

Copilot AI reviewed Jan 2, 2026

View reviewed changes

coderzc marked this pull request as draft January 5, 2026 02:43

coderzc added 6 commits January 15, 2026 14:57

[feat][metrics] Add support for custom metric labels in topic policie…

af97bb0

…s and expose them in Prometheus

[feat][metrics] Enhance custom metric label handling for topic statis…

5e2d51f

…tics

[feat][metrics] Update custom metric label handling to improve consis…

19c898f

…tency in topic statistics

[feat][metrics] Implement custom metric labels management for topics …

cb81da8

…with CLI commands

[feat][metrics] Add option to remove all custom metric labels for topics

9ccf29a

Validate custom metric label keys and update configuration handling

522e42f

coderzc force-pushed the feat_pip_447 branch from bc8eab2 to 522e42f Compare January 15, 2026 12:54

coderzc added 4 commits January 15, 2026 21:01

Update topic policy operations to use CUSTOM_METRIC_LABELS

785428e

Merge branch 'master' into feat_pip_447

a9853c2

Filter custom metric labels by allowed keys in NamespaceStatsAggregator

d72f97a

Enhance custom metric label removal logic in TopicPolicies and CmdTopics

da1b08c

coderzc force-pushed the feat_pip_447 branch from ea01540 to da1b08c Compare January 15, 2026 13:31

coderzc added 7 commits January 16, 2026 00:43

Add PrometheusMetricsLabelsTest.java

1a15c0a

Using topic properties instead of topic policy

3139520

Refactor custom metric label key validation to accept allowed keys di…

30b2bce

…rectly and update usage in NamespacesBase

Refactor allowedTopicPropertiesForMetrics namespace APIs to use async…

b17260f

… operations

Add tests for namespace-level allowedTopicPropertiesForMetrics and fi…

ac4a063

…x config validation order

coderzc self-assigned this Feb 12, 2026

coderzc added type/PIP ready-to-test labels Feb 12, 2026

coderzc closed this Feb 12, 2026

coderzc reopened this Feb 12, 2026

coderzc marked this pull request as ready for review February 12, 2026 02:04

coderzc added 2 commits February 12, 2026 10:08

Update broker.conf and ServiceConfiguration to remove maxCustomMetric…

7039799

…LabelValueLength and clarify allowedTopicPropertiesForMetrics documentation

Remove zookeeper dependency and update license plugin exclusion to ig…

c431957

…nore hidden directories

coderzc force-pushed the feat_pip_447 branch from 28db9d2 to c431957 Compare February 12, 2026 02:17

coderzc added 6 commits February 12, 2026 10:22

Merge branch 'master' into feat_pip_447

d884679

improve code

ab1acc0

Treat null as unset for allowedTopicPropertiesForMetrics in namespace…

74b4dd0

… policies

[improve][pip] Move custom Metrics label extraction logic from TopicS…

e07ad86

…tats to AbstractTopic

revert code

804b972

coderzc force-pushed the feat_pip_447 branch from 4b7fb7b to 804b972 Compare February 12, 2026 08:38

[improve][pip] Rename PolicyName.CUSTOM_METRIC_LABELS to ALLOW_CUSTOM…

30a7c01

…_METRIC_LABELS for consistency

github-actions bot added the PIP label Feb 12, 2026

coderzc added this to the 4.2.0 milestone Feb 12, 2026

coderzc added the type/feature The PR added a new feature or issue requested a new feature label Feb 12, 2026

coderzc requested review from BewareMyPower, Technoboy-, codelipenghui, lhotari, poorbarcode and shibd February 12, 2026 09:42

coderzc force-pushed the feat_pip_447 branch from 0c837f1 to 8c5d6f5 Compare February 12, 2026 10:13

fix doc

2e1cc8a

coderzc force-pushed the feat_pip_447 branch from 8c5d6f5 to 2e1cc8a Compare February 12, 2026 10:31

[improve][pip] Refactor custom metric label validation to throw RestE…

4563d3a

…xception with detailed error messages

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat][monitor] PIP-447 for Customizable Prometheus Labels for Topic Metrics #24991

[feat][monitor] PIP-447 for Customizable Prometheus Labels for Topic Metrics #24991

coderzc commented Nov 16, 2025 •

edited

Loading

Uh oh!

lhotari left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[feat][monitor] PIP-447 for Customizable Prometheus Labels for Topic Metrics #24991

Are you sure you want to change the base?

[feat][monitor] PIP-447 for Customizable Prometheus Labels for Topic Metrics #24991

Conversation

coderzc commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Matching PR in forked repository

Uh oh!

lhotari left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderzc commented Nov 16, 2025 •

edited

Loading