TRACING-6127: feat: add span_kind filter, show p95 latency instead of avg operation duration

[andreasgerstmayr]

Follow-up changes to #1043 (this PR depends on #1043):

• add Span Kind variable to filter by span kind (defaults to SPAN_KIND_SERVER to avoid double-counting)
• rename "Duration" to "Latency", and use P95 histogram quantile instead of average operation duration

[openshift-ci-robot]

@andreasgerstmayr: This pull request references TRACING-6127 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Follow-up changes to #1043 (this PR depends on #1043):

• add Span Kind variable to filter by span kind (defaults to SPAN_KIND_SERVER to avoid double-counting)
• rename "Duration" to "Latency", and use P95 histogram quantile instead of average operation duration
• update error rate unit

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Hi @andreasgerstmayr. Thanks for your PR.

I'm waiting for a rhobs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

Important

Review skipped

Auto reviews are limited based on label configuration.

🚫 Review skipped — only excluded labels are configured. (1)

• work-in-progress

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: e289f225-9716-48f3-8fcf-f019617fea50

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Renamed service panel from "Duration" to "Latency" and switched P95 latency computation to histogram_quantile; updated operations table units/headers and request-rate series name; added a multi-select span_kind dashboard variable and wired span_kind=~"${span_kind}" into variable matchers.

Changes

Cohort / File(s)

Summary

APM dashboard changes pkg/controllers/uiplugin/apm.go

Renamed service panel group metric header (Duration → Latency); replaced P95 duration ratio (_sum/_count) with histogram_quantile(.95, sum(rate(..._bucket...)) by (span_name, le)) and set series name to "P95 Latency"; changed operations table "Error rate" unit to RequestsPerSecondsUnit and request-rate series name to "Request rate"; renamed "Duration" column to "P95 Latency" (ms); added span_kind multi-select dashboard variable (default SPAN_KIND_SERVER, all .*) and added span_kind=~"${span_kind}" to variableMatchers.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main changes: adding span_kind filter and switching from average operation duration to P95 latency, which aligns with the changeset in apm.go.
Description check	✅ Passed	The pull request description clearly relates to the changeset, mentioning span kind variable addition, renaming Duration to Latency, and using P95 histogram quantile instead of average operation duration.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

pkg/controllers/uiplugin/apm.go (1)

101-107: Minor: Error rate unit displays as "req/s" which may be slightly misleading.

Using RequestsPerSecondsUnit for error rate will display as "req/s", but errors aren't requests. This is a minor semantic mismatch. If Perses provides a more generic "per second" unit, that would be more accurate. Otherwise, this is acceptable for consistency with the request rate column.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/controllers/uiplugin/apm.go` around lines 101 - 107, The Error rate
column is using common.RequestsPerSecondsUnit (which renders as "req/s") —
update the Format.Unit for the metric with Header "Error rate" (the block where
Name is "value `#2`" and Format is a *common.Format) to use a more generic
per-second unit if Perses exposes one (e.g., common.PerSecondUnit or similar)
instead of RequestsPerSecondsUnit; if no generic unit exists, leave as-is for
consistency but add a short inline comment explaining the semantic mismatch.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/controllers/uiplugin/apm.go`:
- Around line 101-107: The Error rate column is using
common.RequestsPerSecondsUnit (which renders as "req/s") — update the
Format.Unit for the metric with Header "Error rate" (the block where Name is
"value `#2`" and Format is a *common.Format) to use a more generic per-second unit
if Perses exposes one (e.g., common.PerSecondUnit or similar) instead of
RequestsPerSecondsUnit; if no generic unit exists, leave as-is for consistency
but add a short inline comment explaining the semantic mismatch.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4943b1c9-76cd-4c01-8bef-909e9d321b50

📥 Commits

Reviewing files that changed from the base of the PR and between cf377a3 and a946f4d.

📒 Files selected for processing (1)

• pkg/controllers/uiplugin/apm.go

[jgbernalp]

/ok-to-test

[jgbernalp]

/hold
wait until #1043 is merged, feel free to unhold if it is.

[jgbernalp]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreasgerstmayr, jgbernalp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/controllers/uiplugin/OWNERS [jgbernalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

🧹 Nitpick comments (1)

pkg/controllers/uiplugin/apm.go (1)

183-185: Scope span_kind label-values query to active filters.

Consider including namespace="$namespace", service="$collector", and service_name="$service" in this matcher so variable options stay contextual and avoid broad cluster-wide scans.

Proposed diff

 				labelvalues.PrometheusLabelValues("span_kind",
-					labelvalues.Matchers(`{__name__=~"traces_span_metrics_calls(_total)?"}`),
+					labelvalues.Matchers(`{__name__=~"traces_span_metrics_calls(_total)?", namespace="$namespace", service="$collector", service_name="$service"}`),
 				),

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/controllers/uiplugin/apm.go` around lines 183 - 185, The Prometheus
label-values query for "span_kind" is currently unscoped and can return
cluster-wide values; update the call to labelvalues.PrometheusLabelValues used
with labelvalues.Matchers so the matcher includes the active filter labels
namespace="$namespace", service="$collector", and service_name="$service" (i.e.,
add these matchers alongside `{__name__=~"traces_span_metrics_calls(_total)?"}`)
to ensure span_kind options are contextual to the selected
namespace/collector/service; locate the invocation of
labelvalues.PrometheusLabelValues and adjust the labelvalues.Matchers expression
accordingly.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@pkg/controllers/uiplugin/apm.go`:
- Around line 183-185: The Prometheus label-values query for "span_kind" is
currently unscoped and can return cluster-wide values; update the call to
labelvalues.PrometheusLabelValues used with labelvalues.Matchers so the matcher
includes the active filter labels namespace="$namespace", service="$collector",
and service_name="$service" (i.e., add these matchers alongside
`{__name__=~"traces_span_metrics_calls(_total)?"}`) to ensure span_kind options
are contextual to the selected namespace/collector/service; locate the
invocation of labelvalues.PrometheusLabelValues and adjust the
labelvalues.Matchers expression accordingly.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c8b66647-736c-42fd-82b2-5b86be1ae9d3

📥 Commits

Reviewing files that changed from the base of the PR and between a946f4d and 5306ca9.

📒 Files selected for processing (1)

• pkg/controllers/uiplugin/apm.go

[openshift-ci-robot]

@andreasgerstmayr: This pull request references TRACING-6127 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Follow-up changes to #1043 (this PR depends on #1043):

• add Span Kind variable to filter by span kind (defaults to SPAN_KIND_SERVER to avoid double-counting)
• rename "Duration" to "Latency", and use P95 histogram quantile instead of average operation duration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[andreasgerstmayr]

/retest

[jgbernalp]

/unhold

[jgbernalp]

/test observability-operator-e2e

[jan--f]

/retest
/lgtm