Skip to content

[kafka_consumer] Collect topic & partition disk size via DescribeLogDirs#23002

Draft
piochelepiotr wants to merge 2 commits intomasterfrom
piotr.wolski/kafka-topic-partition-disk-size
Draft

[kafka_consumer] Collect topic & partition disk size via DescribeLogDirs#23002
piochelepiotr wants to merge 2 commits intomasterfrom
piotr.wolski/kafka-topic-partition-disk-size

Conversation

@piochelepiotr
Copy link
Contributor

Summary

  • Add kafka.topic.disk_size and kafka.partition.disk_size metrics using the DescribeLogDirs admin API
  • Metrics are tagged with topic, partition (partition only), broker_id, log_dir, and kafka_cluster_id
  • Per-broker futures are resolved individually so a single broker failure doesn't block the rest

Test plan

  • Unit tests for _collect_log_dir_metadata with mocked describe_log_dirs response
  • E2E test against a multi-broker Kafka cluster to verify metrics and tags
  • Verify error handling when a broker returns an error in its log dir response

🤖 Generated with Claude Code

piochelepiotr and others added 2 commits March 20, 2026 12:15
…gDirs

Add kafka.topic.disk_size and kafka.partition.disk_size metrics using
the DescribeLogDirs admin API to report per-topic and per-partition
on-disk sizes, tagged by log directory.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tag partition.disk_size and topic.disk_size with broker_id from the
DescribeLogDirs response, and resolve per-broker futures individually.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant