[improve][broker] PIP-442: Add memory limits for topic list watcher (part 2)

[lhotari]

PIP-442

Motivation

See PIP-442.

This is part 2 for the PIP-442 implementation. This covers the topic list watcher operations.
The first part of PIP-442 implementation was #24833.

Modifications

• use memory limits in TopicListService / PulsarCommandSenderImpl so that topic list watcher related operations are covered

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[codecov-commenter]

Codecov Report

❌ Patch coverage is 74.17219% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.43%. Comparing base (b1019ce) to head (7fc04d9).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...apache/pulsar/broker/service/TopicListService.java	76.05%	17 Missing and 17 partials ⚠️
...pulsar/broker/service/PulsarCommandSenderImpl.java	33.33%	0 Missing and 2 partials ⚠️
...r/common/semaphore/AsyncDualMemoryLimiterImpl.java	0.00%	2 Missing ⚠️
...he/pulsar/common/semaphore/AsyncSemaphoreImpl.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master   #25070       +/-   ##
=============================================
+ Coverage     38.69%   74.43%   +35.74%     
- Complexity    13166    34067    +20901     
=============================================
  Files          1842     1899       +57     
  Lines        145548   149813     +4265     
  Branches      16917    17418      +501     
=============================================
+ Hits          56313   111511    +55198     
+ Misses        81537    29396    -52141     
- Partials       7698     8906     +1208     

Flag	Coverage Δ
inttests	26.19% <1.32%> (-0.24%)	⬇️
systests	23.01% <2.64%> (-0.01%)	⬇️
unittests	73.97% <74.17%> (+39.49%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...apache/pulsar/broker/resources/TopicResources.java	100.00% <100.00%> (+25.00%)	⬆️
...va/org/apache/pulsar/common/protocol/Commands.java	90.99% <ø> (+25.47%)	⬆️
...ava/org/apache/pulsar/common/topics/TopicList.java	100.00% <100.00%> (+21.05%)	⬆️
...he/pulsar/common/semaphore/AsyncSemaphoreImpl.java	91.21% <0.00%> (+39.51%)	⬆️
...pulsar/broker/service/PulsarCommandSenderImpl.java	85.71% <33.33%> (+24.57%)	⬆️
...r/common/semaphore/AsyncDualMemoryLimiterImpl.java	91.11% <0.00%> (+14.36%)	⬆️
...apache/pulsar/broker/service/TopicListService.java	73.93% <76.05%> (+30.86%)	⬆️

... and 1400 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Denovo1998]

Left some comments.

[Copilot]

Pull request overview

This PR implements memory limits for topic list watcher operations as part of PIP-442. It introduces backpressure mechanisms to prevent broker memory exhaustion when handling concurrent pattern consumer topic list requests by applying memory limiting to topic list watcher creation and update operations.

Key Changes

• Added memory limit acquisition with retry logic for topic list watcher initialization and updates
• Modified topic list watcher to queue updates during initialization to avoid race conditions
• Enhanced test coverage for memory limiting scenarios and concurrent consumer patterns

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncSemaphoreImpl.java	Added toString method to SemaphorePermit for debugging
pulsar-common/src/main/java/org/apache/pulsar/common/semaphore/AsyncDualMemoryLimiterImpl.java	Added toString method to DualMemoryLimiterPermit for debugging
pulsar-broker/src/test/java/org/apache/pulsar/client/api/PatternConsumerTopicWatcherBackPressureMultipleConsumersTest.java	New test validating backpressure with 100 concurrent pattern consumers and 300 topics
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicListWatcherTest.java	Updated test setup to use direct executor and verify watcher initialization completion
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/TopicListServiceTest.java	Added comprehensive tests for permit acquisition retries and memory limiting scenarios
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/TopicListService.java	Implemented memory limiting with retry backoff for watcher operations and queued updates during initialization
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarCommandSenderImpl.java	Modified methods to acquire direct memory permits before sending topic list messages
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/PulsarCommandSender.java	Updated interface to return CompletableFuture and accept error handler for permit acquisition

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.