Skip to content

feat(throttle): allow customized rate limiter#7405

Open
dentiny wants to merge 1 commit into
apache:mainfrom
dentiny:hjiang/allow-customized-rate-limiter
Open

feat(throttle): allow customized rate limiter#7405
dentiny wants to merge 1 commit into
apache:mainfrom
dentiny:hjiang/allow-customized-rate-limiter

Conversation

@dentiny
Copy link
Copy Markdown
Member

@dentiny dentiny commented Apr 17, 2026

Rationale for this change

Hi team, for rate limit there're two metrics I care about (1) what's the blocking wait time; (2) what's the queue depth; these two metrics are important to provide insight on user workload.
I raise a related feature request here: #7312, but requires more change.
A way to unblock me for now is to allow customized rate limiter, similar to what we do at ConcurrentLimitLayer.

What changes are included in this PR?

This PR makes rate limiter generic so users could provide their own implementation; it should be a no-op change.

Are there any user-facing changes?

Current implementation SharedRateLimiter is used as default generic argument, so no API change.

AI Usage Statement

I get opus 4.6 implement the feature, with concurrent limit layer as a reference.

@dentiny dentiny requested a review from Xuanwo as a code owner April 17, 2026 19:08
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels Apr 17, 2026
Copy link
Copy Markdown
Member

@tisonkun tisonkun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm considering that this might make the throttle layer configuration too complex and overly generalized. I’ve been thinking about how much code can actually be reused here and to what extent it should depend on governor.

Since there seems to be very little reusable code, I think two other approaches might be more appropriate:

  1. Let users who want a custom rate limiter write their own layer from scratch. Currently, this throttle layer is heavily dependent on governor's functionality.
  2. Follow the pattern used in observability: write a layer-throttle-base and then provide the specific implementations out of the box.

If we stick with the current approach in this PR, where we depend on governor while trying to allow custom rate limits, there isn't much reusable throttle code anyway. Instead, it just makes the codebase far too generalized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants