Skip to content

feat: add modern rate limiter option using go-github-ratelimit#3236

Open
mkushakov wants to merge 2 commits intointegrations:mainfrom
mkushakov:feat/modern-rate-limiter
Open

feat: add modern rate limiter option using go-github-ratelimit#3236
mkushakov wants to merge 2 commits intointegrations:mainfrom
mkushakov:feat/modern-rate-limiter

Conversation

@mkushakov
Copy link

Summary

Add a new rate_limiter provider configuration option that allows users to choose between two rate limiting strategies for GitHub API calls.

Resolves #2709
References #849, #1226

Motivation

The current provider uses a custom rate limiting implementation with configurable read_delay_ms and write_delay_ms delays. While functional, this approach:

  • Requires users to manually tune delay values
  • Does not automatically respond to GitHub's rate limit headers (Retry-After, X-RateLimit-*)
  • Can be either too aggressive (causing 429 errors) or too conservative (slowing down operations unnecessarily)

The go-github-ratelimit library provides automatic, header-based rate limit handling that respects GitHub's primary and secondary rate limits without manual configuration.

Changes

Config (config.go)

  • Added RateLimiter field to Config struct (accepts "legacy" or "modern")
  • Added ModernRateLimitedHTTPClient function that uses go-github-ratelimit/v2 to create an HTTP client with automatic rate limit handling
  • Updated AuthenticatedHTTPClient and AnonymousHTTPClient to dispatch to the appropriate rate limiter based on the RateLimiter setting

Provider (provider.go)

  • Added rate_limiter schema field with validation ("legacy" or "modern", defaults to "legacy")
  • Wired rate_limiter value into the Config struct in providerConfigure

Dependencies

  • Added github.com/gofri/go-github-ratelimit/v2 v2.0.2

Example Usage

provider "github" {
  token        = var.github_token
  rate_limiter = "modern"  # Use automatic rate limit handling
}

When using "modern", the read_delay_ms, write_delay_ms, and parallel_requests settings are ignored — the library automatically detects and waits for rate limit resets using response headers.

Testing

  • go build ./... — passes
  • go vet ./... — passes
  • The default is "legacy" to maintain backward compatibility — no existing configurations are affected

Add rate_limiter provider option (legacy/modern). Modern uses go-github-ratelimit/v2 for automatic rate limit handling. Resolves integrations#2709, references integrations#849 integrations#1226.
@github-actions
Copy link

👋 Hi! Thank you for this contribution! Just to let you know, our GitHub SDK team does a round of issue and PR reviews twice a week, every Monday and Friday! We have a process in place for prioritizing and responding to your input. Because you are a part of this community please feel free to comment, add to, or pick up any issues/PRs that are labeled with Status: Up for grabs. You & others like you are the reason all of this works! So thank you & happy coding! 🚀

@github-actions github-actions bot added the Type: Feature New feature or request label Feb 27, 2026
Copy link
Collaborator

@deiga deiga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Left a few comments :)

Comment on lines +101 to +107
ValidateFunc: func(val any, key string) (warns []string, errs []error) {
v := val.(string)
if v != "legacy" && v != "modern" {
errs = append(errs, fmt.Errorf("%q must be either 'legacy' or 'modern', got: %s", key, v))
}
return
},
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ValidateFunc: func(val any, key string) (warns []string, errs []error) {
v := val.(string)
if v != "legacy" && v != "modern" {
errs = append(errs, fmt.Errorf("%q must be either 'legacy' or 'modern', got: %s", key, v))
}
return
},
ValidateDiagFunc: validation.ToDiagFunc(validation.StringInSlice([]string{"legacy", "modern"})),

// read_delay_ms, write_delay_ms, and parallel_requests settings are ignored
// because rate limiting is handled automatically by the library.
func ModernRateLimitedHTTPClient(client *http.Client, retryDelay time.Duration, retryableErrors map[int]bool, maxRetries int) *http.Client {
client.Transport = NewEtagTransport(client.Transport)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
client.Transport = NewEtagTransport(client.Transport)
client.Transport = NewEtagTransport(client.Transport)
client.Transport = logging.NewLoggingHTTPTransport(client.Transport)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type: Feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAT]: Bypass preemptive rate limit check by go-github

2 participants