Skip to content

Conversation

@YosefHayim
Copy link

This PR adds token limit handling to prevent API failures when large git diffs exceed the 8k token limit.

Problem

  • GitHub Models API has an 8k token limit for entire requests
  • Large git diffs can exceed this limit, causing API failures
  • Users experience failures when staging large changes

Solution

  • Added token estimation using character-based heuristic (1 token ≈ 4 chars)
  • Implemented truncation logic that preserves UTF-8 boundaries
  • Added intelligent content prioritization when over limit

Implementation Details

  • estimateTokens(): Approximates tokens for any text content
  • truncateToTokenLimit(): Safely truncates text with ellipsis indicator
  • Modified GenerateCommitMessage() to:
    • Estimate tokens for prompt templates + changes + examples
    • Reserve tokens for templates (with buffer)
    • Prioritize examples (20% of remaining tokens) when present
    • Truncate changes to fit remaining budget
    • Display warning when truncation occurs

Benefits

  • Prevents API failures from token overflow
  • Maintains functionality by preserving maximum content
  • User-friendly with clear truncation warnings
  • No external dependencies, follows existing code style
  • Gracefully handles both changes-only and changes+examples scenarios

Testing

  • Code compiles successfully
  • Follows existing project patterns and style
  • Token estimation uses common industry heuristic

Problem:
- GitHub Models API has an 8k token limit for entire requests
- Large git diffs can exceed this limit, causing API failures
- Users experience failures when staging large changes

Solution:
- Added token estimation using character-based heuristic (1 token ≈ 4 chars)
- Implemented truncation logic that preserves UTF-8 boundaries
- Added intelligent content prioritization when over limit

Implementation Details:
- estimateTokens(): Approximates tokens for any text content
- truncateToTokenLimit(): Safely truncates text with ellipsis indicator
- Modified GenerateCommitMessage() to:
  * Estimate tokens for prompt templates + changes + examples
  * Reserve tokens for templates (with buffer)
  * Prioritize examples (20% of remaining tokens) when present
  * Truncate changes to fit remaining budget
  * Display warning when truncation occurs

Benefits:
- Prevents API failures from token overflow
- Maintains functionality by preserving maximum content
- User-friendly with clear truncation warnings
- No external dependencies, follows existing code style
- Gracefully handles both changes-only and changes+examples scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant