Skip to content

feat: implement exponential backoff retry strategy for transient errors#132

Open
noahpodgurski wants to merge 5 commits intomainfrom
exponential-backoff-transient-litellm
Open

feat: implement exponential backoff retry strategy for transient errors#132
noahpodgurski wants to merge 5 commits intomainfrom
exponential-backoff-transient-litellm

Conversation

@noahpodgurski
Copy link
Copy Markdown
Collaborator

@noahpodgurski noahpodgurski commented Apr 6, 2026 •
edited
Loading

What's new

  • Added exponential backoff retry strategy for transient errors coming from LiteLLM

AIPLAT-452

@ti3x
Copy link
Copy Markdown
Contributor

ti3x commented Apr 7, 2026

I noticed we have two identical retry wrappers—_call_litellm_with_retry() in completions.py and litellm_request_with_retry() in utils.py. Just want to check: was that intentional, or would it make sense to use one shared function instead? Either way, the PR looks great!

@noahpodgurski
Copy link
Copy Markdown
Collaborator Author

noahpodgurski commented Apr 7, 2026

I noticed we have two identical retry wrappers—_call_litellm_with_retry() in completions.py and litellm_request_with_retry() in utils.py. Just want to check: was that intentional, or would it make sense to use one shared function instead? Either way, the PR looks great!

Yeah they're separate functions - they do basically the same thing but I kept the one in completion.py unique to make the tests easier to handle

@ti3x
Copy link
Copy Markdown
Contributor

ti3x commented Apr 7, 2026

@noahpodgurski let's hold off merging for now as I am sorting out PT + Paygo fallback

subpath
subpath reviewed Apr 8, 2026
View reviewed changes
subpath
subpath reviewed Apr 8, 2026
View reviewed changes
subpath
subpath reviewed Apr 8, 2026
View reviewed changes
@noahpodgurski noahpodgurski requested a review from a team as a code owner April 8, 2026 13:18
@subpath
Copy link
Copy Markdown
Collaborator

subpath commented Apr 9, 2026

@noahpodgurski also please run make docs

we can even make a pre-commit hook maybe

subpath
subpath reviewed Apr 9, 2026
View reviewed changes
src/mlpa/core/prometheus_metrics.py
"Number of chat requests that included a tools payload.",
["tool_name", "model", "service_type", "purpose"],
),
chat_request_rejections=Counter(
Copy link
Copy Markdown
Collaborator

@subpath subpath Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@noahpodgurski - I think you need to update this too here with error: 5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@subpath subpath subpath approved these changes

@ti3x ti3x ti3x approved these changes

Assignees

No one assigned

Labels

pr-ready

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@noahpodgurski @ti3x @subpath