Skip to content

feat: add LiteLLM as AI gateway provider#559

Open
RheagalFire wants to merge 1 commit into
Huanshere:mainfrom
RheagalFire:feat/add-litellm-provider
Open

feat: add LiteLLM as AI gateway provider#559
RheagalFire wants to merge 1 commit into
Huanshere:mainfrom
RheagalFire:feat/add-litellm-provider

Conversation

@RheagalFire
Copy link
Copy Markdown

Summary

  • Adds LiteLLM as a selectable LLM provider alongside the existing OpenAI-compatible path
  • Users can now access 100+ LLM providers (Anthropic, Google, Azure, Bedrock, Ollama, etc.) directly without needing a separate proxy

Motivation

VideoLingo currently supports OpenAI-compatible APIs via configurable base_url. Users wanting to use Anthropic, Google, or other providers must set up a third-party proxy like OpenRouter (issue #434 workaround). LiteLLM removes this friction: set provider: litellm in config, use a prefixed model string like anthropic/claude-sonnet-4-6 or gemini/gemini-2.0-flash, and LiteLLM routes the request to the right provider automatically. No proxy server required.

Changes

  • config.yaml - added api.provider: 'openai' config key (values: openai, litellm)
  • core/utils/ask_gpt.py - extracted _call_openai() and _call_litellm() helpers; ask_gpt() routes based on provider config
  • core/st_utils/sidebar_setting.py - added provider dropdown in Streamlit sidebar with context-sensitive help text
  • requirements.txt - added litellm>=1.80.0,<1.87.0
  • tests/test_litellm_provider.py - 11 unit tests covering dispatch, drop_params, credential forwarding, base_url handling, JSON format, and provider routing

Tests

1. Unit tests (11/11 passing):

tests/test_litellm_provider.py::TestCallLiteLLM::test_basic_completion PASSED
tests/test_litellm_provider.py::TestCallLiteLLM::test_drop_params_always_true PASSED
tests/test_litellm_provider.py::TestCallLiteLLM::test_api_key_omitted_when_empty PASSED
tests/test_litellm_provider.py::TestCallLiteLLM::test_base_url_forwarded_when_set PASSED
tests/test_litellm_provider.py::TestCallLiteLLM::test_base_url_omitted_when_empty PASSED
tests/test_litellm_provider.py::TestCallLiteLLM::test_json_response_format_forwarded PASSED
tests/test_litellm_provider.py::TestCallLiteLLM::test_response_format_omitted_when_none PASSED
tests/test_litellm_provider.py::TestAskGptProviderRouting::test_litellm_provider_routes_to_litellm PASSED
tests/test_litellm_provider.py::TestAskGptProviderRouting::test_openai_provider_routes_to_openai PASSED
tests/test_litellm_provider.py::TestCallOpenAI::test_ark_base_url_override PASSED
tests/test_litellm_provider.py::TestCallOpenAI::test_v1_appended_when_missing PASSED
============================== 11 passed in 3.21s ==============================

2. Live E2E against Anthropic Claude Sonnet 4.6 (via Azure AI Foundry):

>>> litellm.completion(model='anthropic/claude-sonnet-4-6', messages=[...], drop_params=True)
Response: 4
Model: claude-sonnet-4-6
Usage: prompt_tokens=20, completion_tokens=5, total_tokens=25

3. Live E2E - JSON mode (VideoLingo's primary usage pattern):

>>> litellm.completion(model='anthropic/claude-sonnet-4-6', ..., response_format={'type': 'json_object'})
Response: {"code": 200, "message": "success"}
Model: claude-sonnet-4-6

This proves the full chain: ask_gpt() -> _call_litellm() -> litellm.completion() -> Anthropic via Azure Foundry -> parsed response.

4. Pin resolution verified: litellm>=1.80.0,<1.87.0 resolves cleanly alongside openai>=1.55.3,<2.

Risk / Compatibility

  • Additive only. Existing OpenAI provider path is untouched; default api.provider remains openai.
  • LiteLLM's drop_params=True silently drops provider-unsupported kwargs, preventing cross-provider failures (e.g., response_format on providers that don't support JSON mode).
  • All callers of ask_gpt() work without changes since the function signature is unchanged.

Example usage

1. Config (config.yaml):

# Anthropic via LiteLLM
api:
  provider: 'litellm'
  key: 'sk-ant-...'              # or set ANTHROPIC_API_KEY env var
  model: 'anthropic/claude-sonnet-4-6'
  base_url: ''                    # leave empty; LiteLLM routes automatically
  llm_support_json: true
# Google Gemini via LiteLLM
api:
  provider: 'litellm'
  key: ''                         # set GEMINI_API_KEY env var
  model: 'gemini/gemini-2.0-flash'
  base_url: ''

**2. Python: **

from core.utils import ask_gpt

# With provider set to 'litellm' in config.yaml, all existing calls
# route through LiteLLM automatically:
result = ask_gpt("Translate this to French: Hello world", resp_type="json")

# LiteLLM handles provider routing based on the model string:
#   'anthropic/claude-sonnet-4-6'  -> Anthropic API
#   'gemini/gemini-2.0-flash'     -> Google Gemini API
#   'openai/gpt-4o'               -> OpenAI API
#   'bedrock/anthropic.claude-v2' -> AWS Bedrock

@RheagalFire
Copy link
Copy Markdown
Author

cc @Huanshere

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant