We don’t pass the anthropic base URL to the LiteLLM AnthropicCompletion class

[alex-mcgovern]

Describe the issue

While debugging prompt caching for Cline+Anthropic, we noticed that we aren't actually passing the Anthropic base URL from config / CLI flags to the litellm AnthropicCompletion class.

e.g. poetry run codegate serve --anthropic-url=http://127.0.0.1:8080/
e.g. CODEGATE_PROVIDER_ANTHROPIC_URL=http://127.0.0.1:8080/ poetry run codegate serve

You can workaround this by setting ANTHROPIC_BASE_URL, which is an in-built behaviour of litellm (and also undocumented). @kantord figured this out from this docs link for langsmith

Steps to Reproduce

• export a CODEGATE_PROVIDER_ANTHROPIC_URL (as mentioned in config.yaml.example
  OR
• start the server with --anthropic-url flag

If, for example, you've pointed the URL at something like mitmproxy, you should notice that the requests pass straight through to anthropic, without respecting your config.

Operating System

Linux (Intel)

IDE and Version

VS CODE 1.9.5.3

Extension and Version

Cline v3.2.12

Provider

Anthropic

Model

claude-3-5-sonnet

Codegate version

CodeGate 0.1.7 (as returned by codegate version), 0ed1365

Logs

No response

Additional Context

No response

[blkt]

It looks like we allow the configuration of custom URLs for the Providers we support, but don't pass the configuration struct to the classes upon instantiation, which leads to our client library (litellm) being called with api_base = None. This is likely a problem we have with all Providers, not only with Anthropic.

The solution is to have Providers __init__ method accept an additional struct to then extract overridden base urls and possibly other stuff.

[kantord]

It looks like we allow the configuration of custom URLs for the Providers we support, but don't pass the configuration struct to the classes upon instantiation, which leads to our client library (litellm) being called with api_base = None. This is likely a problem we have with all Providers, not only with Anthropic.

The solution is to have Providers __init__ method accept an additional struct to then extract overridden base urls and possibly other stuff.

yup, this is what I found, albeit I wasn't sure how to pass it to the constructor.

[tan-yong-sheng]

Not sure if this helps. I am using LiteLLM proxy (that is self-hosted on linux cloud). And here's how I set up.

#878 (comment)

If you're referring to litellm python module, then I am not that sure how to do that.