Skip to content

feat: restore XML tool protocol option for compatible providers #10348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
roomote wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

feat: restore XML tool protocol option for compatible providers #10348

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 123 additions & 27 deletions src/utils/__tests__/resolveToolProtocol.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,22 @@ import type { Anthropic } from "@anthropic-ai/sdk"

describe("resolveToolProtocol", () => {
/**
* XML Protocol Deprecation:
* Tool Protocol Resolution:
*
* XML tool protocol has been fully deprecated. All models now use Native
* tool calling. User preferences and model defaults are ignored.
* By default, Native tool calling is used for all providers. However,
* certain providers (LiteLLM, OpenAI Compatible, Ollama, LM Studio)
* allow users to explicitly set XML tool protocol for compatibility.
*
* Precedence:
* 1. Locked Protocol (for resumed tasks that used XML)
* 2. Native (always, for all new tasks)
* 1. Locked Protocol (for resumed tasks - highest priority)
* 2. User Preference (only for specific providers with XML fallback)
* 3. Native (default for all other cases)
*/

describe("Locked Protocol (Precedence Level 0 - Highest Priority)", () => {
it("should return lockedProtocol when provided", () => {
const settings: ProviderSettings = {
toolProtocol: "xml", // Ignored
toolProtocol: "xml", // Ignored when locked
apiProvider: "openai-native",
}
// lockedProtocol overrides everything
Expand All @@ -29,44 +31,54 @@ describe("resolveToolProtocol", () => {

it("should return XML lockedProtocol for resumed tasks that used XML", () => {
const settings: ProviderSettings = {
toolProtocol: "native", // Ignored
toolProtocol: "native", // Ignored when locked
apiProvider: "anthropic",
}
// lockedProtocol forces XML for backward compatibility
const result = resolveToolProtocol(settings, undefined, "xml")
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should fall through to Native when lockedProtocol is undefined", () => {
it("should fall through to user preference for providers with XML fallback when lockedProtocol is undefined", () => {
const settings: ProviderSettings = {
toolProtocol: "xml", // Ignored
toolProtocol: "xml",
apiProvider: "litellm",
}
// undefined lockedProtocol allows user preference for litellm
const result = resolveToolProtocol(settings, undefined, undefined)
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should fall through to Native when lockedProtocol is undefined and provider is not in XML fallback list", () => {
const settings: ProviderSettings = {
toolProtocol: "xml", // Ignored for anthropic
apiProvider: "anthropic",
}
// undefined lockedProtocol should return native
// undefined lockedProtocol should return native for anthropic
const result = resolveToolProtocol(settings, undefined, undefined)
expect(result).toBe(TOOL_PROTOCOL.NATIVE)
})
})

describe("Native Protocol Always Used For New Tasks", () => {
it("should always use native for new tasks", () => {
describe("Native Protocol Default For New Tasks", () => {
it("should use native by default for new tasks", () => {
const settings: ProviderSettings = {
apiProvider: "anthropic",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE)
})

it("should use native even when user preference is XML (user prefs ignored)", () => {
it("should use native even when user preference is XML for non-fallback providers", () => {
const settings: ProviderSettings = {
toolProtocol: "xml", // User wants XML - ignored
toolProtocol: "xml", // User wants XML - ignored for openai-native
apiProvider: "openai-native",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE)
})

it("should use native for OpenAI compatible provider", () => {
it("should use native for OpenAI compatible provider when no toolProtocol specified", () => {
const settings: ProviderSettings = {
apiProvider: "openai",
}
Expand All @@ -75,30 +87,85 @@ describe("resolveToolProtocol", () => {
})
})

describe("XML Fallback Providers", () => {
it("should respect XML preference for LiteLLM provider", () => {
const settings: ProviderSettings = {
toolProtocol: "xml",
apiProvider: "litellm",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should respect XML preference for OpenAI compatible provider", () => {
const settings: ProviderSettings = {
toolProtocol: "xml",
apiProvider: "openai",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should respect XML preference for Ollama provider", () => {
const settings: ProviderSettings = {
toolProtocol: "xml",
apiProvider: "ollama",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should respect XML preference for LM Studio provider", () => {
const settings: ProviderSettings = {
toolProtocol: "xml",
apiProvider: "lmstudio",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should use native for LiteLLM when toolProtocol is not set", () => {
const settings: ProviderSettings = {
apiProvider: "litellm",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE)
})

it("should use native for Ollama when toolProtocol is native", () => {
const settings: ProviderSettings = {
toolProtocol: "native",
apiProvider: "ollama",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE)
})
})

describe("Edge Cases", () => {
it("should handle missing provider name gracefully", () => {
const settings: ProviderSettings = {}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Always native now
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Default to native
})

it("should handle undefined model info gracefully", () => {
const settings: ProviderSettings = {
apiProvider: "openai-native",
}
const result = resolveToolProtocol(settings, undefined)
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Always native now
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Default to native
})

it("should handle empty settings", () => {
const settings: ProviderSettings = {}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Always native now
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Default to native
})
})

describe("Real-world Scenarios", () => {
it("should use Native for OpenAI models", () => {
it("should use Native for OpenAI native models", () => {
const settings: ProviderSettings = {
apiProvider: "openai-native",
}
Expand Down Expand Up @@ -134,25 +201,54 @@ describe("resolveToolProtocol", () => {
const result = resolveToolProtocol(settings, undefined, "xml")
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should allow XML for LiteLLM proxy to models without native tool support", () => {
const settings: ProviderSettings = {
toolProtocol: "xml",
apiProvider: "litellm",
}
// LiteLLM proxying to model without native tools - user can set XML
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.XML)
})

it("should allow XML for local models via Ollama", () => {
const settings: ProviderSettings = {
toolProtocol: "xml",
apiProvider: "ollama",
}
// Local model may not support native tools
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.XML)
})
})

describe("Backward Compatibility - User Preferences Ignored", () => {
it("should ignore user preference for XML", () => {
describe("Provider-specific User Preferences", () => {
it("should ignore user preference for XML on anthropic provider", () => {
const settings: ProviderSettings = {
toolProtocol: "xml", // User explicitly wants XML - ignored for anthropic
apiProvider: "anthropic",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Native is always used for anthropic
})

it("should ignore user preference for XML on openai-native provider", () => {
const settings: ProviderSettings = {
toolProtocol: "xml", // User explicitly wants XML - ignored
toolProtocol: "xml", // User explicitly wants XML - ignored for openai-native
apiProvider: "openai-native",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Native is always used
expect(result).toBe(TOOL_PROTOCOL.NATIVE) // Native is always used for openai-native
})

it("should return native regardless of user preference", () => {
it("should respect user preference for XML on openai compatible provider", () => {
const settings: ProviderSettings = {
toolProtocol: "native", // User preference - ignored but happens to match
apiProvider: "anthropic",
toolProtocol: "xml", // User explicitly wants XML - respected for openai
apiProvider: "openai",
}
const result = resolveToolProtocol(settings)
expect(result).toBe(TOOL_PROTOCOL.NATIVE)
expect(result).toBe(TOOL_PROTOCOL.XML) // XML is allowed for openai compatible
})
})
})
Expand Down
44 changes: 35 additions & 9 deletions src/utils/resolveToolProtocol.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,25 +11,41 @@ type ApiMessageForDetection = Anthropic.MessageParam & {
ts?: number
}

/**
* Providers that can benefit from XML tool protocol as a fallback option.
* These providers may proxy to models that don't fully support native tool calling,
* or may encounter compatibility issues with certain model configurations.
*
* - litellm: Proxies to various LLM backends; some may not support native tools
* - openai: OpenAI-compatible endpoints may have varying tool support
* - ollama: Local models may have limited native tool support
* - lmstudio: Local models may have limited native tool support
*/
const PROVIDERS_WITH_XML_FALLBACK = ["litellm", "openai", "ollama", "lmstudio"] as const

/**
* Resolve the effective tool protocol.
*
* **Deprecation Note (XML Protocol):**
* XML tool protocol has been deprecated. All models now use Native tool calling.
* User/profile preferences (`providerSettings.toolProtocol`) and model defaults
* (`modelInfo.defaultToolProtocol`) are ignored.
* **Tool Protocol Configuration:**
* By default, Native tool calling is used for all providers. However, certain providers
* (LiteLLM, OpenAI Compatible, Ollama, LM Studio) allow users to explicitly set XML
* tool protocol via the `toolProtocol` setting. This is useful when:
* - Proxying through LiteLLM to models that don't support native tools
* - Using local models with limited tool calling capabilities
* - Encountering compatibility issues with specific model configurations
*
* Precedence:
* 1. Locked Protocol (task-level lock for resumed tasks - highest priority)
* 2. Native (always, for all new tasks)
* 2. User Preference (only for providers in PROVIDERS_WITH_XML_FALLBACK)
* 3. Native (default for all other cases)
*
* @param _providerSettings - The provider settings (toolProtocol field is ignored)
* @param providerSettings - The provider settings including optional toolProtocol
* @param _modelInfo - Unused, kept for API compatibility
* @param lockedProtocol - Optional task-locked protocol that takes absolute precedence
* @returns The resolved tool protocol (either "xml" or "native")
*/
export function resolveToolProtocol(
_providerSettings: ProviderSettings,
providerSettings: ProviderSettings,
_modelInfo?: unknown,
lockedProtocol?: ToolProtocol,
): ToolProtocol {
Expand All @@ -39,8 +55,18 @@ export function resolveToolProtocol(
return lockedProtocol
}

// 2. Always return Native protocol for new tasks
// All models now support native tools; XML is deprecated
// 2. For providers with XML fallback support, respect user preference if explicitly set
// This allows users to work around compatibility issues with certain models/configurations
const provider = providerSettings.apiProvider
if (
provider &&
PROVIDERS_WITH_XML_FALLBACK.includes(provider as (typeof PROVIDERS_WITH_XML_FALLBACK)[number]) &&
providerSettings.toolProtocol
) {
return providerSettings.toolProtocol
}

// 3. Default to Native protocol for all new tasks
return TOOL_PROTOCOL.NATIVE
}

Expand Down
Loading