Skip to content

docs: add troubleshooting guide for silent model fallback#795

Closed
patniko wants to merge 1 commit intomainfrom
docs/troubleshoot-model-fallback
Closed

docs: add troubleshooting guide for silent model fallback#795
patniko wants to merge 1 commit intomainfrom
docs/troubleshoot-model-fallback

Conversation

@patniko
Copy link
Contributor

@patniko patniko commented Mar 12, 2026

Summary

Adds a new "Silent Model Fallback / Wrong Model Being Used" section to docs/troubleshooting/debugging.md under Common Issues.

Problem

When users specify an invalid or unavailable model ID in their session configuration, the Copilot CLI silently falls back to a default model (e.g. claude-sonnet-4.6) instead of raising an error. This is confusing because the SDK accepts any string as a model ID without client-side validation.

What's Added

A troubleshooting entry that documents:

  1. Symptoms and root cause of the silent fallback behavior
  2. Workaround: Pre-validate — call listModels() before session creation to verify the model exists
  3. Workaround: Detect at runtime — compare assistant.usage event model against the requested model
  4. Workaround: Query active model — use rpc.model.getCurrent() post-creation to confirm
  5. Workaround: Debug logging — enable verbose logging to surface server-side fallback warnings

All code examples cover the four SDKs (Node.js, Python, Go, .NET) using the existing <details> tab pattern.

Relates to #250

Add a new 'Silent Model Fallback / Wrong Model Being Used' section to the
debugging guide under Common Issues. Documents the behavior where the SDK
silently falls back to a default model when the requested model is unavailable,
and provides workarounds:

- Pre-validate models with listModels() before session creation
- Detect fallback at runtime via assistant.usage events
- Query the active model post-creation via rpc.model.getCurrent()
- Enable debug logging for server-side fallback warnings

Includes code examples for all four SDKs (Node.js, Python, Go, .NET).

Relates to #250

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@patniko patniko requested a review from a team as a code owner March 12, 2026 01:57
Copilot AI review requested due to automatic review settings March 12, 2026 01:57
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds troubleshooting documentation for cases where specifying an invalid/unavailable model ID results in the session using a different model without an SDK error.

Changes:

  • Adds a new “Silent Model Fallback / Wrong Model Being Used” entry under Common Issues.
  • Documents symptoms/root cause and provides multiple workarounds (pre-validate, runtime detection, query active model, debug logging).

You can also share your feedback on Copilot code review. Take the survey.

Comment on lines +458 to +462
2. **Detect fallback at runtime** by comparing the model reported in `assistant.usage` events
against the model you requested:

<details open>
<summary><strong>Node.js / TypeScript</strong></summary>
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description says “All code examples cover the four SDKs (Node.js, Python, Go, .NET)”, but the “Detect fallback at runtime” workaround only includes Node.js and Python. Add Go and .NET examples in the same <details> pattern, or update the PR description/section text to reflect the actual coverage.

Copilot uses AI. Check for mistakes.
Comment on lines +326 to +328
it silently falls back to the first available default model rather than returning an error.
The fallback order is `claude-sonnet-4.6` → `claude-sonnet-4.5` → `claude-haiku-4.5` → etc.
Warnings are logged server-side but are not surfaced to the SDK as errors.
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Listing a specific fallback order is likely to become outdated and can mislead readers if the server behavior changes. Consider rephrasing to a more stable statement (e.g., “falls back to a default available model (often …); exact selection may vary by account/availability”) and avoid implying a guaranteed ordering.

Suggested change
it silently falls back to the first available default model rather than returning an error.
The fallback order is `claude-sonnet-4.6``claude-sonnet-4.5``claude-haiku-4.5` → etc.
Warnings are logged server-side but are not surfaced to the SDK as errors.
it silently falls back to a default available model rather than returning an error. This is often
a `claude-sonnet` or `claude-haiku` variant, but the exact model selected may vary by account,
configuration, and model availability. Warnings are logged server-side but are not surfaced to
the SDK as errors.

Copilot uses AI. Check for mistakes.
```
</details>

3. **Query the active model** after session creation to confirm what model is in use:
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The prose says to “use rpc.model.getCurrent()”, but the Go and .NET sections use different-looking APIs (session.Model.GetCurrent / session.GetModelAsync). To avoid confusion, either (a) adjust the text to “use the SDK-specific equivalent of the model get-current RPC” or (b) align the examples/method naming consistently with the same conceptual API across languages.

Suggested change
3. **Query the active model** after session creation to confirm what model is in use:
3. **Query the active model** after session creation to confirm what model is in use. For each language, use the SDK-specific equivalent of the model get-current RPC:

Copilot uses AI. Check for mistakes.
Comment on lines +417 to +430
models, err := client.ListModels(ctx)
desiredModel := "gpt-5.4"

found := false
var ids []string
for _, m := range models {
ids = append(ids, m.ID)
if m.ID == desiredModel {
found = true
}
}
if !found {
panic(fmt.Sprintf("Model %q is not available. Available: %s",
desiredModel, strings.Join(ids, ", ")))
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This Go example is presented as a snippet, but it relies on fmt/strings without showing imports, and the indentation is not gofmt-style (spaces instead of tabs inside the for and if). Consider either showing a complete, gofmt’d snippet (with imports) or adding a brief note about required imports and formatting to keep the example copy/paste-friendly.

Suggested change
models, err := client.ListModels(ctx)
desiredModel := "gpt-5.4"
found := false
var ids []string
for _, m := range models {
ids = append(ids, m.ID)
if m.ID == desiredModel {
found = true
}
}
if !found {
panic(fmt.Sprintf("Model %q is not available. Available: %s",
desiredModel, strings.Join(ids, ", ")))
// Requires imports "fmt" and "strings".
models, err := client.ListModels(ctx)
desiredModel := "gpt-5.4"
found := false
var ids []string
for _, m := range models {
ids = append(ids, m.ID)
if m.ID == desiredModel {
found = true
}
}
if !found {
panic(fmt.Sprintf("Model %q is not available. Available: %s",
desiredModel, strings.Join(ids, ", ")))

Copilot uses AI. Check for mistakes.
@patniko patniko closed this Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants