docs: add troubleshooting guide for silent model fallback#795
docs: add troubleshooting guide for silent model fallback#795
Conversation
Add a new 'Silent Model Fallback / Wrong Model Being Used' section to the debugging guide under Common Issues. Documents the behavior where the SDK silently falls back to a default model when the requested model is unavailable, and provides workarounds: - Pre-validate models with listModels() before session creation - Detect fallback at runtime via assistant.usage events - Query the active model post-creation via rpc.model.getCurrent() - Enable debug logging for server-side fallback warnings Includes code examples for all four SDKs (Node.js, Python, Go, .NET). Relates to #250 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds troubleshooting documentation for cases where specifying an invalid/unavailable model ID results in the session using a different model without an SDK error.
Changes:
- Adds a new “Silent Model Fallback / Wrong Model Being Used” entry under Common Issues.
- Documents symptoms/root cause and provides multiple workarounds (pre-validate, runtime detection, query active model, debug logging).
You can also share your feedback on Copilot code review. Take the survey.
| 2. **Detect fallback at runtime** by comparing the model reported in `assistant.usage` events | ||
| against the model you requested: | ||
|
|
||
| <details open> | ||
| <summary><strong>Node.js / TypeScript</strong></summary> |
There was a problem hiding this comment.
The PR description says “All code examples cover the four SDKs (Node.js, Python, Go, .NET)”, but the “Detect fallback at runtime” workaround only includes Node.js and Python. Add Go and .NET examples in the same <details> pattern, or update the PR description/section text to reflect the actual coverage.
| it silently falls back to the first available default model rather than returning an error. | ||
| The fallback order is `claude-sonnet-4.6` → `claude-sonnet-4.5` → `claude-haiku-4.5` → etc. | ||
| Warnings are logged server-side but are not surfaced to the SDK as errors. |
There was a problem hiding this comment.
Listing a specific fallback order is likely to become outdated and can mislead readers if the server behavior changes. Consider rephrasing to a more stable statement (e.g., “falls back to a default available model (often …); exact selection may vary by account/availability”) and avoid implying a guaranteed ordering.
| it silently falls back to the first available default model rather than returning an error. | |
| The fallback order is `claude-sonnet-4.6` → `claude-sonnet-4.5` → `claude-haiku-4.5` → etc. | |
| Warnings are logged server-side but are not surfaced to the SDK as errors. | |
| it silently falls back to a default available model rather than returning an error. This is often | |
| a `claude-sonnet` or `claude-haiku` variant, but the exact model selected may vary by account, | |
| configuration, and model availability. Warnings are logged server-side but are not surfaced to | |
| the SDK as errors. |
| ``` | ||
| </details> | ||
|
|
||
| 3. **Query the active model** after session creation to confirm what model is in use: |
There was a problem hiding this comment.
The prose says to “use rpc.model.getCurrent()”, but the Go and .NET sections use different-looking APIs (session.Model.GetCurrent / session.GetModelAsync). To avoid confusion, either (a) adjust the text to “use the SDK-specific equivalent of the model get-current RPC” or (b) align the examples/method naming consistently with the same conceptual API across languages.
| 3. **Query the active model** after session creation to confirm what model is in use: | |
| 3. **Query the active model** after session creation to confirm what model is in use. For each language, use the SDK-specific equivalent of the model get-current RPC: |
| models, err := client.ListModels(ctx) | ||
| desiredModel := "gpt-5.4" | ||
|
|
||
| found := false | ||
| var ids []string | ||
| for _, m := range models { | ||
| ids = append(ids, m.ID) | ||
| if m.ID == desiredModel { | ||
| found = true | ||
| } | ||
| } | ||
| if !found { | ||
| panic(fmt.Sprintf("Model %q is not available. Available: %s", | ||
| desiredModel, strings.Join(ids, ", "))) |
There was a problem hiding this comment.
This Go example is presented as a snippet, but it relies on fmt/strings without showing imports, and the indentation is not gofmt-style (spaces instead of tabs inside the for and if). Consider either showing a complete, gofmt’d snippet (with imports) or adding a brief note about required imports and formatting to keep the example copy/paste-friendly.
| models, err := client.ListModels(ctx) | |
| desiredModel := "gpt-5.4" | |
| found := false | |
| var ids []string | |
| for _, m := range models { | |
| ids = append(ids, m.ID) | |
| if m.ID == desiredModel { | |
| found = true | |
| } | |
| } | |
| if !found { | |
| panic(fmt.Sprintf("Model %q is not available. Available: %s", | |
| desiredModel, strings.Join(ids, ", "))) | |
| // Requires imports "fmt" and "strings". | |
| models, err := client.ListModels(ctx) | |
| desiredModel := "gpt-5.4" | |
| found := false | |
| var ids []string | |
| for _, m := range models { | |
| ids = append(ids, m.ID) | |
| if m.ID == desiredModel { | |
| found = true | |
| } | |
| } | |
| if !found { | |
| panic(fmt.Sprintf("Model %q is not available. Available: %s", | |
| desiredModel, strings.Join(ids, ", "))) |
Summary
Adds a new "Silent Model Fallback / Wrong Model Being Used" section to
docs/troubleshooting/debugging.mdunder Common Issues.Problem
When users specify an invalid or unavailable model ID in their session configuration, the Copilot CLI silently falls back to a default model (e.g.
claude-sonnet-4.6) instead of raising an error. This is confusing because the SDK accepts any string as a model ID without client-side validation.What's Added
A troubleshooting entry that documents:
listModels()before session creation to verify the model existsassistant.usageevent model against the requested modelrpc.model.getCurrent()post-creation to confirmAll code examples cover the four SDKs (Node.js, Python, Go, .NET) using the existing
<details>tab pattern.Relates to #250