Detect VS Code Copilot terminal via COPILOT_AGENT in LLM telemetry detection#54358
Detect VS Code Copilot terminal via COPILOT_AGENT in LLM telemetry detection#54358Copilot wants to merge 2 commits into
COPILOT_AGENT in LLM telemetry detection#54358Conversation
Agent-Logs-Url: https://github.com/dotnet/sdk/sessions/e2f05d30-d845-49b1-b504-60ae3ed82c32 Co-authored-by: baronfel <573979+baronfel@users.noreply.github.com>
COPILOT_AGENT in LLM telemetry detection
There was a problem hiding this comment.
Pull request overview
Updates the SDK’s LLM telemetry environment detection so VS Code Copilot terminal sessions are recognized via the newly exposed COPILOT_AGENT environment variable, preventing Copilot usage from being missed in telemetry.
Changes:
- Extended the Copilot detection rule in
LLMEnvironmentDetectorForTelemetryto treatCOPILOT_AGENTas a Copilot signal. - Updated the inline comment documenting Copilot detection inputs.
- Added a targeted telemetry test case verifying
COPILOT_AGENT=1resolves to"copilot".
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
src/Cli/dotnet/Telemetry/LLMEnvironmentDetectorForTelemetry.cs |
Adds COPILOT_AGENT to the Copilot environment detection rule and updates the rule comment accordingly. |
test/dotnet.Tests/TelemetryTests/TelemetryCommonPropertiesTests.cs |
Adds a new theory data case asserting COPILOT_AGENT triggers "copilot" detection. |
| new EnvironmentDetectionRuleWithResult<string>("copilot", new AnyMatchEnvironmentRule( | ||
| new BooleanEnvironmentRule("GITHUB_COPILOT_CLI_MODE"), | ||
| new AnyPresentEnvironmentRule("GH_COPILOT_WORKING_DIRECTORY", "COPILOT_CLI"))), | ||
| new AnyPresentEnvironmentRule("GH_COPILOT_WORKING_DIRECTORY", "COPILOT_CLI", "COPILOT_AGENT"))), |
There was a problem hiding this comment.
Should the new env variable be a boolean rule?
https://github.com/microsoft/vscode/pull/316267/changes seems to be setting it to '1'.
| { new Dictionary<string, string> { { "GITHUB_COPILOT_CLI_MODE", "true" } }, "copilot" }, | ||
| { new Dictionary<string, string> { { "GH_COPILOT_WORKING_DIRECTORY", "/repo" } }, "copilot" }, | ||
| { new Dictionary<string, string> { { "COPILOT_CLI", "1" } }, "copilot" }, | ||
| { new Dictionary<string, string> { { "COPILOT_AGENT", "1" } }, "copilot" }, |
There was a problem hiding this comment.
@baronfel do we want to differentiate between the CLI and VSCode usage in our telemetry?
There was a problem hiding this comment.
For this field IMO no - it should attribute to copilot. For your other PR we can attribute to the 'host' (vscode, etc) and I think we should update your PR to check for this signal too.
There was a problem hiding this comment.
Are you thinking we'll take the other PR? That one feels more hacky and harder to justify in servicing. Hence my thought of splitting this up with the simpler env check.
There was a problem hiding this comment.
Yeah, I was thinking we'd polish it up and take it. Harder to justify in servicing, sure - but fine for the 11 and 10.0.4xx releases.
|
This PR has been labeled with
|
The LLM environment detector was not identifying Copilot terminal sessions in VS Code. VS Code now exposes
COPILOT_AGENT, so Copilot usage could be missed in telemetry without this update.LLM telemetry detection
copilotrule inLLMEnvironmentDetectorForTelemetryto treatCOPILOT_AGENTas a Copilot signal, alongside existing Copilot env vars.Targeted test coverage
COPILOT_AGENT=1resolves to"copilot".