feat: token usage cm#256
Closed
Priyanka-Microsoft wants to merge 7 commits into
Closed
Conversation
Adds the small foundation needed for the rest of the App Insights
wiring on this branch:
- New `libs/logging/event_utils.py` with `track_event_if_configured`,
a tiny gated wrapper around `azure.monitor.events.extension.track_event`
that no-ops when `APPLICATIONINSIGHTS_CONNECTION_STRING` is unset.
Lazy-imports the SDK and swallows export errors so telemetry can
never break a request. Single warn-once latch keeps logs clean in
unit tests / local dev.
- New `libs/logging/span_filters.py` with two custom `SpanProcessor`
implementations:
* `DropASGIResponseBodySpanProcessor` strips `http.response.body`
child spans (one per streamed chunk) so streamed downloads / SSE
do not flood Application Insights.
* `DropCosmosDependencySpanProcessor` drops Cosmos DB dependency
spans (matched on `db.system==cosmosdb` and the
`documents.azure.com` peer host) so the Application Map is not
dominated by per-call Cosmos operations.
- Adds `azure-monitor-events-extension` and (explicitly)
`opentelemetry-instrumentation-fastapi` to `pyproject.toml`, then
refreshes `uv.lock` via `uv lock` (no hand edits).
No call sites change in this commit; the new helpers are wired into
`Application.initialize` and the routers in the next two commits.
Implements groundwork for AC #1, #2, #3, #4 of AB#37816.
Work item: https://dev.azure.com/CSACTOSOL/CSA%20Solutioning/_workitems/edit/37816
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ation.initialize Glues the helpers from the previous commit into the FastAPI startup path so AC #1, #2, #3, #4 of AB#37816 are satisfied at the code layer: `Application.initialize` - New `_configure_azure_monitor`: when `APPLICATIONINSIGHTS_CONNECTION_STRING` is set, calls `azure.monitor.opentelemetry.configure_azure_monitor` with `enable_live_metrics=True` and the two custom span processors (`DropASGIResponseBodySpanProcessor`, `DropCosmosDependencySpanProcessor`) added in the previous commit. Connection string is read from env, never logged. - New `_instrument_fastapi`: attaches `FastAPIInstrumentor.instrument_app(self.app, excluded_urls="health,startup")` AFTER all routers are registered so every business route is instrumented but the liveness / startup probes do not flood Application Insights with empty request rows. - Both helpers fail-soft: any exception during telemetry setup is logged and swallowed, so a misconfigured App Insights instance can never block backend startup. - Both helpers are no-ops when the connection string is unset, so local dev / unit tests behave exactly as they did before. `Application_Base` - Adds a `_NOISY_LOGGER_PACKAGES` hard-suppression list (`azure.core.pipeline.policies.http_logging_policy`, `azure.cosmos`, `opentelemetry.sdk`, `azure.monitor.opentelemetry.exporter.export._base`) clamped to WARNING regardless of the operator-supplied `AZURE_LOGGING_PACKAGES`. Without this clamp, the App Insights logs view is dominated by per- request HTTP policy logs and per-call Cosmos diagnostics. - The clamp is one-way: never lowers a level the operator has explicitly raised below WARNING. `.env.example` - Documents `APPLICATIONINSIGHTS_CONNECTION_STRING` (commented) so local dev mirrors the Bicep-injected production env. Validation - `uv run pytest src/tests --no-cov -c /dev/null` -> 44 passed (with `PYTHONPATH=src/app`, matching the existing repo convention). - Manual `Application()` smoke with the env var unset -> single INFO line, 26 routes registered, no exceptions. - Manual `Application()` smoke with a fake conn string -> Azure Monitor configured, FastAPIInstrumentor attached, exporter fails-soft on DNS resolution, app continues. Implements AC #1, #2, #3, #4 of AB#37816. Work item: https://dev.azure.com/CSACTOSOL/CSA%20Solutioning/_workitems/edit/37816 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds Application Insights custom-event emission and per-request span annotation to every business handler in: - routers/router_process.py - routers/router_files.py - routers/router_debug.py For each handler: - On success, emits a `<RouteName>Success` event via `track_event_if_configured` with the relevant domain ids (process_id, file_id, filename, counts, kill_state, ...). - On exception, emits a `<RouteName>Error` event carrying `error`, `error_type`, and (when available) `status_code`. Then calls `current_span.record_exception(e)` and `set_status(StatusCode.ERROR)` so the failure is also visible in the App Insights end-to-end transaction view. - Stamps domain ids onto the active span via `set_attribute` so custom queries on `requests | extend customDimensions` can pivot by `process_id` / `file_id` directly without re-parsing message strings. Two small private helpers `_annotate_span` and `_record_exception_on_span` are duplicated into both `router_process.py` and `router_files.py` (rather than placed in `libs/logging/`) to keep the OpenTelemetry import surface visible at the call site and to avoid a new public helper that the rest of the codebase might accidentally depend on. `routers/http_probes.py` is left untouched — its routes match the `excluded_urls="health,startup"` configured on `FastAPIInstrumentor` in the previous commit and so are intentionally not telemetered. The pre-existing `ILoggerService.log_info(f"...")` / `log_error(...)` calls are preserved as-is. The new code emitted by this commit uses `%s` lazy formatting in its log strings. Validation - `uv run pytest src/tests --no-cov -c /dev/null` -> 44 passed (with `PYTHONPATH=src/app`). - `Application()` smoke-init -> 26 routes registered, all routers import cleanly. Implements AC #3 of AB#37816 (telemetry visibility) and contributes to AC #4 (continuous log collection). Work item: https://dev.azure.com/CSACTOSOL/CSA%20Solutioning/_workitems/edit/37816 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ts gating
Adds the first dedicated test package for the new
`libs/logging/event_utils.py` helper at
`src/tests/logging/test_event_utils.py` (mirroring the existing
`src/tests/<area>/test_<thing>.py` layout — no new tooling, no new
config).
Coverage:
- `test_no_op_when_connection_string_unset`: with the env var absent,
the helper does NOT call `track_event` and the SDK module is never
imported (verified via a `sys.modules` fake).
- `test_warning_fires_only_once_per_process`: subsequent unconfigured
calls are silent (one-shot warning latch).
- `test_unconfigured_warning_message_is_stable`: pins the exact
warning template that the rest of the system asserts against.
- `test_forwards_to_track_event_when_configured`: with the env var
set, the helper forwards the (name, properties) pair through to
`azure.monitor.events.extension.track_event`.
- `test_none_properties_normalised_to_empty_dict`: `properties=None`
surfaces as `{}` to the SDK so `customDimensions` is always an
object.
- `test_whitespace_only_connection_string_is_treated_as_unset`:
belt-and-braces gating against accidentally-blank env values.
- `test_sdk_exception_is_swallowed`: a `RuntimeError` thrown by the
SDK is caught and logged; telemetry must never break a request.
The tests use a `sys.modules` fake for `azure.monitor.events.extension`
so they never import the real Azure Monitor SDK at runtime — keeps
the suite hermetic, fast (<1s), and CI-friendly.
Validation
- `uv run pytest src/tests --no-cov -c /dev/null` -> 51 passed
(44 pre-existing + 7 new), with `PYTHONPATH=src/app`.
Implements verification scope of AC #3 of AB#37816.
Work item: https://dev.azure.com/CSACTOSOL/CSA%20Solutioning/_workitems/edit/37816
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ts AVM module The AVM `insights/component@0.6.0` module already wires Application Insights to Log Analytics through the `workspaceResourceId` parameter (workspace-based App Insights). Adding a separate `diagnosticSettings` entry pointing at the SAME workspace causes the platform-level `AppAvailabilityResults`, `AppRequests`, etc. tables to be ingested twice — once via the workspace-based App Insights pipeline and again via the diagnostic-settings forwarder. Removes the redundant line from both: - `infra/main.bicep` (line 305) - `infra/main_custom.bicep` (line 283) The custom-event / OpenTelemetry path the rest of this branch wires up sends data straight to App Insights via `APPLICATIONINSIGHTS_CONNECTION_STRING`, which is already injected into the backend-api and processor container apps by the same Bicep files (no change required to those env-var blocks). Validation - `bicep build infra/main.bicep --outfile <tmp>` -> OK, no diagnostics. - `bicep build infra/main_custom.bicep --outfile <tmp>` -> OK, only a pre-existing BCP334 warning at line 694 (unrelated to this edit). Implements AC #4 of AB#37816 (continuous, non-duplicated log collection). Reference: microsoft/Conversation-Knowledge-Mining-Solution-Accelerator#811 Work item: https://dev.azure.com/CSACTOSOL/CSA%20Solutioning/_workitems/edit/37816 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… timeout/connect events Aligns the property shape of the Timeout/Connect error events with every other error event emitted by the router (which all include both `error` and `error_type`). Without `error`, an App Insights query like `customEvents | where customDimensions.error contains "..."` silently misses these two failure paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Priyanka-Microsoft
requested review from
Avijit-Microsoft,
Dongbumlee,
Prajwal-Microsoft,
Roopan-Microsoft,
Vinay-Microsoft,
aniaroramsft,
dgp10801,
nchandhi,
sethsteenken and
toherman-msft
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end LLM token usage observability for the solution by instrumenting the processor and backend API to emit App Insights custom events/OpenTelemetry spans, persisting aggregated usage to Cosmos DB, and providing Azure Workbooks/KQL + IaC to deploy dashboards.
Changes:
- Add processor-side token usage tracking (per call + aggregated summaries), with optional persistence into
ProcessStatusin Cosmos DB. - Add backend-api App Insights custom event helper + OpenTelemetry configuration (including span noise suppression) and emit router-level success/error events.
- Add infra assets for token-usage workbooks (Bicep module + exported workbook JSONs + deploy script + reusable KQL queries) and enable monitoring by default in main Bicep templates.
Reviewed changes
Copilot reviewed 37 out of 39 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| src/processor/src/utils/token_usage_tracker.py | New token usage aggregation + extraction helpers + App Insights event emission. |
| src/processor/src/utils/event_utils.py | New processor-side track_event_if_configured helper. |
| src/processor/src/utils/agent_telemetry.py | Persist aggregated token usage fields into ProcessStatus. |
| src/processor/src/steps/migration_processor.py | Instantiate/register token tracker and emit/persist summary at workflow end. |
| src/processor/src/steps/documentation/orchestration/documentation_orchestrator.py | Pass token tracker into groupchat orchestrator. |
| src/processor/src/steps/design/orchestration/design_orchestrator.py | Pass token tracker into groupchat orchestrator. |
| src/processor/src/steps/convert/orchestration/yaml_convert_orchestrator.py | Pass token tracker into groupchat orchestrator. |
| src/processor/src/steps/analysis/orchestration/analysis_orchestrator.py | Pass token tracker into groupchat orchestrator. |
| src/processor/src/steps/analysis/models/step_param.py | Add user_id to processor task param for attribution. |
| src/processor/src/services/queue_service.py | Populate user_id into Analysis_TaskParam. |
| src/processor/src/main.py | Configure Azure Monitor OTEL exporter at startup (if env configured). |
| src/processor/src/main_service.py | Configure Azure Monitor OTEL exporter at startup (if env configured). |
| src/processor/src/libs/base/orchestrator_base.py | Resolve token tracker from DI and register agent->model mapping; increase ResultGenerator max tokens. |
| src/processor/src/libs/agent_framework/groupchat_orchestrator.py | Capture token usage during streaming/backfill and surface summary on orchestration result. |
| src/processor/src/libs/agent_framework/azure_openai_response_retry.py | Add token usage diagnostics and broaden retry classification for certain errors. |
| src/processor/pyproject.toml | Add Azure Monitor event/OTEL dependencies. |
| src/processor/Dockerfile | Change dependency install to run uv lock during image build. |
| src/frontend/Dockerfile | Increase Node build memory via NODE_OPTIONS. |
| src/backend-api/uv.lock | Update backend-api lockfile for new telemetry dependencies. |
| src/backend-api/src/tests/logging/test_event_utils.py | Add unit tests for backend track_event_if_configured. |
| src/backend-api/src/tests/logging/init.py | New test package marker. |
| src/backend-api/src/app/routers/router_process.py | Emit custom events + annotate spans + record exceptions for process routes. |
| src/backend-api/src/app/routers/router_files.py | Emit custom events + annotate spans + record exceptions for file routes. |
| src/backend-api/src/app/routers/router_debug.py | Emit custom events + record exceptions for debug route. |
| src/backend-api/src/app/libs/logging/span_filters.py | Add custom span processors intended to suppress noisy spans. |
| src/backend-api/src/app/libs/logging/event_utils.py | Add backend track_event_if_configured wrapper with lazy import + one-shot warning. |
| src/backend-api/src/app/libs/logging/init.py | Document logging/telemetry helpers package. |
| src/backend-api/src/app/libs/base/application_base.py | Clamp noisy logger packages to WARNING by default. |
| src/backend-api/src/app/application.py | Configure Azure Monitor + instrument FastAPI; apply span processors and excluded URLs. |
| src/backend-api/src/app/.env.example | Document App Insights env var and basic logging options. |
| src/backend-api/pyproject.toml | Add Azure Monitor event extension + FastAPI OTEL instrumentation dependency. |
| infra/modules/tokenUsageWorkbook.bicep | New workbook module for token usage dashboard. |
| infra/main.parameters.json | Enable monitoring/telemetry defaults in sample parameters. |
| infra/main.bicep | Default enableMonitoring to true; add token usage workbook module. |
| infra/main_custom.bicep | Default enableMonitoring to true; add token usage workbook module. |
| infra/dashboards/workbook-gke-content.json | Add exported workbook JSON for GKE. |
| infra/dashboards/workbook-eks-content.json | Add exported workbook JSON for EKS. |
| infra/dashboards/token-usage-queries.kql | Add reusable KQL snippets for token usage analysis. |
| infra/dashboards/deploy-workbooks.ps1 | Add PowerShell automation to deploy the workbooks via Azure CLI REST. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| import logging | ||
| import threading | ||
| from dataclasses import dataclass, field |
| from mem0 import AsyncMemory | ||
| from pydantic import BaseModel, ValidationError | ||
|
|
||
| from utils.token_usage_tracker import TokenUsageTracker, extract_usage_from_response, _parse_usage_object |
| from mem0 import AsyncMemory | ||
| from pydantic import BaseModel, ValidationError | ||
|
|
||
| from utils.token_usage_tracker import TokenUsageTracker, extract_usage_from_response, _parse_usage_object |
Comment on lines
+303
to
+306
| token_usage_by_agent: dict[str, dict[str, Any]] = Field( | ||
| default_factory=dict, | ||
| description="Token usage per agent: {agent_name: {input_tokens, output_tokens, total_tokens, call_count, model_deployment_name}}", | ||
| ) |
Comment on lines
+215
to
+226
| # "The model produced invalid content" is a transient error from Azure OpenAI | ||
| # when the model output fails content/schema validation — worth retrying. | ||
| # "No tool call found" is a 400 error when the conversation has orphaned | ||
| # function call outputs with no matching tool call request. | ||
| if any( | ||
| s in msg | ||
| for s in [ | ||
| "model produced invalid content", | ||
| "invalid content", | ||
| "no tool call found", | ||
| ] | ||
| ): |
Comment on lines
+16
to
+17
| "azure-monitor-events-extension==0.1.0", | ||
| "azure-monitor-opentelemetry==1.8.7", |
| @@ -0,0 +1 @@ | |||
| {"version":"Notebook/1.0","items":[{"type":1,"content":{"json":"# LLM Token Usage Dashboard\n\nThis workbook provides comprehensive visibility into LLM token consumption across agents, models, workflow steps, and users.\n\n---"},"name":"header"},{"type":9,"content":{"version":"KqlParameterItem/1.0","parameters":[{"id":"time-range-param","version":"KqlParameterItem/1.0","name":"TimeRange","type":4,"isRequired":true,"value":{"durationMs":1500000,"endTime":"2026-05-21T06:08:00.000Z"},"typeSettings":{"selectableValues":[{"durationMs":3600000},{"durationMs":14400000},{"durationMs":86400000},{"durationMs":259200000},{"durationMs":604800000},{"durationMs":2592000000}],"allowCustom":true},"label":"Time Range"}],"style":"pills","queryType":0,"resourceType":"microsoft.insights/components"},"name":"parameters"},{"type":1,"content":{"json":"## Overall Token Usage Summary"},"name":"summary-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| summarize \n total_input = sum(toint(customDimensions.total_input_tokens)),\n total_output = sum(toint(customDimensions.total_output_tokens)),\n total = sum(toint(customDimensions.total_tokens)),\n total_calls = sum(toint(customDimensions.total_calls)),\n processes = dcount(tostring(customDimensions.process_id))","size":4,"title":"Token Usage Totals","queryType":0,"resourceType":"microsoft.insights/components","visualization":"tiles","tileSettings":{"titleContent":{"columnMatch":"Column1","formatter":1},"leftContent":{"columnMatch":"total","formatter":12,"formatOptions":{"palette":"auto"},"numberFormat":{"unit":0,"options":{"style":"decimal","maximumFractionDigits":0}}},"showBorder":true}},"name":"summary-tiles"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| extend process_id = tostring(customDimensions.process_id),\n total_input = toint(customDimensions.total_input_tokens),\n total_output = toint(customDimensions.total_output_tokens),\n total = toint(customDimensions.total_tokens),\n call_count = toint(customDimensions.total_calls)\n| project timestamp, process_id, total_input, total_output, total, call_count\n| order by timestamp desc","size":0,"title":"Token Usage by Process","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"blue"}}]}},"name":"summary-table"},{"type":1,"content":{"json":"## Per-Agent Token Usage"},"name":"agent-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Agent_Token_Usage\"\n| where timestamp {TimeRange}\n| extend agent_name = tostring(customDimensions.agent_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by agent_name\n| order by total desc","size":0,"title":"Token Consumption by Agent","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"blue"}}]}},"customWidth":"50","name":"agent-table"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Agent_Token_Usage\"\n| where timestamp {TimeRange}\n| extend agent_name = tostring(customDimensions.agent_name),\n total_tokens = toint(customDimensions.total_tokens)\n| summarize total = sum(total_tokens) by agent_name\n| order by total desc","size":0,"title":"Token Distribution by Agent","queryType":0,"resourceType":"microsoft.insights/components","visualization":"piechart"},"customWidth":"50","name":"agent-chart"},{"type":1,"content":{"json":"## Per-Model Token Usage"},"name":"model-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Model_Token_Usage\"\n| where timestamp {TimeRange}\n| extend model_name = tostring(customDimensions.model_deployment_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by model_name\n| order by total desc","size":0,"title":"Token Consumption by Model","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"green"}}]}},"customWidth":"50","name":"model-table"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Model_Token_Usage\"\n| where timestamp {TimeRange}\n| extend model_name = tostring(customDimensions.model_deployment_name),\n total_tokens = toint(customDimensions.total_tokens)\n| summarize total = sum(total_tokens) by model_name\n| order by total desc","size":0,"title":"Token Distribution by Model","queryType":0,"resourceType":"microsoft.insights/components","visualization":"piechart"},"customWidth":"50","name":"model-chart"},{"type":1,"content":{"json":"## Per-Step (Team) Token Usage"},"name":"step-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Step_Token_Usage\"\n| where timestamp {TimeRange}\n| extend step_name = tostring(customDimensions.step_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by step_name\n| order by total desc","size":0,"title":"Token Consumption by Workflow Step","queryType":0,"resourceType":"microsoft.insights/components","visualization":"barchart","chartSettings":{"xAxis":"step_name","yAxis":"total","group":"step_name"}},"customWidth":"50","name":"step-chart"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Step_Token_Usage\"\n| where timestamp {TimeRange}\n| extend step_name = tostring(customDimensions.step_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by step_name\n| order by total desc","size":0,"title":"Step Usage Details","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"orange"}}]}},"customWidth":"50","name":"step-table"},{"type":1,"content":{"json":"## Per-User Token Usage"},"name":"user-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| extend process_id = tostring(customDimensions.process_id),\n total_tokens = toint(customDimensions.total_tokens),\n user_id = tostring(customDimensions.user_id)\n| summarize total = sum(total_tokens), runs = count() by user_id\n| order by total desc","size":0,"title":"Token Usage by User","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"purple"}}]}},"name":"user-table"},{"type":1,"content":{"json":"## Token Usage Trends"},"name":"trend-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage\"\n| where timestamp {TimeRange}\n| extend total_tokens = toint(customDimensions.total_tokens)\n| summarize hourly_tokens = sum(total_tokens), calls = count() by bin(timestamp, 1h)\n| order by timestamp asc","size":0,"title":"Hourly Token Consumption","queryType":0,"resourceType":"microsoft.insights/components","visualization":"linechart","chartSettings":{"xAxis":"timestamp","yAxis":"hourly_tokens","showLegend":true}},"name":"trend-chart"},{"type":1,"content":{"json":"## Estimated Cost\n\n> Cost estimates use GPT-4o pricing: **$2.50 / 1M input tokens**, **$10.00 / 1M output tokens**. Adjust for your actual model pricing."},"name":"cost-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| extend process_id = tostring(customDimensions.process_id),\n input_tokens = toint(customDimensions.total_input_tokens),\n output_tokens = toint(customDimensions.total_output_tokens)\n| extend estimated_cost_usd = round((input_tokens / 1000000.0 * 2.50) + (output_tokens / 1000000.0 * 10.0), 4)\n| project timestamp, process_id, input_tokens, output_tokens, estimated_cost_usd\n| order by estimated_cost_usd desc","size":0,"title":"Estimated Cost per Process (USD)","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"estimated_cost_usd","formatter":3,"formatOptions":{"palette":"redBright"}}]}},"name":"cost-table"},{"type":1,"content":{"json":"## Individual LLM Call Log"},"name":"calls-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage\"\n| where timestamp {TimeRange}\n| extend agent_name = tostring(customDimensions.agent_name),\n step_name = tostring(customDimensions.step_name),\n model = tostring(customDimensions.model_deployment_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n process_id = tostring(customDimensions.process_id)\n| project timestamp, process_id, agent_name, step_name, model, input_tokens, output_tokens, total_tokens\n| order by timestamp desc\n| take 200","size":0,"title":"Recent LLM Calls (last 200)","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total_tokens","formatter":3,"formatOptions":{"palette":"blue"}}]}},"name":"calls-table"}],"isLocked":false,"fallbackResourceIds":["/subscriptions/1d5876cd-7603-407a-96d2-ae5ca9a9c5f3/resourcegroups/rg-pricmglogp33/providers/microsoft.insights/components/appi-pricmglogp33usmqm"]} | |||
| @@ -0,0 +1 @@ | |||
| {"version":"Notebook/1.0","items":[{"type":1,"content":{"json":"# LLM Token Usage Dashboard\n\nThis workbook provides comprehensive visibility into LLM token consumption across agents, models, workflow steps, and users.\n\n---"},"name":"header"},{"type":9,"content":{"version":"KqlParameterItem/1.0","parameters":[{"id":"time-range-param","version":"KqlParameterItem/1.0","name":"TimeRange","type":4,"isRequired":true,"value":{"durationMs":1800000,"endTime":"2026-05-21T06:50:00.000Z"},"typeSettings":{"selectableValues":[{"durationMs":3600000},{"durationMs":14400000},{"durationMs":86400000},{"durationMs":259200000},{"durationMs":604800000},{"durationMs":2592000000}],"allowCustom":true},"label":"Time Range"}],"style":"pills","queryType":0,"resourceType":"microsoft.insights/components"},"name":"parameters"},{"type":1,"content":{"json":"## Overall Token Usage Summary"},"name":"summary-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| summarize \n total_input = sum(toint(customDimensions.total_input_tokens)),\n total_output = sum(toint(customDimensions.total_output_tokens)),\n total = sum(toint(customDimensions.total_tokens)),\n total_calls = sum(toint(customDimensions.total_calls)),\n processes = dcount(tostring(customDimensions.process_id))","size":4,"title":"Token Usage Totals","queryType":0,"resourceType":"microsoft.insights/components","visualization":"tiles","tileSettings":{"titleContent":{"columnMatch":"Column1","formatter":1},"leftContent":{"columnMatch":"total","formatter":12,"formatOptions":{"palette":"auto"},"numberFormat":{"unit":0,"options":{"style":"decimal","maximumFractionDigits":0}}},"showBorder":true}},"name":"summary-tiles"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| extend process_id = tostring(customDimensions.process_id),\n total_input = toint(customDimensions.total_input_tokens),\n total_output = toint(customDimensions.total_output_tokens),\n total = toint(customDimensions.total_tokens),\n call_count = toint(customDimensions.total_calls)\n| project timestamp, process_id, total_input, total_output, total, call_count\n| order by timestamp desc","size":0,"title":"Token Usage by Process","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"blue"}}]}},"name":"summary-table"},{"type":1,"content":{"json":"## Per-Agent Token Usage"},"name":"agent-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Agent_Token_Usage\"\n| where timestamp {TimeRange}\n| extend agent_name = tostring(customDimensions.agent_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by agent_name\n| order by total desc","size":0,"title":"Token Consumption by Agent","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"blue"}}]}},"customWidth":"50","name":"agent-table"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Agent_Token_Usage\"\n| where timestamp {TimeRange}\n| extend agent_name = tostring(customDimensions.agent_name),\n total_tokens = toint(customDimensions.total_tokens)\n| summarize total = sum(total_tokens) by agent_name\n| order by total desc","size":0,"title":"Token Distribution by Agent","queryType":0,"resourceType":"microsoft.insights/components","visualization":"piechart"},"customWidth":"50","name":"agent-chart"},{"type":1,"content":{"json":"## Per-Model Token Usage"},"name":"model-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Model_Token_Usage\"\n| where timestamp {TimeRange}\n| extend model_name = tostring(customDimensions.model_deployment_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by model_name\n| order by total desc","size":0,"title":"Token Consumption by Model","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"green"}}]}},"customWidth":"50","name":"model-table"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Model_Token_Usage\"\n| where timestamp {TimeRange}\n| extend model_name = tostring(customDimensions.model_deployment_name),\n total_tokens = toint(customDimensions.total_tokens)\n| summarize total = sum(total_tokens) by model_name\n| order by total desc","size":0,"title":"Token Distribution by Model","queryType":0,"resourceType":"microsoft.insights/components","visualization":"piechart"},"customWidth":"50","name":"model-chart"},{"type":1,"content":{"json":"## Per-Step (Team) Token Usage"},"name":"step-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Step_Token_Usage\"\n| where timestamp {TimeRange}\n| extend step_name = tostring(customDimensions.step_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by step_name\n| order by total desc","size":0,"title":"Token Consumption by Workflow Step","queryType":0,"resourceType":"microsoft.insights/components","visualization":"barchart","chartSettings":{"xAxis":"step_name","yAxis":"total","group":"step_name"}},"customWidth":"50","name":"step-chart"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Step_Token_Usage\"\n| where timestamp {TimeRange}\n| extend step_name = tostring(customDimensions.step_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n calls = toint(customDimensions.call_count)\n| summarize total_input = sum(input_tokens),\n total_output = sum(output_tokens),\n total = sum(total_tokens),\n total_calls = sum(calls)\n by step_name\n| order by total desc","size":0,"title":"Step Usage Details","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"orange"}}]}},"customWidth":"50","name":"step-table"},{"type":1,"content":{"json":"## Per-User Token Usage"},"name":"user-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| extend process_id = tostring(customDimensions.process_id),\n total_tokens = toint(customDimensions.total_tokens),\n user_id = tostring(customDimensions.user_id)\n| summarize total = sum(total_tokens), runs = count() by user_id\n| order by total desc","size":0,"title":"Token Usage by User","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total","formatter":3,"formatOptions":{"palette":"purple"}}]}},"name":"user-table"},{"type":1,"content":{"json":"## Token Usage Trends"},"name":"trend-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage\"\n| where timestamp {TimeRange}\n| extend total_tokens = toint(customDimensions.total_tokens)\n| summarize hourly_tokens = sum(total_tokens), calls = count() by bin(timestamp, 1h)\n| order by timestamp asc","size":0,"title":"Hourly Token Consumption","queryType":0,"resourceType":"microsoft.insights/components","visualization":"linechart","chartSettings":{"xAxis":"timestamp","yAxis":"hourly_tokens","showLegend":true}},"name":"trend-chart"},{"type":1,"content":{"json":"## Estimated Cost\n\n> Cost estimates use GPT-4o pricing: **$2.50 / 1M input tokens**, **$10.00 / 1M output tokens**. Adjust for your actual model pricing."},"name":"cost-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage_Summary\"\n| where timestamp {TimeRange}\n| extend process_id = tostring(customDimensions.process_id),\n input_tokens = toint(customDimensions.total_input_tokens),\n output_tokens = toint(customDimensions.total_output_tokens)\n| extend estimated_cost_usd = round((input_tokens / 1000000.0 * 2.50) + (output_tokens / 1000000.0 * 10.0), 4)\n| project timestamp, process_id, input_tokens, output_tokens, estimated_cost_usd\n| order by estimated_cost_usd desc","size":0,"title":"Estimated Cost per Process (USD)","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"estimated_cost_usd","formatter":3,"formatOptions":{"palette":"redBright"}}]}},"name":"cost-table"},{"type":1,"content":{"json":"## Individual LLM Call Log"},"name":"calls-header"},{"type":3,"content":{"version":"KqlItem/1.0","query":"customEvents\n| where name == \"LLM_Token_Usage\"\n| where timestamp {TimeRange}\n| extend agent_name = tostring(customDimensions.agent_name),\n step_name = tostring(customDimensions.step_name),\n model = tostring(customDimensions.model_deployment_name),\n input_tokens = toint(customDimensions.input_tokens),\n output_tokens = toint(customDimensions.output_tokens),\n total_tokens = toint(customDimensions.total_tokens),\n process_id = tostring(customDimensions.process_id)\n| project timestamp, process_id, agent_name, step_name, model, input_tokens, output_tokens, total_tokens\n| order by timestamp desc\n| take 200","size":0,"title":"Recent LLM Calls (last 200)","queryType":0,"resourceType":"microsoft.insights/components","visualization":"table","gridSettings":{"formatters":[{"columnMatch":"total_tokens","formatter":3,"formatOptions":{"palette":"blue"}}]}},"name":"calls-table"}],"isLocked":false,"fallbackResourceIds":["/subscriptions/1d5876cd-7603-407a-96d2-ae5ca9a9c5f3/resourcegroups/rg-pricmglogp33/providers/microsoft.insights/components/appi-pricmglogp33usmqm"]} | |||
Comment on lines
93
to
95
| @description('Optional. Enable monitoring applicable resources, aligned with the Well Architected Framework recommendations. This setting enables Application Insights and Log Analytics and configures all the resources applicable resources to send logs. Defaults to false.') | ||
| param enableMonitoring bool = false | ||
| param enableMonitoring bool = true | ||
|
|
Comment on lines
86
to
88
| @description('Optional. Enable monitoring applicable resources, aligned with the Well Architected Framework recommendations. This setting enables Application Insights and Log Analytics and configures all the resources applicable resources to send logs. Defaults to false.') | ||
| param enableMonitoring bool = false | ||
| param enableMonitoring bool = true | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a complete solution for deploying and visualizing LLM token usage in Azure. It adds deployment automation, a comprehensive Azure Workbook dashboard for EKS, and a set of reusable KQL queries for Application Insights. These changes make it much easier to monitor, analyze, and estimate the cost of LLM token consumption across agents, models, steps, and users.
Deployment automation:
deploy-workbooks.ps1, a PowerShell script that automates the deployment of LLM token usage dashboards (workbooks) for both GKE and EKS clusters into a specified Azure resource group. It takes parameters for resource group, Application Insights resource ID, and location, and uses the Azure CLI to create the workbooks.Dashboards and analytics:
workbook-eks-content.json, a detailed Azure Workbook definition for EKS that visualizes LLM token usage. The dashboard includes summary tiles, per-agent/model/step/user breakdowns, token usage trends, estimated cost calculations, and a log of recent LLM calls, all parameterized by time range.token-usage-queries.kql, a set of KQL queries for Application Insights/Log Analytics. These queries provide insights into overall, per-agent, per-model, per-step, and per-user token usage, individual call logs, hourly trends, and estimated costs.Does this introduce a breaking change?
Golden Path Validation
Deployment Validation
What to Check
Verify that the following are valid
Other Information