Skip to content

Add Enterprise Usage Analyzer (Next.js UI for CSM dashboard + grounded chat)#63

Open
Yonatan Edelstein (YoniEdelstein) wants to merge 7 commits into
mainfrom
yedelstein/enterprise-usage-analyzer-ui
Open

Add Enterprise Usage Analyzer (Next.js UI for CSM dashboard + grounded chat)#63
Yonatan Edelstein (YoniEdelstein) wants to merge 7 commits into
mainfrom
yedelstein/enterprise-usage-analyzer-ui

Conversation

@YoniEdelstein
Copy link
Copy Markdown

Summary

Adds enterprise-usage-analyzer/ — a standalone Next.js 16 app that gives CSM / Sales / leadership a single screen per enterprise client with KPIs, charts, AI-generated qualitative insights, and a chat panel that can re-query BigQuery on demand.

Local-only (no deploy in this PR). Queries BigQuery via the user's own gcloud ADC and Anthropic for qualitative summaries / chat. Those are the only two destinations data leaves the machine for — no analytics, no Sentry, no third-party fonts, Next.js telemetry disabled.

What you get on screen

For a chosen client + date range (default last 30 days, max 12 months):

  • KPI tiles — Registered Users, Active Users in range, Active Users that Used Tokens, Total Tokens Consumed, plus engagement-bucket breakdowns
  • Time-series charts with day / week / month granularity toggle:
    • Active users over time
    • Token consumption stacked by model
    • Generation count stacked by gen_type
    • Dual-axis chart (bars = generations, line = token cost)
  • Qualitative insights (LLM-generated, fused from usage + call transcripts):
    client health (Green / Yellow / Red), sentiment, feature requests, blockers, new features tried, recommended next actions, risk flags. Every claim is cited; call-backed claims include verbatim quote spans copied character-for-character from the transcripts so the CSM can validate at a glance.
  • Users table — sortable, searchable, CSV-exportable
  • Per-user-per-model credit consumption pivot — sortable, CSV-exportable, model column order is byte-stable across runs
  • Sliding right-side chat panel — bound to the loaded client + range, with 4 BigQuery-backed tools (query_users, query_actions, query_calls, aggregate). Parallel tool execution, iteration cap = 6, threads keyed by client+range so switching clients gives a fresh thread but coming back restores history.

Architecture notes worth flagging at review

  • All BigQuery queries are parameterized (@param style); the per-org CTE applies the McCann-style normalization rules in one place (src/lib/bigquery/normalize.ts) so the dashboard, insights, and chat tools all see the same roster.
  • LLM output is forced through Anthropic tool-use (tool_choice set to submit_insights) and validated with Zod — no free-form JSON parsing.
  • Chat history is in-memory Zustand only (no localStorage) — refreshing the browser wipes threads. That's intentional for the local-only scope but flagged as a polish item.

Repo layout

enterprise-usage-analyzer/
  src/
    app/                  Next.js App Router pages + API routes
      api/{kpis,series,users,pivot,calls,insights,chat,export/...}
    components/           Dashboard + chat UI
    lib/
      bigquery/           Parameterized BQ queries + normalize CTE
      chat/               4 tools, snapshot, prompt, tool loop
      insights/           Context builder, prompt, schema, client
  scripts/smoke-test.ts   End-to-end smoke test
  README.md               Setup + privacy boundary + acceptance test

Test plan

  • cd enterprise-usage-analyzer && pnpm install
  • Authenticate to BigQuery: gcloud auth application-default login
  • Create .env.local from .env.example and fill in ANTHROPIC_API_KEY
  • pnpm dev, open http://localhost:3000
  • Pick McCann_Paris, range 2026-03-012026-05-01, click Analyze
  • Verify KPI tiles, all 4 charts, both tables render and CSV exports work
  • Verify the Insights section renders with verbatim quotes attached to call-backed items
  • Open the chat panel, ask: "Who are the top 3 users by tokens this period?" — verify it calls query_users and answers in markdown
  • Follow up: "What gen_types has the top user run?" — verify it picks up the lt_id from context and calls query_actions or aggregate

Made with Cursor

Standalone Next.js 16 app under enterprise-usage-analyzer/ that gives CSM /
Sales / leadership a single screen per enterprise client, plus a grounded
chat panel for ad-hoc follow-ups.

Local-only, queries BigQuery via the user's ADC + Anthropic for qualitative
summaries and chat. No third-party services beyond those two.

Highlights:
- KPI tiles + day/week/month time-series charts (active users, tokens by
  model, generations by gen_type, dual-axis bars + cost line)
- Sortable + exportable users table and per-user-per-model pivot table
- LLM insights section that fuses usage data with call transcripts and
  cites every claim — call-backed claims include verbatim quote spans
- Sliding right-side chat panel with 4 BigQuery-backed tools
  (query_users, query_actions, query_calls, aggregate), parallel tool
  execution, capped at 6 iterations
Bigger pill, indigo accent + ring, sparkles + chat icons, and explicit
'Ask the data' label so the entry point isn't easy to miss.
- KpiTiles: drop 'Active Users (in range)', 'Avg Tokens / Active
  Token-User', '% Registered Active'. The three remaining tiles
  (Registered Users, Active Token-Users, Total Tokens) render in
  a 3-up grid instead of 6-up.
- PivotTable: each cell now shows 'tokens (generations)' instead of
  just tokens, e.g. '1,044 (2)'. Total column gets the same treatment.
  The 'Model normalization' diagnostic table gets a new Generations
  column. Backed by COUNT(1) in the per-user-per-model SQL.
- Model mapping: rename 'Genrate Video with LTX2' to 'Genrate Video
  with LTX2.3' to match the underlying model_short_name 'ltx-2.3'.
  Add new column 'Genrate Video Kling 3.0' mapped from kling-3.0
  (previously fell into 'other').
Mirrors the LTX2 -> LTX2.3 rename done earlier. ltx-2.3-fast and
ltx-2-fast both still flow into this column (merge preserved per
earlier accepted convention).
Folds internal Slack chatter into the briefing alongside BigQuery usage data
and Gong call transcripts. Same trust-layer treatment as transcripts:
verbatim quotes are mandatory for any slack-cited claim.

Channel convention (hybrid resolver):
- Main channel (default #ltx-studio-enterprise) is pulled for every client
  and substring-filtered by client mention.
- Per-client channel #customer-<slug> is pulled in full; slug = lowercase +
  non-alphanumerics -> hyphens. 'Bent Image Labs' -> customer-bent-image-labs,
  'McCann_Paris' -> customer-mccann-paris.
- Thread replies expanded for any matching parent (capped, configurable).

Slack module (src/lib/slack/):
- client.ts: minimal Web API wrapper with 429 retry, no @slack/web-api dep
- channels.ts: 10-min cached channel discovery, name->id resolver, permalink
  builder (no per-message chat.getPermalink call)
- users.ts: lazy U-id -> display-name resolver with in-process cache
- normalize.ts: client name -> channel slug + mention patterns
- fetch.ts: hybrid fetcher orchestrating the two channel paths
- cache.ts: 60s TTL keyed by (client, start, end)

Insights pipeline (src/lib/insights/):
- build-context.ts: Slack pulled in the same Promise.all as usage+calls,
  per-message and total char caps applied
- schema.ts: adds slack:<channel>:<ts> citation pattern + slack_quotes[]
  array on every item, plus top-level slack_message_count
- prompt.ts: dedicated 'Slack messages' section with same verbatim-quote
  rules as transcripts (no ellipsis, drop citation if no literal span
  supports the claim)
- index.ts: surfaces a slack_used summary and slack context_summary back to
  the UI

UI (src/components/insights/):
- SlackQuotes.tsx: purple-bordered blockquote, mirroring the amber Quotes
- SlackUsed.tsx: 'Slack messages used as evidence' card with permalinks
- Citations.tsx: third pill colour (purple) for slack:* ids
- InsightItem.tsx: renders slackQuotes alongside quotes
- Insights.tsx: wires slack quotes into every section; renders cleanly when
  Slack is not configured (no UI clutter)

Chat panel:
- types.ts: TOOL_NAMES gets query_slack
- tools.ts: query_slack tool with text_search, channel_scope, date overrides,
  include_threads, limit
- snapshot.ts: Slack channel discovery + per-channel message counts for the
  selected client included in the per-turn snapshot
- prompt.ts: tool list expanded to 5; explicit verbatim-quote rules for
  slack quotes; cross-source fusion guidance
- ToolCallChip.tsx: purple chip for query_slack

Graceful fallback: when SLACK_USER_TOKEN is unset, all Slack code paths
silently disable (no error). The UI hides Slack-related cards/badges so the
dashboard looks identical to the pre-Slack version. The chat tool returns
not_configured: true so the LLM tells the user honestly that Slack isn't
connected, rather than claiming 'no chatter found'.

Smoke test extended: scripts/smoke-test.ts verifies the not-configured path
when no token is present, and reports per-channel counts + the most recent
message when a token is provided.

Required user-token scopes: channels:read, channels:history, groups:read,
groups:history, users:read, team:read. The token must already be a member
of the channels you want to read (no auto-invite).

README updated with the Slack section and project layout.

Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a fixtures-backed code path for the Slack data source so the agent
can be demo'd end-to-end without a live SLACK_USER_TOKEN. When
SLACK_FIXTURES_PATH is set in .env.local, fetchRelevantSlackMessages
reads channel snapshots from a local JSON file instead of calling
slack.com/api. Everything downstream (insights citations, slack_quotes,
chat query_slack tool, UI pills) behaves identically to live mode — so
flipping to a real token later requires zero code changes.

Slack module:
- client.ts: slackEnabled() now returns true when SLACK_FIXTURES_PATH is
  set; new slackFixturesEnabled() helper for the loader.
- fetch.ts: loadFromFixtures() branch at the top of
  fetchRelevantSlackMessages, gated by slackFixturesEnabled(). Reuses
  the same client/customer channel resolution, mention-pattern
  substring matching, date-window filtering, and SlackMessageWithMatch
  shape as the live path. File is mtime-cached in-process and parsed
  lazily.

Fixture build pipeline:
- scripts/build-slack-fixtures.ts: parser that consumes raw MCP
  detailed-format dumps in data/raw-mcp/*.txt and emits
  data/slack-fixtures.json. Handles both the wrapped {"messages": "..."}
  envelope and plain-text variants. Filename (sans .txt) becomes the
  channel name, channel id is picked up from the "Channel: #x (Cxxx)"
  header. Strips Reactions:/Files:/Thread: footers and join/leave
  noise.
- .gitignore: /data/slack-fixtures.json and /data/raw-mcp/ are now
  ignored — real internal Slack content must never end up in the repo.
- .env.example: documents the SLACK_FIXTURES_PATH alternative to
  SLACK_USER_TOKEN.

Insights:
- index.ts: bump Anthropic max_tokens from 8192 to 16384. With Slack
  layered on top of usage + transcripts the model occasionally hit the
  cap mid-tool-call; 16k gives comfortable headroom while staying
  under the SDK's streaming-required threshold.

UI:
- Insights.tsx: drop the "Slack messages used as evidence" card. The
  Slack quotes attached to each insight item are sufficient evidence;
  the standalone list was visually noisy without adding new info.

How to demo without a token:
  pnpm tsx scripts/build-slack-fixtures.ts
  echo 'SLACK_FIXTURES_PATH=./data/slack-fixtures.json' >> .env.local
  pnpm dev

Co-authored-by: Cursor <cursoragent@cursor.com>
… handoff

Two issues surfaced when running the full Slack-aware insights on Meta
(a chatty customer channel with long forwarded legal/email blocks):

1. Schema validation failed because the LLM produced verbatim Slack
   quotes >500 chars — those quotes were real (multi-paragraph legal
   responses, status updates that genuinely don't trim down to a single
   sentence without losing the point). Raised the per-quote cap from
   500 → 1200 chars for both call and slack quotes. Prompt still
   STRONGLY prefers short evidentiary spans (~200–400 chars / 1–3
   sentences); the longer ceiling is only there as a release valve for
   naturally long quotes.

2. Anthropic hit max_tokens mid tool-call when the Slack budget was
   layered onto a transcript-heavy client, producing a truncated
   partial JSON that Zod rejected with cryptic "expected object,
   received string" errors. Tightened the input budget rather than
   raising max_tokens (which would have pushed us above the SDK's
   streaming-required threshold):
   - TOTAL_SLACK_CHAR_CAP: 40k → 22k chars
   - TOTAL_TRANSCRIPT_CAP: 80k → 60k chars
   - quotes per item: 1–3 → 1–2 (prefer 1 strong span)

With these settings, Meta insights now generate cleanly in ~3.5 min
end-to-end (8 calls + 29 Slack messages from #customer-meta and
#ltx-studio-enterprise) with the expected health / sentiment / feature
requests / blockers sections and verbatim Slack quotes attached.

README: added a "Demo mode (fixtures — no Slack token required)"
subsection explaining the SLACK_FIXTURES_PATH workflow + how to obtain
the demo `data/slack-fixtures.json` (out-of-band from the PR author).
The file itself is intentionally gitignored because it carries real
internal Slack content from #ltx-studio-enterprise and the per-customer
channels (deal sizes, named external contacts, legal text). Updated the
graceful-fallback paragraph so the "neither token nor fixtures set"
case is explicit.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant