Add Enterprise Usage Analyzer (Next.js UI for CSM dashboard + grounded chat)#63
Open
Yonatan Edelstein (YoniEdelstein) wants to merge 7 commits into
Open
Add Enterprise Usage Analyzer (Next.js UI for CSM dashboard + grounded chat)#63Yonatan Edelstein (YoniEdelstein) wants to merge 7 commits into
Yonatan Edelstein (YoniEdelstein) wants to merge 7 commits into
Conversation
Standalone Next.js 16 app under enterprise-usage-analyzer/ that gives CSM / Sales / leadership a single screen per enterprise client, plus a grounded chat panel for ad-hoc follow-ups. Local-only, queries BigQuery via the user's ADC + Anthropic for qualitative summaries and chat. No third-party services beyond those two. Highlights: - KPI tiles + day/week/month time-series charts (active users, tokens by model, generations by gen_type, dual-axis bars + cost line) - Sortable + exportable users table and per-user-per-model pivot table - LLM insights section that fuses usage data with call transcripts and cites every claim — call-backed claims include verbatim quote spans - Sliding right-side chat panel with 4 BigQuery-backed tools (query_users, query_actions, query_calls, aggregate), parallel tool execution, capped at 6 iterations
Bigger pill, indigo accent + ring, sparkles + chat icons, and explicit 'Ask the data' label so the entry point isn't easy to miss.
- KpiTiles: drop 'Active Users (in range)', 'Avg Tokens / Active Token-User', '% Registered Active'. The three remaining tiles (Registered Users, Active Token-Users, Total Tokens) render in a 3-up grid instead of 6-up. - PivotTable: each cell now shows 'tokens (generations)' instead of just tokens, e.g. '1,044 (2)'. Total column gets the same treatment. The 'Model normalization' diagnostic table gets a new Generations column. Backed by COUNT(1) in the per-user-per-model SQL. - Model mapping: rename 'Genrate Video with LTX2' to 'Genrate Video with LTX2.3' to match the underlying model_short_name 'ltx-2.3'. Add new column 'Genrate Video Kling 3.0' mapped from kling-3.0 (previously fell into 'other').
Mirrors the LTX2 -> LTX2.3 rename done earlier. ltx-2.3-fast and ltx-2-fast both still flow into this column (merge preserved per earlier accepted convention).
Folds internal Slack chatter into the briefing alongside BigQuery usage data and Gong call transcripts. Same trust-layer treatment as transcripts: verbatim quotes are mandatory for any slack-cited claim. Channel convention (hybrid resolver): - Main channel (default #ltx-studio-enterprise) is pulled for every client and substring-filtered by client mention. - Per-client channel #customer-<slug> is pulled in full; slug = lowercase + non-alphanumerics -> hyphens. 'Bent Image Labs' -> customer-bent-image-labs, 'McCann_Paris' -> customer-mccann-paris. - Thread replies expanded for any matching parent (capped, configurable). Slack module (src/lib/slack/): - client.ts: minimal Web API wrapper with 429 retry, no @slack/web-api dep - channels.ts: 10-min cached channel discovery, name->id resolver, permalink builder (no per-message chat.getPermalink call) - users.ts: lazy U-id -> display-name resolver with in-process cache - normalize.ts: client name -> channel slug + mention patterns - fetch.ts: hybrid fetcher orchestrating the two channel paths - cache.ts: 60s TTL keyed by (client, start, end) Insights pipeline (src/lib/insights/): - build-context.ts: Slack pulled in the same Promise.all as usage+calls, per-message and total char caps applied - schema.ts: adds slack:<channel>:<ts> citation pattern + slack_quotes[] array on every item, plus top-level slack_message_count - prompt.ts: dedicated 'Slack messages' section with same verbatim-quote rules as transcripts (no ellipsis, drop citation if no literal span supports the claim) - index.ts: surfaces a slack_used summary and slack context_summary back to the UI UI (src/components/insights/): - SlackQuotes.tsx: purple-bordered blockquote, mirroring the amber Quotes - SlackUsed.tsx: 'Slack messages used as evidence' card with permalinks - Citations.tsx: third pill colour (purple) for slack:* ids - InsightItem.tsx: renders slackQuotes alongside quotes - Insights.tsx: wires slack quotes into every section; renders cleanly when Slack is not configured (no UI clutter) Chat panel: - types.ts: TOOL_NAMES gets query_slack - tools.ts: query_slack tool with text_search, channel_scope, date overrides, include_threads, limit - snapshot.ts: Slack channel discovery + per-channel message counts for the selected client included in the per-turn snapshot - prompt.ts: tool list expanded to 5; explicit verbatim-quote rules for slack quotes; cross-source fusion guidance - ToolCallChip.tsx: purple chip for query_slack Graceful fallback: when SLACK_USER_TOKEN is unset, all Slack code paths silently disable (no error). The UI hides Slack-related cards/badges so the dashboard looks identical to the pre-Slack version. The chat tool returns not_configured: true so the LLM tells the user honestly that Slack isn't connected, rather than claiming 'no chatter found'. Smoke test extended: scripts/smoke-test.ts verifies the not-configured path when no token is present, and reports per-channel counts + the most recent message when a token is provided. Required user-token scopes: channels:read, channels:history, groups:read, groups:history, users:read, team:read. The token must already be a member of the channels you want to read (no auto-invite). README updated with the Slack section and project layout. Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a fixtures-backed code path for the Slack data source so the agent
can be demo'd end-to-end without a live SLACK_USER_TOKEN. When
SLACK_FIXTURES_PATH is set in .env.local, fetchRelevantSlackMessages
reads channel snapshots from a local JSON file instead of calling
slack.com/api. Everything downstream (insights citations, slack_quotes,
chat query_slack tool, UI pills) behaves identically to live mode — so
flipping to a real token later requires zero code changes.
Slack module:
- client.ts: slackEnabled() now returns true when SLACK_FIXTURES_PATH is
set; new slackFixturesEnabled() helper for the loader.
- fetch.ts: loadFromFixtures() branch at the top of
fetchRelevantSlackMessages, gated by slackFixturesEnabled(). Reuses
the same client/customer channel resolution, mention-pattern
substring matching, date-window filtering, and SlackMessageWithMatch
shape as the live path. File is mtime-cached in-process and parsed
lazily.
Fixture build pipeline:
- scripts/build-slack-fixtures.ts: parser that consumes raw MCP
detailed-format dumps in data/raw-mcp/*.txt and emits
data/slack-fixtures.json. Handles both the wrapped {"messages": "..."}
envelope and plain-text variants. Filename (sans .txt) becomes the
channel name, channel id is picked up from the "Channel: #x (Cxxx)"
header. Strips Reactions:/Files:/Thread: footers and join/leave
noise.
- .gitignore: /data/slack-fixtures.json and /data/raw-mcp/ are now
ignored — real internal Slack content must never end up in the repo.
- .env.example: documents the SLACK_FIXTURES_PATH alternative to
SLACK_USER_TOKEN.
Insights:
- index.ts: bump Anthropic max_tokens from 8192 to 16384. With Slack
layered on top of usage + transcripts the model occasionally hit the
cap mid-tool-call; 16k gives comfortable headroom while staying
under the SDK's streaming-required threshold.
UI:
- Insights.tsx: drop the "Slack messages used as evidence" card. The
Slack quotes attached to each insight item are sufficient evidence;
the standalone list was visually noisy without adding new info.
How to demo without a token:
pnpm tsx scripts/build-slack-fixtures.ts
echo 'SLACK_FIXTURES_PATH=./data/slack-fixtures.json' >> .env.local
pnpm dev
Co-authored-by: Cursor <cursoragent@cursor.com>
… handoff Two issues surfaced when running the full Slack-aware insights on Meta (a chatty customer channel with long forwarded legal/email blocks): 1. Schema validation failed because the LLM produced verbatim Slack quotes >500 chars — those quotes were real (multi-paragraph legal responses, status updates that genuinely don't trim down to a single sentence without losing the point). Raised the per-quote cap from 500 → 1200 chars for both call and slack quotes. Prompt still STRONGLY prefers short evidentiary spans (~200–400 chars / 1–3 sentences); the longer ceiling is only there as a release valve for naturally long quotes. 2. Anthropic hit max_tokens mid tool-call when the Slack budget was layered onto a transcript-heavy client, producing a truncated partial JSON that Zod rejected with cryptic "expected object, received string" errors. Tightened the input budget rather than raising max_tokens (which would have pushed us above the SDK's streaming-required threshold): - TOTAL_SLACK_CHAR_CAP: 40k → 22k chars - TOTAL_TRANSCRIPT_CAP: 80k → 60k chars - quotes per item: 1–3 → 1–2 (prefer 1 strong span) With these settings, Meta insights now generate cleanly in ~3.5 min end-to-end (8 calls + 29 Slack messages from #customer-meta and #ltx-studio-enterprise) with the expected health / sentiment / feature requests / blockers sections and verbatim Slack quotes attached. README: added a "Demo mode (fixtures — no Slack token required)" subsection explaining the SLACK_FIXTURES_PATH workflow + how to obtain the demo `data/slack-fixtures.json` (out-of-band from the PR author). The file itself is intentionally gitignored because it carries real internal Slack content from #ltx-studio-enterprise and the per-customer channels (deal sizes, named external contacts, legal text). Updated the graceful-fallback paragraph so the "neither token nor fixtures set" case is explicit. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
enterprise-usage-analyzer/— a standalone Next.js 16 app that gives CSM / Sales / leadership a single screen per enterprise client with KPIs, charts, AI-generated qualitative insights, and a chat panel that can re-query BigQuery on demand.Local-only (no deploy in this PR). Queries BigQuery via the user's own
gcloudADC and Anthropic for qualitative summaries / chat. Those are the only two destinations data leaves the machine for — no analytics, no Sentry, no third-party fonts, Next.js telemetry disabled.What you get on screen
For a chosen client + date range (default last 30 days, max 12 months):
client health (Green / Yellow / Red), sentiment, feature requests, blockers, new features tried, recommended next actions, risk flags. Every claim is cited; call-backed claims include verbatim quote spans copied character-for-character from the transcripts so the CSM can validate at a glance.
query_users,query_actions,query_calls,aggregate). Parallel tool execution, iteration cap = 6, threads keyed by client+range so switching clients gives a fresh thread but coming back restores history.Architecture notes worth flagging at review
@paramstyle); the per-org CTE applies the McCann-style normalization rules in one place (src/lib/bigquery/normalize.ts) so the dashboard, insights, and chat tools all see the same roster.tool_choiceset tosubmit_insights) and validated with Zod — no free-form JSON parsing.localStorage) — refreshing the browser wipes threads. That's intentional for the local-only scope but flagged as a polish item.Repo layout
Test plan
cd enterprise-usage-analyzer && pnpm installgcloud auth application-default login.env.localfrom.env.exampleand fill inANTHROPIC_API_KEYpnpm dev, openhttp://localhost:3000McCann_Paris, range2026-03-01→2026-05-01, click Analyzequery_usersand answers in markdownquery_actionsoraggregateMade with Cursor