Skip to content

Commit 1cfc296

Browse files
ericallamsamejr
andauthored
feat(ai): LLM metrics tracking and AI span inspector (#3213)
- Automatic LLM cost enrichment for AI SDK spans (streamText, generateText, generateObject) or any other spans that use semantic gen_ai attributes with support for 145+ models - New AI span inspector sidebar showing model, tokens, cost, messages, tool calls, and response text - LLM metrics dual-write to ClickHouse `llm_metrics_v1` table for analytics - LLM metrics built-in dashboard (unlinked at the moment) - Provider cost fallback — uses gateway/OpenRouter reported costs from `providerMetadata` when registry pricing is unavailable - Prefix-stripping for gateway/OpenRouter model names (e.g. `mistral/mistral-large-3` matches `mistral-large-3` pricing) - Admin dashboard for managing LLM model pricing (list, create, edit, delete, search, test pattern matching) - Missing models detection page — queries ClickHouse for unpriced models with sample spans and Claude Code-ready prompts for adding pricing - AI span seed script (`pnpm run db:seed:ai-spans`) with 51 spans across 12 provider systems for local dev testing - UI fixes: `completionTokens`/`promptTokens` aliases, `ai.response.object` display for generateObject, cache read/write token breakdown ## Screenshots: <img width="1030" height="104" alt="CleanShot 2026-03-17 at 16 48 54@2x" src="https://github.com/user-attachments/assets/bc8fccda-e48b-4d0c-bfb1-e620064e5979" /> <img width="1094" height="1512" alt="CleanShot 2026-03-17 at 16 49 23@2x" src="https://github.com/user-attachments/assets/c2424569-d07e-4d67-a436-e8250043a1ee" /> <img width="1074" height="1412" alt="CleanShot 2026-03-17 at 16 49 18@2x" src="https://github.com/user-attachments/assets/22342ac4-4769-45d1-a328-a24fb9a82a50" /> <img width="1012" height="2292" alt="CleanShot 2026-03-17 at 16 39 01@2x" src="https://github.com/user-attachments/assets/59e327d1-6652-4293-8be0-bb8326e5fbc5" /> <img width="3680" height="2392" alt="CleanShot 2026-03-15 at 08 29 38@2x" src="https://github.com/user-attachments/assets/1f77beb8-de67-495b-b890-bcdb8d7f1fe8" /> --------- Co-authored-by: James Ritchie <james@trigger.dev>
1 parent 411803e commit 1cfc296

File tree

75 files changed

+15771
-45
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+15771
-45
lines changed
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@trigger.dev/core": patch
3+
---
4+
5+
Propagate run tags to span attributes so they can be extracted server-side for LLM cost attribution metadata.
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: feature
4+
---
5+
6+
Add automatic LLM cost calculation for spans with GenAI semantic conventions. When a span arrives with `gen_ai.response.model` and token usage data, costs are calculated from an in-memory pricing registry backed by Postgres and dual-written to both span attributes (`trigger.llm.*`) and a new `llm_metrics_v1` ClickHouse table that captures usage, cost, performance (TTFC, tokens/sec), and behavioral (finish reason, operation type) metrics.

apps/webapp/app/assets/icons/AiProviderIcons.tsx

Lines changed: 177 additions & 0 deletions
Large diffs are not rendered by default.
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
export function AnthropicLogoIcon({ className }: { className?: string }) {
2+
return (
3+
<svg
4+
className={className}
5+
viewBox="0 0 24 24"
6+
fill="currentColor"
7+
xmlns="http://www.w3.org/2000/svg"
8+
>
9+
<path d="M17.3041 3.541h-3.6718l6.696 16.918H24Zm-10.6082 0L0 20.459h3.7442l1.3693-3.5527h7.0052l1.3693 3.5528h3.7442L10.5363 3.5409Zm-.3712 10.2232 2.2914-5.9456 2.2914 5.9456Z" />
10+
</svg>
11+
);
12+
}

apps/webapp/app/components/code/QueryResultsChart.tsx

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1209,9 +1209,19 @@ function createYAxisFormatter(
12091209
formatDurationMilliseconds(value * 1000, { style: "short" });
12101210
}
12111211

1212+
if (format === "durationNs") {
1213+
return (value: number): string =>
1214+
formatDurationMilliseconds(value / 1_000_000, { style: "short" });
1215+
}
1216+
12121217
if (format === "costInDollars" || format === "cost") {
12131218
return (value: number): string => {
12141219
const dollars = format === "cost" ? value / 100 : value;
1220+
if (dollars === 0) return "$0";
1221+
if (Math.abs(dollars) >= 1000) return `$${(dollars / 1000).toFixed(1)}K`;
1222+
if (Math.abs(dollars) >= 1) return `$${dollars.toFixed(2)}`;
1223+
if (Math.abs(dollars) >= 0.01) return `$${dollars.toFixed(4)}`;
1224+
if (Math.abs(dollars) >= 0.0001) return `$${dollars.toFixed(6)}`;
12151225
return formatCurrencyAccurate(dollars);
12161226
};
12171227
}

apps/webapp/app/components/code/TSQLResultsTable.tsx

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,11 @@ function getFormattedValue(value: unknown, column: OutputColumnMetadata): string
8181
return formatDurationMilliseconds(value * 1000, { style: "short" });
8282
}
8383
break;
84+
case "durationNs":
85+
if (typeof value === "number") {
86+
return formatDurationMilliseconds(value / 1_000_000, { style: "short" });
87+
}
88+
break;
8489
case "cost":
8590
if (typeof value === "number") {
8691
return formatCurrencyAccurate(value / 100);
@@ -282,6 +287,12 @@ function getDisplayLength(value: unknown, column: OutputColumnMetadata): number
282287
return formatted.length;
283288
}
284289
return 10;
290+
case "durationNs":
291+
if (typeof value === "number") {
292+
const formatted = formatDurationMilliseconds(value / 1_000_000, { style: "short" });
293+
return formatted.length;
294+
}
295+
return 10;
285296
case "cost":
286297
case "costInDollars":
287298
// Currency format: "$1,234.56"
@@ -598,6 +609,15 @@ function CellValue({
598609
);
599610
}
600611
return <span>{String(value)}</span>;
612+
case "durationNs":
613+
if (typeof value === "number") {
614+
return (
615+
<span className="tabular-nums">
616+
{formatDurationMilliseconds(value / 1_000_000, { style: "short" })}
617+
</span>
618+
);
619+
}
620+
return <span>{String(value)}</span>;
601621
case "cost":
602622
if (typeof value === "number") {
603623
return <span className="tabular-nums">{formatCurrencyAccurate(value / 100)}</span>;
Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
import { CubeIcon } from "@heroicons/react/20/solid";
2+
import * as Ariakit from "@ariakit/react";
3+
import { type ReactNode, useMemo } from "react";
4+
import { AppliedFilter } from "~/components/primitives/AppliedFilter";
5+
import {
6+
ComboBox,
7+
SelectItem,
8+
SelectList,
9+
SelectPopover,
10+
SelectProvider,
11+
SelectTrigger,
12+
} from "~/components/primitives/Select";
13+
import { useSearchParams } from "~/hooks/useSearchParam";
14+
import { appliedSummary, FilterMenuProvider } from "~/components/runs/v3/SharedFilters";
15+
import { tablerIcons } from "~/utils/tablerIcons";
16+
import tablerSpritePath from "~/components/primitives/tabler-sprite.svg";
17+
import { AnthropicLogoIcon } from "~/assets/icons/AnthropicLogoIcon";
18+
19+
const shortcut = { key: "m" };
20+
21+
export type ModelOption = {
22+
model: string;
23+
system: string;
24+
};
25+
26+
interface ModelsFilterProps {
27+
possibleModels: ModelOption[];
28+
}
29+
30+
function modelIcon(system: string, model: string): ReactNode {
31+
// For gateway/openrouter, derive provider from model prefix
32+
let provider = system.split(".")[0];
33+
if (provider === "gateway" || provider === "openrouter") {
34+
if (model.includes("/")) {
35+
provider = model.split("/")[0].replace(/-/g, "");
36+
}
37+
}
38+
39+
// Special case: Anthropic uses a custom SVG icon
40+
if (provider === "anthropic") {
41+
return <AnthropicLogoIcon className="size-4 shrink-0" />;
42+
}
43+
44+
const iconName = `tabler-brand-${provider}`;
45+
if (tablerIcons.has(iconName)) {
46+
return (
47+
<svg className="size-4 shrink-0 stroke-[1.5]">
48+
<use xlinkHref={`${tablerSpritePath}#${iconName}`} />
49+
</svg>
50+
);
51+
}
52+
53+
return <CubeIcon className="size-4 shrink-0" />;
54+
}
55+
56+
export function ModelsFilter({ possibleModels }: ModelsFilterProps) {
57+
const { values, replace, del } = useSearchParams();
58+
const selectedModels = values("models");
59+
60+
if (selectedModels.length === 0 || selectedModels.every((v) => v === "")) {
61+
return (
62+
<FilterMenuProvider>
63+
{(search, setSearch) => (
64+
<ModelsDropdown
65+
trigger={
66+
<SelectTrigger
67+
icon={<CubeIcon className="size-4" />}
68+
variant="secondary/small"
69+
shortcut={shortcut}
70+
tooltipTitle="Filter by model"
71+
>
72+
<span className="ml-0.5">Models</span>
73+
</SelectTrigger>
74+
}
75+
searchValue={search}
76+
clearSearchValue={() => setSearch("")}
77+
possibleModels={possibleModels}
78+
/>
79+
)}
80+
</FilterMenuProvider>
81+
);
82+
}
83+
84+
return (
85+
<FilterMenuProvider>
86+
{(search, setSearch) => (
87+
<ModelsDropdown
88+
trigger={
89+
<Ariakit.Select render={<div className="group cursor-pointer focus-custom" />}>
90+
<AppliedFilter
91+
label="Model"
92+
icon={<CubeIcon className="size-4" />}
93+
value={appliedSummary(selectedModels)}
94+
onRemove={() => del(["models"])}
95+
variant="secondary/small"
96+
/>
97+
</Ariakit.Select>
98+
}
99+
searchValue={search}
100+
clearSearchValue={() => setSearch("")}
101+
possibleModels={possibleModels}
102+
/>
103+
)}
104+
</FilterMenuProvider>
105+
);
106+
}
107+
108+
function ModelsDropdown({
109+
trigger,
110+
clearSearchValue,
111+
searchValue,
112+
onClose,
113+
possibleModels,
114+
}: {
115+
trigger: ReactNode;
116+
clearSearchValue: () => void;
117+
searchValue: string;
118+
onClose?: () => void;
119+
possibleModels: ModelOption[];
120+
}) {
121+
const { values, replace } = useSearchParams();
122+
123+
const handleChange = (values: string[]) => {
124+
clearSearchValue();
125+
replace({ models: values });
126+
};
127+
128+
const filtered = useMemo(() => {
129+
return possibleModels.filter((m) => {
130+
return m.model?.toLowerCase().includes(searchValue.toLowerCase());
131+
});
132+
}, [searchValue, possibleModels]);
133+
134+
return (
135+
<SelectProvider value={values("models")} setValue={handleChange} virtualFocus={true}>
136+
{trigger}
137+
<SelectPopover
138+
className="min-w-0 max-w-[min(360px,var(--popover-available-width))]"
139+
hideOnEscape={() => {
140+
if (onClose) {
141+
onClose();
142+
return false;
143+
}
144+
return true;
145+
}}
146+
>
147+
<ComboBox placeholder="Filter by model..." value={searchValue} />
148+
<SelectList>
149+
{filtered.map((m) => (
150+
<SelectItem key={m.model} value={m.model} icon={modelIcon(m.system, m.model)}>
151+
{m.model}
152+
</SelectItem>
153+
))}
154+
{filtered.length === 0 && <SelectItem disabled>No models found</SelectItem>}
155+
</SelectList>
156+
</SelectPopover>
157+
</SelectProvider>
158+
);
159+
}

apps/webapp/app/components/runs/v3/RunIcon.tsx

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,25 @@ import {
33
HandRaisedIcon,
44
InformationCircleIcon,
55
RectangleStackIcon,
6+
SparklesIcon,
67
Squares2X2Icon,
78
TableCellsIcon,
89
TagIcon,
10+
WrenchIcon,
911
} from "@heroicons/react/20/solid";
12+
import { AnthropicLogoIcon } from "~/assets/icons/AnthropicLogoIcon";
13+
import {
14+
AnthropicIcon,
15+
AzureIcon,
16+
CerebrasIcon,
17+
DeepseekIcon,
18+
GeminiIcon,
19+
LlamaIcon,
20+
MistralIcon,
21+
OpenAIIcon,
22+
PerplexityIcon,
23+
XAIIcon,
24+
} from "~/assets/icons/AiProviderIcons";
1025
import { AttemptIcon } from "~/assets/icons/AttemptIcon";
1126
import { TaskIcon } from "~/assets/icons/TaskIcon";
1227
import { cn } from "~/utils/cn";
@@ -112,6 +127,31 @@ export function RunIcon({ name, className, spanName }: TaskIconProps) {
112127
return <FunctionIcon className={cn(className, "text-error")} />;
113128
case "streams":
114129
return <StreamsIcon className={cn(className, "text-text-dimmed")} />;
130+
case "hero-sparkles":
131+
return <SparklesIcon className={cn(className, "text-text-dimmed")} />;
132+
case "hero-wrench":
133+
return <WrenchIcon className={cn(className, "text-text-dimmed")} />;
134+
case "tabler-brand-anthropic":
135+
case "ai-provider-anthropic":
136+
return <AnthropicIcon className={cn(className, "text-text-dimmed")} />;
137+
case "ai-provider-openai":
138+
return <OpenAIIcon className={cn(className, "text-text-dimmed")} />;
139+
case "ai-provider-gemini":
140+
return <GeminiIcon className={cn(className, "text-text-dimmed")} />;
141+
case "ai-provider-llama":
142+
return <LlamaIcon className={cn(className, "text-text-dimmed")} />;
143+
case "ai-provider-deepseek":
144+
return <DeepseekIcon className={cn(className, "text-text-dimmed")} />;
145+
case "ai-provider-xai":
146+
return <XAIIcon className={cn(className, "text-text-dimmed")} />;
147+
case "ai-provider-perplexity":
148+
return <PerplexityIcon className={cn(className, "text-text-dimmed")} />;
149+
case "ai-provider-cerebras":
150+
return <CerebrasIcon className={cn(className, "text-text-dimmed")} />;
151+
case "ai-provider-mistral":
152+
return <MistralIcon className={cn(className, "text-text-dimmed")} />;
153+
case "ai-provider-azure":
154+
return <AzureIcon className={cn(className, "text-text-dimmed")} />;
115155
}
116156

117157
return <InformationCircleIcon className={cn(className, "text-text-dimmed")} />;

apps/webapp/app/components/runs/v3/SpanTitle.tsx

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ import { TaskEventStyle } from "@trigger.dev/core/v3";
33
import type { TaskEventLevel } from "@trigger.dev/database";
44
import { Fragment } from "react";
55
import { cn } from "~/utils/cn";
6+
import { tablerIcons } from "~/utils/tablerIcons";
7+
import tablerSpritePath from "~/components/primitives/tabler-sprite.svg";
68

79
type SpanTitleProps = {
810
message: string;
@@ -45,6 +47,15 @@ function SpanAccessory({
4547
/>
4648
);
4749
}
50+
case "pills": {
51+
return (
52+
<div className="flex items-center gap-1">
53+
{accessory.items.map((item, index) => (
54+
<SpanPill key={index} text={item.text} icon={item.icon} />
55+
))}
56+
</div>
57+
);
58+
}
4859
default: {
4960
return (
5061
<div className={cn("flex gap-1")}>
@@ -59,6 +70,21 @@ function SpanAccessory({
5970
}
6071
}
6172

73+
function SpanPill({ text, icon }: { text: string; icon?: string }) {
74+
const hasIcon = icon && tablerIcons.has(icon);
75+
76+
return (
77+
<span className="inline-flex items-center gap-0.5 rounded-full border border-charcoal-700 bg-charcoal-850 px-1.5 py-px text-xxs text-text-dimmed">
78+
{hasIcon && (
79+
<svg className="size-3 stroke-[1.5] text-text-dimmed/70">
80+
<use xlinkHref={`${tablerSpritePath}#${icon}`} />
81+
</svg>
82+
)}
83+
<span className="truncate">{text}</span>
84+
</span>
85+
);
86+
}
87+
6288
export function SpanCodePathAccessory({
6389
accessory,
6490
className,

0 commit comments

Comments
 (0)