-
-
Notifications
You must be signed in to change notification settings - Fork 360
Add OpenTelemetry observability support #556
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
- Add instrumentation module with span building for chat completions and tool calls - Support session tracking across conversation turns - Add configurable metadata prefix for custom trace attributes - Implement NullTracer pattern for graceful fallback when OTel not installed - Fail loudly with helpful error when tracing enabled but OTel unavailable - Add comprehensive documentation for LangSmith and other backends - Include full test coverage for instrumentation
Instrumentation.enabled? and Instrumentation.tracer now accept an optional config parameter, allowing Chat to pass its context-specific config instead of always using the global RubyLLM.config. This ensures that per-context tracing configuration is respected.
Convert AR record ID to string when passing to Chat#session_id to ensure type consistency. Chat defaults to UUID (string) when no session_id is provided, so AR adapters should also pass strings.
OTel supports string, int, float, and bool attribute values natively. Only stringify complex objects that don't fit these types. This improves filtering and querying capabilities in observability backends.
Replace attr_reader :metadata with a method that returns @metadata.dup to prevent external mutation of internal state. Callers must use with_metadata to modify the hash.
Add tracing_langsmith_compat config option (default: false) that: - Adds langsmith.span.kind attribute to spans - Adds input.value/output.value for LangSmith panels - Auto-sets tracing_metadata_prefix to 'langsmith.metadata' Standard gen_ai.* attributes are always emitted regardless of this setting. This allows users of other OTel backends to avoid the LangSmith-specific noise while LangSmith users get full integration. Also replaces the made-up gen_ai.prompt/gen_ai.completion fallback attributes with proper gen_ai.tool.input/gen_ai.tool.output for tool spans.
Use "#{self.class.name}:#{id}" format (e.g., "ChatSession:123")
instead of just the ID. This prevents session_id collisions when
multiple models use acts_as_chat, ensuring traces are correctly
grouped per-model in observability backends.
Update What Gets Traced section to show gen_ai.tool.input/output (always emitted) vs input.value/output.value (LangSmith-only). Add LangSmith-specific attribute tables to both Chat and Tool sections.
- Remove tracer memoization to handle OTel reconfiguration - Warn and disable (not crash) when OTel gems missing (follows OTel's 'safe by default' pattern) - Revert metadata prefix when disabling langsmith_compat - JSON encode Hash/Array metadata values instead of .to_s garbage
Remove otel_safe_value method - just pass values through and let the OpenTelemetry SDK handle type conversion. No need to reinvent the wheel.
- Use attr_reader for tracing_langsmith_compat (custom setter defined) - Consolidate build_request_attributes params into config hash - Use allow/have_received pattern for logger spy tests - Fix Model::Info double in tests
74e06eb to
62ee78b
Compare
- Unified complete() to always wrap in span (streaming + non-streaming) - Extracted add_request_span_attributes and add_response_span_attributes helpers - Added streaming documentation section - Added 4 streaming instrumentation tests
- Rename gen_ai.system to gen_ai.provider.name - Rename gen_ai.tool.input to gen_ai.tool.call.arguments - Rename gen_ai.tool.output to gen_ai.tool.call.result - Change tool operation name from 'tool' to 'execute_tool' - Add links to OTEL GenAI spec in observability docs See: https://opentelemetry.io/docs/specs/semconv/gen-ai/
- Introduced 'json' library to enhance metadata handling capabilities.
- Updated content logging to output prompts and completions as JSON arrays. - Introduced `gen_ai.input.messages` and `gen_ai.output.messages` attributes for structured message logging. - Refactored `build_message_attributes` and `build_completion_attributes` methods to generate OTEL-compliant message formats. - Updated tests to validate new JSON structure for input and output messages.
…observability - Updated span names in the complete and execute_tool methods to include specific identifiers for better traceability. - Enhanced logging clarity by dynamically generating span names based on model ID and tool call name.
|
@crmne I'd love to get this merged. Let me know how I can support you. |
- Revised span naming conventions in the documentation for `chat.ask()` and tool invocation to include specific identifiers, enhancing traceability and alignment with industry standards.
|
I really like OpenTelemetry and i think something like this is super helpful. But I think the instrumentation belongs somewhere else. OpenTelemetry instruments rails (active record, action view, action mailers, etc), sidekiq, sinatra, rake, etc, and the instrumentation code doesn't live in the original code. The instrumentation code is installed separately and configured separately. See how OpenTelemetry instruments various libraries. I think the right approach is to follow OpenTelemetry's approach. This way it will be consistent with OpenTelemetry, and it will keep RubyLLM clean, free of OpenTelemetry code. |
Summary
Add OpenTelemetry-based observability to RubyLLM, enabling users to send traces to any compatible backend (LangSmith, Honeycomb, New Relic, etc.).
Follows the OpenTelemetry Semantic Conventions for GenAI.
What's Included
chat {model_id}spans (e.g.,chat gpt-4o) with model, provider, token counts, and GenAI semantic convention attributesexecute_tool {tool_name}child spans (e.g.,execute_tool get_weather) with tool name, arguments, and resultsClassName:id) to prevent collisions across modelschat.with_metadata(user_id: 123)for filtering/debugging. Native types (int, float, bool) preserved for better backend filteringtracing_langsmith_compatmode adds LangSmith-specific attributes (langsmith.span.kind,input.value/output.valuepanels)RubyLLM.contextConfigurationErrorwith helpful message if tracing enabled but OpenTelemetry gems not installedConfiguration
Not in Scope
Intentionally deferred to keep this PR focused: