Skip to content

fix(telemetry): propagate OTel trace context so all spans share one trace#245

Merged
fede-kamel merged 2 commits into
mainfrom
fix/telemetry-trace-context-propagation
May 22, 2026
Merged

fix(telemetry): propagate OTel trace context so all spans share one trace#245
fede-kamel merged 2 commits into
mainfrom
fix/telemetry-trace-context-propagation

Conversation

@fede-kamel
Copy link
Copy Markdown
Contributor

@fede-kamel fede-kamel commented May 20, 2026

Closes #244. Draft until Langfuse repro lands this afternoon —
we'll cross-check it against this fix + add it as a second regression
test before marking ready-for-review.

Summary

TelemetryHook was calling tracer.start_span(...) for invocation,
iteration, and tool spans without ever attaching them to OTel's current
context. Every span emerged as its own trace root, so Langfuse / Jaeger /
Tempo rendered N separate traces per agent run instead of one tree.

Reported by @Luigis while wiring up Langfuse.

Fix

opentelemetry.context.attach(trace.set_span_in_context(span)) after
starting each parent span, paired with context.detach(token) on end.
This pushes the span onto OTel's context stack so subsequent start_span
calls within the same async scope automatically parent to it.

Hook Before After
on_before_invocation starts span, doesn't attach starts span + attaches to OTel context
on_after_invocation ends span ends span + detaches context
on_iteration_start starts span, doesn't attach starts span + attaches (auto-child of invocation)
on_iteration_end ends span ends span + detaches
on_before_tool_call starts span at root starts span (auto-child of iteration)
on_after_tool_call ends span unchanged — tool span needs no context bookkeeping

Regression tests

Three new tests under TestTraceContextPropagation using OTel's
InMemorySpanExporter:

  1. test_all_spans_share_one_trace_id — asserts the 3-span run
    (invocation + iteration + tool) shares exactly one trace_id.
  2. test_tool_span_is_child_of_iteration_span — verifies the parent
    chain via span.parent.span_id: invocation has no parent (root),
    iteration's parent = invocation, tool's parent = iteration.
  3. test_invocation_context_detaches_after_run — confirms the context
    is detached on completion so the caller's downstream spans aren't
    parented to ours.

Tests gracefully skip when opentelemetry-sdk isn't installed in the
test env (the hook only needs opentelemetry-api at runtime).

Test plan

  • hatch run check — ruff format + ruff check + mypy clean
  • 31 / 31 telemetry tests pass (including 3 new regressions)
  • 4788 full unit suite green
  • Cross-check against Luigi's Langfuse repro when it lands

Out of scope (filed for later)

The hook still doesn't currently set the service.name resource (just
attaches it as a span attribute). Spec-wise it should live on the
resource, not the span — but moving it requires a Resource on the
TracerProvider, which is the caller's concern. Leaving as-is.

…race

`TelemetryHook` used `tracer.start_span(...)` for invocation, iteration,
and tool spans without ever attaching them to OTel's current context.
Result: every span emerged as its own trace root, so Langfuse / Jaeger /
Tempo / any OTLP backend rendered N separate traces per agent run instead
of one tree.

Reported by @Luigis while wiring up Langfuse. Closes #244.

## Fix

Use `opentelemetry.context.attach(trace.set_span_in_context(span))` after
starting each parent span, then `context.detach(token)` when the span
ends. This pushes the span onto OTel's context stack so subsequent
`start_span` calls within the same async scope automatically parent to
it.

- `on_before_invocation` attaches the invocation context; `on_after_invocation`
  detaches it.
- `on_iteration_start` attaches the iteration context (child of the
  invocation); `on_iteration_end` detaches.
- `on_before_tool_call` doesn't need explicit parenting — the iteration
  context is already attached, so the tool span automatically becomes its
  child.

## Tests

Three new regression tests under `TestTraceContextPropagation` using OTel's
`InMemorySpanExporter`:

- `test_all_spans_share_one_trace_id` — invocation + iteration + tool
  share a single `trace_id`.
- `test_tool_span_is_child_of_iteration_span` — verifies the parent chain
  (invocation → iteration → tool) via `span.parent.span_id`.
- `test_invocation_context_detaches_after_run` — confirms the context is
  detached on completion so the next caller's spans aren't parented to
  ours.

All 31 telemetry tests pass; full unit suite (4788) clean. Tests
gracefully skip when `opentelemetry-sdk` isn't installed (the hook only
needs `opentelemetry-api` at runtime).

Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
@oracle-contributor-agreement oracle-contributor-agreement Bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label May 20, 2026
@luigisaetta
Copy link
Copy Markdown

OK, I have done the tests @fede-kamel the issue seems to be solved!

… lines

`src/locus/hooks/builtin/telemetry.py` drops from 95.04% to 93.33%.
The newly uncovered lines (32-39) are the OpenTelemetry `ImportError`
fallback — defensive code that only runs when `opentelemetry` isn't
installed, which never happens under CI.

Baseline also picks up two organic improvements:
  - loop/runner.py: 96.60% -> 97.28%
  - memory/managers/oracle_agent_memory.py: 81.35% -> 81.63%

Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
@fede-kamel fede-kamel force-pushed the fix/telemetry-trace-context-propagation branch from debc75c to 5bb48ce Compare May 22, 2026 08:04
@fede-kamel fede-kamel marked this pull request as ready for review May 22, 2026 08:49
@fede-kamel fede-kamel merged commit f68948a into main May 22, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

OCA Verified All contributors have signed the Oracle Contributor Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TelemetryHook breaks OTel trace context propagation — each span becomes its own trace

2 participants