Skip to content

Conversation

@AAgnihotry
Copy link
Contributor

@AAgnihotry AAgnihotry commented Jan 17, 2026

Summary

  • Enabled LiveTrackingSpanProcessor for both job_exporter and studio_web_tracking_exporter for real-time span updates
  • Only create LLM Ops exporters when actually needed (job or Studio Web context exists)
  • Fixed evaluation hanging issue caused by missing exporter parameters in evaluate() calls
  • Removed unnecessary logging statements
  • Updated all unit tests to pass required exporter parameters

Changes

Core Implementation

  • cli_eval.py:
    • Only create studio_web_tracking_exporter when should_register_progress_reporter is True
    • Create separate job_exporter only when ctx.job_id exists
    • Only add job_exporter to trace_manager for batch export
  • _evaluate.py: Updated signature to accept optional exporters (can be None)
  • _runtime.py:
    • Accept optional exporters in constructor
    • Only create LiveTrackingSpanProcessor when exporters are not None
    • This prevents sending traces to LLM Ops during local development
  • _progress_reporter.py: Removed extra logging statements

Test Fixes

  • test_evaluate.py:
    • Added LlmOpsHttpExporter imports and updated test calls with exporter parameters
    • Added new test to verify local execution works without any exporters
  • test_eval_tracing_integration.py: Updated all 10 UiPathEvalRuntime instantiations with mock exporters
  • test_eval_runtime_metadata.py: Updated all 8 UiPathEvalRuntime instantiations with exporters

Version

  • Bumped version to 2.5.9

Technical Details

When Exporters Are Created

  • job_exporter: Only when ctx.job_id exists (running in Orchestrator job)
  • studio_web_tracking_exporter: Only when should_register_progress_reporter is True (running in Studio Web with UIPATH_PROJECT_ID)
  • Local development: Neither exporter is created, no traces sent to LLM Ops

Real-time Tracking

  • Both exporters have LiveTrackingSpanProcessor enabled for real-time tracking when they exist
  • Only job_exporter is added to trace_manager (for batch export)
  • studio_web_tracking_exporter is NOT added to trace_manager to avoid duplicates with manual exports from StudioWebProgressReporter
  • Shared instance pattern ensures StudioWebProgressReporter's trace_id updates are reflected in LiveTrackingSpanProcessor

Test Results

  • All 1751 tests passing
  • 98% code coverage for eval tests
  • New test confirms local execution works without exporters

🤖 Generated with Claude Code

Development Package

  • Use uipath pack --nolock to get the latest dev build from this PR (requires version range).
  • Add this package as a dependency in your pyproject.toml:
[project]
dependencies = [
  # Exact version:
  "uipath==2.5.9.dev1011453984",

  # Any version from PR
  "uipath>=2.5.9.dev1011450000,<2.5.9.dev1011460000"
]

[[tool.uv.index]]
name = "testpypi"
url = "https://test.pypi.org/simple/"
publish-url = "https://test.pypi.org/legacy/"
explicit = true

[tool.uv.sources]
uipath = { index = "testpypi" }

[tool.uv]
override-dependencies = [
    "uipath>=2.5.9.dev1011450000,<2.5.9.dev1011460000",
]

@github-actions github-actions bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository labels Jan 17, 2026
@AAgnihotry AAgnihotry force-pushed the feat/dt5 branch 4 times, most recently from 5517bc3 to e63d1e3 Compare January 17, 2026 01:11
@AAgnihotry AAgnihotry added the build:dev Create a dev build from the pr label Jan 17, 2026
@AAgnihotry
Copy link
Contributor Author

image

live_tracking_processor = LiveTrackingSpanProcessor(live_tracking_exporter)
self.trace_manager.tracer_span_processors.append(live_tracking_processor)
self.trace_manager.tracer_provider.add_span_processor(live_tracking_processor)
# Job exporter tracking processor for real-time span updates
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not do all of this in cli_eval.py instead?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While making the change, please move LiveTrackingSpanProcessor to a separate file..

trace_manager: UiPathTraceManager,
eval_context: UiPathEvalContext,
event_bus: EventBus,
job_exporter: LlmOpsHttpExporter | None,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not do this -- it's a breaking change for code conversion. Moving initialization logic to cli_eval will avoid this change..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build:dev Create a dev build from the pr test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-llamaindex Triggers tests in the uipath-llamaindex-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants