Add Braintrust tracing as experimental drop-in environment variants

[cdreetz]

Summary

Adds opt-in Braintrust tracing to verifiers as experimental drop-in replacements for the core environment classes, living entirely under verifiers/envs/experimental/braintrust_tracing/. The original core files (environment.py, multiturn_env.py, tool_env.py, stateful_tool_env.py) are unchanged from main — no tracing code touches production paths.

When BRAINTRUST_API_KEY is set, every rollout emits a trace with child spans for setup, turns, model requests, tool calls, and scoring. When unset, all tracing is a no-op with near-zero overhead.

Usage

# Instead of:
from verifiers.envs.stateful_tool_env import StatefulToolEnv

# Use the tracing variant:
from verifiers.envs.experimental.braintrust_tracing.stateful_tool_env import StatefulToolEnv

pip install 'verifiers[braintrust]'
export BRAINTRUST_API_KEY="sk-..."
export VF_BRAINTRUST_PROJECT="my-project"  # optional, defaults to "verifiers"

Structure

verifiers/envs/experimental/braintrust_tracing/
├── __init__.py                # re-exports all four classes
├── braintrust_tracing.py      # self-contained tracing layer
├── environment.py             # Environment with tracing
├── multiturn_env.py           # MultiTurnEnv with tracing
├── tool_env.py                # ToolEnv with tracing
└── stateful_tool_env.py       # StatefulToolEnv with tracing

Inheritance chain within the experimental folder: Environment → MultiTurnEnv → ToolEnv → StatefulToolEnv. Each class inherits from its experimental sibling (not from the original vf.* classes), so the full tracing instrumentation propagates through the hierarchy.

braintrust_tracing.py module

A self-contained tracing layer with:

• Lazy singleton logger initialization (thread-safe, import-safe)
• All errors swallowed so tracing never kills an eval
• _safe() / _safe_response() serialization helpers for Pydantic models and arbitrary objects
• contextvars.ContextVar for passing the rollout span from _run_rollout_state → rollout() across the await boundary (concurrent-rollout safe)
• Run-level tags: each generate() call auto-generates a unique tag (run-<epoch>-<short_uuid>) attached to every root span (rollout, group, generate), enabling filter/group by eval run in Braintrust. Tags stored in a ContextVar for concurrent safety. Can also be set manually via set_run_tags(['my-tag']).

Instrumented spans

• environment.py — get_model_response (LLM spans with token counts), _run_rollout_state (rollout + scoring spans), _run_group_states (group spans with per-rollout child spans), generate (run tag lifecycle + flush on exit)
• multiturn_env.py — rollout() setup spans, per-turn spans (with model/env duration breakdown), timeout logging
• tool_env.py — call_tool() tool spans with args/result/duration; env_response() passes state= to call_tool for span parenting
• stateful_tool_env.py — env_response() passes state=state to call_tool, enabling tool tracing for all StatefulToolEnv subclasses

pyproject.toml changes

• braintrust added as optional dep group [braintrust]
• [[tool.ty.overrides]] added for verifiers/envs/experimental/braintrust_tracing/** to suppress unresolved-import (braintrust is optional, not installed in CI)

Review & Testing Checklist

• Experimental files will drift from main: The five files under experimental/braintrust_tracing/ are full copies of their core counterparts with tracing added. Any future changes to the core environment.py, multiturn_env.py, tool_env.py, or stateful_tool_env.py must be manually mirrored to the experimental copies. Verify this maintenance burden is acceptable vs. a mixin/decorator approach.
• Import chain correctness: Each experimental class inherits from its experimental sibling (e.g., ToolEnv(MultiTurnEnv) where MultiTurnEnv is the local tracing variant). But decorators like @vf.stop, @vf.cleanup and types like vf.State, vf.Error still come from the original verifiers package. Verify these cross-references work correctly at runtime.
• call_tool public API change: call_tool(**kwargs) reads kwargs.get("state") for span parenting, and env_response() passes state=state. Any third-party subclass that overrides call_tool and calls super().call_tool(...) without forwarding state will lose tool span nesting. Verify for downstream environments (e.g., MiniBrowseEnv).
• End-to-end verification: Install with pip install -e '.[braintrust]', set BRAINTRUST_API_KEY, import StatefulToolEnv from the experimental path, and run an eval. Confirm nested spans (rollout → setup → turns → model/tool calls) appear in Braintrust with correct run-level tags.

Notes

• All pre-commit checks (ruff, ruff-format, ty) pass locally.
• This is a parallel/alternative to the Datadog telemetry in How hard is it to integrate non-LLM environment feedbacks into this framework? #9, targeting a different observability backend.
• Traces previously confirmed correct in Braintrust for grouped 5×3 evals.

---

Note

Medium Risk
Adds a large new experimental environment stack plus an optional braintrust dependency; while isolated from existing paths, it introduces new concurrency/contextvar-based tracing logic and substantial duplicated code that could drift from core behavior.

Overview
Adds an opt-in experimental Braintrust tracing implementation under verifiers/envs/experimental/braintrust_tracing/, providing drop-in Environment/MultiTurnEnv/ToolEnv/StatefulToolEnv variants that emit nested spans for rollout setup, per-turn execution, model requests (with token/duration metrics), tool calls, and scoring.

Introduces a self-contained braintrust_tracing.py layer with lazy logger init gated by BRAINTRUST_API_KEY, run-level tagging via ContextVar, best-effort serialization, and defensive no-op/error-swallowing behavior; the experimental environment.py wires these hooks into generate(), per-rollout/group execution, and request/tool paths.

Updates packaging to add a braintrust optional extra (verifiers[braintrust]), adds ty overrides to ignore optional imports in the experimental folder, and refreshes uv.lock to include braintrust and its transitive dependencies.

^{Reviewed by Cursor Bugbot for commit 5558e02. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Mutable default in ContextVar creates shared state risk

Low Severity

The _run_tags ContextVar uses default=[], which is a single mutable list object shared across all contexts that haven't explicitly called .set(). While current callers always copy the returned list before use, any future code that does _run_tags.get().append(...) would silently corrupt the shared default, affecting all contexts. The safe pattern is to use a sentinel or immutable default (e.g., default=()) and convert to a list where needed.

 

^{Reviewed by Cursor Bugbot for commit b0460f6. Configure here.}

[cursor]

Rollout span leaks when rollout or scoring raises

Medium Severity

In _run_rollout_state, the calls to self.rollout(), self.rubric.score_rollout(), and self.rubric.cleanup() are not wrapped in a try/finally that ensures _bt.rollout_completed() is called. If any of these raise an exception, the Braintrust rollout span is never closed, leading to leaked open spans. The same pattern applies in _run_group_states where _traced_rollout doesn't protect against exceptions from self.rollout().

 

^{Reviewed by Cursor Bugbot for commit b0460f6. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5558e02. Configure here.}

[cursor]

Group scoring path missing Braintrust scoring child spans

Low Severity

In _run_rollout_state, scoring is wrapped with scoring_started/scoring_completed spans, but _run_group_states performs group scoring without any corresponding Braintrust scoring spans. Since group scoring is the default path (used when independent_scoring=False), most evaluations will be missing the "scoring" child span in their traces, making the tracing hierarchy inconsistent.

Additional Locations (1)

• verifiers/envs/experimental/braintrust_tracing/environment.py#L739-L752

 

^{Reviewed by Cursor Bugbot for commit 5558e02. Configure here.}