Skip to content

feat(engine): env-var switch for async-first models experiment#280

Draft
eric-tramel wants to merge 6 commits intomainfrom
async/async-facade
Draft

feat(engine): env-var switch for async-first models experiment#280
eric-tramel wants to merge 6 commits intomainfrom
async/async-facade

Conversation

@eric-tramel
Copy link
Contributor

@eric-tramel eric-tramel commented Feb 2, 2026

Summary

Opt-in async execution path for the cell-by-cell generation pipeline, gated behind DATA_DESIGNER_ASYNC_ENGINE=1. Replaces thread-pool concurrency with native asyncio for LLM calls while keeping the sync path as the unchanged default.

What changed

  • Async ModelFacade methodsacompletion(), agenerate(), agenerate_text_embeddings() on ModelFacade, using LiteLLM Router's native async APIs. MCP tool calls offloaded via asyncio.to_thread().
  • acatch_llm_exceptions — async mirror of the existing sync decorator, sharing the same error-mapping logic.
  • AsyncConcurrentExecutor — drop-in async replacement for ConcurrentThreadExecutor. Uses TaskGroup + Semaphore for bounded concurrency, asyncio.Event for early shutdown. Runs on a persistent background event loop (same pattern as mcp/io.py) to avoid breaking libraries that maintain internal async state across calls.
  • ColumnWiseBuilder — branches on env var: _fan_out_with_async() vs existing _fan_out_with_threads().
  • agenerate() on generators — async path in ColumnGeneratorWithModelChatCompletion.
  • Benchmark — updated with async mock support for --mode compare.

Benchmark results

With mock LLMs (zero-latency stubs), sync and async produce identical output (SHA256 content hash match across all iterations). No measurable speedup is expected here — the async path's advantage comes from real network-bound LLM calls where await yields the event loop during I/O wait. With mocks there's no I/O to overlap.

Validated end-to-end with the pdf_qa recipe using real inference (Opus 4.5 + MCP tool calls).

How to test

# Benchmark (correctness)
uv run python scripts/benchmarks/benchmark_engine_v2.py --mode compare --iterations 3 --num-records 64

# Existing tests (no regressions)
uv run pytest packages/data-designer-engine/tests -x -q

# Real inference
DATA_DESIGNER_ASYNC_ENGINE=1 uv run python docs/assets/recipes/mcp_and_tooluse/pdf_qa.py --model-alias <your-model> --num-records 2

Test plan

  • 1086 engine tests pass
  • Benchmark content match: identical hash sync vs async
  • pdf_qa recipe completes with real inference, no event loop errors
  • Async path confirmed via log markers (one per LLM column)

@eric-tramel eric-tramel self-assigned this Feb 2, 2026
@eric-tramel eric-tramel added the enhancement New feature or request label Feb 2, 2026
eric-tramel and others added 3 commits February 2, 2026 14:49
Adds an opt-in async execution path (DATA_DESIGNER_ASYNC_ENGINE=1) for
the cell-by-cell generation pipeline. Replaces thread-pool concurrency
with native asyncio TaskGroup + Semaphore for bounded concurrent LLM
calls, while keeping the sync path as the default.

Key changes:
- ModelFacade: acompletion(), agenerate_text_embeddings(), agenerate()
- acatch_llm_exceptions decorator (async mirror of catch_llm_exceptions)
- AsyncConcurrentExecutor with persistent background event loop
- ColumnWiseBuilder branches on env var to fan out via async or threads
- Benchmark updated with async mock support

Co-Authored-By: Remi <noreply@anthropic.com>
Resolved conflicts:
- llm_completion.py: kept agenerate() async method + main's new
  _extract_reasoning_content(), TraceType handling, and
  extract_reasoning_content config. Updated agenerate() to match
  main's trace handling patterns.
- column_wise_builder.py: kept DATA_DESIGNER_ASYNC_ENGINE env var +
  adopted main's get_library_version() replacing importlib.metadata.

Co-Authored-By: Remi <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant