Skip to content

feat: send concurrency metrics to LLM for async optimization#2146

Draft
KRRT7 wants to merge 1 commit intomainfrom
cf-async-concurrency-context-v2
Draft

feat: send concurrency metrics to LLM for async optimization#2146
KRRT7 wants to merge 1 commit intomainfrom
cf-async-concurrency-context-v2

Conversation

@KRRT7
Copy link
Copy Markdown
Collaborator

@KRRT7 KRRT7 commented May 8, 2026

Summary

  • For async functions, runs the concurrency benchmark before the optimization request so the LLM receives runtime metrics proving the function blocks the event loop
  • Sends concurrency_metrics (ratio, sequential/concurrent time) in the /optimize API payload
  • Caches the benchmark result to avoid running it twice (reused during baseline establishment)
  • Sync function flow unchanged (parallel test-gen + optimization)

Motivation

The e2e-python (async-optimization) test is flaky because the LLM only receives is_async: true — a bare boolean with no context about what to optimize. With a concurrency_ratio ≈ 1.0, the LLM gets concrete evidence that the function doesn't parallelize, steering it toward the correct fix (e.g. time.sleepasyncio.sleep).

Backend dependency

The backend needs to consume the new concurrency_metrics field from the payload and include it in the LLM prompt. Without that change, this field is safely ignored.

Test plan

  • Verify CI prek passes (ruff + mypy on changed lines)
  • Unit tests pass (async + optimizer suites)
  • Backend consumes concurrency_metrics in prompt construction
  • Re-run async e2e test to confirm improved reliability

For async functions, run the concurrency benchmark before submitting
the optimization request so the LLM receives runtime proof that the
function blocks (concurrency_ratio ≈ 1.0). This steers the model
toward correct async optimizations (e.g. time.sleep → asyncio.sleep).

Sync functions keep the existing parallel test-gen + optimization flow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant