feat: annotation queue testset export#4261
Open
ashrafchowdury wants to merge 31 commits intomainfrom
Open
Conversation
Adds PRD, RFC, and README to docs/designs/testset-annotation-queue/. The design covers: - Per-scenario "Add to Testset" button in the annotate tab - "Done with queue" screen supporting both trace and testcase queues - All-annotations tab with row selection and upgraded commit modal - New AddToTestsetModal component with EntityPicker integration - Controller actions for addScenariosToTestset (trace + testcase paths) - Default target testset heuristics and last-used persistence https://claude.ai/code/session_01B2uQKidAr1KJ4CR9sroY2H
Key corrections: - Replace new AddToTestsetModal with EntityCommitModal + renderModeContent - Replace useState for selected testset with pendingTestsetSelectionAtom in annotationSessionController — survives re-renders and can be read imperatively by addScenariosToTestset without closure capture - selectedScenarioIdsAtom for row selection (also atom, not useState) - openAddToTestsetModal seeds pendingTestsetSelectionAtom from default - addScenariosToTestset reads target testset from atom, not payload - Add state atom summary table documenting all atom lifecycles https://claude.ai/code/session_01B2uQKidAr1KJ4CR9sroY2H
Inputs (agData.inputs) spread into N columns — one per input key. Outputs (agData.outputs) always map to a single "output" column regardless of value shape, matching extractOutputs() behaviour in trace/utils/selectors.ts which treats outputs as a leaf. Annotation values add one column per evaluator slug.
…n reset 1. Trace annotation resolution: use scenarioAnnotationsAtomFamily(scenarioId) not a raw query by traceId. The atom handles step-based resolution to avoid cross-queue bleed (as documented in the controller). 2. Scope label when scope="all": addToTestsetScenarioIds() is empty for the "all" case — read actual count from scenarioIds() instead of falling back to the "all" string. 3. Row selection reset: addScenariosToTestset clears selectedScenarioIdsAtom after a successful export so stale selections don't persist.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an "Add to Testset" feature: controller atoms/actions and types for modal/export jobs, background export implementation with row builders/remapping for traces/testcases, UI wiring and modal integration, annotation column child support, and related testset API adjustments. ChangesAdd to Testset Feature
sequenceDiagram Estimated Code Review Effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly Related PRs
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
docs/designs/testset-annotation-queue/PRD.md (1)
42-48: 💤 Low valueAdd blank lines around tables for markdown compliance.
The tables in this document are missing surrounding blank lines, which causes markdownlint warnings. This affects tables at lines 42, 89, 107, and 118.
📝 Example fix for the competitive reference table
## Competitive Reference ### LangSmith + | Capability | Supported | |------------|-----------| | Create or extend a testset from both test cases and traces | Yes | | Add a single scenario to a testset | Yes | | Add to a testset from the focused/annotate tab | Yes | | Select table rows and add them to a testset | Yes | + LangSmith supports all four capabilities; Agenta currently supports none of them in full.docs/designs/testset-annotation-queue/RFC.md (1)
18-57: 💤 Low valueAdd language specifiers to fenced code blocks.
Several code blocks in this RFC lack language specifiers, which affects syntax highlighting and readability. The architecture diagram at line 18 and examples at lines 244, 361, 366, 385, 391, 485, 500, 517 should specify a language (use
textorplaintextfor ASCII diagrams).📝 Example fix for ASCII diagram
-``` +```text User action (button click) │ ▼
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 22636d88-f5f8-408b-a088-51bb814f81e5
📒 Files selected for processing (11)
docs/designs/testset-annotation-queue/PRD.mddocs/designs/testset-annotation-queue/README.mddocs/designs/testset-annotation-queue/RFC.mdweb/packages/agenta-annotation-ui/src/components/AnnotationSession/FocusView.tsxweb/packages/agenta-annotation-ui/src/components/AnnotationSession/ScenarioListView.tsxweb/packages/agenta-annotation-ui/src/components/AnnotationSession/SessionNavigation.tsxweb/packages/agenta-annotation-ui/src/components/AnnotationSession/index.tsxweb/packages/agenta-entity-ui/src/modals/commit/components/EntityCommitContent.tsxweb/packages/agenta-entity-ui/src/selection/components/UnifiedEntityPicker/types.tsweb/packages/agenta-entity-ui/src/selection/components/UnifiedEntityPicker/variants/CascadingVariant.tsxweb/packages/agenta-entity-ui/src/selection/hooks/modes/useCascadingMode.ts
dosubot
Bot
added
size:XXL
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
web/packages/agenta-annotation-ui/src/components/AnnotationSession/ScenarioListView.tsx (1)
552-569:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winKeep the testcase fallback key separate from the metric path.
AnnotationOutputKeyCellpassesoutputKeyintouseAnnotationCellFallback(), and after Line 912 that value is often the full mapped path (attributes.ag.data.outputs.score, etc.). The fallback hook indexes the testcase container object by that same string, so grouped annotation cells now miss synced testcase values and render—until an annotation exists.Possible fix
const AnnotationOutputKeyCell = memo(function AnnotationOutputKeyCell({ scenarioId, def, outputKey, + fallbackOutputKey, fallbackDataKey, }: { scenarioId: string def: AnnotationColumnDef outputKey: string + fallbackOutputKey?: string fallbackDataKey?: string | null }) { const runId = useAtomValue(annotationSessionController.selectors.activeRunId()) ?? undefined const PopoverWrapper = useMetricPopoverWrapper() const {fallbackValue, isPending} = useAnnotationCellFallback( scenarioId, fallbackDataKey, - outputKey, + fallbackOutputKey ?? outputKey, )render: (_value: unknown, record: ScenarioTableRow) => ( <AnnotationOutputKeyCell scenarioId={record.scenarioId} def={outputColumn.annotationDef} outputKey={outputColumn.annotationDef.path ?? outputColumn.title} + fallbackOutputKey={outputColumn.title} fallbackDataKey={def.fallbackDataKey} /> ),Also applies to: 899-913
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: e691f741-e404-4527-b6c9-1dbc7e659952
📒 Files selected for processing (12)
web/packages/agenta-annotation-ui/src/components/AnnotationSession/ScenarioListView.tsxweb/packages/agenta-annotation/src/index.tsweb/packages/agenta-annotation/src/state/controllers/annotationSessionController.tsweb/packages/agenta-annotation/src/state/controllers/index.tsweb/packages/agenta-annotation/src/state/index.tsweb/packages/agenta-annotation/src/state/testsetSync.tsweb/packages/agenta-annotation/src/state/types.tsweb/packages/agenta-entities/src/testset/api/api.tsweb/packages/agenta-entities/src/testset/api/index.tsweb/packages/agenta-entities/src/testset/index.tsweb/packages/agenta-entity-ui/src/modals/commit/components/EntityCommitModal.tsxweb/packages/agenta-entity-ui/src/modals/commit/state.ts
💤 Files with no reviewable changes (2)
- web/packages/agenta-entity-ui/src/modals/commit/state.ts
- web/packages/agenta-entity-ui/src/modals/commit/components/EntityCommitModal.tsx
✅ Files skipped from review due to trivial changes (1)
- web/packages/agenta-annotation/src/state/index.ts
Contributor
Railway Preview Environment
|
ardaerzin
requested changes
May 6, 2026
…xt acceptance to provide entity link easily on the message for faster and better navigation
- refactor code - rendering json data in beautified mode by default
ardaerzin
approved these changes
May 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's new
Fixes
QA scope (Annotation-queue)