test(ce-dispatch): add skill-creator-style eval pack + battle-test results

plugins/compound-engineering/evals/ce-dispatch-workspace/.gitignore

@@ -0,0 +1,16 @@
+# Workspace for skill-creator-style eval runs of ce-dispatch.
+#
+# Raw transcripts and tool-call dumps can be large and noisy.
+# Keep grading.json, benchmark.json, eval_metadata.json, and
+# the prompt/output skeleton checked in as battle-test evidence;
+# exclude the heavy raw transcript dumps.
+
+# Heavy raw transcript text (regenerable from the runner)
+**/transcript-raw.md
+**/transcript-raw.json
+
+# Streaming chunks captured during runs
+**/*.stream.jsonl
+
+# Large-file outputs that exceed 1 MB
+**/outputs/*.bin

plugins/compound-engineering/evals/ce-dispatch-workspace/REPORT.md

@@ -0,0 +1,101 @@
+# Battle-test report: `ce-dispatch` (single-unit sync MVP)
+
+**Skill under test:** `plugins/compound-engineering/skills/ce-dispatch/` (rewrite from PR #4)
+**Date:** 2026-05-04
+**Framework:** Anthropic skill-creator eval protocol (https://github.com/anthropics/skills/tree/main/skills/skill-creator)
+**Executor model:** `anthropic/claude-opus-4.7` (resolved to `claude-4.7-opus-20260416`, served via Bedrock through OpenRouter)
+**Grader model:** `anthropic/claude-opus-4.7`
+**Runner:** `evals/scripts/run_eval_pack.py` (Path-A: direct OpenRouter; mimics skill-creator's eval protocol — system+user prompt with skill loaded vs. baseline, JSON-graded against quantitative expectations, aggregated to `benchmark.json`)
+
+## TL;DR
+
+The skill provides a large, discriminating lift over the no-skill baseline on every test prompt. After one round of eval-design refinement, **with-skill passes 24/24 expectations across 4 prompts**. The remaining gap to baseline is `+44 pp` (95% → 51% iteration-1, then 100% on the refined eval-2 in iteration-2).
+
+| Configuration | Pass rate (24 expectations across 4 evals) |
+|---|---|
+| `with_skill` | **100% (24/24)** after iteration-2 assertion fix; **95% (23/24)** in iteration-1 |
+| `without_skill` baseline | 51% iteration-1 / 56% with iteration-2 fix |
+| **delta** | **+44 to +49 pp** |
+
+## Evals run
+
+Four prompts cover the four meaningful skill surfaces of the MVP rewrite:
+
+| ID | Name | Surface tested | Iter-1 result |
+|---|---|---|---|
+| 1 | `happy-path-single-unit-dispatch` | Phase 0–3: orientation + agent-identity + comment-protocol + ce-plugin block + metadata footer + single `gh issue create` | **9/9 (100%)** vs. 4/9 baseline |
+| 2 | `phase-4-respond-review-pr` | Phase 4 four-option menu + PR-review routing | 4/5 (80%) iter-1 → **5/5 (100%) iter-2** vs. 3/5 → 4/5 baseline |
+| 3 | `phase-4-respond-reply-to-agent-comment` | Phase 4 + `[orchestrator -> agent] <ts>` comment-protocol prefix on reply | **5/5 (100%)** vs. 3/5 baseline |
+| 4 | `phase-4-respond-mark-unit-complete` | Phase 4 + `gh issue close` + worktree archival prompt + PR-state verification gate | **5/5 (100%)** vs. 2/5 baseline |
+
+Eval pack lives at `plugins/compound-engineering/skills/ce-dispatch/evals/evals.json` (per Anthropic's expected layout, co-located with the skill).
+
+## Findings
+
+### Skill-level
+
+No skill bugs surfaced. The skill correctly:
+
+- Renders all six required prompt-template sections (`<orientation>`, `<agent-identity>`, `<comment-protocol>`, `<ce-plugin>`, `<output-contract>`, metadata footer) with the exact shapes the contract test enforces.
+- Scopes content from the correct unit when picking U2 of a multi-unit plan (does not bleed in U1 or U3 content).
+- Surfaces exactly four Phase 4 options (no six-option monitor menu, no dependency graph, no auto-review).
+- Routes review work through `/ce-code-review` via the platform's skill-invocation primitive rather than inlining a fresh review prompt.
+- Uses `gh pr list --state all` consistently, preventing the "merged PRs are invisible" footgun.
+- Uses the `[orchestrator -> <agent-name>] <ISO 8601 UTC>` prefix shape for replies, matching the comment-protocol the dispatched prompt expects.
+- Verifies PR is in `MERGED` state (via `gh pr view --json state`) before closing the issue, instead of trusting the user's word.
+- Tells the user to archive the Conductor workspace manually rather than attempting to do it itself.
+
+### Eval-pack-level
+
+Round 1 surfaced one eval design issue (caught by the grader's `eval_feedback`, exactly as Anthropic's process intends):
+
+- **Eval 2, expectation 2** prescribed `gh pr diff <number>` even when the agent delegated review to `/ce-code-review`. The agent's behavior (delegating the diff fetch to the sub-skill) was correct; the assertion was over-specified.
+- **Fix applied in iteration-2:** loosened to "if delegating to `/ce-code-review`, direct `gh pr diff` is optional; if inspecting itself, it remains required."
+- **Iteration-2 result on the refined assertion:** 5/5 with-skill, 4/5 baseline. The grader's `eval_feedback` confirms: *"Expectations are clear and well-targeted at the common failure modes; the output cleanly satisfied all of them."*
+
+### What this method does and does not test
+
+**Tests (well, with strong signal):**
+- Skill prose correctness — does the agent produce the right structure, in the right order, with the right routing decisions, when the skill is loaded.
+- Discrimination — does loading the skill genuinely change behavior compared to a vanilla agent. Yes: +44 to +49 pp across the pack.
+- Eval design — the grader's `eval_feedback` flags assertions that pass for the wrong reasons.
+
+**Does not test:**
+- Real tool use. The runner is single-shot Chat Completions; the agent describes commands rather than executing them. The contract test (`tests/skills/ce-dispatch-contract.test.ts`, 63 cases) covers the actual loader/template-render path.
+- Multi-turn conversation across the comment protocol (asking for clarification → STOPping → resuming on reply). Would require multiple linked LLM calls; out of scope for this round.
+- Real `gh issue create`/`gh issue close`/`gh pr view` execution. The agent surfaces commands; verifying they actually work end-to-end requires live GitHub credentials and a real Conductor workspace, which is the user's manual end-to-end test plan in PR #4.
+
+## Cost
+
+| Phase | Tokens | Cost (USD) |
+|---|---|---|
+| Iteration 1 (8 executor + 8 grader runs) | ~135 k | ~$1.20 |
+| Iteration 2 (2 executor + 2 grader runs, eval-2 only) | ~30 k | ~$0.30 |
+| **Total** | **~165 k** | **~$1.50** |
+
+Well under my pre-flight estimate of $7–20.
+
+## How to reproduce
+
+```bash
+# From repo root, on branch mvp/ce-dispatch-evals:
+export OPENROUTER_API_KEY=<your-key>
+
+# Run the full pack (iteration-1):
+python3 plugins/compound-engineering/skills/ce-dispatch/evals/scripts/run_eval_pack.py
+
+# Run a single eval, e.g. after iterating evals.json:
+python3 plugins/compound-engineering/skills/ce-dispatch/evals/scripts/run_eval_pack.py --iteration 2 --eval-id 2
+
+# Substitute a different model:
+python3 plugins/compound-engineering/skills/ce-dispatch/evals/scripts/run_eval_pack.py --executor-model anthropic/claude-sonnet-4.5
+
+# Dry-run (no API calls; renders prompts only):
+python3 plugins/compound-engineering/skills/ce-dispatch/evals/scripts/run_eval_pack.py --dry-run
+```
+
+Outputs land at `plugins/compound-engineering/evals/ce-dispatch-workspace/iteration-<N>/`. The runner produces, per-run, the `eval_metadata.json`, `outputs/output.md`, `outputs/metrics.json`, `transcript.md`, `transcript-raw.json`, `timing.json`, and `grading.json` files specified by Anthropic's `references/schemas.md`. Per-iteration aggregates land at `iteration-<N>/benchmark.json` and `benchmark.md`.
+
+## Conclusion
+
+**PR #4 is ready for review.** The MVP rewrite produces structurally correct, discriminating output across all four meaningful skill surfaces, with no signal that further skill changes are needed. The remaining validation step is the user's manual end-to-end test in a real Conductor workspace, which only the user can drive.

plugins/compound-engineering/evals/ce-dispatch-workspace/iteration-1/1-happy-path-single-unit-dispatch/eval_metadata.json

@@ -0,0 +1,20 @@
+{
+  "eval_id": 1,
+  "eval_name": "happy-path-single-unit-dispatch",
+  "prompt": "I have a multi-unit plan at evals/files/sample-multi-unit-plan.md and I want to dispatch ONLY unit U2 to a Conductor workspace. The workspace I just created is at /Users/ryan/conductor/workspaces/api-gateway/jackson. Render the GitHub issue body you would create (do not actually create the issue), including all template sections, and tell me the final `gh issue create` command you would run. Use the dispatch defaults: branch prefix `dispatch/`, base branch `main`, labels `ce-dispatch`.",
+  "expected_output": "A complete dispatch issue body (rendered from the dispatch-prompt-template), populated for unit U2 of the rate-limit plan. Includes <orientation>, <agent-identity>, <task>, <files>, <patterns>, <approach>, <constraints>, <testing>, <verify>, <output-contract>, <comment-protocol>, <ce-plugin>, plus the metadata footer with `unit_id: U2`, `agent_name: jackson`, `worktree_path: /Users/ryan/conductor/workspaces/api-gateway/jackson`. Plus the literal `gh issue create` command line.",
+  "files": [
+    "evals/files/sample-multi-unit-plan.md"
+  ],
+  "expectations": [
+    "The output renders a complete <orientation> section listing repo-relative orientation files (README/AGENTS.md/plan path/architecture doc/pattern files) \u2014 not inlined content, just paths.",
+    "The output renders an <agent-identity> section with `agent-name: jackson` (the dirname of the worktree path) and `worktree-path: /Users/ryan/conductor/workspaces/api-gateway/jackson`.",
+    "The output renders a <comment-protocol> section that includes the literal prefix shape `[<agent-name> -> orchestrator]` and an explicit STOP-and-wait directive after asking a clarification.",
+    "The output renders a <ce-plugin> block whose body is a numbered nine-step compound-engineering loop in this order: read orientation -> /ce-work -> implement -> /ce-code-review -> /ce-compound (optional) -> /ce-commit-push-pr -> comment with PR URL -> stop and wait -> /ce-resolve-pr-feedback on ping.",
+    "The output's <task>, <files>, <patterns>, <approach>, <constraints>, <testing>, and <verify> sections are populated from U2 of the supplied plan (token-bucket middleware on /api/v1/messages), NOT from U1 (token-bucket primitive) or U3 (per-tenant override).",
+    "The metadata footer (HTML comment) uses `unit_id: U2` (singular), NOT `unit_ids:` (plural), and includes `agent_name: jackson`, `worktree_path: /Users/ryan/conductor/workspaces/api-gateway/jackson`, and either `dependencies: U1` or no `dependencies:` line at all (the MVP drops the field but listing this unit's stated dependency is acceptable in prose).",
+    "The output proposes exactly ONE `gh issue create` invocation, not multiple. The command must include `--label ce-dispatch` and a body file or heredoc carrying the rendered prompt \u2014 not a partial body or a JSON dump.",
+    "The output does NOT contain a dependency-graph rendering, a parallel-safety check, a six-option monitor menu, or any reference to `dispatch_mode` / `dispatch_auto_review` (these were removed in the single-unit sync MVP).",
+    "The output does NOT actually invoke `gh issue create` (since the user asked for a dry-run rendering); the agent surfaces the command but defers execution."
+  ]
+}

plugins/compound-engineering/evals/ce-dispatch-workspace/iteration-1/1-happy-path-single-unit-dispatch/with_skill/grading.json

@@ -0,0 +1,77 @@
+{
+  "expectations": [
+    {
+      "text": "The output renders a complete <orientation> section listing repo-relative orientation files (README/AGENTS.md/plan path/architecture doc/pattern files) \u2014 not inlined content, just paths.",
+      "passed": true,
+      "evidence": "<orientation> section lists README.md, AGENTS.md, CLAUDE.md, docs/architecture.md, services/api-gateway/src/middleware/auth.ts, token_bucket.ts, etc. as paths."
+    },
+    {
+      "text": "The output renders an <agent-identity> section with `agent-name: jackson` (the dirname of the worktree path) and `worktree-path: /Users/ryan/conductor/workspaces/api-gateway/jackson`.",
+      "passed": true,
+      "evidence": "<agent-identity>\\n- agent-name: `jackson`\\n- worktree-path: `/Users/ryan/conductor/workspaces/api-gateway/jackson`"
+    },
+    {
+      "text": "The output renders a <comment-protocol> section that includes the literal prefix shape `[<agent-name> -> orchestrator]` and an explicit STOP-and-wait directive after asking a clarification.",
+      "passed": true,
+      "evidence": "`**[jackson -> orchestrator] <ISO 8601 UTC timestamp>**` and 'STOP. Do not proceed past the open question. Do not start related work. Wait for an...'"
+    },
+    {
+      "text": "The output renders a <ce-plugin> block whose body is a numbered nine-step compound-engineering loop in this order: read orientation -> /ce-work -> implement -> /ce-code-review -> /ce-compound (optional) -> /ce-commit-push-pr -> comment with PR URL -> stop and wait -> /ce-resolve-pr-feedback on ping.",
+      "passed": true,
+      "evidence": "Nine numbered steps: 1. Read orientation, 2. /ce-work, 3. Implement, 4. /ce-code-review, 5. /ce-compound, 6. /ce-commit-push-pr, 7. Append comment with PR URL, 8. Stop and wait, 9. On ping /ce-resolve-pr-feedback."
+    },
+    {
+      "text": "The output's <task>, <files>, <patterns>, <approach>, <constraints>, <testing>, and <verify> sections are populated from U2 of the supplied plan (token-bucket middleware on /api/v1/messages), NOT from U1 (token-bucket primitive) or U3 (per-tenant override).",
+      "passed": true,
+      "evidence": "<task>: 'Add a `rateLimitMiddleware` that calls `TokenBucket.consume(1)` keyed on the JWT subject claim...'; constraints warn against starting U3's per-tenant override."
+    },
+    {
+      "text": "The metadata footer (HTML comment) uses `unit_id: U2` (singular), NOT `unit_ids:` (plural), and includes `agent_name: jackson`, `worktree_path: /Users/ryan/conductor/workspaces/api-gateway/jackson`, and either `dependencies: U1` or no `dependencies:` line at all (the MVP drops the field but listing this unit's stated dependency is acceptable in prose).",
+      "passed": true,
+      "evidence": "Metadata: 'unit_id: U2\\nagent_name: jackson\\nworktree_path: /Users/ryan/conductor/workspaces/api-gateway/jackson' with no dependencies line."
+    },
+    {
+      "text": "The output proposes exactly ONE `gh issue create` invocation, not multiple. The command must include `--label ce-dispatch` and a body file or heredoc carrying the rendered prompt \u2014 not a partial body or a JSON dump.",
+      "passed": true,
+      "evidence": "Single `gh issue create --title ... --body-file \"$SCRATCH/issue-body.md\" --label ce-dispatch`"
+    },
+    {
+      "text": "The output does NOT contain a dependency-graph rendering, a parallel-safety check, a six-option monitor menu, or any reference to `dispatch_mode` / `dispatch_auto_review` (these were removed in the single-unit sync MVP).",
+      "passed": true,
+      "evidence": "No mention of dependency graph, parallel-safety, six-option monitor menu, or dispatch_mode/dispatch_auto_review fields."
+    },
+    {
+      "text": "The output does NOT actually invoke `gh issue create` (since the user asked for a dry-run rendering); the agent surfaces the command but defers execution.",
+      "passed": true,
+      "evidence": "'The `gh issue create` command I would run' and 'A few things I'd flag to you before actually running the command' \u2014 command shown, not executed."
+    }
+  ],
+  "summary": {
+    "passed": 9,
+    "failed": 0,
+    "total": 9,
+    "pass_rate": 1.0
+  },
+  "eval_feedback": {
+    "suggestions": [
+      "Expectation [5] is slightly ambiguous on whether a `dependencies:` line is forbidden or merely optional \u2014 the phrasing 'either `dependencies: U1` or no `dependencies:` line at all' could be tightened."
+    ],
+    "overall": "Eval is well-scoped and the expectations map cleanly to observable artifacts in the rendered output."
+  },
+  "execution_metrics": {
+    "tool_calls": {},
+    "total_tool_calls": 0,
+    "total_steps": 1,
+    "files_created": [
+      "outputs/output.md"
+    ],
+    "errors_encountered": 0,
+    "output_chars": 14609,
+    "transcript_chars": 39091
+  },
+  "timing": {
+    "executor_duration_seconds": 69.05,
+    "grader_duration_seconds": 20.02,
+    "total_duration_seconds": 89.07
+  }
+}

plugins/compound-engineering/evals/ce-dispatch-workspace/iteration-1/1-happy-path-single-unit-dispatch/with_skill/outputs/metrics.json

@@ -0,0 +1,11 @@
+{
+  "tool_calls": {},
+  "total_tool_calls": 0,
+  "total_steps": 1,
+  "files_created": [
+    "outputs/output.md"
+  ],
+  "errors_encountered": 0,
+  "output_chars": 14609,
+  "transcript_chars": 39091
+}