Release v0.1.4

.github/copilot-instructions.md

@@ -54,7 +54,9 @@ Only the following commands are in scope:
 
 - `agentops init`
 - `agentops eval run --config <run.yaml> [--output <dir>]`
+- `agentops eval compare --runs <ID1>,<ID2>[,ID3,...] [--output <dir>]`
 - `agentops report --in <results.json> [--out <report.md>]`
+- `agentops config cicd [--force] [--dir <path>]`
 
 Do not add new commands or flags unless explicitly discussed.
 
@@ -80,7 +82,7 @@ See `docs/how-it-works.md` for the full source-code map and architecture diagram
 - Keep CLI command handlers **thin** (`cli/app.py`) — only parse args and call `services/`
 - Place business logic in:
   - `core/` — config loading, Pydantic models, thresholds, report generation. **Must have zero Azure SDK imports and zero network calls.**
-  - `services/` — orchestration (runner), Foundry publishing, workspace init, report regen
+  - `services/` — orchestration (runner), comparison, CI/CD workflow generation, Foundry publishing, workspace init, report regen
   - `backends/` — execution backends (Foundry, subprocess). Each implements the `Backend` protocol from `base.py`.
 - Use `pathlib.Path` everywhere (no raw string paths)
 - No side effects at import time
@@ -130,6 +132,7 @@ The Foundry backend (`backends/foundry_backend.py`) is the largest and most comp
 - Auto-derive Azure OpenAI endpoint from the project endpoint via `_derive_openai_endpoint_from_project()` — users should not need to set `AZURE_OPENAI_ENDPOINT` manually.
 - Agent invocation supports both reference-based and threads-based API calls.
 - Evaluator names map from class names to builtins: `SimilarityEvaluator` → `builtin.similarity`.
+- Cloud evaluator routing uses frozensets: `_EVALUATORS_NEEDING_GROUND_TRUTH`, `_EVALUATORS_NEEDING_CONTEXT`, `_EVALUATORS_NEEDING_TOOL_CALLS`, `_EVALUATORS_NEEDING_TOOL_DEFS_ONLY`, `_EVALUATORS_NEEDING_OUTPUT_ITEMS`. NLP evaluators with required init params use `_NLP_DEFAULT_INIT_PARAMS`.
 
 ### Environment Variables
 
@@ -208,6 +211,10 @@ When cloud evaluation is used, a `cloud_evaluation.json` is also produced contai
   - Foundry backend helpers (`test_foundry_backend.py`)
   - Subprocess backend (`test_subprocess_backend.py`)
   - Initializer (`test_initializer.py`)
+  - CI/CD workflow generation (`test_cicd.py`)
+  - CLI command behavior (`test_cli_commands.py`)
+  - Eval comparison logic (`test_comparison.py`)
+  - OTLP telemetry instrumentation (`test_telemetry.py`)
 - Integration test for:
   - `agentops eval run` end-to-end using a fake subprocess backend (`test_eval_run_integration.py`)
 - Tests must assert correct **exit codes**
@@ -248,9 +255,18 @@ When generating or modifying code:
 - Azure SDK imports must be **lazy** (inside functions, not top-level)
 - Never hardcode Azure API versions — let the SDK handle versioning
 - Keep user-facing log output clean — no warning cascades or retry noise
-- When adding evaluator support, update both cloud (`_cloud_evaluator_data_mapping` + `_cloud_evaluator_needs_model`) and local paths
+- When adding evaluator support, add the builtin name to the correct frozenset in `foundry_backend.py` (`_EVALUATORS_NEEDING_GROUND_TRUTH`, `_EVALUATORS_NEEDING_CONTEXT`, `_EVALUATORS_NEEDING_TOOL_CALLS`, `_EVALUATORS_NEEDING_TOOL_DEFS_ONLY`, or `_EVALUATORS_NEEDING_OUTPUT_ITEMS`), update `_NLP_DEFAULT_INIT_PARAMS` if init params are required, and update both cloud (`_cloud_evaluator_data_mapping` + `_cloud_evaluator_needs_model`) and local paths
 - All new logic must have corresponding unit tests in `tests/unit/`
 - Always mock Azure SDK calls in tests — tests must run without credentials
 - The `core/` package must remain free of Azure imports and I/O
 - Follow the request flow: CLI → Services → Backends → Core (never skip layers)
 - If a change is user-visible, add an entry to `CHANGELOG.md` under `[Unreleased]` (Keep a Changelog format)
+
+### OTLP Telemetry
+
+- `utils/telemetry.py` provides optional OTLP trace emission for evaluation runs
+- Activated by `AGENTOPS_OTLP_ENDPOINT` env var — zero overhead when unset
+- All OpenTelemetry imports must be **lazy** (inside functions in `utils/telemetry.py`)
+- `opentelemetry-sdk` is an optional runtime dependency — not declared in `pyproject.toml`
+- Span schema: CICD semconv (`cicd.pipeline.*`) for pipeline structure, GenAI semconv (`gen_ai.*`) for agent calls, `agentops.eval.*` for evaluator scores
+- When adding new spans, follow the three-layer pattern in `telemetry.py`

.github/workflows/_build.yml

@@ -1,16 +1,19 @@
 # AgentOps Toolkit — Reusable Build Workflow
 #
 # Workflows:
-#   1. ci.yml          — Lint + test on every push/PR; publish dev builds to TestPyPI on develop
-#   2. _build.yml      — Reusable build (test + package), called by staging and release
-#   3. staging.yml     — Staging: release/* branch → TestPyPI → verify
-#   4. release.yml     — Production: v* tag → TestPyPI → verify → PyPI → GitHub Release
+#   1. ci.yml          — Lint + test on every push/PR; build VSIX validation
+#   2. _build.yml      — Reusable Python build (test + package), called by staging and release
+#   3. staging.yml     — Staging: release/* → TestPyPI + VSIX pre-release
+#   4. release.yml     — Production: v* tag → PyPI + VSIX stable + GitHub Release
 #   5. cut-release.yml — Manual dispatch: create release branch + PR from develop
 #
 # Called by staging.yml and release.yml via workflow_call.
-# Runs tests, builds the package (version via setuptools-scm), and uploads
+# Runs tests, builds the Python package (version via setuptools-scm), and uploads
 # the dist/ artifacts for downstream jobs.
 #
+# Note: VSIX packaging is handled directly in ci/staging/release workflows
+# (requires Node.js + @vscode/vsce), not in this Python-focused reusable build.
+#
 # Usage in caller workflows:
 #   jobs:
 #     build:

.github/workflows/ci.yml

@@ -3,8 +3,8 @@
 # Workflows:
 #   1. ci.yml          — Lint + test on every push/PR; publish dev builds to TestPyPI on develop
 #   2. _build.yml      — Reusable build (test + package), called by staging and release
-#   3. staging.yml     — Staging: release/* branch → TestPyPI → verify
-#   4. release.yml     — Production: v* tag → TestPyPI → verify → PyPI → GitHub Release
+#   3. staging.yml     — Staging: release/* branch → TestPyPI → verify; VSIX pre-release → Marketplace
+#   4. release.yml     — Production: v* tag → TestPyPI → verify → PyPI → GitHub Release; VSIX stable → Marketplace
 #   5. cut-release.yml — Manual dispatch: create release branch + PR from develop
 
 name: CI
@@ -186,3 +186,104 @@ jobs:
           echo "- TestPyPI: https://test.pypi.org/project/agentops-toolkit/${{ steps.version.outputs.version }}/" >> "$GITHUB_STEP_SUMMARY"
           echo "" >> "$GITHUB_STEP_SUMMARY"
           echo "Install: \`pip install agentops-toolkit==${{ steps.version.outputs.version }} --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/\`" >> "$GITHUB_STEP_SUMMARY"
+
+  # Validate that the VSIX extension packages correctly
+  build-vsix:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Full history for version derivation
+
+      - name: Sync VSIX version from git tag
+        run: |
+          # Use global tag sort (not git describe) to find the latest tag
+          # across ALL branches, not just reachable ones from HEAD.
+          LAST_TAG=$(git tag -l 'v*' --sort=-v:refname | head -1)
+          LAST_TAG=${LAST_TAG:-v0.0.0}
+          LAST_VERSION=${LAST_TAG#v}
+          IFS='.' read -r MAJOR MINOR PATCH <<< "$LAST_VERSION"
+          if git describe --tags --exact-match HEAD >/dev/null 2>&1; then
+            BASE_VERSION="$LAST_VERSION"
+          else
+            BASE_VERSION="$MAJOR.$MINOR.$((PATCH + 1))"
+          fi
+          jq --arg v "$BASE_VERSION" '.version = $v' \
+            plugins/agentops/package.json > plugins/agentops/package.json.tmp
+          mv plugins/agentops/package.json.tmp plugins/agentops/package.json
+          echo "VSIX version set to $BASE_VERSION (from tag $LAST_TAG)"
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Install vsce
+        run: npm install -g @vscode/vsce
+
+      - name: Package VSIX (dry run)
+        working-directory: plugins/agentops
+        run: vsce package -o agentops-skills.vsix
+
+      - name: Show VSIX info
+        run: |
+          ls -la plugins/agentops/*.vsix
+          echo "✅ VSIX packaging validated"
+
+      - name: Upload VSIX artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: vsix
+          path: plugins/agentops/*.vsix
+
+  # Publish VSIX pre-release to Marketplace on every push to develop (not PRs)
+  publish-vsix-dev:
+    if: github.event_name == 'push' && github.ref == 'refs/heads/develop'
+    needs: [lint, test, build-vsix]
+    runs-on: ubuntu-latest
+    environment: staging
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Full history for version derivation
+
+      - name: Sync VSIX version from git tag
+        run: |
+          # Use global tag sort (not git describe) to find the latest tag
+          # across ALL branches, not just reachable ones from HEAD.
+          LAST_TAG=$(git tag -l 'v*' --sort=-v:refname | head -1)
+          LAST_TAG=${LAST_TAG:-v0.0.0}
+          LAST_VERSION=${LAST_TAG#v}
+          IFS='.' read -r MAJOR MINOR PATCH <<< "$LAST_VERSION"
+          if git describe --tags --exact-match HEAD >/dev/null 2>&1; then
+            BASE_VERSION="$LAST_VERSION"
+          else
+            BASE_VERSION="$MAJOR.$MINOR.$((PATCH + 1))"
+          fi
+          jq --arg v "$BASE_VERSION" '.version = $v' \
+            plugins/agentops/package.json > plugins/agentops/package.json.tmp
+          mv plugins/agentops/package.json.tmp plugins/agentops/package.json
+          echo "VSIX version set to $BASE_VERSION (from tag $LAST_TAG)"
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: "20"
+
+      - name: Install vsce
+        run: npm install -g @vscode/vsce
+
+      - name: Package VSIX (pre-release)
+        working-directory: plugins/agentops
+        run: vsce package --pre-release -o agentops-skills.vsix
+
+      - name: Publish pre-release to VS Code Marketplace
+        continue-on-error: true  # Tolerate "already exists" for dev builds
+        working-directory: plugins/agentops
+        run: vsce publish --pre-release --packagePath agentops-skills.vsix -p "${{ secrets.VSCE_PAT }}"
+
+      - name: Summary
+        run: |
+          echo "## ✅ VSIX pre-release published to Marketplace" >> "$GITHUB_STEP_SUMMARY"
+          echo "" >> "$GITHUB_STEP_SUMMARY"
+          echo "Extension: [AgentOps Toolkit](https://marketplace.visualstudio.com/items?itemName=AgentOpsToolkit.agentops-toolkit)" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/cut-release.yml

@@ -1,14 +1,15 @@
 # AgentOps Toolkit — Cut Release
 #
 # Workflows:
-#   1. ci.yml          — Lint + test on every push/PR
+#   1. ci.yml          — Lint + test on every push/PR; VSIX build validation
 #   2. _build.yml      — Reusable build (test + package), called by staging and release
-#   3. staging.yml     — Staging: release/* branch → TestPyPI → verify
-#   4. release.yml     — Production: v* tag → TestPyPI → verify → PyPI → GitHub Release
+#   3. staging.yml     — Staging: release/* → TestPyPI → verify; VSIX pre-release → Marketplace
+#   4. release.yml     — Production: v* tag → TestPyPI → verify → PyPI → GH Release; VSIX stable → Marketplace
 #   5. cut-release.yml — Manual dispatch: create release branch + PR from develop
 #
 # One-click release branch creation. Triggered manually from the Actions tab.
-# Creates a release branch from develop, updates CHANGELOG.md, and opens a PR to main.
+# Creates a release branch from develop, updates CHANGELOG.md, syncs the
+# VS Code extension version in package.json, and opens a PR to main.
 # The branch push then triggers staging.yml automatically.
 #
 # Usage:
@@ -72,14 +73,21 @@ jobs:
           # Replace [Unreleased] with versioned section, add fresh Unreleased above
           sed -i "s/## \[Unreleased\]/## [Unreleased]\n\n## [${{ env.version }}] - $DATE/" CHANGELOG.md
 
+      - name: Sync VS Code extension version
+        run: |
+          jq --arg v "${{ env.version }}" '.version = $v' \
+            plugins/agentops/package.json > plugins/agentops/package.json.tmp
+          mv plugins/agentops/package.json.tmp plugins/agentops/package.json
+          echo "VSIX version set to ${{ env.version }}"
+
       - name: Configure git
         run: |
           git config user.name "github-actions[bot]"
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
 
       - name: Commit and push
         run: |
-          git add CHANGELOG.md
+          git add CHANGELOG.md plugins/agentops/package.json
           git commit -m "chore: prepare release ${{ env.version }}"
           git push origin "release/v${{ env.version }}"
 
@@ -98,22 +106,24 @@ jobs:
           ### What happened
           - Branch \`release/v${{ env.version }}\` created from \`develop\`
           - \`CHANGELOG.md\` updated: \`[Unreleased]\` → \`[${{ env.version }}]\`
-          - Staging pipeline triggered automatically (build → TestPyPI → verify)
+          - \`plugins/agentops/package.json\` version synced to \`${{ env.version }}\`
+          - Staging pipeline triggered automatically (build → TestPyPI + VSIX pre-release → verify)
 
           ### Next steps
           1. Wait for the **Staging** pipeline to pass
           2. Review and approve this PR
           3. Merge to \`main\`
           4. Tag and push: \`git tag v${{ env.version }} && git push origin v${{ env.version }}\`
-          5. Approve the PyPI publish in the **Release** workflow
+          5. Approve the PyPI publish and VSIX stable publish in the **Release** workflow
           6. Sync develop: \`git checkout develop && git merge main && git push origin develop\`
 
           ### Checklist
-          - [ ] Staging pipeline passes (build + TestPyPI + verify)
+          - [ ] Staging pipeline passes (build + TestPyPI + VSIX pre-release + verify)
           - [ ] CHANGELOG entries reviewed
           - [ ] PR approved and merged to main
           - [ ] Tag \`v${{ env.version }}\` pushed
           - [ ] PyPI publish approved
+          - [ ] VSIX stable publish approved
           - [ ] develop synced from main"
 
       - name: Summary
@@ -122,6 +132,7 @@ jobs:
           echo "" >> "$GITHUB_STEP_SUMMARY"
           echo "- Branch: \`release/v${{ env.version }}\`" >> "$GITHUB_STEP_SUMMARY"
           echo "- CHANGELOG updated with version **${{ env.version }}**" >> "$GITHUB_STEP_SUMMARY"
+          echo "- VS Code extension version synced to **${{ env.version }}**" >> "$GITHUB_STEP_SUMMARY"
           echo "- PR opened: \`release/v${{ env.version }}\` → \`main\`" >> "$GITHUB_STEP_SUMMARY"
           echo "- Staging pipeline triggered automatically" >> "$GITHUB_STEP_SUMMARY"
           echo "" >> "$GITHUB_STEP_SUMMARY"