Skip to content

Feat/kiro cli executor with tool usage#330

Open
perezjoseph wants to merge 6 commits into
KeygraphHQ:mainfrom
perezjoseph:feat/kiro-cli-executor-with-tool-usage
Open

Feat/kiro cli executor with tool usage#330
perezjoseph wants to merge 6 commits into
KeygraphHQ:mainfrom
perezjoseph:feat/kiro-cli-executor-with-tool-usage

Conversation

@perezjoseph
Copy link
Copy Markdown

feat: Add Kiro CLI executor backend with tool usage logging

Summary

Adds a new executor backend (kiro-cli) that runs Shannon agents via the Kiro CLI in headless mode, as an alternative to the Claude Agent SDK. Includes per-invocation tool usage tracking exposed through a Temporal workflow query.

Changes

Kiro CLI Executor (apps/worker/src/ai/kiro-cli-executor.ts)

  • Full executor implementation: agent JSON generation, subprocess spawning, exit code mapping, error classification
  • Generates .kiro/agents/<name>.json with model, prompt, MCP servers, and hooks
  • Heartbeat signaling to prevent Temporal activity timeouts during long-running agents
  • Credits extraction from kiro-cli stderr output
  • Structured output support via prompt augmentation (queue-writing instructions for vuln agents)
  • Tool usage logging via pre/post hook scripts that write JSONL

Tool Usage Tracking

  • Hook scripts capture every tool invocation (pre + post events) to ~/.kiro/agents/tool-usage.jsonl
  • readAndLogToolUsage() parses JSONL and returns both summary and per-invocation records
  • Workflow query getToolUsage accepts optional detailed boolean arg:
    • false (default): returns Record<string, ToolUsageSummaryMetrics> (counts, failures)
    • true: returns Record<string, DetailedToolUsageMetrics> (includes invocations[] with tool name, timestamp, success)
  • Cumulative tool invocations/failures exposed as Temporal search attributes

CLI Credential Validation

  • validateCredentials() now recognizes SHANNON_EXECUTOR_BACKEND=kiro-cli and only requires KIRO_API_KEY
  • buildEnvFlags() excludes conflicting Bedrock/Vertex env vars when kiro-cli backend is active
  • Worker preflight (preflight.ts) has matching kiro-cli bypass

Dockerfile

  • Installs kiro-cli binary in builder stage via official installer
  • Copies binary to runtime stage

Infrastructure

  • Agent JSON supports MCP server configuration, custom hooks, and model tier resolution
  • Global agents directory (~/.kiro/agents/) used for writable hook scripts inside Docker (repo mounted :ro)
  • Playwright skill installation for browser-based testing agents

Configuration

# .env
SHANNON_EXECUTOR_BACKEND=kiro-cli
KIRO_API_KEY=ksk_your_key_here

Querying Tool Usage (Temporal)

# Summary (default)
temporal workflow query \
  --workflow-id <id> \
  --name getToolUsage \
  --address localhost:7233

# Detailed (per-invocation records)
temporal workflow query \
  --workflow-id <id> \
  --name getToolUsage \
  --input 'true' \
  --address localhost:7233 \
  --output json

Testing

  • Full 5-phase pipeline scan against a live target (10 agents, 2394 tool invocations tracked)
  • Verified getToolUsage query returns correct data in both summary and detailed modes
  • Build passes (pnpm run build, pnpm run check)

jpilier added 6 commits April 23, 2026 16:03
- Add Claude SDK and Kiro CLI executor implementations with comprehensive test coverage
- Create executor interface abstraction for pluggable AI backend support
- Implement container and preflight service tests for reliability validation
- Add vitest configuration for worker package testing
- Extend environment variable forwarding for AWS credentials and Kiro API configuration
- Add SHANNON_EXECUTOR_BACKEND mode detection supporting kiro-cli backend
- Create .kiro/agents directory structure for agent definitions mount point
- Add kiro-cli binary installation to Docker build with Go security tool configuration
- Mount kiro-agents workspace overlay for writable agent definitions in containers
- Update .gitignore to exclude .kiro and .vscode directories
- Add licenses.txt documentation file
- Update dependencies in pnpm-lock.yaml
…obal agent directory

- Update agent JSON prompt paths to use relative file:// URIs (file://./ prefix)
- Write agents to global ~/.kiro/agents/ directory for reliable kiro-cli resolution
- Add best-effort fallback writes to project-level .kiro/agents/ for host-side visibility
- Change verify-queue-file output extension from .js to .mjs
- Remove unused execSync import from kiro-cli-executor
- Add cpSync import for potential file operations
- Update generateAgentJson to handle both global and project directories with retry logic
- Update generateQueueValidationHooks to write to global directory with project fallback
- Add detailed comments explaining Docker bind-mount strategy and agent resolution
…output

- Update CREDITS_PATTERN regex to match both "Credits used:" and "▸ Credits:" formats
- Rename extractCreditsUsed parameter from stdout to text for clarity
- Check stderr first for credits output, then fall back to stdout
- Enhance JSDoc to document kiro-cli's stderr credit format and accepted patterns
- Handle kiro-cli's new credit output format in stderr with bullet point prefix
- Add optional onHeartbeat callback to ExecutorOptions interface for orchestrator liveness signaling
- Implement heartbeat loop in spawnKiroCli that emits elapsed time and process ID every 30 seconds
- Pass onHeartbeat callback through agent execution pipeline from Temporal activities to CLI executor
- Clean up heartbeat timer on process completion to prevent resource leaks
- Prevents Temporal heartbeat starvation when multiple kiro-cli subprocesses saturate the event loop with I/O callbacks
# Conflicts:
#	jpilier/week1/shannon/Dockerfile
#	jpilier/week1/shannon/apps/cli/src/env.ts
#	jpilier/week1/shannon/apps/worker/src/services/preflight.ts
#	jpilier/week1/shannon/apps/worker/src/temporal/activities.ts
#	jpilier/week1/shannon/pnpm-lock.yaml
- Install kiro-cli binary in Docker image for headless mode execution
- Add SHANNON_EXECUTOR_BACKEND and KIRO_API_KEY environment variables
- Implement credential validation for kiro-cli backend mode
- Exclude conflicting provider variables when using kiro-cli backend
- Add ToolUsageSummary type and tool invocation tracking to ClaudePromptResult
- Implement KiroCliExecutor with tool usage logging via pre/post-tool hooks
- Add mergeHooks utility to combine multiple hook configurations
- Implement readAndLogToolUsage to parse and aggregate tool execution metrics
- Add generateToolUsageLoggerScript to create hook scripts for tool tracking
- Update metrics tracking to record tool usage events with timestamps and duration
- Extend audit logging to capture tool invocation details and performance data
- Add comprehensive test coverage for hook merging and tool usage extraction
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants