[delight] User Experience Analysis - 2026-05-21

[github-actions[bot]]

User Experience Analysis Report - 2026-05-21

Executive Summary

Today's analysis focused on:

• 2 documentation files (threat-detection.md, engines.md)
• 2 CLI commands (validate, logs)
• 2 workflow message configurations (smoke-project, refiner)
• 1 validation file (runtime_import_validation.go)

Overall Quality: Professional with targeted improvement opportunities

Key Finding: Documentation files demonstrate excellent technical accuracy and structure, but could benefit from more explicit enterprise context framing and practical decision-making guidance in introductory sections.

Quality Highlights ✅

Example 1: Threat Detection Documentation Structure

• File: docs/src/content/docs/reference/threat-detection.md
• What works well: Excellent progressive disclosure with clear security architecture diagram, comprehensive configuration options organized by complexity, and practical examples for each use case
• Quote/Reference: Lines 14-36 present a clear ASCII diagram showing the security architecture flow, followed by well-organized sections from basic to advanced configuration

Example 2: CLI Command Help Text Quality

• File: pkg/cli/logs_command.go
• What works well: Extensive practical examples (lines 52-100) covering all major use cases, well-organized into categories (basic usage, date filtering, content filtering, output options), professional tone throughout
• Strong pattern: Each flag has multiple contextual examples showing real-world usage patterns

Improvement Opportunities 💡

High Priority

🎯 Actionable Tasks

Here are 2 targeted improvement tasks, each affecting a single file:

Task 1: Add Enterprise Context to Engines Documentation

File to Modify: docs/src/content/docs/reference/engines.md

Current Experience

Lines 1-28 provide a comparison table of available engines with technical details, but lacks introductory context explaining when and why an enterprise team would choose one engine over another.

Quality Issue

Design Principle: Clarity and Precision

Enterprise decision-makers need strategic guidance before diving into technical configuration details. The current opening jumps directly into a feature comparison table without explaining the decision framework.

Teams evaluating gh-aw need to understand:

1. Total cost of ownership per engine
2. Compliance and data residency considerations
3. Integration with existing enterprise tooling
4. Team skill requirements

Proposed Improvement

Add a new "Choosing an Engine for Enterprise Use" section before the feature comparison table.

Before:

---
title: AI Engines (aka Coding Agents)
description: Complete guide to AI engines...
---

GitHub Agentic Workflows use [AI Engines]... to interpret and execute natural language instructions.

## Available Coding Agents

Set `engine:` in your workflow frontmatter...

After:

---
title: AI Engines (aka Coding Agents)
description: Complete guide to AI engines...
---

GitHub Agentic Workflows use [AI Engines]... to interpret and execute natural language instructions.

## Choosing an Engine for Enterprise Use

When selecting an engine for enterprise workflows, consider these factors:

**Integration and Authentication**: Copilot integrates natively with GitHub Enterprise authentication and respects repository permissions. Claude, Codex, and Gemini require separate API keys and may need approval for corporate procurement.

**Cost and Budget**: Each engine has different pricing models. Copilot costs are typically included in GitHub Enterprise licenses, while Claude, Codex, and Gemini bill per API request. Estimate costs based on your expected workflow frequency and complexity.

**Compliance and Data Residency**: For organizations with data residency requirements, verify where each engine processes data. GitHub Copilot supports regional endpoints (see `api-target` configuration), while third-party engines may have different data handling policies.

**Feature Requirements**: Review the feature comparison table below to ensure your chosen engine supports required capabilities like custom agents (`engine.agent`), turn limits (`max-turns`), or specific tool integrations.

**Recommendation for Getting Started**: Use Copilot (the default) for initial evaluation unless your organization already has established relationships with Anthropic, OpenAI, or Google for AI services.

## Available Coding Agents

Set `engine:` in your workflow frontmatter...

Why This Matters

• User Impact: Reduces time-to-decision for enterprise teams evaluating gh-aw by providing a clear decision framework before technical details
• Quality Factor: Clarity and Precision - helps users understand why before how
• Frequency: This is often the first documentation page enterprise decision-makers consult when evaluating engines

Success Criteria

• Changes made to docs/src/content/docs/reference/engines.md only
• Section added before "Available Coding Agents" heading
• Addresses cost, compliance, integration, and feature selection
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: docs/src/content/docs/reference/engines.md
• No changes to other files required
• Can be completed independently

---

Task 2: Improve Error Message Clarity in Runtime Import Validation

File to Modify: pkg/workflow/runtime_import_validation.go

Current Experience

Lines 127-129 generate a security error message when a file path escapes the .github folder:

if err != nil || relativePath == ".." || strings.HasPrefix(relativePath, ".."+string(filepath.Separator)) || filepath.IsAbs(relativePath) {
    validationErrors = append(validationErrors, fmt.Sprintf("%s: Security: Path must be within .github folder (resolves to: %s)", filePath, relativePath))
    continue
}

Quality Issue

Design Principle: Clarity and Precision

Error message "Security: Path must be within .github folder (resolves to: ../something)" is cryptic for users who may not understand path traversal security implications.

When users encounter this error during compilation, they need to understand:

1. Why this security restriction exists
2. What the actual vs. expected path resolution is
3. How to fix the import path

Proposed Improvement

Enhance error message with clearer explanation and actionable guidance:

Before:

if err != nil || relativePath == ".." || strings.HasPrefix(relativePath, ".."+string(filepath.Separator)) || filepath.IsAbs(relativePath) {
    validationErrors = append(validationErrors, fmt.Sprintf("%s: Security: Path must be within .github folder (resolves to: %s)", filePath, relativePath))
    continue
}

After:

if err != nil || relativePath == ".." || strings.HasPrefix(relativePath, ".."+string(filepath.Separator)) || filepath.IsAbs(relativePath) {
    validationErrors = append(validationErrors, fmt.Sprintf(
        "%s: Invalid path - runtime-import files must be within the .github/ folder for security reasons.\n"+
        "  Current path resolves to: %s\n"+
        "  Fix: Use a relative path from .github/ (e.g., 'workflows/shared/file.md' not '../file.md')",
        filePath, relativePath))
    continue
}

Why This Matters

• User Impact: Users can self-correct path errors immediately without needing to consult documentation or support
• Quality Factor: Clarity and Precision - explains the security rationale and provides specific fix guidance
• Frequency: This error occurs whenever users attempt to import files from outside .github/, which is a common mistake when organizing shared workflow content

Success Criteria

• Changes made to pkg/workflow/runtime_import_validation.go only
• Error message explains security rationale
• Error message shows actual path resolution
• Error message provides concrete fix example
• Quality rating improves from ⚠️ to ✅

Scope Constraint

• Single file only: pkg/workflow/runtime_import_validation.go
• No changes to other files required
• Can be completed independently

---

Medium Priority

Observation 3: Workflow Message Tone (Informational - No Task Created)

• Files: .github/workflows/smoke-project.md, .github/workflows/refiner.md
• Current State: Both workflows use professional, informative message templates with appropriate emoji usage
• Assessment: Messages appropriately balance technical precision with human-readable status updates. Emoji usage (🧪, 🔍, ✅, ❌) adds visual scanning value without being excessive
• No action needed: These workflows demonstrate good professional communication patterns

Files Reviewed

Documentation

• docs/src/content/docs/reference/threat-detection.md - Rating: ✅ Professional
• docs/src/content/docs/reference/engines.md - Rating: ⚠️ Needs minor work (missing enterprise context)

CLI Commands

• pkg/cli/validate_command.go - Rating: ✅ Professional
• pkg/cli/logs_command.go - Rating: ✅ Professional

Workflow Messages

• .github/workflows/smoke-project.md - Rating: ✅ Professional
• .github/workflows/refiner.md - Rating: ✅ Professional

Validation Code

• pkg/workflow/runtime_import_validation.go - Rating: ⚠️ Needs minor work (error message clarity)

Metrics

• Files Analyzed: 7
• Quality Distribution:
  • ✅ Professional: 5
  • ⚠️ Needs Minor Work: 2
  • ❌ Needs Significant Work: 0

Design Principles Applied

1. Clarity and Precision: Engines documentation needs decision-making context; error messages need actionable guidance
2. Professional Communication: All files maintain appropriate enterprise tone
3. Efficiency and Productivity: CLI help text provides extensive practical examples for quick reference
4. Trust and Reliability: Validation code provides clear security boundaries
5. Documentation Quality: Threat detection documentation uses excellent progressive disclosure patterns

References:

• §26237083809

📊 User experience analysis by Delight · ● 1.3M · ◷

• expires on May 24, 2026, 3:59 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Delight.

A newer discussion is available at Discussion #34467.