📚 Documentation Noob Test Report - February 10, 2026

[github-actions]

Test Date: 2026-02-10
Pages Analyzed: 8 key documentation pages including homepage, quick start, creating workflows, CLI commands, and reference documentation
Overall Impression: The documentation provides comprehensive information, but critical broken internal links prevent the site from building properly, creating a major barrier for new users trying to understand the system through cross-references.

---

🔴 Critical Issues (Block Getting Started)

1. Broken Internal Links Prevent Documentation Build

Severity: CRITICAL - Blocks documentation deployment
Impact: New users cannot access a properly built documentation site

The documentation build fails with 100+ broken internal link errors when link validation is enabled. The starlight-links-validator plugin reports errors like:

05:35:06 ▶ gh-aw/setup/quick-start/
05:35:06   ├─ /gh-aw/reference/workflow-structure/ - invalid link
05:35:06   └─ /gh-aw/troubleshooting/common-issues/ - invalid link

Details:

• The astro.config.mjs configures starlightLinksValidator with strict validation (errorOnRelativeLinks: true, errorOnLocalLinks: true)
• All referenced files actually exist in src/content/docs/reference/ and other directories
• Links use correct format: /gh-aw/reference/workflow-structure/
• Build only succeeds when link validator is disabled

Root Cause: Likely a configuration issue with the link validator plugin or base path handling in the Astro/Starlight build process.

Recommendations:

1. Immediate Fix: Investigate why the link validator is reporting false positives for valid internal links
2. Short-term: Temporarily disable link validation or configure it to be warning-only instead of error
3. Long-term: Fix the underlying issue with link resolution in the build pipeline
4. Testing: Add a CI check that validates internal links work correctly

---

2. Prerequisites Section Unclear for Complete Beginners

Location: /setup/quick-start/ - Prerequisites section
Issue: Assumes too much prior knowledge

The quick start guide lists prerequisites but doesn't provide enough context for complete beginners:

Problems:

1. "AI Account" is vague - No guidance on which to choose, no cost information, no signup links
2. "GitHub Actions enabled" - No clear instructions on how to check or enable it
3. WSL requirement on Windows - Not explained why it's needed or how to install it

Recommendation: Expand the prerequisites section with:

• Direct links to signup pages for each AI service
• Cost information and "recommended for beginners" guidance
• Step-by-step instructions for checking/enabling GitHub Actions
• Clear explanation of WSL requirement with installation link

---

3. "add-wizard" Command Used Before Explanation

Location: /setup/quick-start/ - Step 2
Issue: The quick start jumps directly to gh aw add-wizard without explaining:

• What "add-wizard" means or does
• That it's an interactive command (beginners might get confused when prompts appear)
• The source format githubnext/agentics/daily-repo-status (what does this path mean?)
• What happens if the command fails

Recommendation: Add context before the command explaining it's interactive and will guide the user through setup with prompts.

---

🟡 Confusing Areas (Slow Down Learning)

1. "Workflow Markdown File" Concept Not Introduced Early Enough

Location: Throughout quick start guide
Issue: The concept of "workflow markdown files" is mentioned multiple times but never clearly explained for beginners. Terms like "workflow markdown file" and "Copies the workflow markdown file to .github/workflows/" appear before the user understands what this means.

Recommendation: Add a brief explanation box early in the quick start explaining that gh-aw uses human-readable markdown files (.md) that get compiled into GitHub Actions YAML automatically.

---

2. "Compile" Step Seems Magical Without Context

Location: /setup/quick-start/ - Step 4 (Customize)
Issue: The guide says "Regenerate the workflow YAML by running gh aw compile" but doesn't explain:

• What "compile" means in this context
• Why you need to compile
• What happens during compilation
• The relationship between .md and .lock.yml files

Recommendation: Add a tip box explaining the compile process and the relationship between the markdown source and generated YAML files.

---

3. Secrets Setup Instructions Too Sparse

Location: /setup/quick-start/ - Step 2, point 3
Issue: "Set up required secrets" with link to reference docs isn't beginner-friendly

Problems:

• Link points to engine reference docs (too technical for quick start)
• No inline guidance on where to find API keys
• No explanation of what these secrets are for
• Missing step-by-step instructions for setting up repository secrets

Recommendation: Add an expandable section with step-by-step instructions for setting up secrets for each AI engine.

---

4. CLI Commands Page Missing Quick Copy Examples

Location: /setup/cli/
Issue: While the CLI page lists common commands nicely, it doesn't have quick copy-paste examples for common workflows

Positive: The "🚀 Most Common Commands" section at the top is excellent and beginner-friendly!

Improvement Opportunity: Add a "Quick Reference Card" section with copy-pasteable command sequences for common tasks.

---

5. "Creating Workflows" Page Lacks Guidance on Method Selection

Location: /setup/creating-workflows/
Issue: The page shows 3 methods (Coding Agent, AI Chatbot, Dictation) but doesn't help beginners choose which one to use

Recommendation: Add a decision tree at the top helping users pick the right method based on their needs and experience level.

---

🟢 What Worked Well

1. Video Demonstrations

The embedded videos (e.g., "Install the extension, add a workflow, and trigger a run from the CLI") are excellent! They show exactly what to expect.

2. Estimated Time Indicators

"⏱️ Estimated time: 10 minutes" at the top of guides is very helpful for beginners to set expectations.

3. Most Common Commands Section

The CLI page's "🚀 Most Common Commands" with links to detailed documentation is perfect for beginners. Clear, concise, and not overwhelming.

4. TIP and NOTE Callouts

The use of GitHub-style alerts (tip, note, warning) throughout the docs makes important information stand out effectively.

5. Alternative Installation Method

The quick start mentions the standalone installer script as an alternative - this is great for users who might have issues with gh extension install.

6. Step-by-Step Structure

The numbered steps in the quick start guide are clear and easy to follow.

---

📋 Recommendations Summary

Priority 1 - Critical Fixes (Blocks Users)

1. ✅ Fix broken internal links issue (link validation failing)
2. ✅ Expand prerequisites section with links and explanations
3. ✅ Add beginner context for "workflow markdown files" concept

Priority 2 - Quick Wins (Improve Learning)

4. ✅ Add "What does compile do?" explanation
5. ✅ Expand secrets setup instructions with step-by-step guide
6. ✅ Add quick reference card to CLI commands page
7. ✅ Add decision tree to "Creating Workflows" page

Priority 3 - Polish (Nice to Have)

8. ✅ Add more inline examples throughout documentation
9. ✅ Create a troubleshooting flowchart for common issues
10. ✅ Add a "Glossary" link in the quick start for terms like "compile", "frontmatter", "lock file"

---

🎯 Test Coverage

Pages Analyzed

1. ✅ Homepage (/gh-aw/)
2. ✅ Quick Start (/setup/quick-start/)
3. ✅ Creating Workflows (/setup/creating-workflows/)
4. ✅ CLI Commands (/setup/cli/)
5. ✅ Reference pages (via source files)

Beginner Perspective Applied

• Assumed no prior knowledge of GitHub Actions
• Assumed no prior knowledge of AI workflows
• Assumed familiarity with basic Git/GitHub but not advanced topics
• Tested mental model: "Can a complete beginner follow these steps without getting confused?"

---

📊 Overall Assessment

Strengths:

• Comprehensive documentation with good structure
• Excellent use of videos and visual aids
• Clear numbered steps in quick start guide
• Good use of callout boxes for tips and warnings

Weaknesses:

• Critical broken links issue prevents proper build
• Some concepts introduced before they're explained
• Prerequisites assume too much knowledge
• Missing contextual explanations for technical terms

Recommended Next Steps:

1. Fix link validation issue immediately
2. Expand prerequisites and add more context for beginners
3. Add quick-win improvements (glossary links, expandable sections)
4. Consider adding a "Complete Beginner's Tutorial" that walks through every concept from scratch

---

Testing completed by: Automated Documentation Noob Testing Workflow
Documentation version: Built from main branch

---

Note: This was intended to be a discussion, but discussions could not be created due to permissions issues. This issue was created as a fallback.

AI generated by Documentation Noob Tester

• expires on Feb 17, 2026, 5:43 AM UTC