feat: docs: frontmatter key for documentation site ingestion

[johnlindquist]

Summary

Add a new frontmatter key docs: that fetches and includes entire documentation sites as agent context using into.md.

Proposed Syntax

---
harness: claude
model: sonnet
docs: https://docs.astro.build/
---

Help me build an Astro site with these requirements...

How It Would Work

1. Parse docs: URL(s) from frontmatter
2. Fetch the root page through into.md
3. Optionally spider linked pages (configurable depth)
4. Concatenate all converted markdown as context
5. Inject before the prompt body (similar to context: globs)

Use Cases

• Instant Expert Agents: Create an agent that "knows" a specific library's docs
• Version-Specific Context: Point to /v2/ vs /v3/ docs for the right version
• Framework Helpers: Build agents specialized in Next.js, Remix, SvelteKit, etc.

Configuration Options

docs:
  url: https://docs.example.com/
  depth: 2          # How many levels to spider (default: 1)
  include: /api/**  # URL patterns to include
  exclude: /blog/** # URL patterns to skip
  cache: 24h        # Cache duration

Implementation Notes

• New frontmatter key in src/types.ts and src/schema.ts
• Spider logic with depth limiting
• Respect robots.txt? Or trust user intent?
• Token budget awareness (docs can be huge)

Questions for Discussion

• How do we handle docs that exceed context limits? Summarize? Truncate? Error?
• Should we show a progress indicator while spidering?
• Is docs: the right key name, or something like web-context:?

Related

This is part of exploring into.md integration opportunities.