Skip to content

Add robots.txt and llms.txt to counterfact.dev#2113

Merged
pmcelhaney merged 2 commits into
mainfrom
copilot/create-llms-and-robots-files
May 23, 2026
Merged

Add robots.txt and llms.txt to counterfact.dev#2113
pmcelhaney merged 2 commits into
mainfrom
copilot/create-llms-and-robots-files

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 23, 2026

Summary

Adds two discovery files to site/public/ so they are served at the root of https://counterfact.dev. The goal is to increase findability for both search crawlers and LLMs surfacing relevant tools to developers.

  • robots.txt — open to all crawlers, no Disallow rules
  • llms.txt — follows llmstxt.org spec: title, blockquote description, extended prose for target audience, and curated link sections for Documentation, Patterns, and Source Code
Original Prompt

Create these two files and add them to the web site. The goal to increase the chances that people interested in the kinds of problems Counterfact solves will find the web site.

Manual acceptance tests

  • https://counterfact.dev/robots.txt is reachable and contains User-agent: * / Allow: / with no Disallow directives
  • https://counterfact.dev/llms.txt is reachable and begins with # Counterfact
  • All links in llms.txt resolve (docs and patterns pages return 200)
  • cd site && npm run build completes without errors and dist/robots.txt + dist/llms.txt appear in the output

Tasks

  • Added site/public/robots.txt (allow all bots)
  • Added site/public/llms.txt with project description, Documentation section (5 links), Patterns section (11 links), and Source Code section

Copilot AI linked an issue May 23, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Add LLMs.txt and robots.txt to website Add robots.txt and llms.txt to counterfact.dev May 23, 2026
Copilot AI requested a review from pmcelhaney May 23, 2026 17:50
@pmcelhaney pmcelhaney marked this pull request as ready for review May 23, 2026 18:10
@pmcelhaney pmcelhaney enabled auto-merge May 23, 2026 18:10
@pmcelhaney pmcelhaney added this pull request to the merge queue May 23, 2026
Merged via the queue into main with commit 4d6f1c7 May 23, 2026
9 of 12 checks passed
@pmcelhaney pmcelhaney deleted the copilot/create-llms-and-robots-files branch May 23, 2026 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create an LLMs.txt and robots.txt

2 participants