Prevent indexing of non-stable docs versions and publish stable-only crawl policy

[Copilot]

☑️ Resolves

This sparked from a discussion in another ticket where users were seeing old versions docs in the first results of some search engines. Usually Google is smart enough, but sometimes it's not great.
I think we should maybe prevent indexing old docs and only let stable poke through ?

Search engines are indexing multiple version paths (/latest, /stable, /N/), creating duplicate/outdated results when canonical tags are ignored. This change makes non-stable pages explicitly non-indexable and adds a root crawl policy that keeps only server/stable crawlable.

• Shared HTML meta policy

  • Added _shared_assets/templates/layout.html to inject:

  <meta name="robots" content="noindex, follow" />

  on generated Sphinx pages across manuals.

• Deploy-time stable exception + robots policy

  • In .github/workflows/sphinxbuild.yml (deploy job), after staged artifacts are applied:
    • strip noindex from server/stable/**/*.html (stable remains indexable),
    • write root robots.txt:

  User-agent: *
  Allow: /server/stable/
  Disallow: /server/

• Link-check behavior

  • Keep the existing canonical-link stripping before lychee runs.
  • Removed the additional noindex stripping in link-check prep because it is unnecessary for lychee.

🖼️ Screenshots

N/A (workflow + template-only changes; no rendered page layout/content changes)

✅ Checklist

• I have built the documentation locally and reviewed the output
• Screenshots are included for visual changes
• I have not moved or renamed pages (or added a redirect if I did)
• I have run codespell or similar and addressed any spelling issues

[github-actions]

📖 Documentation Preview

🔍 Open preview →

No RST documentation pages changed in this PR.

Last updated: Thu, 07 May 2026 17:24:10 GMT