Add skill library audit reference to writing-skills

[vishnujayvel]

Summary

• Adds auditing-existing-skills.md — a new reference file for reviewing existing skill libraries against quality standards
• Updates SKILL.md description to include "auditing a skill library" as a trigger condition
• Adds cross-reference to the new audit doc in SKILL.md overview section

Motivation

writing-skills teaches how to create skills with TDD and how to test them with pressure scenarios. But there's no guidance for auditing an existing library of skills — finding structural issues, trigger conflicts, and quality gaps that accumulate over time.

This gap became apparent while auditing 26 personal skills against Anthropic's official best practices guide. The audit found:

• 3 skills with no YAML frontmatter (won't activate at all)
• 13 skills missing description fields (poor routing)
• 11 skills over 500 lines (context waste)
• 4 skills claiming "interview prep" (routing conflicts — Claude couldn't reliably pick the right one)
• 2 broken file references

None of these would be caught by the existing TDD creation workflow, since each skill was created correctly in isolation — the problems emerged from the library growing organically.

What's in the audit reference

1. When to Audit — triggers (10+ skills, routing problems, new domain skill, quarterly hygiene)
2. 5-Phase Checklist:
   • Phase 1: Structure Scan (frontmatter, name, description, line count)
   • Phase 2: Description Quality (CSO compliance, anti-patterns)
   • Phase 3: Trigger Overlap Detection (the highest-impact step)
   • Phase 4: Progressive Disclosure Check (ref links, depth)
   • Phase 5: Context Budget Check (total description size)
3. Automated scan scripts (bash one-liners for each phase)
4. Audit Report Template (structured output format)
5. Real-world example (26-skill audit findings)
6. Connection to TDD (audit = REFACTOR phase applied to entire library)

Design decisions

• Separate reference file (not inline in SKILL.md) — follows the repo's existing pattern where heavy reference content goes in separate files (like anthropic-best-practices.md, persuasion-principles.md)
• Bash scripts for automation — keeps it platform-agnostic, no dependencies
• Focuses on structural/routing issues — complements (doesn't duplicate) the behavioral TDD testing in testing-skills-with-subagents.md

TDD applicability note

The writing-skills Iron Law states "NO SKILL WITHOUT A FAILING TEST FIRST" for new skills and edits. However, testing-skills-with-subagents.md (lines 18-29) explicitly carves out reference material:

Don't test:

• Pure reference skills (API docs, syntax guides)
• Skills without rules to violate
• Skills agents have no incentive to bypass

This contribution is a pure reference document — a checklist and scan procedures. There are no discipline rules to enforce, no compliance to pressure-test, and no incentive for agents to bypass an audit checklist. It falls in the same category as anthropic-best-practices.md (which also ships without TDD pressure tests).

The SKILL.md edit is minimal (added one trigger phrase to description + one cross-reference line) and doesn't change any behavioral rules.

Test plan

• Verified the audit checklist against Anthropic's official anthropic-best-practices.md (already bundled in writing-skills)
• Applied the full 5-phase audit to 26 real skills — all issues found and fixed
• Confirmed reference file classification per testing-skills-with-subagents.md exemption criteria
• Manual review: read the reference and follow it on a sample skill directory

Generated with Claude Code

Summary by CodeRabbit

• Documentation
  • Added a comprehensive guide for auditing existing skill libraries with a five‑phase checklist (structure, description quality, trigger overlap, progressive disclosure, context budget) and an audit report template.
  • Included practical examples, ownership/resolution guidance, and automated checks/scripts for Phase 1 scans.
  • Updated skill-writing docs frontmatter/overview to reference the new auditing guidance.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

---

📝 Walkthrough

Walkthrough

SKILL.md now links to a new auditing guide. A new file, skills/writing-skills/auditing-existing-skills.md, was added providing a five‑phase audit workflow for skill libraries: structure scans, description quality, trigger overlap detection, progressive disclosure checks, and context‑budget verification.

Changes

Cohort / File(s)	Summary
Skills Writing Documentation skills/writing-skills/SKILL.md	Frontmatter description and Overview updated to reference the new auditing guide; adds an "Auditing existing skills" pointer.
New Auditing Guide skills/writing-skills/auditing-existing-skills.md	New comprehensive guide with a five‑phase audit checklist (Structure Scan, Description Quality, Trigger Overlap Detection, Progressive Disclosure Check, Context Budget Check), automated Phase 1 shell checks, guidance for overlap resolution and ownership, an audit report template, example workflow, and links to related materials.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 I hopped through lines and checked each clause,

Five bright phases laid without a pause,
Triggers pruned and descriptions neat,
Context counted, every budget complete,
I left the skill garden ready to cause a clap.

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: adding a new audit reference document to the writing-skills folder and updating SKILL.md to reference it.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (1)

skills/writing-skills/auditing-existing-skills.md (1)

128-135: This reference file is itself missing a table of contents.

Line 134 of the checklist states: "Reference files have table of contents if >100 lines." auditing-existing-skills.md is ~200 lines and has no ToC, which means it violates its own Phase 4 recommendation.

✨ Suggested addition after line 4

## Contents

1. [When to Audit](`#when-to-audit`)
2. [Audit Checklist](`#audit-checklist`)
   - [Phase 1: Structure Scan](`#phase-1-structure-scan`)
   - [Phase 2: Description Quality](`#phase-2-description-quality`)
   - [Phase 3: Trigger Overlap Detection](`#phase-3-trigger-overlap-detection`)
   - [Phase 4: Progressive Disclosure Check](`#phase-4-progressive-disclosure-check`)
   - [Phase 5: Context Budget Check](`#phase-5-context-budget-check`)
3. [Audit Report Template](`#audit-report-template`)
4. [Real-World Example](`#real-world-example`)
5. [Connection to TDD](`#connection-to-tdd`)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@skills/writing-skills/auditing-existing-skills.md` around lines 128 - 135,
The file auditing-existing-skills.md violates its own rule requiring a table of
contents for reference files >100 lines; add a "Contents" section near the top
(e.g., after the initial intro/line 4) listing the major headings (When to
Audit, Audit Checklist with Phase 1–5 anchors, Audit Report Template, Real-World
Example, Connection to TDD) and ensure anchor IDs match existing headings (e.g.,
`#when-to-audit`, `#audit-checklist`, `#phase-1-structure-scan`,
`#phase-2-description-quality`, `#phase-3-trigger-overlap-detection`,
`#phase-4-progressive-disclosure-check`, `#phase-5-context-budget-check`) so all
links resolve correctly and the file now complies with the "Reference files have
table of contents if >100 lines" rule.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@skills/writing-skills/auditing-existing-skills.md`:
- Around line 25-34: The three unlabelled fenced code blocks containing the
checklist starting with "For each skill directory:" (and the equivalent
checklist blocks later in the file) are triggering MD040; add a language
identifier (use ```text) to each of those fenced code blocks so they read
```text ... ``` to silence the markdownlint warning and preserve formatting for
the checklist blocks that begin with "For each skill directory:" and the other
two identical checklist sections.
- Around line 140-148: The chars counter is overcounting because it uses grep
-A20 and captures lines beyond the description; update the chars assignment
inside the for dir loop that reads SKILL.md so it extracts only the description
field's value (match the line that begins with "description:" and strip the
"description:" prefix and any leading whitespace) before piping to wc -c; adjust
the code that sets the chars variable (the line assigning to chars) accordingly
so the total truly reflects only the description characters.
- Around line 66-82: The Phase 2 checklist entry that says "Describes WHAT it
does (action verb, not 'A skill for…')" and the anti-pattern fix example "Manage
X. Use when [triggers]." contradict SKILL.md's CSO rule that descriptions must
only state when to use the skill (not what it does) and must avoid
workflow-summary phrases; update the checklist and anti-patterns table so the
checklist line reads something like "Describes WHEN to use it (trigger phrases
users say) — do NOT describe WHAT it does or workflow", and change the example
fix to only show trigger-first phrasing (e.g., "Use when [triggers]. Avoid
'Manage X' or other action-summary clauses") and add a short note referencing
SKILL.md's CSO guidance to reinforce that descriptions must not summarize
workflow.
- Around line 37-60: The script prints "OK" unconditionally even when earlier
checks emitted FAIL; add a per-skill failure flag (e.g., failed=0) before the
frontmatter/name/description checks, set failed=1 whenever any grep/head check
fails (the frontmatter head -1 check and the grep checks for "^name:" and
"^description:"), and only print "OK: $skill ($lines lines)" when failed is
still 0; keep the existing continue for missing SKILL.md and treat WARNs (line
count) as non-fatal so they do not flip the failure flag.

---

Nitpick comments:
In `@skills/writing-skills/auditing-existing-skills.md`:
- Around line 128-135: The file auditing-existing-skills.md violates its own
rule requiring a table of contents for reference files >100 lines; add a
"Contents" section near the top (e.g., after the initial intro/line 4) listing
the major headings (When to Audit, Audit Checklist with Phase 1–5 anchors, Audit
Report Template, Real-World Example, Connection to TDD) and ensure anchor IDs
match existing headings (e.g., `#when-to-audit`, `#audit-checklist`,
`#phase-1-structure-scan`, `#phase-2-description-quality`,
`#phase-3-trigger-overlap-detection`, `#phase-4-progressive-disclosure-check`,
`#phase-5-context-budget-check`) so all links resolve correctly and the file now
complies with the "Reference files have table of contents if >100 lines" rule.

[vishnujayvel]

All 4 actionable comments + the nitpick about the missing table of contents have been addressed in 04d6e30. Thanks for the thorough review — the CSO contradiction catch (Phase 2 saying "WHAT" when SKILL.md explicitly says "NOT what it does") was especially good.

[coderabbitai]

🧹 Nitpick comments (1)

skills/writing-skills/auditing-existing-skills.md (1)

109-118: Phase 3 script: noisy description extraction and no missing-file guard.

grep -A5 "^description:" … | head -6 captures the description: line plus up to five trailing lines (YAML closing ---, blank lines, section headings, body content). An auditor scanning for trigger-phrase overlap will see that noise mixed in with the actual description values.

There is also no [ ! -f … ] && continue guard, unlike Phase 1 and Phase 5, so a skill directory without a SKILL.md silently emits an empty block rather than a skip notice.

♻️ Suggested improvement

-for dir in */; do
-  skill="${dir%/}"
-  desc=$(grep -A5 "^description:" "$dir/SKILL.md" | head -6)
-  echo "=== $skill ==="
-  echo "$desc"
-  echo
-done
+for dir in */; do
+  skill="${dir%/}"
+  [ ! -f "$dir/SKILL.md" ] && echo "SKIP: $skill — no SKILL.md" && continue
+  desc=$(grep "^description:" "$dir/SKILL.md" | sed 's/^description:[[:space:]]*//')
+  echo "=== $skill ==="
+  echo "$desc"
+  echo
+done

This mirrors the Phase 5 sed extraction and keeps the output limited to description values — exactly what an auditor needs to spot overlapping trigger phrases.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@skills/writing-skills/auditing-existing-skills.md` around lines 109 - 118,
The extraction loop for Phase 3 is noisy and lacks a missing-file guard: replace
the grep-then-head pipeline in the for loop with the same sed-based extraction
used in Phase 5 to capture only the description value (avoid YAML delimiters,
blank lines, and headings) and add a guard like [ ! -f "$dir/SKILL.md" ] &&
continue before extracting; target the for loop and the grep -A5 "^description:"
usage so the script prints a skip notice for missing SKILL.md files and outputs
only the clean description text.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@skills/writing-skills/auditing-existing-skills.md`:
- Around line 109-118: The extraction loop for Phase 3 is noisy and lacks a
missing-file guard: replace the grep-then-head pipeline in the for loop with the
same sed-based extraction used in Phase 5 to capture only the description value
(avoid YAML delimiters, blank lines, and headings) and add a guard like [ ! -f
"$dir/SKILL.md" ] && continue before extracting; target the for loop and the
grep -A5 "^description:" usage so the script prints a skip notice for missing
SKILL.md files and outputs only the clean description text.

[vishnujayvel]

Fixed the Phase 3 script nitpick in 1c50c71 — now mirrors Phase 5's sed-based extraction and includes the missing-file guard. All three scripts (Phase 1, 3, 5) are now consistent.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@skills/writing-skills/auditing-existing-skills.md`:
- Around line 172-196: Update the audit template's status labels to match the
automation output by replacing "PASS | NEEDS_IMPROVEMENT | FAIL" with "OK | WARN
| FAIL | SKIP" (and add a brief note mapping old terms if desired); specifically
edit the header/status line in the "Skill Audit Report — [Date]" template where
"PASS | NEEDS_IMPROVEMENT | FAIL" appears so it exactly reflects the automation
outputs "OK, FAIL, WARN, SKIP" and avoid any ambiguous mapping guidance in the
body.
- Around line 91-97: The "Too vague" row in the anti-patterns table currently
shows a capabilities example ("Extract text from PDFs, fill forms") which
violates the rule in the doc to "Describe ONLY WHEN to use it — NOT what it
does"; update the Fix column for the "Too vague" row to a trigger-oriented
phrase (e.g., start with "Use when..." or "Invoke when...") that describes the
user situation that should trigger the skill (for example: "Use when user asks
to extract text from PDFs or to auto-fill form data"), and ensure this change is
applied to the anti-pattern table entry labelled "Too vague" so auditors will
produce trigger-only descriptions.

[obra]

Thanks for this! v5.0.0 just shipped and substantially reworked writing-skills, so this PR is no longer a clean apply against main. Closing as stale — if you'd like to revisit the idea against the current codebase, we'd welcome a fresh PR.

— Claude (claude-opus-4-6, Claude Code 2.1.71, session 64908a66-5c98-4c79-b0f7-1aa7eac2dcb0)