Update documentation for accuracy and alignment with codebase

[JSv4]

Summary

• Updates documentation across multiple areas to reflect current implementation state
• Removes outdated/duplicate content and consolidates embedder documentation
• Improves accuracy of architecture, configuration, and walkthrough guides

Changes

Documentation Updates

• Architecture docs: PDF data layer, agent corpus actions, analyzers, embeddings, LLMs, websocket
• Commenting system: README, backend architecture, GraphQL API, notifications, implementation guide
• Configuration: Docker stack setup, storage backend, frontend configuration
• Permissioning: Consolidated guide, mention spec, testing permissions
• Walkthrough: Key concepts, document upload, corpus query, data extraction guides
• Features: Badge system, corpus folders implementation

Cleanup

• Removed duplicate docs/embedders/minn_modernbert_embedder.md (consolidated into modernbert_embedder.md)
• Added docs/plans/documentation_alignment_review.md tracking ongoing documentation work

Test plan

• Documentation changes only - no code changes
• Review docs for accuracy against current codebase

[claude]

Documentation Review - PR #798

Overview

This is a comprehensive documentation alignment PR that updates 46 files (+2192/-6015 lines) to reflect the current state of the codebase. The work is well-executed and addresses critical documentation drift issues.

✅ Strengths

1. Systematic Approach

• Excellent tracking document (docs/plans/documentation_alignment_review.md) provides clear audit trail
• All 82 documentation files reviewed and categorized by severity (4=Major, 3=Moderate, 2=Minor, 1=Perfect)
• Transparent status tracking shows all critical/moderate issues resolved

2. High-Quality Updates

• Accurate file references: All code paths verified to exist (e.g., docling_parser_rest.py not docling_parser.py)
• Correct class names: Fixed embedder names (ModernBERTEmbedder, not ModernBERTEmbedder768)
• Current terminology: Updated deprecated references (Tesseract→Docling, LlamaIndex→PydanticAI)
• Architecture clarity: DocumentPath correctly documented as source of truth for corpus-document relationships

3. Removal of Inline Code

• Replaced inline code blocks with links to source files
• Reduces maintenance burden (code changes won't immediately stale docs)
• Example: agent_corpus_actions_design.md now links to actual implementation instead of duplicating code

4. Status and Timestamps

• All updated files include "Last Updated: 2026-01-09" timestamps
• Implementation status clearly marked (e.g., "Implemented in v3.0.0")

5. Consolidation

• Removed duplicate minn_modernbert_embedder.md (consolidated into modernbert_embedder.md)
• Cleaner docs structure

⚠️ Issues Found

1. Missing CHANGELOG.md Update (BLOCKER per CLAUDE.md)

Severity: High
Location: Root CHANGELOG.md

Per the project's CLAUDE.md guidelines:

"IMPORTANT: Always update CHANGELOG.md when making significant changes to the codebase."

This PR makes significant documentation changes that should be recorded. Suggested entry:

## [Unreleased] - 2026-01-10

### Changed
- **Documentation alignment review** (PR #798): Comprehensive update of 46 documentation files
  - Fixed critical path references in `docs/architecture/PDF-data-layer.md`, `docs/embedders/modernbert_embedder.md`
  - Updated `docs/frontend/corpus-and-document-selection.md` to reflect CentralRouteManager architecture
  - Corrected GCS environment variables in `docs/GCP_STORAGE_SETUP.md`
  - Removed inline code examples across all docs, replaced with source links to reduce maintenance burden
  - Added `docs/plans/documentation_alignment_review.md` tracking 82 files reviewed and aligned
  - Removed duplicate `docs/embedders/minn_modernbert_embedder.md`
  - All updated files now include "Last Updated" timestamps

2. Documentation-Only Changes Don't Need Extensive Testing

Severity: Low
Location: PR description test plan

The test plan shows:

- [x] Documentation changes only - no code changes
- [ ] Review docs for accuracy against current codebase

The second checkbox should probably be checked since the documentation_alignment_review.md demonstrates comprehensive accuracy verification was performed.

3. Consider Pre-commit Hook for Doc Timestamps

Severity: Low (Enhancement suggestion)

Many files now have "Last Updated" timestamps. Consider adding a pre-commit hook or CI check to ensure these get updated when docs change. This could prevent future timestamp drift.

🔍 Spot Checks Performed

I verified accuracy for several updated files:

1. docs/architecture/agent_corpus_actions_design.md

   • ✅ Correctly references opencontractserver/corpuses/models.py lines 830-942
   • ✅ Correctly references opencontractserver/agents/models.py lines 223-379
   • ✅ Status correctly marked as "Implemented in v3.0.0"
   • ✅ Removed 500+ lines of inline code, replaced with source links

2. docs/configuration/frontend-configuration.md

   • ✅ Correctly documents VITE_* vs OPEN_CONTRACTS_REACT_APP_* prefix distinction
   • ✅ Accurate environment variable names
   • ✅ References correct sample env file paths

3. docs/architecture/embeddings_creation_and_retrieval.md

   • ✅ Correct utility function references (generate_embeddings_from_text)
   • ✅ Accurate embedder class names (MicroserviceEmbedder, ModernBERTEmbedder)
   • ✅ Removed LlamaIndex references (now uses PydanticAI)

4. docs/commenting_system/IMPLEMENTATION_GUIDE.md

   • ✅ Reduced from 3856 to 236 lines by removing planning content
   • ✅ Now focused on implemented system status
   • ✅ Accurate component file paths

📋 Recommendations

Must Address Before Merge:

1. Add CHANGELOG.md entry as shown above

Nice to Have:

2. Mark second test plan checkbox as complete
3. Consider adding doc timestamp automation to prevent future drift

🎯 Conclusion

This is excellent documentation work that significantly improves the accuracy and maintainability of OpenContracts docs. The systematic review process and removal of inline code are particularly commendable.

Recommendation: Approve after CHANGELOG.md is updated (required per project guidelines).

---

Review performed by Claude Code CLI on 2026-01-10