Index markdown headers to enable section validation and deep-linking

[phernandez]

Summary

Indexing markdown section headers (## Heading) in the parser would unlock two capabilities from a single change:

1. Section validation — schemas could declare required sections and validate that observations appear in the correct section
2. Deep-linking — permalinks could point to sections within documents (e.g., memory://note#section)

Current State

• The parser is completely section-agnostic — observations and relations are extracted by pattern matching regardless of which ## section they appear under
• The validator receives flat (category, content) tuples with no section context
• Section headers like ## Observations, ## Relations, ## Background are purely organizational/visual

Proposed Changes

Parser

• Track which section each observation/relation was extracted from
• Index section headers as searchable/queryable metadata
• Preserve section context through the parse → validate pipeline

Schema Format Extension

The picoschema format could extend with a sections key:

schema:
  name: string
  role?: string
sections:
  - Background
  - Methods
  - Findings

Deep-linking

With headers indexed, permalinks could reference specific sections:

• memory://coffee-brewing-methods#methods
• memory://auth-service#observations

Context

• Schema system core (parser, resolver, validator, inference, diff) has 100+ tests at high coverage
• Schema router integration tests completed (8 tests in tests/api/v2/test_schema_router.py)
• Router search_fn bug was fixed during test work — was returning search index metadata instead of full entity_metadata

🤖 Generated with Claude Code

[claude]

Claude finished @phernandez's task in 2m 38s —— View job

---

Issue Triage Complete

Type: Enhancement / Feature Request
Priority: Medium
Complexity: Complex (architectural changes)
Components: Parser, Schema System, MCP Tools
Label Applied: enhancement

---

Summary

This is a well-structured feature request to add markdown section header indexing, enabling:

1. Section validation - Schemas could enforce that observations appear in correct sections
2. Deep-linking - Permalinks could reference specific sections (e.g., memory://note#section)

Related Issues

Closely Related:

• Section and Observation Anchor Wikilinks Support #135 - "Section and Observation Anchor Wikilinks Support" - Proposes [[entity#section#observation]] syntax. Issue Index markdown headers to enable section validation and deep-linking #547 provides the parser infrastructure needed to support section-aware features like Section and Observation Anchor Wikilinks Support #135.

Possibly Related:

• [FEATURE] Support schema for notes #375 - "[FEATURE] Support schema for notes" - General schema system. Section validation would build upon this.

Current Architecture

The parser (entity_parser.py:113-136) is currently section-agnostic:

• Extracts observations/relations via pattern matching without section context
• Uses markdown-it with custom plugins (observation_plugin, relation_plugin)
• Returns flat EntityContent with no section metadata
• No section tracking in Observation or Relation schemas

Required Changes

1. Parser Layer (entity_parser.py)

   • Track section headers during markdown-it token parsing
   • Associate observations/relations with their containing section
   • Extend EntityContent to include section metadata

2. Schema Layer (markdown/schemas.py)

   • Add optional section field to Observation and Relation models
   • Create section header index structure
   • Support for picoschema format extension (if applicable)

3. Deep-linking Support

   • Extend permalink resolution to handle #section fragments
   • Update MCP tools (read_note, build_context) to support section-level addressing
   • Database schema migration to store section metadata

Complexity Assessment

Complex due to:

• Core parsing logic with extensive test coverage (17+ test files in tests/markdown/)
• Database schema migration required
• Multi-layer impact: parser → schemas → database → MCP tools → API
• Backwards compatibility considerations
• Coordination with related issues (Section and Observation Anchor Wikilinks Support #135, [FEATURE] Support schema for notes #375)

Recommendations

1. Clarify Schema System: The issue mentions a schema system with "parser, resolver, validator, inference, diff" and 100+ tests. Verify if this refers to existing Pydantic schemas or planned features from [FEATURE] Support schema for notes #375.

2. Coordinate with Section and Observation Anchor Wikilinks Support #135: These issues are highly synergistic - consider implementing together.

3. Phased Approach:

   • Phase 1: Add section tracking to parser (non-breaking - optional section field)
   • Phase 2: Implement deep-linking with #section fragments
   • Phase 3: Add section validation once schema system is ready

4. Consider Observation IDs: Issue Section and Observation Anchor Wikilinks Support #135 proposes observation-level anchors (#id-0001). Worth considering in the same effort.

Next Steps

This issue is now properly triaged and ready for:

• Discussion on implementation approach
• Prioritization in the roadmap
• Assignment when ready to implement

---