OAK-12089: LuceneNg (Lucene 9.12.2) Implementation - Phase 1 & 2

[bhabegger]

LuceneNg (Lucene 9.12.2) Implementation - Phase 1 & 2

This PR implements a new Lucene 9 index module (oak-search-luceneNg) for Apache Jackrabbit Oak, targeting the OAK-12089 epic.

🎯 Goals of This PR

Goal	Description	Status
New Module	Create oak-search-luceneNg module with Lucene 9.12.2	✅ Complete
Write Path	Implement document indexing via IndexEditor	✅ Complete
Storage	Oak-native storage with chunked blob support	✅ Complete
Read Path	Implement query execution with full-text search	✅ Complete
Property Queries	Support equality constraints on indexed properties	✅ Complete
Full-Text Search	Support analyzed text queries with tokenization	✅ Complete
Test Coverage	Comprehensive unit and integration tests	✅ Complete (53/53 tests pass)
Build Integration	Maven build, OSGi bundles, Apache RAT compliance	✅ Complete

📊 Implementation Status & Roadmap

✅ Phase 1: Write Path (Complete)

Component	Status	Tests	Notes
LuceneNgIndexEditor	✅ Done	7 tests	Indexes string properties (single & multi-value)
OakDirectory	✅ Done	16 tests	Lucene Directory backed by Oak NodeStore
Chunked I/O	✅ Done	5 tests	Efficient large file handling with 1MB chunks
IndexWriter lifecycle	✅ Done	7 tests	Shared writer pattern for correct commit semantics

✅ Phase 2: Read Path - Basic Queries (Complete)

Component	Status	Tests	Notes
LuceneNgIndex	✅ Done	2 tests	QueryIndex implementation with cost calculation
Full-text queries	✅ Done	2 tests	Visitor pattern, tokenization, phrase/term queries
Property queries	✅ Done	5 tests	Exact-match equality constraints
LuceneNgCursor	✅ Done	7 tests	Result iteration with score support
Query planner integration	✅ Done	2 tests	Cost-based index selection (cost = 2.0)

🚧 Phase 2: Read Path - Advanced (Planned)

Feature	Priority	Complexity	Notes
Range queries	High	Medium	Support <, >, <=, >= operators
Boolean queries	High	Medium	Complex AND/OR/NOT combinations
Sorting	Medium	Medium	ORDER BY support
Aggregation rules	Medium	High	Property aggregation across node types
Highlighting	Low	Medium	rep:excerpt support
Faceting	Low	High	rep:facet support

⏳ Phase 3: Migration & Production (Future)

Feature	Priority	Complexity	Notes
Hot migration	High	High	Migrate from Lucene 4.7 without downtime
Index compatibility	High	High	Read existing lucene indexes
Performance benchmarks	High	Medium	Compare with legacy Lucene
AEM integration testing	High	High	Validate in AEM environment
Documentation	Medium	Low	Usage guides, migration docs

📦 What's Included

New Module Structure

oak-search-luceneNg/
├── src/main/java/
│   └── org/apache/jackrabbit/oak/plugins/index/luceneNg/
│       ├── LuceneNgIndex.java              # Query execution
│       ├── LuceneNgIndexEditor.java        # Document indexing
│       ├── LuceneNgCursor.java             # Result iteration
│       ├── LuceneNgIndexTracker.java       # Index lifecycle
│       ├── LuceneNgIndexDefinition.java    # Index metadata
│       ├── IndexSearcherHolder.java        # Search resource management
│       └── directory/
│           ├── OakDirectory.java           # Lucene Directory implementation
│           ├── OakIndexInput.java          # Read operations
│           └── OakIndexOutput.java         # Write operations
└── src/test/java/
    ├── LuceneNgComparisonTest.java         # Property query validation
    ├── IntegrationTest.java                # End-to-end tests
    ├── IndexingFunctionalTest.java         # Indexing edge cases
    └── directory/                          # Storage layer tests

Key Features

Query Support:

• ✅ Full-text search with StandardAnalyzer tokenization
• ✅ Property equality queries (@property = 'value')
• ✅ Proper cost-based query planning
• ✅ Score-based result ranking

Indexing:

• ✅ String properties (single and multi-value)
• ✅ Full-text aggregation to :fulltext field
• ✅ Exact-match fields for property queries
• ✅ 32KB term length handling

Storage:

• ✅ Oak NodeStore integration via :data child node
• ✅ Chunked blob storage (1MB chunks)
• ✅ Concurrent read/write support
• ✅ Memory-efficient streaming

🧪 Test Results

All 53 tests pass:

• ✅ 16 OakDirectory tests (storage layer)
• ✅ 7 IndexingFunctionalTest (write path)
• ✅ 5 LuceneNgComparisonTest (property queries)
• ✅ 5 IntegrationTest (end-to-end)
• ✅ 20 additional unit tests (components, tracking, etc.)

Build:

mvn clean install
[INFO] Tests run: 53, Failures: 0, Errors: 0, Skipped: 0
[INFO] BUILD SUCCESS

🔍 Technical Highlights

1. Proper Full-Text Query Building

Implements visitor pattern matching legacy Lucene behavior:

• Tokenizes query text using StandardAnalyzer
• Builds PhraseQuery for multi-token terms
• Handles FullTextAnd, FullTextOr, FullTextTerm expressions

2. Shared IndexWriter Pattern

Root editor creates IndexWriter, child editors share it:

• Prevents data loss from multiple writers
• Correct commit semantics across node tree
• Proper resource cleanup

3. Dynamic NodeBuilder Access

Avoids staleness issues during commits:

private NodeBuilder getDirectoryBuilder() {
    return definitionBuilder.child(INDEX_DATA_CHILD_NAME);
}

4. Field Strategy

• StringField: Exact matching for property queries (not analyzed)
• TextField: Analyzed text for full-text search (FieldNames.FULLTEXT)
• Path storage: Stored field for cursor results

🔗 Related Issues

• OAK-12089: Epic for Lucene 9 migration
• Builds on exploration work from earlier branches

📝 Notes for Reviewers

1. Module isolation: New module doesn't affect existing lucene/elastic modules
2. Dependency embedding: Lucene 9.12.2 libs embedded to avoid conflicts
3. Test independence: All tests use in-memory storage, no external dependencies
4. Apache compliance: All files have Apache license headers, RAT check passes

✅ Checklist

• All tests pass
• Apache RAT license check passes
• Code follows Oak patterns (QueryIndex, IndexEditor, Cursor)
• No backwards compatibility issues (new module, opt-in)
• Documentation in code comments
• Test coverage for all major code paths

---

Ready for review! This PR establishes the foundation for Lucene 9 support in Oak. Phase 2 advanced features and Phase 3 migration can be tackled in subsequent PRs.

[reschke]

package info seems to be missing. It this an internal package?

[reschke]

why embed?