Redesign file persistence pipeline to eliminate race conditions

[meanmail]

Overview

The plugin has a race condition between asynchronous file content persistence and YAML serialization. Current workaround uses blocking waitForPersistingTasks() which can freeze the EDT. The persistence pipeline needs architectural redesign to be fully non-blocking while maintaining data consistency.

Problem

Current Architecture

1. File contents are persisted asynchronously on pooled threads (executeOnPooledThread at LearningObjectsStorageManager.kt:62)
2. YAML serialization happens on EDT and requires all file contents to be persisted
3. Current solution: waitForPersistingTasks() (lines 140-152) blocks on Future.get() until all persistence completes

Issues with Current Approach

fun waitForPersistingTasks() {
    persistingTasks.forEach { it.get() } // Blocking wait!
}

• Blocks the calling thread (often EDT)
• Can cause UI freezes during course save operations
• No timeout handling
• No cancellation support

Diagnostic Wrapper

Lines 58-61 track content changes during persistence, indicating awareness of the race condition, but the fix is to just block and wait.

Key Files

• intellij-plugin/hs-core/src/org/hyperskill/academy/learning/storage/LearningObjectsStorageManager.kt
• intellij-plugin/hs-core/src/org/hyperskill/academy/learning/yaml/YamlFormatSynchronizer.kt
• intellij-plugin/hs-core/src/org/hyperskill/academy/learning/framework/impl/FrameworkLessonManagerImpl.kt

What Makes This Hard

1. Concurrency complexity: Coordinating async file persistence with EDT-bound YAML serialization
2. State synchronization: File content can change during persistence operation
3. Multiple storage backends: YAML, SQLite, InMemory - each with different threading characteristics
4. EDT constraints: Cannot perform slow operations on EDT, but YAML writing currently requires synchronized state
5. Cancellation: Must handle project disposal during persistence
6. Consistency guarantees: Must ensure YAML always represents completely persisted state
7. Performance: Solution must not regress performance compared to async persistence

Requirements

Architecture Design

• Design a fully non-blocking persistence pipeline
• Implement proper async coordination between persistence and serialization
• Consider using:
  • CompletableFuture instead of raw Future for better composition
  • ReadAction.nonBlocking() / WriteAction.nonBlocking() for VFS operations
  • Message bus listeners to trigger YAML save after persistence completes
  • Coroutines for async flow control (if appropriate for the platform version)

Implementation

• Remove all blocking Future.get() calls
• Ensure no EDT violations
• Handle all edge cases:
  • File content changes during persistence
  • Project disposal mid-persistence
  • Multiple concurrent persistence requests
  • Storage backend failures
• Add proper cancellation support
• Maintain data consistency across all storage backends

Testing

• Add comprehensive tests for concurrent scenarios:
  • Rapid file modifications + frequent saves
  • Project close during persistence
  • Storage backend failures
• Performance test: persistence should not regress
• EDT test: no blocking operations on EDT

Success Criteria

• Zero EDT violations (verify with EDT violation detection enabled)
• No blocking waits anywhere in persistence pipeline
• All concurrent test scenarios pass
• Performance equal to or better than current implementation
• Clean shutdown even during active persistence

Estimated Time

6-8 hours (complex async coordination, multiple storage backends, comprehensive testing)