perf: progressive file loading for non-blocking startup

[seflue]

Summary

This PR introduces progressive file loading to eliminate startup blocking on large org directories. Instead of parsing all files synchronously (which could freeze the editor for multiple seconds with larger orgfile collections), files now load in batches with yields between them, keeping the UI responsive.

Motivation

As a tmux user who frequently opens and closes Neovim instances, and with a large collection of org files, startup blocking was a constant friction point. As the maintainer of telescope-orgmode.nvim, I also needed a public API that enables progressive file loading - allowing plugins to show immediate results to users, even on first startup. The goal is a smooth, non-blocking experience throughout.

This PR focuses on the core loading infrastructure. Follow-up PRs for progressive agenda loading may come later.

Key improvements

• Prioritized buffer loading: Files already open in buffers load first
• Batched loading with yields: Remaining files load progressively with 1ms yields between batches
• Async clock preloading: Clock headline search runs asynchronously in batches after files load, with sync fallback if accessed early
• New plugin API: Public methods for plugins to load files progressively with callbacks

Related Issues

TODO Will be added later.

Changes

• Add OrgFiles:scan() for fast file metadata retrieval without parsing
• Add OrgFiles:load_progressive() with batching, yielding, and callbacks
• Add OrgFiles:request_load() as unified, idempotent loading entry point
• Add OrgFiles:get_clocked_headline_async() for non-blocking clock search
• Preload clock asynchronously after files load (with sync fallback if accessed early)
• Add public API methods that can be used by plugins:
  • scan_files()
  • load_files()
  • on_file_loaded()
  • on_files_loaded()
  • is_files_loaded()
  • get_files_progress()
• Add :Org profiling command for performance debugging
• Add comprehensive tests for progressive loading and clock initialization

Checklist

I confirm that I have:

• Followed the
  Conventional Commits
  specification (e.g., feat: add new feature, fix: correct bug,
  docs: update documentation).
• My PR title also follows the conventional commits specification.
• Updated relevant documentation, if necessary.
• Thoroughly tested my changes.
• Added tests (if applicable) and verified existing tests pass with
  make test.
• Checked for breaking changes and documented them, if any.

[kristijanhusak]

Thanks for the PR! A lot of things are going on here. I did a quick overview of the PR, but didn't go into the details. I'd like to avoid adding all of this if possible.
Since the emphasis seems to be on preventing the blocking of the editor, can you test if this patch with master branch does anything for your use case?

diff --git a/lua/orgmode/files/init.lua b/lua/orgmode/files/init.lua
index 93dad13..29d129a 100644
--- a/lua/orgmode/files/init.lua
+++ b/lua/orgmode/files/init.lua
@@ -70,7 +70,7 @@ function OrgFiles:load(force)
       end
       return orgfile
     end)
-  end, self:_files(true), 50):next(function()
+  end, self:_files(true), 30):next(function()
     self.load_state = 'loaded'
     return self
   end)
diff --git a/lua/orgmode/utils/promise.lua b/lua/orgmode/utils/promise.lua
index 06227b7..a3d384b 100644
--- a/lua/orgmode/utils/promise.lua
+++ b/lua/orgmode/utils/promise.lua
@@ -366,7 +366,9 @@ function Promise.map(callback, list, concurrency)
         :next(function(...)
           results[i] = ...
           processing = processing - 1
-          processNext()
+          vim.defer_fn(function()
+            processNext()
+          end, 1)
         end)
         :catch(function(...)
           reject(...)

You can tweak the number in files/init.lua to see if there are any changes if the number is bigger or smaller, just for testing purposes.