-
-
Notifications
You must be signed in to change notification settings - Fork 16
perf: Add parallel processing infrastructure #1297
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
perf: Add parallel processing infrastructure #1297
Conversation
|
I'm sorry, I think this will add a lot of extra complexity to this project. I had this idea in mind for quit a while, and I do understand how this would drastically improve the speed of this project. But before we go this path I would love to learn more about how you think the new method would make sure we do not end up with race conditions. How do different processes share results etc. This whole topic is very new to me, I'm open to learn new things but as I will be the one maintaining this, I need to have full understanding of this topic before we go this route. |
|
Initially I thought I will provide the integration part with another PR, but does not make much sense, right?
So I will add it here as additional commits. |
05acc57 to
15a1216
Compare
Adds comprehensive parallel processing infrastructure for documentation builds using pcntl_fork with copy-on-write memory semantics. New Classes: - Build/Parallel: CpuDetector, ProcessManager, ParallelSettings - Compiler/Parallel: ParallelCompiler, DocumentCompilationResult - Renderer/Parallel: ForkingRenderer, DocumentNavigationProvider, StaticDocumentIterator - Pipeline: SingleForkPipeline Features: - Cross-platform CPU detection (Linux, macOS, BSD) - Phase-based parallel compilation (Collection → Merge → Resolution → Finalization) - Parallel Twig rendering with pre-computed navigation - Non-blocking wait with configurable timeout (default 300s) - Graceful fallback to sequential processing when pcntl unavailable - Secure temp file handling with signal-based cleanup
15a1216 to
29a23ed
Compare
|
As I mentioned in my previous comment, I’m having a hard time understanding the conceptual model behind this change and how it’s intended to be used. The approach introduces a significant amount of additional complexity, and at this point I don’t feel confident that I would be able to maintain this over the long term. Another concern for me is the impact this would have on users of the project. The changes affect core parts of the parsing, compiling, and rendering flow. While this may work when looking at this repository in isolation, there are several downstream projects that rely on this code in different ways — sometimes at a very low level, bypassing the usual commands and handlers. From my perspective, it’s difficult to assess the full impact on those integrations. Because of that, I’d really like to ask again that changes of this scope and complexity are discussed before introducing new subsystems like this. You’re clearly investing time to help improve the project, and that’s appreciated. At the same time, I and a few other contributors are responsible for maintaining this codebase for many years to come, and that heavily influences how we evaluate changes. I want to be clear that this isn’t a reflection on the quality of your work or the intent behind it, nor on whether it could improve performance. The concern is primarily about long-term maintainability, conceptual clarity, and ecosystem impact. |
Summary
Adds comprehensive parallel processing infrastructure for documentation builds using
pcntl_forkwith copy-on-write memory semantics.New Classes
CpuDetectorProcessManagerParallelSettingsParallelCompilerDocumentCompilationResultForkingRendererDocumentNavigationProviderStaticDocumentIteratorDirtyDocumentProviderSingleForkPipelineParallel Compiler Phases
The
ParallelCompilersplits compilation into phases based on shared state dependencies:Features
Core Utilities:
/proc/cpuinfo,nproc,sysctlParallel Compiler:
Parallel Renderer:
DirtyDocumentProviderGraceful Fallback
All components fall back to sequential processing when:
pcntlextension is unavailableTest Coverage
CpuDetectorTest: 4 tests covering core detection and max workersProcessManagerTest: 12 tests covering temp file management and process waitTest Plan
Documentation
Developer documentation available in
docs/parallel-processing.mdcovering: