Skip to content

Conversation

@CybotTM
Copy link
Contributor

@CybotTM CybotTM commented Jan 23, 2026

Summary

Adds comprehensive parallel processing infrastructure for documentation builds using pcntl_fork with copy-on-write memory semantics.

New Classes

Package Class Purpose
Build/Parallel CpuDetector Cross-platform CPU core detection (Linux, macOS, BSD)
ProcessManager Forked process management with timeout and cleanup
ParallelSettings Configuration for parallel processing (-1=disabled, 0=auto, N=workers)
Compiler/Parallel ParallelCompiler Phase-based parallel compilation with 4 phases
DocumentCompilationResult Container for per-batch results that survive serialization
Renderer/Parallel ForkingRenderer Parallel Twig rendering using pcntl_fork with COW
DocumentNavigationProvider Pre-computed prev/next navigation for parallel contexts
StaticDocumentIterator Document iterator with pre-computed relationships
DirtyDocumentProvider Interface for incremental rendering integration
Pipeline SingleForkPipeline Simple fork-once architecture (parse→compile→render per worker)

Parallel Compiler Phases

The ParallelCompiler splits compilation into phases based on shared state dependencies:

  1. Collection (parallel, priority ≥4900): Write to ProjectNode, don't read cross-document
  2. Merge (sequential, O(n)): Merge DocumentCompilationResults into ProjectNode
  3. Resolution (parallel, priority 1000-4500): Read from complete ProjectNode, write to documents
  4. Finalization (sequential, priority <1000): Cross-document mutations

Features

Core Utilities:

  • Cross-platform CPU detection via /proc/cpuinfo, nproc, sysctl
  • Non-blocking wait with configurable timeout (default 300s)
  • Automatic SIGKILL for stuck processes
  • Secure temp file creation with 0600 permissions
  • Signal handlers for cleanup on SIGTERM/SIGINT

Parallel Compiler:

  • Phase-based parallel execution with automatic fallback
  • Toctree relationship preservation across process boundaries
  • Document entry reference fixing after serialization
  • Incremental cache state merging from child processes

Parallel Renderer:

  • Copy-on-write memory sharing for parsed AST
  • Pre-computed navigation for correct prev/next links
  • Optional incremental rendering via DirtyDocumentProvider
  • Graceful fallback to sequential rendering

Graceful Fallback

All components fall back to sequential processing when:

  • pcntl extension is unavailable
  • Document count is below threshold (MIN_DOCS_FOR_PARALLEL = 10)
  • Worker count is set to 1 or -1 (disabled)
  • Fork fails at runtime

Test Coverage

  • CpuDetectorTest: 4 tests covering core detection and max workers
  • ProcessManagerTest: 12 tests covering temp file management and process wait

Test Plan

  • All existing tests pass
  • New unit tests for CpuDetector (4 tests)
  • New unit tests for ProcessManager (12 tests)
  • Code style checks pass
  • Manual testing with large documentation sets

Documentation

Developer documentation available in docs/parallel-processing.md covering:

  • Architecture overview
  • Phase-based compilation
  • Integration guide
  • Troubleshooting

@CybotTM CybotTM marked this pull request as draft January 23, 2026 15:46
@CybotTM CybotTM changed the title [WIP] perf: Add parallel processing infrastructure perf: Add parallel processing infrastructure Jan 23, 2026
@CybotTM CybotTM marked this pull request as ready for review January 23, 2026 16:30
@jaapio
Copy link
Member

jaapio commented Jan 23, 2026

I'm sorry, I think this will add a lot of extra complexity to this project. I had this idea in mind for quit a while, and I do understand how this would drastically improve the speed of this project. But before we go this path I would love to learn more about how you think the new method would make sure we do not end up with race conditions. How do different processes share results etc.

This whole topic is very new to me, I'm open to learn new things but as I will be the one maintaining this, I need to have full understanding of this topic before we go this route.

@CybotTM
Copy link
Contributor Author

CybotTM commented Jan 23, 2026

Initially I thought I will provide the integration part with another PR, but does not make much sense, right?

Note: This is a WIP PR. The utilities are complete but integration with a parallel renderer is planned for a follow-up PR.

So I will add it here as additional commits.
And the bigger picture is still: https://cybottm.github.io/render-guides/ - render complete TYPO3 Core Changelog in under 1 minute.

@CybotTM CybotTM marked this pull request as draft January 24, 2026 08:18
@CybotTM CybotTM force-pushed the perf/parallel-processing-infrastructure branch 4 times, most recently from 05acc57 to 15a1216 Compare January 26, 2026 08:43
Adds comprehensive parallel processing infrastructure for documentation builds
using pcntl_fork with copy-on-write memory semantics.

New Classes:
- Build/Parallel: CpuDetector, ProcessManager, ParallelSettings
- Compiler/Parallel: ParallelCompiler, DocumentCompilationResult
- Renderer/Parallel: ForkingRenderer, DocumentNavigationProvider, StaticDocumentIterator
- Pipeline: SingleForkPipeline

Features:
- Cross-platform CPU detection (Linux, macOS, BSD)
- Phase-based parallel compilation (Collection → Merge → Resolution → Finalization)
- Parallel Twig rendering with pre-computed navigation
- Non-blocking wait with configurable timeout (default 300s)
- Graceful fallback to sequential processing when pcntl unavailable
- Secure temp file handling with signal-based cleanup
@CybotTM CybotTM force-pushed the perf/parallel-processing-infrastructure branch from 15a1216 to 29a23ed Compare January 26, 2026 08:54
@CybotTM CybotTM marked this pull request as ready for review January 26, 2026 09:21
@jaapio
Copy link
Member

jaapio commented Jan 26, 2026

As I mentioned in my previous comment, I’m having a hard time understanding the conceptual model behind this change and how it’s intended to be used. The approach introduces a significant amount of additional complexity, and at this point I don’t feel confident that I would be able to maintain this over the long term.

Another concern for me is the impact this would have on users of the project. The changes affect core parts of the parsing, compiling, and rendering flow. While this may work when looking at this repository in isolation, there are several downstream projects that rely on this code in different ways — sometimes at a very low level, bypassing the usual commands and handlers. From my perspective, it’s difficult to assess the full impact on those integrations.

Because of that, I’d really like to ask again that changes of this scope and complexity are discussed before introducing new subsystems like this. You’re clearly investing time to help improve the project, and that’s appreciated. At the same time, I and a few other contributors are responsible for maintaining this codebase for many years to come, and that heavily influences how we evaluate changes.

I want to be clear that this isn’t a reflection on the quality of your work or the intent behind it, nor on whether it could improve performance. The concern is primarily about long-term maintainability, conceptual clarity, and ecosystem impact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants