-
-
Notifications
You must be signed in to change notification settings - Fork 16
feat: Add incremental build infrastructure for dependency tracking #1296
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat: Add incremental build infrastructure for dependency tracking #1296
Conversation
|
Can you help me to understand why we need this? What are you trying to achieve with this PR? What is the problem we are solving? How is this going to work? It seems to be part of a more complicated system that is not finished yet? |
|
@jaapio, so the added documentation does not answer your questions?
Incremental builds. You do not need to render the whole documentation if only one or some file changes.
Incremental builds.
It takes only 0.x seconds to update documentation.
By checking timestamps and hashes if source files. And render only what is necessary to render.
It is finished. It just must be utilized by the consuming app. |
|
Initially I thought I will provide the integration part with another PR, but does not make much sense, right?
So I will add it here as additional commits. |
ed4e101 to
9636daa
Compare
Add core classes for tracking document changes and dependencies: - DependencyGraph: Bidirectional graph for import/dependent relationships with cycle detection, BFS propagation, and resource exhaustion protection - DocumentExports: Immutable container for document anchors, titles, citations with comprehensive input validation - ChangeDetector: Fast change detection using mtime + content hash strategy - ContentHasher: File/content hashing with xxh128 (fast) or sha256 fallback - ChangeDetectionResult: Simple DTO for categorizing file changes Security hardening included: - Resource limits (100k docs, 2M edges, 1k imports/doc) - Path validation with control character rejection - Hash format validation - Depth-limited traversal protection
Container service that holds DependencyGraph and DocumentExports during compilation. Supports: - Current and previous exports for change detection - Hash algorithm tracking for cache compatibility - Serialization for persistence between builds - Input directory configuration Enforces MAX_EXPORTS limit (100k) consistent with DependencyGraph.
Add compiler passes that integrate incremental build with document compilation: - NodeTraversalTrait: Depth-limited (100 levels) recursive node traversal to prevent stack overflow from malicious documents - DependencyGraphPass (priority 9): Extracts imports from TocTree, Doc refs, and Include directives to build dependency graph - ExportsCollectorPass (priority 10): Collects anchors, titles, citations from compiled documents with path traversal protection Both passes run after standard compilation to capture final document state.
Register ContentHasher and IncrementalBuildState as services. Compiler passes are auto-discovered via the existing glob pattern.
9636daa to
4d1c5f2
Compare
43f76d3 to
78a2681
Compare
Document the incremental build infrastructure including: - Architecture overview and component responsibilities - How to use IncrementalBuildState in consumer applications - Persistence format and cache invalidation strategies - Security considerations and resource limits
78a2681 to
6bb3265
Compare
Extract higher-level incremental build classes from render-guides: - PropagationResult: DTO for dirty propagation results - CacheVersioning: Cache version validation (PHP version, format version) - DirtyPropagator: Propagates dirty state through dependency graph - GlobalInvalidationDetector: Detects when full rebuild is needed - IncrementalBuildCache: Main cache orchestrator with sharded storage Security hardening includes: - MAX_EXPORTS and MAX_OUTPUT_PATHS limits (100k each) - Input validation for all JSON data - Path validation for sharded storage - SplQueue for O(1) queue operations All classes have comprehensive unit tests (75 new tests).
6bb3265 to
fce5c30
Compare
Summary
Adds core infrastructure for incremental documentation builds, enabling consumers to:
Core Components
DependencyGraphDocumentExportsChangeDetectorContentHasherIncrementalBuildStateDependencyGraphPassExportsCollectorPassOrchestration Layer
IncrementalBuildCacheDirtyPropagatorGlobalInvalidationDetectorPropagationResultCacheVersioningSecurity Hardening
fromArray()deserializationDocumentation
incremental-builds.rst- Consumer guide with usage examples and integration patternsincremental-builds-architecture.rst- Maintainer guide with architecture, security model, and extension instructionsUsage
Consumer applications integrate by:
IncrementalBuildCachefrom cache (if exists)ChangeDetectorto determine which documents need re-renderingDirtyPropagator::propagate()to find affected dependents (with export comparison)GlobalInvalidationDetectorto check if full rebuild is neededIncrementalBuildCache::save()for next buildSee
docs/developers/incremental-builds.rstfor detailed documentation.Test Plan