perf(pm): skip missing clean deps roots#2970
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the clean_deps function to skip missing node_modules directories and adds a corresponding test case to verify this behavior. The review feedback highlights a cross-platform compatibility issue regarding hardcoded path separators in existing logic and suggests further performance optimizations by reordering the existence checks and deferring the construction of the valid_packages set.
| } | ||
|
|
||
| for nm_dir in &nm_dirs { | ||
| if !crate::fs::try_exists(nm_dir).await? { |
There was a problem hiding this comment.
Note that the cleaning logic following this check (specifically remove_unused_packages and its dependency path_to_pkg_name) contains hardcoded forward slashes (e.g., node_modules/ at line 336) and lacks path separator normalization when checking valid_packages. On Windows, where the path separator is , these checks will fail, effectively making the dependency cleaning a no-op. While this issue predates the current PR, it significantly impacts the correctness of the feature being optimized.
| } | ||
|
|
||
| for nm_dir in &nm_dirs { | ||
| if !crate::fs::try_exists(nm_dir).await? { |
There was a problem hiding this comment.
This existence check is redundant for workspace node_modules directories, as they are already verified to exist at line 156 before being added to nm_dirs. For the root node_modules directory, it would be more efficient to perform this check at line 153 when it is first added to the vector. Additionally, the construction of the valid_packages set (lines 146-149) could be deferred until after verifying that at least one node_modules directory exists, which would further improve performance in the 'missing node_modules' scenario.
📊 pm-bench-phases ·
|
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 8.99s | 0.23s | 10.46s | 10.16s | 758M | 344.2K |
| utoo-next | 8.10s | 0.10s | 10.51s | 12.26s | 967M | 124.6K |
| utoo-npm | 8.00s | 0.15s | 10.99s | 12.55s | 993M | 121.0K |
| utoo | 8.45s | 1.33s | 11.61s | 12.58s | 930M | 149.0K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 14.1K | 17.7K | 1.20G | 6M | 1.88G | 1.76G | 1M |
| utoo-next | 117.8K | 80.6K | 1.17G | 5M | 1.73G | 1.72G | 2M |
| utoo-npm | 132.7K | 90.3K | 1.17G | 5M | 1.73G | 1.72G | 2M |
| utoo | 124.5K | 90.0K | 1.17G | 6M | 1.72G | 1.72G | 2M |
p1_resolve
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 1.89s | 0.01s | 4.09s | 1.07s | 536M | 168.0K |
| utoo-next | 2.96s | 0.02s | 5.49s | 1.99s | 615M | 79.0K |
| utoo-npm | 3.00s | 0.03s | 5.56s | 2.00s | 626M | 77.9K |
| utoo | 2.41s | 0.03s | 6.07s | 1.65s | 647M | 121.3K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 7.8K | 4.8K | 203M | 3M | 108M | - | 1M |
| utoo-next | 69.8K | 85.8K | 201M | 2M | 7M | 3M | 2M |
| utoo-npm | 70.4K | 89.7K | 201M | 2M | 7M | 3M | 2M |
| utoo | 15.2K | 18.8K | 204M | 3M | 7M | 3M | 2M |
p3_cold_install
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 7.31s | 0.28s | 6.37s | 9.99s | 607M | 212.5K |
| utoo-next | 6.85s | 1.71s | 5.07s | 11.16s | 499M | 61.1K |
| utoo-npm | 7.09s | 1.78s | 5.19s | 11.09s | 507M | 60.2K |
| utoo | 5.69s | 0.17s | 5.13s | 11.39s | 485M | 59.8K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 4.5K | 6.7K | 1.00G | 3M | 1.77G | 1.77G | 1M |
| utoo-next | 109.3K | 48.8K | 1000M | 3M | 1.72G | 1.72G | 3M |
| utoo-npm | 107.3K | 48.6K | 1000M | 3M | 1.72G | 1.72G | 3M |
| utoo | 120.3K | 77.0K | 1000M | 3M | 1.72G | 1.72G | 3M |
p4_warm_link
| PM | wall | ±σ | user | sys | RSS | pgMinor |
|---|---|---|---|---|---|---|
| bun | 3.36s | 0.06s | 0.19s | 2.42s | 134M | 31.6K |
| utoo-next | 2.33s | 0.05s | 0.54s | 3.89s | 80M | 18.4K |
| utoo-npm | 2.31s | 0.08s | 0.53s | 3.88s | 80M | 18.3K |
| utoo | 2.33s | 0.11s | 0.56s | 4.00s | 63M | 13.9K |
| PM | vCtx | iCtx | netRX | netTX | cache | node_mod | lock |
|---|---|---|---|---|---|---|---|
| bun | 254 | 23 | 27K | 25K | 1.88G | 1.75G | 1M |
| utoo-next | 43.7K | 19.1K | 2K | 6K | 1.72G | 1.72G | 3M |
| utoo-npm | 44.1K | 20.9K | 2K | 19K | 1.72G | 1.72G | 3M |
| utoo | 54.8K | 25.4K | 2K | 26K | 1.73G | 1.72G | 3M |
npmmirror.com: no output captured.
GHA run 1 readRun: https://github.com/utooland/utoo/actions/runs/26017207217
Conclusion: reject as a perf fold. The missing- |
|
Closing this PM performance experiment after the investigation phase. The benchmark data and conclusions are preserved in the PR body/comments; we will split the validated pieces into smaller reviewable PRs for the formal ship path. |
Summary
Small AB experiment for p3/p4 install overhead.
clean_depscurrently always enters stale-entry cleanup and unused-package globbing for the rootnode_modules, even when the directory does not exist. In phase-isolated p3/p4, the prepare step removesnode_modulesbefore each run, so this path should be a no-op.This change skips each
node_modulesroot when it is missing before running stale cleanup and package globbing. Existing workspacenode_modulesdirectories are still discovered and cleaned when present.Hypothesis
This should remove a small amount of cold/warm install filesystem probing and glob setup in p3/p4. Expected impact is small; keep only if same-run p3/p4 wall or ctx moves positively.
Validation
Passed:
cargo fmtcargo test -p utoo-pm service::clean::tests::test_clean_deps_skips_missing_node_modulescargo clippy -p utoo-pm --all-targets -- -D warnings --no-depsFull workspace clippy is not repeated here; it is currently blocked in this worktree by the known
pack-core/next.jsAPI mismatch observed on #2969 validation.Benchmark Plan
Label-trigger GHA bench, compare p3/p4 against same-run
utoo-next,utoo-npm, andbun.