Skip to content

perf(pm): skip missing clean deps roots#2970

Closed
elrrrrrrr wants to merge 1 commit into
perf/pm-resolver-demand-bfsfrom
exp/pm-skip-missing-clean-deps-b33d922
Closed

perf(pm): skip missing clean deps roots#2970
elrrrrrrr wants to merge 1 commit into
perf/pm-resolver-demand-bfsfrom
exp/pm-skip-missing-clean-deps-b33d922

Conversation

@elrrrrrrr
Copy link
Copy Markdown
Contributor

Summary

Small AB experiment for p3/p4 install overhead.

clean_deps currently always enters stale-entry cleanup and unused-package globbing for the root node_modules, even when the directory does not exist. In phase-isolated p3/p4, the prepare step removes node_modules before each run, so this path should be a no-op.

This change skips each node_modules root when it is missing before running stale cleanup and package globbing. Existing workspace node_modules directories are still discovered and cleaned when present.

Hypothesis

This should remove a small amount of cold/warm install filesystem probing and glob setup in p3/p4. Expected impact is small; keep only if same-run p3/p4 wall or ctx moves positively.

Validation

Passed:

  • cargo fmt
  • cargo test -p utoo-pm service::clean::tests::test_clean_deps_skips_missing_node_modules
  • cargo clippy -p utoo-pm --all-targets -- -D warnings --no-deps

Full workspace clippy is not repeated here; it is currently blocked in this worktree by the known pack-core/next.js API mismatch observed on #2969 validation.

Benchmark Plan

Label-trigger GHA bench, compare p3/p4 against same-run utoo-next, utoo-npm, and bun.

@elrrrrrrr elrrrrrrr added A-Pkg Manager Area: Package Manager benchmark Run pm-bench on PR labels May 18, 2026
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the clean_deps function to skip missing node_modules directories and adds a corresponding test case to verify this behavior. The review feedback highlights a cross-platform compatibility issue regarding hardcoded path separators in existing logic and suggests further performance optimizations by reordering the existence checks and deferring the construction of the valid_packages set.

}

for nm_dir in &nm_dirs {
if !crate::fs::try_exists(nm_dir).await? {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Note that the cleaning logic following this check (specifically remove_unused_packages and its dependency path_to_pkg_name) contains hardcoded forward slashes (e.g., node_modules/ at line 336) and lacks path separator normalization when checking valid_packages. On Windows, where the path separator is , these checks will fail, effectively making the dependency cleaning a no-op. While this issue predates the current PR, it significantly impacts the correctness of the feature being optimized.

}

for nm_dir in &nm_dirs {
if !crate::fs::try_exists(nm_dir).await? {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This existence check is redundant for workspace node_modules directories, as they are already verified to exist at line 156 before being added to nm_dirs. For the root node_modules directory, it would be more efficient to perform this check at line 153 when it is first added to the vector. Additionally, the construction of the valid_packages set (lines 146-149) could be deferred until after verifying that at least one node_modules directory exists, which would further improve performance in the 'missing node_modules' scenario.

@github-actions
Copy link
Copy Markdown

📊 pm-bench-phases · 78453b3 · linux (ubuntu-latest)

Workflow run — ant-design

PMs: utoo (this branch) · utoo-npm (latest published) · bun (latest)

npmjs.org

p0_full_cold

PM wall ±σ user sys RSS pgMinor
bun 8.99s 0.23s 10.46s 10.16s 758M 344.2K
utoo-next 8.10s 0.10s 10.51s 12.26s 967M 124.6K
utoo-npm 8.00s 0.15s 10.99s 12.55s 993M 121.0K
utoo 8.45s 1.33s 11.61s 12.58s 930M 149.0K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 14.1K 17.7K 1.20G 6M 1.88G 1.76G 1M
utoo-next 117.8K 80.6K 1.17G 5M 1.73G 1.72G 2M
utoo-npm 132.7K 90.3K 1.17G 5M 1.73G 1.72G 2M
utoo 124.5K 90.0K 1.17G 6M 1.72G 1.72G 2M

p1_resolve

PM wall ±σ user sys RSS pgMinor
bun 1.89s 0.01s 4.09s 1.07s 536M 168.0K
utoo-next 2.96s 0.02s 5.49s 1.99s 615M 79.0K
utoo-npm 3.00s 0.03s 5.56s 2.00s 626M 77.9K
utoo 2.41s 0.03s 6.07s 1.65s 647M 121.3K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 7.8K 4.8K 203M 3M 108M - 1M
utoo-next 69.8K 85.8K 201M 2M 7M 3M 2M
utoo-npm 70.4K 89.7K 201M 2M 7M 3M 2M
utoo 15.2K 18.8K 204M 3M 7M 3M 2M

p3_cold_install

PM wall ±σ user sys RSS pgMinor
bun 7.31s 0.28s 6.37s 9.99s 607M 212.5K
utoo-next 6.85s 1.71s 5.07s 11.16s 499M 61.1K
utoo-npm 7.09s 1.78s 5.19s 11.09s 507M 60.2K
utoo 5.69s 0.17s 5.13s 11.39s 485M 59.8K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 4.5K 6.7K 1.00G 3M 1.77G 1.77G 1M
utoo-next 109.3K 48.8K 1000M 3M 1.72G 1.72G 3M
utoo-npm 107.3K 48.6K 1000M 3M 1.72G 1.72G 3M
utoo 120.3K 77.0K 1000M 3M 1.72G 1.72G 3M

p4_warm_link

PM wall ±σ user sys RSS pgMinor
bun 3.36s 0.06s 0.19s 2.42s 134M 31.6K
utoo-next 2.33s 0.05s 0.54s 3.89s 80M 18.4K
utoo-npm 2.31s 0.08s 0.53s 3.88s 80M 18.3K
utoo 2.33s 0.11s 0.56s 4.00s 63M 13.9K
PM vCtx iCtx netRX netTX cache node_mod lock
bun 254 23 27K 25K 1.88G 1.75G 1M
utoo-next 43.7K 19.1K 2K 6K 1.72G 1.72G 3M
utoo-npm 44.1K 20.9K 2K 19K 1.72G 1.72G 3M
utoo 54.8K 25.4K 2K 26K 1.73G 1.72G 3M

npmmirror.com: no output captured.

@elrrrrrrr
Copy link
Copy Markdown
Contributor Author

GHA run 1 read

Run: https://github.com/utooland/utoo/actions/runs/26017207217

phase utoo wall utoo ctx same-run utoo-next same-run utoo-npm same-run bun read
p3_cold_install 5.69s ±0.17 120.3K / 77.0K 6.85s, 109.3K / 48.8K 7.09s, 107.3K / 48.6K 7.31s, 4.5K / 6.7K wall looks positive, but ctx regresses materially
p4_warm_link 2.33s ±0.11 54.8K / 25.4K 2.33s, 43.7K / 19.1K 2.31s, 44.1K / 20.9K 3.36s, 254 / 23 no wall win and ctx worse

Conclusion: reject as a perf fold. The missing-node_modules clean skip does not reduce the p4 no-op path, and p3 wall alone is not enough when vCtx/iCtx are worse than same-run next/npm. The p3 wall improvement is likely run noise rather than the clean step being a real lever.

@elrrrrrrr
Copy link
Copy Markdown
Contributor Author

Closing this PM performance experiment after the investigation phase. The benchmark data and conclusions are preserved in the PR body/comments; we will split the validated pieces into smaller reviewable PRs for the formal ship path.

@elrrrrrrr elrrrrrrr closed this May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-Pkg Manager Area: Package Manager benchmark Run pm-bench on PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant