Skip to content

Refocus on wall-clock time and add cached clean benchmark for COMPILATION_CACHING#2

Merged
AvdLee merged 2 commits intomainfrom
wall-time-refocus
Mar 23, 2026
Merged

Refocus on wall-clock time and add cached clean benchmark for COMPILATION_CACHING#2
AvdLee merged 2 commits intomainfrom
wall-time-refocus

Conversation

@AvdLee
Copy link
Copy Markdown
Owner

@AvdLee AvdLee commented Mar 23, 2026

Summary

  • Refocus all skills on wall-clock build time as the primary success metric. Cumulative task-time savings are diagnostic evidence, not proof of wait-time improvement. Every recommendation now states its expected wall-clock impact using standardized impact language.
  • Add cached clean benchmark that measures clean builds with a warm compilation cache. Standard clean builds show cache-population overhead without cache-hit benefit, making COMPILATION_CACHING appear to regress performance. The new phase deletes DerivedData (not the system compilation cache) between runs, capturing the realistic developer workflow: branch switching, pulling changes, and Clean Build Folder.

Changes across 15 files

Scripts:

  • benchmark_builds.py: auto-detects COMPILATION_CACHING = YES and runs cached clean builds using a controlled DerivedData path. New --no-cached-clean flag to opt out.
  • generate_optimization_report.py: shows "Cached Clean" column in baseline table when data is present; adds timing summary section for cached clean builds.

Schema:

  • build-benchmark.schema.json: bumped to 1.2.0 with optional cached_clean in runs/summary and "cached-clean" build type.

Skills & references:

  • All six skill SKILL.md files updated with wall-clock-first prioritization and cached clean awareness.
  • recommendation-format.md: added required wait_time_impact field.
  • benchmark-artifacts.md: replaced "trust us" COMPILATION_CACHING caveat with cached clean documentation.
  • build-settings-best-practices.md: added measurement note to COMPILATION_CACHING entry.
  • orchestration-report-template.md: expanded baseline table, verification section, and community row template for cached clean.
  • README.md: community results table includes Cached Clean columns.

Test plan

  • Run benchmark_builds.py on a project without COMPILATION_CACHING and verify output matches schema 1.1.0 (no cached_clean key)
  • Run benchmark_builds.py on a project with COMPILATION_CACHING = YES and verify cached clean runs appear in the artifact
  • Run generate_optimization_report.py with both artifact types and verify the Cached Clean column appears only when data exists
  • Verify --no-cached-clean flag skips the cached clean phase even when COMPILATION_CACHING is detected

AvdLee and others added 2 commits March 23, 2026 13:59
Standard clean builds show cache-population overhead without the
corresponding cache-hit benefit, making COMPILATION_CACHING appear
to regress performance. The new "cached clean" phase measures clean
builds with a warm compilation cache by deleting DerivedData (not
the system compilation cache) between runs. This captures the
realistic developer workflow: branch switching, pulling changes,
and Clean Build Folder.

Changes:
- benchmark_builds.py: auto-detect COMPILATION_CACHING and run
  cached clean builds using a controlled DerivedData path
- Schema bumped to 1.2.0 with optional cached_clean runs/summary
- Report generator shows Cached Clean column when data is present
- All skill docs and references updated to use the cached clean
  metric as the primary comparison for COMPILATION_CACHING impact
@AvdLee AvdLee merged commit 1c390da into main Mar 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant