Refocus on wall-clock time and add cached clean benchmark for COMPILATION_CACHING#2
Merged
Refocus on wall-clock time and add cached clean benchmark for COMPILATION_CACHING#2
Conversation
Standard clean builds show cache-population overhead without the corresponding cache-hit benefit, making COMPILATION_CACHING appear to regress performance. The new "cached clean" phase measures clean builds with a warm compilation cache by deleting DerivedData (not the system compilation cache) between runs. This captures the realistic developer workflow: branch switching, pulling changes, and Clean Build Folder. Changes: - benchmark_builds.py: auto-detect COMPILATION_CACHING and run cached clean builds using a controlled DerivedData path - Schema bumped to 1.2.0 with optional cached_clean runs/summary - Report generator shows Cached Clean column when data is present - All skill docs and references updated to use the cached clean metric as the primary comparison for COMPILATION_CACHING impact
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
COMPILATION_CACHINGappear to regress performance. The new phase deletes DerivedData (not the system compilation cache) between runs, capturing the realistic developer workflow: branch switching, pulling changes, and Clean Build Folder.Changes across 15 files
Scripts:
benchmark_builds.py: auto-detectsCOMPILATION_CACHING = YESand runs cached clean builds using a controlled DerivedData path. New--no-cached-cleanflag to opt out.generate_optimization_report.py: shows "Cached Clean" column in baseline table when data is present; adds timing summary section for cached clean builds.Schema:
build-benchmark.schema.json: bumped to1.2.0with optionalcached_cleanin runs/summary and"cached-clean"build type.Skills & references:
recommendation-format.md: added requiredwait_time_impactfield.benchmark-artifacts.md: replaced "trust us" COMPILATION_CACHING caveat with cached clean documentation.build-settings-best-practices.md: added measurement note to COMPILATION_CACHING entry.orchestration-report-template.md: expanded baseline table, verification section, and community row template for cached clean.README.md: community results table includes Cached Clean columns.Test plan
benchmark_builds.pyon a project withoutCOMPILATION_CACHINGand verify output matches schema 1.1.0 (nocached_cleankey)benchmark_builds.pyon a project withCOMPILATION_CACHING = YESand verify cached clean runs appear in the artifactgenerate_optimization_report.pywith both artifact types and verify the Cached Clean column appears only when data exists--no-cached-cleanflag skips the cached clean phase even when COMPILATION_CACHING is detected