Restructure benchmark into training/build cost vs steady-state execution

Phase 1: one-time costs (Python __pycache__, JBang export, AOT training)
Phase 2: steady-state avg of 5 runs (FAT JAR+AOT, FAT JAR, JBang, Python)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: brunoborges

.github/workflows/benchmark.yml

@@ -28,18 +28,12 @@ jobs:
         with:
           python-version: '3.13'
 
-      - name: Build fat JAR
-        run: jbang export fatjar --force --output html-generators/generate.jar html-generators/generate.java
-
-      - name: Build AOT cache
-        run: java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
-
       - name: Run benchmark
         shell: bash
         run: |
           JAR="html-generators/generate.jar"
           AOT="html-generators/generate.aot"
-          RUNS=6
+          STEADY_RUNS=5
 
           snippet_count=$(find content -name '*.json' | wc -l | tr -d ' ')
           java_ver=$(java -version 2>&1 | head -1 | sed 's/.*"\(.*\)".*/\1/')
@@ -53,40 +47,53 @@ jobs:
             echo "$t"
           }
 
-          bench() {
-            local label="$1"; shift
-            local times=()
-            for ((i = 1; i <= RUNS; i++)); do
-              t=$(measure "$@")
-              times+=("$t")
-            done
-            local cold="${times[0]}"
+          avg_runs() {
+            local n="$1"; shift
             local sum=0
-            for ((i = 1; i < RUNS; i++)); do
-              sum=$(awk "BEGIN {print $sum + ${times[$i]}}")
+            for ((i = 1; i <= n; i++)); do
+              local t
+              t=$(measure "$@")
+              sum=$(awk "BEGIN {print $sum + $t}")
             done
-            local warm
-            warm=$(awk "BEGIN {printf \"%.2f\", $sum / ($RUNS - 1)}")
-            echo "${label}|${cold}|${warm}"
+            awk "BEGIN {printf \"%.2f\", $sum / $n}"
           }
 
           echo "Running benchmark on $os_name (Java $java_ver, $snippet_count snippets)..."
 
-          aot_result=$(bench "Fat JAR + AOT" java -XX:AOTCache="$AOT" -jar "$JAR")
-          jar_result=$(bench "Fat JAR" java -jar "$JAR")
-          jbang_result=$(bench "JBang" jbang html-generators/generate.java)
-          python_result=$(bench "Python" python3 html-generators/generate.py)
+          # --- Phase 1: Training / build cost ---
+          rm -f "$JAR" "$AOT"
+          find html-generators -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true
+
+          PY_TRAIN=$(measure python3 html-generators/generate.py)
+          JBANG_EXPORT=$(measure jbang export fatjar --force --output "$JAR" html-generators/generate.java)
+          AOT_TRAIN=$(measure java -XX:AOTCacheOutput="$AOT" -jar "$JAR")
+
+          # --- Phase 2: Steady-state execution ---
+          PY_STEADY=$(avg_runs $STEADY_RUNS python3 html-generators/generate.py)
+          JBANG_STEADY=$(avg_runs $STEADY_RUNS jbang html-generators/generate.java)
+          JAR_STEADY=$(avg_runs $STEADY_RUNS java -jar "$JAR")
+          AOT_STEADY=$(avg_runs $STEADY_RUNS java -XX:AOTCache="$AOT" -jar "$JAR")
 
           # Write to GitHub Actions Job Summary
           {
             echo "## Benchmark Results — \`$os_name\`"
             echo ""
-            echo "Java $java_ver · $snippet_count snippets · $RUNS runs (1 cold + $((RUNS - 1)) warm)"
+            echo "Java $java_ver · $snippet_count snippets"
             echo ""
-            echo "| Method | Cold Start | Warm Average |"
-            echo "|--------|-----------|-------------|"
-            for result in "$aot_result" "$jar_result" "$jbang_result" "$python_result"; do
-              IFS='|' read -r label cold warm <<< "$result"
-              echo "| **$label** | ${cold}s | ${warm}s |"
-            done
+            echo "### Phase 1: Training / Build Cost (one-time)"
+            echo ""
+            echo "| Step | Time | What it does |"
+            echo "|------|------|-------------|"
+            echo "| Python first run | ${PY_TRAIN}s | Interprets source, creates \`__pycache__\` bytecode |"
+            echo "| JBang export | ${JBANG_EXPORT}s | Compiles source + bundles dependencies into fat JAR |"
+            echo "| AOT training run | ${AOT_TRAIN}s | Runs JAR once to record class loading, produces \`.aot\` cache |"
+            echo ""
+            echo "### Phase 2: Steady-State Execution (avg of $STEADY_RUNS runs)"
+            echo ""
+            echo "| Method | Avg Time |"
+            echo "|--------|---------|"
+            echo "| **Fat JAR + AOT** | **${AOT_STEADY}s** |"
+            echo "| **Fat JAR** | ${JAR_STEADY}s |"
+            echo "| **JBang** | ${JBANG_STEADY}s |"
+            echo "| **Python** | ${PY_STEADY}s |"
           } >> "$GITHUB_STEP_SUMMARY"

html-generators/benchmark/README.md

@@ -1,31 +1,48 @@
 # Generator Benchmarks
 
-Performance comparison of the four ways to run the HTML generator, measured on 95 snippets across 10 categories.
+Performance comparison of execution methods for the HTML generator, measured on 95 snippets across 10 categories.
 
-## Results
+## Phase 1: Training / Build Cost (one-time)
 
-| Method | Cold Start | Warm Average | Notes |
-|--------|-----------|-------------|-------|
-| **Fat JAR + AOT** (`java -XX:AOTCache`) | 0.47s | **0.43s** | Fastest overall; requires one-time cache build |
-| **Fat JAR** (`java -jar`) | 0.43s | 0.55s | No setup needed |
-| **JBang** (`jbang generate.java`) | 0.93s | 0.98s | Includes JBang overhead |
-| **Python** (`python3 generate.py`) | 0.37s | 1.60s | Fast cold start; slowest warm |
+These are one-time setup costs, comparable across languages.
 
-- **Cold start**: First run after clearing caches / fresh process
-- **Warm average**: Mean of 5 subsequent runs
+| Step | Time | What it does |
+|------|------|-------------|
+| Python first run | 2.34s | Interprets source, creates `__pycache__` bytecode |
+| JBang export | 2.92s | Compiles source + bundles dependencies into fat JAR |
+| AOT training run | 3.14s | Runs JAR once to record class loading, produces `.aot` cache |
+
+## Phase 2: Steady-State Execution (avg of 5 runs)
+
+After one-time setup, these are the per-run execution times.
+
+| Method | Avg Time | Notes |
+|--------|---------|-------|
+| **Fat JAR + AOT** | **0.35s** | Fastest; pre-loaded classes from AOT cache |
+| **Fat JAR** | 0.50s | JVM class loading on every run |
+| **JBang** | 1.19s | Includes JBang launcher overhead |
+| **Python** | 1.37s | Uses cached `__pycache__` bytecode |
+
+## How It Works
+
+- **Python** caches compiled bytecode in `__pycache__/` after the first run, similar to how Java's AOT cache works.
+- **Java AOT** (JEP 483) snapshots ~3,300 pre-loaded classes from a training run into a `.aot` file, eliminating class loading overhead on subsequent runs.
+- **JBang** compiles and caches internally but adds launcher overhead on every invocation.
+- **Fat JAR** (`java -jar`) loads and links all classes from scratch each time.
 
 ## AOT Cache Setup
 
 ```bash
-# One-time: build the cache (~21 MB, platform-specific)
+# One-time: build the fat JAR
+jbang export fatjar --force --output html-generators/generate.jar html-generators/generate.java
+
+# One-time: build the AOT cache (~21 MB, platform-specific)
 java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
 
-# Use it
+# Steady-state: run with AOT cache
 java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar
 ```
 
-The AOT cache uses Java 25 CDS (JEP 483) to pre-load classes from a training run. It is platform-specific (CPU arch + JDK version).
-
 ## Environment
 
 | | |
@@ -37,10 +54,6 @@ The AOT cache uses Java 25 CDS (JEP 483) to pre-load classes from a training run
 | **Python** | 3.14.3 |
 | **OS** | Darwin |
 
-## Methodology
-
-Each method was timed 6 times using `/usr/bin/time -p`. The first run is reported as "cold start" and the remaining 5 runs are averaged for "warm average". Between each run, `site/index.html` was reset via `git checkout` to ensure the generator runs fully each time.
-
 ## Reproduce
 
 ```bash