Skip to content

Commit 5cd95bc

Browse files
brunoborgesCopilot
brunoborges
and
Copilot
committed
Move benchmark artifacts to html-generators/benchmark/ and add run.sh
- Replace build-cds.sh and build-aot.sh with a single run.sh script - run.sh rebuilds AOT cache, benchmarks all 4 methods (6 runs each), and optionally updates BENCHMARK.md with --update - Commit generate.aot (no longer gitignored) - Refresh BENCHMARK.md with 95-snippet results Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 3347ec4 commit 5cd95bc

File tree

5 files changed

+244
-70
lines changed

5 files changed

+244
-70
lines changed

‎html-generators/BENCHMARK.md‎

Lines changed: 0 additions & 50 deletions
This file was deleted.

‎html-generators/benchmark/BENCHMARK.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Generator Benchmarks
2+
3+
Performance comparison of the four ways to run the HTML generator, measured on 95 snippets across 10 categories.
4+
5+
## Results
6+
7+
| Method | Cold Start | Warm Average | Notes |
8+
|--------|-----------|-------------|-------|
9+
| **Fat JAR + AOT** (`java -XX:AOTCache`) | 0.47s | **0.43s** | Fastest overall; requires one-time cache build |
10+
| **Fat JAR** (`java -jar`) | 0.43s | 0.55s | No setup needed |
11+
| **JBang** (`jbang generate.java`) | 0.93s | 0.98s | Includes JBang overhead |
12+
| **Python** (`python3 generate.py`) | 0.37s | 1.60s | Fast cold start; slowest warm |
13+
14+
- **Cold start**: First run after clearing caches / fresh process
15+
- **Warm average**: Mean of 5 subsequent runs
16+
17+
## AOT Cache Setup
18+
19+
```bash
20+
# One-time: build the cache (~21 MB, platform-specific)
21+
java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
22+
23+
# Use it
24+
java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar
25+
```
26+
27+
The AOT cache uses Java 25 CDS (JEP 483) to pre-load classes from a training run. It is platform-specific (CPU arch + JDK version).
28+
29+
## Environment
30+
31+
| | |
32+
|---|---|
33+
| **CPU** | Apple M1 Max |
34+
| **RAM** | 32 GB |
35+
| **Java** | OpenJDK 25.0.1 (Temurin) |
36+
| **JBang** | 0.136.0 |
37+
| **Python** | 3.14.3 |
38+
| **OS** | Darwin |
39+
40+
## Methodology
41+
42+
Each method was timed 6 times using `/usr/bin/time -p`. The first run is reported as "cold start" and the remaining 5 runs are averaged for "warm average". Between each run, `site/index.html` was reset via `git checkout` to ensure the generator runs fully each time.
43+
44+
## Reproduce
45+
46+
```bash
47+
./html-generators/benchmark/run.sh # print results to stdout
48+
./html-generators/benchmark/run.sh --update # also update this file
49+
```

‎html-generators/benchmark/run.sh‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
#!/usr/bin/env bash
2+
# Benchmark the four ways to run the HTML generator.
3+
#
4+
# Setup: rebuilds the AOT cache before benchmarking.
5+
# Runs: 6 iterations per method (1 cold + 5 warm).
6+
# Output: table printed to stdout; optionally updates BENCHMARK.md.
7+
#
8+
# Usage:
9+
# ./html-generators/benchmark/run.sh # run all methods
10+
# ./html-generators/benchmark/run.sh --update # also update BENCHMARK.md
11+
12+
set -euo pipefail
13+
cd "$(git rev-parse --show-toplevel)"
14+
15+
JAR="html-generators/generate.jar"
16+
AOT="html-generators/generate.aot"
17+
RUNS=6
18+
UPDATE_MD=false
19+
[[ "${1:-}" == "--update" ]] && UPDATE_MD=true
20+
21+
# ---------------------------------------------------------------------------
22+
# Helpers
23+
# ---------------------------------------------------------------------------
24+
reset_site() { git checkout -- site/index.html 2>/dev/null || true; }
25+
26+
# Run a command $RUNS times, collect real times into $TIMES array
27+
bench() {
28+
local label="$1"; shift
29+
TIMES=()
30+
for ((i = 1; i <= RUNS; i++)); do
31+
reset_site
32+
local t
33+
t=$( { /usr/bin/time -p "$@" > /dev/null; } 2>&1 | awk '/^real/ {print $2}' )
34+
TIMES+=("$t")
35+
done
36+
local cold="${TIMES[0]}"
37+
local sum=0
38+
for ((i = 1; i < RUNS; i++)); do
39+
sum=$(echo "$sum + ${TIMES[$i]}" | bc)
40+
done
41+
local warm
42+
warm=$(echo "scale=2; $sum / ($RUNS - 1)" | bc | sed 's/^\./0./')
43+
printf "| %-42s | %5ss | **%5ss** |\n" "$label" "$cold" "$warm"
44+
# export for BENCHMARK.md update
45+
eval "${2//[^a-zA-Z]/_}_COLD=$cold"
46+
eval "${2//[^a-zA-Z]/_}_WARM=$warm"
47+
}
48+
49+
# ---------------------------------------------------------------------------
50+
# Environment
51+
# ---------------------------------------------------------------------------
52+
CPU=$(sysctl -n machdep.cpu.brand_string 2>/dev/null || lscpu 2>/dev/null | awk -F: '/Model name/ {gsub(/^ +/,"",$2); print $2}' || echo "unknown")
53+
RAM=$(sysctl -n hw.memsize 2>/dev/null | awk '{printf "%d GB", $1/1024/1024/1024}' || free -h 2>/dev/null | awk '/Mem:/ {print $2}' || echo "unknown")
54+
JAVA_VER=$(java -version 2>&1 | head -1 | sed 's/.*"\(.*\)".*/\1/')
55+
JBANG_VER=$(jbang version 2>/dev/null || echo "n/a")
56+
PYTHON_VER=$(python3 --version 2>/dev/null | awk '{print $2}' || echo "n/a")
57+
OS=$(uname -s)
58+
SNIPPET_COUNT=$(find content -name '*.json' | wc -l | tr -d ' ')
59+
60+
echo ""
61+
echo "Environment: $CPU · $RAM · Java $JAVA_VER · $OS"
62+
echo "Snippets: $SNIPPET_COUNT across 10 categories"
63+
echo ""
64+
65+
# ---------------------------------------------------------------------------
66+
# Setup: rebuild AOT cache
67+
# ---------------------------------------------------------------------------
68+
echo "=== Setup: building AOT cache ==="
69+
java -XX:AOTCacheOutput="$AOT" -jar "$JAR" > /dev/null 2>&1
70+
echo "Cache: $AOT ($(du -h "$AOT" | cut -f1 | tr -d ' '))"
71+
reset_site
72+
echo ""
73+
74+
# ---------------------------------------------------------------------------
75+
# Benchmark
76+
# ---------------------------------------------------------------------------
77+
echo "=== Benchmark ($RUNS runs each: 1 cold + $((RUNS - 1)) warm) ==="
78+
echo ""
79+
printf "| %-42s | %7s | %10s |\n" "Method" "Cold" "Warm Avg"
80+
printf "|%-44s|%9s|%12s|\n" "--------------------------------------------" "---------" "------------"
81+
82+
AOT_COLD="" ; AOT_WARM=""
83+
JAR_COLD="" ; JAR_WARM=""
84+
JBANG_COLD="" ; JBANG_WARM=""
85+
PY_COLD="" ; PY_WARM=""
86+
87+
# Fat JAR + AOT
88+
TIMES=()
89+
for ((i = 1; i <= RUNS; i++)); do
90+
reset_site
91+
t=$( { /usr/bin/time -p java -XX:AOTCache="$AOT" -jar "$JAR" > /dev/null; } 2>&1 | awk '/^real/ {print $2}' )
92+
TIMES+=("$t")
93+
done
94+
AOT_COLD="${TIMES[0]}"
95+
sum=0; for ((i = 1; i < RUNS; i++)); do sum=$(echo "$sum + ${TIMES[$i]}" | bc); done
96+
AOT_WARM=$(echo "scale=2; $sum / ($RUNS - 1)" | bc | sed 's/^\./0./')
97+
printf "| %-42s | %5ss | **%5ss** |\n" "**Fat JAR + AOT** (\`java -XX:AOTCache\`)" "$AOT_COLD" "$AOT_WARM"
98+
99+
# Fat JAR
100+
TIMES=()
101+
for ((i = 1; i <= RUNS; i++)); do
102+
reset_site
103+
t=$( { /usr/bin/time -p java -jar "$JAR" > /dev/null; } 2>&1 | awk '/^real/ {print $2}' )
104+
TIMES+=("$t")
105+
done
106+
JAR_COLD="${TIMES[0]}"
107+
sum=0; for ((i = 1; i < RUNS; i++)); do sum=$(echo "$sum + ${TIMES[$i]}" | bc); done
108+
JAR_WARM=$(echo "scale=2; $sum / ($RUNS - 1)" | bc | sed 's/^\./0./')
109+
printf "| %-42s | %5ss | **%5ss** |\n" "**Fat JAR** (\`java -jar\`)" "$JAR_COLD" "$JAR_WARM"
110+
111+
# JBang
112+
TIMES=()
113+
for ((i = 1; i <= RUNS; i++)); do
114+
reset_site
115+
t=$( { /usr/bin/time -p jbang html-generators/generate.java > /dev/null; } 2>&1 | awk '/^real/ {print $2}' )
116+
TIMES+=("$t")
117+
done
118+
JBANG_COLD="${TIMES[0]}"
119+
sum=0; for ((i = 1; i < RUNS; i++)); do sum=$(echo "$sum + ${TIMES[$i]}" | bc); done
120+
JBANG_WARM=$(echo "scale=2; $sum / ($RUNS - 1)" | bc | sed 's/^\./0./')
121+
printf "| %-42s | %5ss | **%5ss** |\n" "**JBang** (\`jbang generate.java\`)" "$JBANG_COLD" "$JBANG_WARM"
122+
123+
# Python
124+
TIMES=()
125+
for ((i = 1; i <= RUNS; i++)); do
126+
reset_site
127+
t=$( { /usr/bin/time -p python3 html-generators/generate.py > /dev/null; } 2>&1 | awk '/^real/ {print $2}' )
128+
TIMES+=("$t")
129+
done
130+
PY_COLD="${TIMES[0]}"
131+
sum=0; for ((i = 1; i < RUNS; i++)); do sum=$(echo "$sum + ${TIMES[$i]}" | bc); done
132+
PY_WARM=$(echo "scale=2; $sum / ($RUNS - 1)" | bc | sed 's/^\./0./')
133+
printf "| %-42s | %5ss | **%5ss** |\n" "**Python** (\`python3 generate.py\`)" "$PY_COLD" "$PY_WARM"
134+
135+
echo ""
136+
reset_site
137+
138+
# ---------------------------------------------------------------------------
139+
# Optionally update BENCHMARK.md
140+
# ---------------------------------------------------------------------------
141+
if $UPDATE_MD; then
142+
MD="html-generators/benchmark/BENCHMARK.md"
143+
cat > "$MD" <<EOF
144+
# Generator Benchmarks
145+
146+
Performance comparison of the four ways to run the HTML generator, measured on $SNIPPET_COUNT snippets across 10 categories.
147+
148+
## Results
149+
150+
| Method | Cold Start | Warm Average | Notes |
151+
|--------|-----------|-------------|-------|
152+
| **Fat JAR + AOT** (\`java -XX:AOTCache\`) | ${AOT_COLD}s | **${AOT_WARM}s** | Fastest overall; requires one-time cache build |
153+
| **Fat JAR** (\`java -jar\`) | ${JAR_COLD}s | ${JAR_WARM}s | No setup needed |
154+
| **JBang** (\`jbang generate.java\`) | ${JBANG_COLD}s | ${JBANG_WARM}s | Includes JBang overhead |
155+
| **Python** (\`python3 generate.py\`) | ${PY_COLD}s | ${PY_WARM}s | Fast cold start; slowest warm |
156+
157+
- **Cold start**: First run after clearing caches / fresh process
158+
- **Warm average**: Mean of $((RUNS - 1)) subsequent runs
159+
160+
## AOT Cache Setup
161+
162+
\`\`\`bash
163+
# One-time: build the cache (~21 MB, platform-specific)
164+
java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
165+
166+
# Use it
167+
java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar
168+
\`\`\`
169+
170+
The AOT cache uses Java 25 CDS (JEP 483) to pre-load classes from a training run. It is platform-specific (CPU arch + JDK version).
171+
172+
## Environment
173+
174+
| | |
175+
|---|---|
176+
| **CPU** | $CPU |
177+
| **RAM** | $RAM |
178+
| **Java** | OpenJDK $JAVA_VER (Temurin) |
179+
| **JBang** | $JBANG_VER |
180+
| **Python** | $PYTHON_VER |
181+
| **OS** | $OS |
182+
183+
## Methodology
184+
185+
Each method was timed $RUNS times using \`/usr/bin/time -p\`. The first run is reported as "cold start" and the remaining $((RUNS - 1)) runs are averaged for "warm average". Between each run, \`site/index.html\` was reset via \`git checkout\` to ensure the generator runs fully each time.
186+
187+
## Reproduce
188+
189+
\`\`\`bash
190+
./html-generators/benchmark/run.sh # print results to stdout
191+
./html-generators/benchmark/run.sh --update # also update this file
192+
\`\`\`
193+
EOF
194+
echo "Updated $MD"
195+
fi

‎html-generators/build-cds.sh‎

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎html-generators/generate.aot‎

21.5 MB
Binary file not shown.

0 commit comments

Comments
 (0)