Skip to content

Commit 5c9d966

Browse files
brunoborgesCopilot
brunoborges
and
Copilot
committed
Add OG card generator to benchmark workflow and run.sh
Benchmark both HTML generator (generate.java/py) and OG card generator (generateog.java/py) across all phases: training, steady-state, and CI cold start. Updates workflow, local run.sh, and README. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 804d62f commit 5c9d966

File tree

3 files changed

+156
-17
lines changed

3 files changed

+156
-17
lines changed

‎.github/workflows/benchmark.yml‎

Lines changed: 69 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
name: Benchmark Generator
1+
name: Benchmark Generators
22

33
on:
44
workflow_dispatch:
@@ -28,11 +28,16 @@ jobs:
2828
with:
2929
python-version: '3.13'
3030

31+
- name: Install Python dependencies
32+
run: pip install pyyaml cairosvg
33+
3134
- name: Run benchmark
3235
shell: bash
3336
run: |
3437
JAR="html-generators/generate.jar"
3538
AOT="html-generators/generate.aot"
39+
OG_JAR="html-generators/generateog.jar"
40+
OG_AOT="html-generators/generateog.aot"
3641
STEADY_RUNS=5
3742
3843
snippet_count=$(find content -name '*.json' | wc -l | tr -d ' ')
@@ -61,41 +66,75 @@ jobs:
6166
echo "Running benchmark on $os_name (Java $java_ver, $snippet_count snippets)..."
6267
6368
# --- Phase 1: Training / build cost ---
64-
rm -f "$JAR" "$AOT"
69+
rm -f "$JAR" "$AOT" "$OG_JAR" "$OG_AOT"
6570
find html-generators -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true
6671
72+
# HTML generator
6773
PY_TRAIN=$(measure python3 html-generators/generate.py)
6874
JBANG_EXPORT=$(measure jbang export fatjar --force --output "$JAR" html-generators/generate.java)
6975
AOT_TRAIN=$(measure java -XX:AOTCacheOutput="$AOT" -jar "$JAR")
7076
77+
# OG generator
78+
OG_PY_TRAIN=$(measure python3 html-generators/generateog.py)
79+
OG_JBANG_EXPORT=$(measure jbang export fatjar --force --output "$OG_JAR" html-generators/generateog.java)
80+
OG_AOT_TRAIN=$(measure java -XX:AOTCacheOutput="$OG_AOT" -jar "$OG_JAR")
81+
7182
# --- Phase 2: Steady-state execution ---
83+
# HTML generator
7284
PY_STEADY=$(avg_runs $STEADY_RUNS python3 html-generators/generate.py)
7385
JBANG_STEADY=$(avg_runs $STEADY_RUNS jbang html-generators/generate.java)
7486
JAR_STEADY=$(avg_runs $STEADY_RUNS java -jar "$JAR")
7587
AOT_STEADY=$(avg_runs $STEADY_RUNS java -XX:AOTCache="$AOT" -jar "$JAR")
7688
89+
# OG generator
90+
OG_PY_STEADY=$(avg_runs $STEADY_RUNS python3 html-generators/generateog.py)
91+
OG_JBANG_STEADY=$(avg_runs $STEADY_RUNS jbang html-generators/generateog.java)
92+
OG_JAR_STEADY=$(avg_runs $STEADY_RUNS java -jar "$OG_JAR")
93+
OG_AOT_STEADY=$(avg_runs $STEADY_RUNS java -XX:AOTCache="$OG_AOT" -jar "$OG_JAR")
94+
7795
# Write to GitHub Actions Job Summary
7896
{
7997
echo "## Benchmark Results — \`$os_name\`"
8098
echo ""
8199
echo "Java $java_ver · $snippet_count snippets"
82100
echo ""
83-
echo "### Phase 1: Training / Build Cost (one-time)"
101+
echo "### HTML Generator"
102+
echo ""
103+
echo "#### Phase 1: Training / Build Cost (one-time)"
84104
echo ""
85105
echo "| Step | Time | What it does |"
86106
echo "|------|------|-------------|"
87107
echo "| Python first run | ${PY_TRAIN}s | Interprets source, creates \`__pycache__\` bytecode |"
88108
echo "| JBang export | ${JBANG_EXPORT}s | Compiles source + bundles dependencies into fat JAR |"
89109
echo "| AOT training run | ${AOT_TRAIN}s | Runs JAR once to record class loading, produces \`.aot\` cache |"
90110
echo ""
91-
echo "### Phase 2: Steady-State Execution (avg of $STEADY_RUNS runs)"
111+
echo "#### Phase 2: Steady-State Execution (avg of $STEADY_RUNS runs)"
92112
echo ""
93113
echo "| Method | Avg Time |"
94114
echo "|--------|---------|"
95115
echo "| **Fat JAR + AOT** | **${AOT_STEADY}s** |"
96116
echo "| **Fat JAR** | ${JAR_STEADY}s |"
97117
echo "| **JBang** | ${JBANG_STEADY}s |"
98118
echo "| **Python** | ${PY_STEADY}s |"
119+
echo ""
120+
echo "### OG Card Generator"
121+
echo ""
122+
echo "#### Phase 1: Training / Build Cost (one-time)"
123+
echo ""
124+
echo "| Step | Time | What it does |"
125+
echo "|------|------|-------------|"
126+
echo "| Python first run | ${OG_PY_TRAIN}s | Interprets source, generates SVG+PNG via cairosvg |"
127+
echo "| JBang export | ${OG_JBANG_EXPORT}s | Compiles source + bundles Batik dependencies into fat JAR |"
128+
echo "| AOT training run | ${OG_AOT_TRAIN}s | Runs JAR once to record class loading, produces \`.aot\` cache |"
129+
echo ""
130+
echo "#### Phase 2: Steady-State Execution (avg of $STEADY_RUNS runs)"
131+
echo ""
132+
echo "| Method | Avg Time |"
133+
echo "|--------|---------|"
134+
echo "| **Fat JAR + AOT** | **${OG_AOT_STEADY}s** |"
135+
echo "| **Fat JAR** | ${OG_JAR_STEADY}s |"
136+
echo "| **JBang** | ${OG_JBANG_STEADY}s |"
137+
echo "| **Python** | ${OG_PY_STEADY}s |"
99138
} >> "$GITHUB_STEP_SUMMARY"
100139
101140
# ---------------------------------------------------------------------------
@@ -119,10 +158,12 @@ jobs:
119158

120159
- uses: jbangdev/setup-jbang@main
121160

122-
- name: Build fat JAR and AOT cache
161+
- name: Build fat JARs and AOT caches
123162
run: |
124163
jbang export fatjar --force --output html-generators/generate.jar html-generators/generate.java
125164
java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
165+
jbang export fatjar --force --output html-generators/generateog.jar html-generators/generateog.java
166+
java -XX:AOTCacheOutput=html-generators/generateog.aot -jar html-generators/generateog.jar
126167
127168
- name: Upload JAR and AOT
128169
uses: actions/upload-artifact@v4
@@ -131,6 +172,8 @@ jobs:
131172
path: |
132173
html-generators/generate.jar
133174
html-generators/generate.aot
175+
html-generators/generateog.jar
176+
html-generators/generateog.aot
134177
135178
ci-cold-start:
136179
needs: build-jar
@@ -151,6 +194,9 @@ jobs:
151194
with:
152195
python-version: '3.13'
153196

197+
- name: Install Python dependencies
198+
run: pip install pyyaml cairosvg
199+
154200
- name: Download JAR and AOT
155201
uses: actions/download-artifact@v4
156202
with:
@@ -172,21 +218,38 @@ jobs:
172218
}
173219
174220
# Everything is cold: no __pycache__, no JBang cache, fresh JVM
221+
222+
# HTML generator
175223
PY_CI=$(measure python3 html-generators/generate.py)
176224
JAR_CI=$(measure java -jar html-generators/generate.jar)
177225
AOT_CI=$(measure java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar)
178226
227+
# OG generator
228+
OG_PY_CI=$(measure python3 html-generators/generateog.py)
229+
OG_JAR_CI=$(measure java -jar html-generators/generateog.jar)
230+
OG_AOT_CI=$(measure java -XX:AOTCache=html-generators/generateog.aot -jar html-generators/generateog.jar)
231+
179232
{
180233
echo "## CI Cold Start — \`$os_name\`"
181234
echo ""
182235
echo "Java $java_ver · $snippet_count snippets"
183236
echo ""
184-
echo "Fresh runner, no caches. JAR and AOT restored from artifact"
237+
echo "Fresh runner, no caches. JARs and AOT restored from artifact"
185238
echo "(simulates actions cache restore in deploy workflow)."
186239
echo ""
240+
echo "### HTML Generator"
241+
echo ""
187242
echo "| Method | Time |"
188243
echo "|--------|------|"
189244
echo "| **Fat JAR + AOT** | **${AOT_CI}s** |"
190245
echo "| **Fat JAR** | ${JAR_CI}s |"
191246
echo "| **Python** | ${PY_CI}s |"
247+
echo ""
248+
echo "### OG Card Generator"
249+
echo ""
250+
echo "| Method | Time |"
251+
echo "|--------|------|"
252+
echo "| **Fat JAR + AOT** | **${OG_AOT_CI}s** |"
253+
echo "| **Fat JAR** | ${OG_JAR_CI}s |"
254+
echo "| **Python** | ${OG_PY_CI}s |"
192255
} >> "$GITHUB_STEP_SUMMARY"

‎html-generators/benchmark/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Generator Benchmarks
22

3-
Performance comparison of execution methods for the HTML generator, measured on 95 snippets across 10 categories.
3+
Performance comparison of execution methods for the HTML and OG card generators, measured on 112 snippets across 11 categories.
44

55
## CI Benchmark (GitHub Actions)
66

‎html-generators/benchmark/run.sh‎

Lines changed: 86 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
#!/usr/bin/env bash
2-
# Benchmark the HTML generator across languages and execution methods.
2+
# Benchmark the HTML and OG card generators across languages and execution methods.
33
#
44
# Phase 1: Training/build cost (one-time setup)
55
# - Python first run (creates __pycache__)
@@ -21,6 +21,8 @@ cd "$(git rev-parse --show-toplevel)"
2121

2222
JAR="html-generators/generate.jar"
2323
AOT="html-generators/generate.aot"
24+
OG_JAR="html-generators/generateog.jar"
25+
OG_AOT="html-generators/generateog.aot"
2426
STEADY_RUNS=5
2527
UPDATE_MD=false
2628
[[ "${1:-}" == "--update" ]] && UPDATE_MD=true
@@ -66,7 +68,7 @@ SNIPPET_COUNT=$(find content -name '*.json' | wc -l | tr -d ' ')
6668

6769
echo ""
6870
echo "Environment: $CPU · $RAM · Java $JAVA_VER · $OS"
69-
echo "Snippets: $SNIPPET_COUNT across 10 categories"
71+
echo "Snippets: $SNIPPET_COUNT across 11 categories"
7072
echo ""
7173

7274
# ---------------------------------------------------------------------------
@@ -77,6 +79,7 @@ echo ""
7779

7880
# Clean up any cached state
7981
rm -f html-generators/generate.aot html-generators/generate.jar
82+
rm -f html-generators/generateog.aot html-generators/generateog.jar
8083
find html-generators -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true
8184

8285
# Python first run (populates __pycache__)
@@ -91,6 +94,19 @@ echo " JBang export (creates fat JAR): ${JBANG_EXPORT}s"
9194
AOT_TRAIN=$(measure java -XX:AOTCacheOutput="$AOT" -jar "$JAR")
9295
echo " AOT training run (creates .aot): ${AOT_TRAIN}s"
9396

97+
echo ""
98+
echo "--- OG Card Generator ---"
99+
echo ""
100+
101+
OG_PY_TRAIN=$(measure python3 html-generators/generateog.py)
102+
echo " Python first run (creates __pycache__): ${OG_PY_TRAIN}s"
103+
104+
OG_JBANG_EXPORT=$(measure jbang export fatjar --force --output "$OG_JAR" html-generators/generateog.java)
105+
echo " JBang export (creates fat JAR): ${OG_JBANG_EXPORT}s"
106+
107+
OG_AOT_TRAIN=$(measure java -XX:AOTCacheOutput="$OG_AOT" -jar "$OG_JAR")
108+
echo " AOT training run (creates .aot): ${OG_AOT_TRAIN}s"
109+
94110
echo ""
95111

96112
# ---------------------------------------------------------------------------
@@ -111,6 +127,22 @@ echo " Fat JAR: ${JAR_STEADY}s"
111127
AOT_STEADY=$(avg_runs $STEADY_RUNS java -XX:AOTCache="$AOT" -jar "$JAR")
112128
echo " Fat JAR + AOT: ${AOT_STEADY}s"
113129

130+
echo ""
131+
echo "--- OG Card Generator ---"
132+
echo ""
133+
134+
OG_PY_STEADY=$(avg_runs $STEADY_RUNS python3 html-generators/generateog.py)
135+
echo " Python (warm): ${OG_PY_STEADY}s"
136+
137+
OG_JBANG_STEADY=$(avg_runs $STEADY_RUNS jbang html-generators/generateog.java)
138+
echo " JBang (from source): ${OG_JBANG_STEADY}s"
139+
140+
OG_JAR_STEADY=$(avg_runs $STEADY_RUNS java -jar "$OG_JAR")
141+
echo " Fat JAR: ${OG_JAR_STEADY}s"
142+
143+
OG_AOT_STEADY=$(avg_runs $STEADY_RUNS java -XX:AOTCache="$OG_AOT" -jar "$OG_JAR")
144+
echo " Fat JAR + AOT: ${OG_AOT_STEADY}s"
145+
114146
echo ""
115147

116148
# ---------------------------------------------------------------------------
@@ -133,6 +165,20 @@ echo " Fat JAR: ${JAR_CI}s"
133165
AOT_CI=$(measure java -XX:AOTCache="$AOT" -jar "$JAR")
134166
echo " Fat JAR + AOT: ${AOT_CI}s"
135167

168+
echo ""
169+
echo "--- OG Card Generator ---"
170+
echo ""
171+
172+
find html-generators -name '__pycache__' -type d -exec rm -rf {} + 2>/dev/null || true
173+
OG_PY_CI=$(measure python3 html-generators/generateog.py)
174+
echo " Python (no __pycache__): ${OG_PY_CI}s"
175+
176+
OG_JAR_CI=$(measure java -jar "$OG_JAR")
177+
echo " Fat JAR: ${OG_JAR_CI}s"
178+
179+
OG_AOT_CI=$(measure java -XX:AOTCache="$OG_AOT" -jar "$OG_JAR")
180+
echo " Fat JAR + AOT: ${OG_AOT_CI}s"
181+
136182
echo ""
137183

138184
# ---------------------------------------------------------------------------
@@ -145,7 +191,9 @@ if $UPDATE_MD; then
145191
146192
Local benchmark results from \`run.sh\`. These will differ from CI because of OS file caching and warm \`__pycache__/\`.
147193
148-
## Phase 1: Training / Build Cost (one-time)
194+
## HTML Generator
195+
196+
### Phase 1: Training / Build Cost (one-time)
149197
150198
These are one-time setup costs, comparable across languages.
151199
@@ -155,7 +203,7 @@ These are one-time setup costs, comparable across languages.
155203
| JBang export | ${JBANG_EXPORT}s | Compiles source + bundles dependencies into fat JAR |
156204
| AOT training run | ${AOT_TRAIN}s | Runs JAR once to record class loading, produces \`.aot\` cache |
157205
158-
## Phase 2: Steady-State Execution (avg of $STEADY_RUNS runs)
206+
### Phase 2: Steady-State Execution (avg of $STEADY_RUNS runs)
159207
160208
After one-time setup, these are the per-run execution times.
161209
@@ -166,7 +214,7 @@ After one-time setup, these are the per-run execution times.
166214
| **JBang** | ${JBANG_STEADY}s | Includes JBang launcher overhead |
167215
| **Python** | ${PY_STEADY}s | Uses cached \`__pycache__\` bytecode |
168216
169-
## Phase 3: CI Cold Start (simulated locally)
217+
### Phase 3: CI Cold Start (simulated locally)
170218
171219
Clears \`__pycache__/\` and JBang cache, then measures a single run. On a local machine the OS file cache still helps, so these numbers are faster than true CI.
172220
@@ -177,6 +225,33 @@ Clears \`__pycache__/\` and JBang cache, then measures a single run. On a local
177225
| **JBang** | ${JBANG_CI}s | Must compile source before running |
178226
| **Python** | ${PY_CI}s | No \`__pycache__\`; full interpretation |
179227
228+
## OG Card Generator
229+
230+
### Phase 1: Training / Build Cost (one-time)
231+
232+
| Step | Time | What it does |
233+
|------|------|-------------|
234+
| Python first run | ${OG_PY_TRAIN}s | Generates SVG+PNG via cairosvg |
235+
| JBang export | ${OG_JBANG_EXPORT}s | Compiles source + bundles Batik dependencies into fat JAR |
236+
| AOT training run | ${OG_AOT_TRAIN}s | Runs JAR once to record class loading, produces \`.aot\` cache |
237+
238+
### Phase 2: Steady-State Execution (avg of $STEADY_RUNS runs)
239+
240+
| Method | Avg Time | Notes |
241+
|--------|---------|-------|
242+
| **Fat JAR + AOT** | **${OG_AOT_STEADY}s** | Fastest; pre-loaded classes from AOT cache |
243+
| **Fat JAR** | ${OG_JAR_STEADY}s | JVM class loading on every run |
244+
| **JBang** | ${OG_JBANG_STEADY}s | Includes JBang launcher overhead |
245+
| **Python** | ${OG_PY_STEADY}s | Uses cached \`__pycache__\` bytecode |
246+
247+
### Phase 3: CI Cold Start (simulated locally)
248+
249+
| Method | Time | Notes |
250+
|--------|------|-------|
251+
| **Fat JAR + AOT** | **${OG_AOT_CI}s** | AOT cache ships pre-loaded classes |
252+
| **Fat JAR** | ${OG_JAR_CI}s | JVM class loading from scratch |
253+
| **Python** | ${OG_PY_CI}s | No \`__pycache__\`; full interpretation |
254+
180255
## How each method works
181256
182257
- **Python** caches compiled bytecode in \`__pycache__/\` after the first run, similar to how Java's AOT cache works. But this cache is local-only and not available in CI.
@@ -187,14 +262,15 @@ Clears \`__pycache__/\` and JBang cache, then measures a single run. On a local
187262
## AOT Cache Setup
188263
189264
\`\`\`bash
190-
# One-time: build the fat JAR
265+
# HTML generator
191266
jbang export fatjar --force --output html-generators/generate.jar html-generators/generate.java
192-
193-
# One-time: build the AOT cache (~21 MB, platform-specific)
194267
java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
195-
196-
# Steady-state: run with AOT cache
197268
java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar
269+
270+
# OG card generator
271+
jbang export fatjar --force --output html-generators/generateog.jar html-generators/generateog.java
272+
java -XX:AOTCacheOutput=html-generators/generateog.aot -jar html-generators/generateog.jar
273+
java -XX:AOTCache=html-generators/generateog.aot -jar html-generators/generateog.jar
198274
\`\`\`
199275
200276
## Environment

0 commit comments

Comments
 (0)