Skip to content

Commit 709c3f4

Browse files
brunoborgesCopilot
brunoborges
and
Copilot
committed
Move local benchmark results to LOCAL.md, keep README for CI docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 96992d6 commit 709c3f4

File tree

3 files changed

+84
-83
lines changed

3 files changed

+84
-83
lines changed

‎html-generators/benchmark/LOCAL.md‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Local Benchmark Results
2+
3+
Local benchmark results from `run.sh`. These will differ from CI because of OS file caching and warm `__pycache__/`.
4+
5+
## Phase 1: Training / Build Cost (one-time)
6+
7+
These are one-time setup costs, comparable across languages.
8+
9+
| Step | Time | What it does |
10+
|------|------|-------------|
11+
| Python first run | 1.98s | Interprets source, creates `__pycache__` bytecode |
12+
| JBang export | 2.19s | Compiles source + bundles dependencies into fat JAR |
13+
| AOT training run | 2.92s | Runs JAR once to record class loading, produces `.aot` cache |
14+
15+
## Phase 2: Steady-State Execution (avg of 5 runs)
16+
17+
After one-time setup, these are the per-run execution times.
18+
19+
| Method | Avg Time | Notes |
20+
|--------|---------|-------|
21+
| **Fat JAR + AOT** | **0.32s** | Fastest; pre-loaded classes from AOT cache |
22+
| **Fat JAR** | 0.44s | JVM class loading on every run |
23+
| **JBang** | 1.08s | Includes JBang launcher overhead |
24+
| **Python** | 1.26s | Uses cached `__pycache__` bytecode |
25+
26+
## Phase 3: CI Cold Start (simulated locally)
27+
28+
Clears `__pycache__/` and JBang cache, then measures a single run. On a local machine the OS file cache still helps, so these numbers are faster than true CI.
29+
30+
| Method | Time | Notes |
31+
|--------|------|-------|
32+
| **Fat JAR + AOT** | **0.46s** | AOT cache ships pre-loaded classes |
33+
| **Fat JAR** | 0.40s | JVM class loading from scratch |
34+
| **JBang** | 3.25s | Must compile source before running |
35+
| **Python** | 0.16s | No `__pycache__`; full interpretation |
36+
37+
## How each method works
38+
39+
- **Python** caches compiled bytecode in `__pycache__/` after the first run, similar to how Java's AOT cache works. But this cache is local-only and not available in CI.
40+
- **Java AOT** (JEP 483) snapshots ~3,300 pre-loaded classes from a training run into a `.aot` file, eliminating class loading overhead on subsequent runs. The `.aot` file is stored in the GitHub Actions cache.
41+
- **JBang** compiles and caches internally but adds launcher overhead on every invocation.
42+
- **Fat JAR** (`java -jar`) loads and links all classes from scratch each time.
43+
44+
## AOT Cache Setup
45+
46+
```bash
47+
# One-time: build the fat JAR
48+
jbang export fatjar --force --output html-generators/generate.jar html-generators/generate.java
49+
50+
# One-time: build the AOT cache (~21 MB, platform-specific)
51+
java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
52+
53+
# Steady-state: run with AOT cache
54+
java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar
55+
```
56+
57+
## Environment
58+
59+
| | |
60+
|---|---|
61+
| **CPU** | Apple M1 Max |
62+
| **RAM** | 32 GB |
63+
| **Java** | OpenJDK 25.0.1 (Temurin) |
64+
| **JBang** | 0.136.0 |
65+
| **Python** | 3.14.3 |
66+
| **OS** | Darwin |
67+
68+
## Reproduce
69+
70+
```bash
71+
./html-generators/benchmark/run.sh # print results to stdout
72+
./html-generators/benchmark/run.sh --update # also update this file
73+
```

‎html-generators/benchmark/README.md‎

Lines changed: 1 addition & 71 deletions
Original file line numberDiff line numberDiff line change
@@ -37,74 +37,4 @@ Python's `__pycache__/` serves a similar purpose — it caches compiled bytecode
3737

3838
## Local Benchmark
3939

40-
The local benchmark script runs all three phases on your development machine. Local results will differ from CI because of OS file caching and warm `__pycache__/`.
41-
42-
### Phase 1: Training / Build Cost (one-time)
43-
44-
These are one-time setup costs, comparable across languages.
45-
46-
| Step | Time | What it does |
47-
|------|------|-------------|
48-
| Python first run | 1.98s | Interprets source, creates `__pycache__` bytecode |
49-
| JBang export | 2.19s | Compiles source + bundles dependencies into fat JAR |
50-
| AOT training run | 2.92s | Runs JAR once to record class loading, produces `.aot` cache |
51-
52-
### Phase 2: Steady-State Execution (avg of 5 runs)
53-
54-
After one-time setup, these are the per-run execution times.
55-
56-
| Method | Avg Time | Notes |
57-
|--------|---------|-------|
58-
| **Fat JAR + AOT** | **0.32s** | Fastest; pre-loaded classes from AOT cache |
59-
| **Fat JAR** | 0.44s | JVM class loading on every run |
60-
| **JBang** | 1.08s | Includes JBang launcher overhead |
61-
| **Python** | 1.26s | Uses cached `__pycache__` bytecode |
62-
63-
### Phase 3: CI Cold Start (simulated locally)
64-
65-
Clears `__pycache__/` and JBang cache, then measures a single run. On a local machine the OS file cache still helps, so these numbers are faster than true CI.
66-
67-
| Method | Time | Notes |
68-
|--------|------|-------|
69-
| **Fat JAR + AOT** | **0.46s** | AOT cache ships pre-loaded classes |
70-
| **Fat JAR** | 0.40s | JVM class loading from scratch |
71-
| **JBang** | 3.25s | Must compile source before running |
72-
| **Python** | 0.16s | No `__pycache__`; full interpretation |
73-
74-
### How each method works
75-
76-
- **Python** caches compiled bytecode in `__pycache__/` after the first run, similar to how Java's AOT cache works. But this cache is local-only and not available in CI.
77-
- **Java AOT** (JEP 483) snapshots ~3,300 pre-loaded classes from a training run into a `.aot` file, eliminating class loading overhead on subsequent runs. The `.aot` file is stored in the GitHub Actions cache.
78-
- **JBang** compiles and caches internally but adds launcher overhead on every invocation.
79-
- **Fat JAR** (`java -jar`) loads and links all classes from scratch each time.
80-
81-
### AOT Cache Setup
82-
83-
```bash
84-
# One-time: build the fat JAR
85-
jbang export fatjar --force --output html-generators/generate.jar html-generators/generate.java
86-
87-
# One-time: build the AOT cache (~21 MB, platform-specific)
88-
java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar
89-
90-
# Steady-state: run with AOT cache
91-
java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar
92-
```
93-
94-
### Environment
95-
96-
| | |
97-
|---|---|
98-
| **CPU** | Apple M1 Max |
99-
| **RAM** | 32 GB |
100-
| **Java** | OpenJDK 25.0.1 (Temurin) |
101-
| **JBang** | 0.136.0 |
102-
| **Python** | 3.14.3 |
103-
| **OS** | Darwin |
104-
105-
### Reproduce
106-
107-
```bash
108-
./html-generators/benchmark/run.sh # print results to stdout
109-
./html-generators/benchmark/run.sh --update # also update local results in this file
110-
```
40+
See [LOCAL.md](LOCAL.md) for local benchmark results and instructions to run on your own machine.

‎html-generators/benchmark/run.sh‎

Lines changed: 10 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
#
1515
# Usage:
1616
# ./html-generators/benchmark/run.sh # print results to stdout
17-
# ./html-generators/benchmark/run.sh --update # also update README.md
17+
# ./html-generators/benchmark/run.sh --update # also update LOCAL.md
1818

1919
set -euo pipefail
2020
cd "$(git rev-parse --show-toplevel)"
@@ -136,14 +136,14 @@ echo " Fat JAR + AOT: ${AOT_CI}s"
136136
echo ""
137137

138138
# ---------------------------------------------------------------------------
139-
# Optionally update README.md
139+
# Optionally update LOCAL.md
140140
# ---------------------------------------------------------------------------
141141
if $UPDATE_MD; then
142-
MD="html-generators/benchmark/README.md"
142+
MD="html-generators/benchmark/LOCAL.md"
143143
cat > "$MD" <<EOF
144-
# Generator Benchmarks
144+
# Local Benchmark Results
145145
146-
Performance comparison of execution methods for the HTML generator, measured on $SNIPPET_COUNT snippets across 10 categories.
146+
Local benchmark results from \`run.sh\`. These will differ from CI because of OS file caching and warm \`__pycache__/\`.
147147
148148
## Phase 1: Training / Build Cost (one-time)
149149
@@ -166,11 +166,9 @@ After one-time setup, these are the per-run execution times.
166166
| **JBang** | ${JBANG_STEADY}s | Includes JBang launcher overhead |
167167
| **Python** | ${PY_STEADY}s | Uses cached \`__pycache__\` bytecode |
168168
169-
## Phase 3: CI Cold Start (fresh runner, no caches)
169+
## Phase 3: CI Cold Start (simulated locally)
170170
171-
Simulates a CI environment where every run is the first run.
172-
Python has no \`__pycache__\`, JBang has no compilation cache.
173-
Java AOT benefits from the pre-built \`.aot\` file restored from actions cache.
171+
Clears \`__pycache__/\` and JBang cache, then measures a single run. On a local machine the OS file cache still helps, so these numbers are faster than true CI.
174172
175173
| Method | Time | Notes |
176174
|--------|------|-------|
@@ -179,10 +177,10 @@ Java AOT benefits from the pre-built \`.aot\` file restored from actions cache.
179177
| **JBang** | ${JBANG_CI}s | Must compile source before running |
180178
| **Python** | ${PY_CI}s | No \`__pycache__\`; full interpretation |
181179
182-
## How It Works
180+
## How each method works
183181
184-
- **Python** caches compiled bytecode in \`__pycache__/\` after the first run, similar to how Java's AOT cache works.
185-
- **Java AOT** (JEP 483) snapshots ~3,300 pre-loaded classes from a training run into a \`.aot\` file, eliminating class loading overhead on subsequent runs.
182+
- **Python** caches compiled bytecode in \`__pycache__/\` after the first run, similar to how Java's AOT cache works. But this cache is local-only and not available in CI.
183+
- **Java AOT** (JEP 483) snapshots ~3,300 pre-loaded classes from a training run into a \`.aot\` file, eliminating class loading overhead on subsequent runs. The \`.aot\` file is stored in the GitHub Actions cache.
186184
- **JBang** compiles and caches internally but adds launcher overhead on every invocation.
187185
- **Fat JAR** (\`java -jar\`) loads and links all classes from scratch each time.
188186

0 commit comments

Comments
 (0)