Skip to content

Latest commit

 

History

History
73 lines (53 loc) · 2.8 KB

LOCAL.md

File metadata and controls

73 lines (53 loc) · 2.8 KB

Local Benchmark Results

Local benchmark results from run.sh. These will differ from CI because of OS file caching and warm __pycache__/.

Phase 1: Training / Build Cost (one-time)

These are one-time setup costs, comparable across languages.

Step Time What it does
Python first run 1.98s Interprets source, creates __pycache__ bytecode
JBang export 2.19s Compiles source + bundles dependencies into fat JAR
AOT training run 2.92s Runs JAR once to record class loading, produces .aot cache

Phase 2: Steady-State Execution (avg of 5 runs)

After one-time setup, these are the per-run execution times.

Method Avg Time Notes
Fat JAR + AOT 0.32s Fastest; pre-loaded classes from AOT cache
Fat JAR 0.44s JVM class loading on every run
JBang 1.08s Includes JBang launcher overhead
Python 1.26s Uses cached __pycache__ bytecode

Phase 3: CI Cold Start (simulated locally)

Clears __pycache__/ and JBang cache, then measures a single run. On a local machine the OS file cache still helps, so these numbers are faster than true CI.

Method Time Notes
Fat JAR + AOT 0.46s AOT cache ships pre-loaded classes
Fat JAR 0.40s JVM class loading from scratch
JBang 3.25s Must compile source before running
Python 0.16s No __pycache__; full interpretation

How each method works

  • Python caches compiled bytecode in __pycache__/ after the first run, similar to how Java's AOT cache works. But this cache is local-only and not available in CI.
  • Java AOT (JEP 483) snapshots ~3,300 pre-loaded classes from a training run into a .aot file, eliminating class loading overhead on subsequent runs. The .aot file is stored in the GitHub Actions cache.
  • JBang compiles and caches internally but adds launcher overhead on every invocation.
  • Fat JAR (java -jar) loads and links all classes from scratch each time.

AOT Cache Setup

# One-time: build the fat JAR
jbang export fatjar --force --output html-generators/generate.jar html-generators/generate.java

# One-time: build the AOT cache (~21 MB, platform-specific)
java -XX:AOTCacheOutput=html-generators/generate.aot -jar html-generators/generate.jar

# Steady-state: run with AOT cache
java -XX:AOTCache=html-generators/generate.aot -jar html-generators/generate.jar

Environment
CPU Apple M1 Max
RAM 32 GB
Java OpenJDK 25.0.1 (Temurin)
JBang 0.136.0
Python 3.14.3
OS Darwin

Reproduce

./html-generators/benchmark/run.sh            # print results to stdout
./html-generators/benchmark/run.sh --update    # also update this file