Skip to content

[GR-21590] Update imports and benchmark CI configs #801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
graalvmbot wants to merge 9 commits into
base: master
Choose a base branch
from
Open

[GR-21590] Update imports and benchmark CI configs #801

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4925942
Update unittest tags
timfel May 4, 2026
80ce89d
Get rid of all CE benchmarks - that is just not a supported configura…
timfel May 4, 2026
246b2c0
Get rid of multi-tier, 3-compiler-threads, and panama benchmarks
timfel May 4, 2026
50d8eaf
Minor convenience for rota-bench-regression-analysis skill
timfel May 5, 2026
d5f6655
Skip test involving native arrays in sandboxed mode
timfel May 6, 2026
8d8e6ae
Fix expected recursion limit in tests for GraalPy
timfel May 6, 2026
b5ba78c
Fix PolyBench cext package venv path
timfel May 6, 2026
841a300
Handle singleton C API decrefs
timfel May 6, 2026
fe52218
untag recursive test on windows
timfel May 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .agents/skills/rota-bench-regression-analysis/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,4 +86,5 @@ git diff --stat GOOD..BAD

## Guardrails
- If the script or you can't find `bench-cli`, ask the user to provide it from the `bench-server` repo.
- You may offer to clone the repo and create the cli for the user. The repo is on the same bitbucket as graalpython, the project is `INFRA` and the repo is called `bench-server`
- Don't submit more than 5 bisect jobs. If there are more in the "to bisect" list, pick 5 that look the most serious and leave the rest as "to bisect".
35 changes: 1 addition & 34 deletions ci.jsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -303,9 +303,7 @@
// not specified as the first arg to `bench_task`.
local bench_task_dict = {
[bench]: bench_task(bench) + platform_spec(no_jobs) + bench_variants({
"vm_name:graalvm_ce_default" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalvm_ee_default" : {"linux:amd64:jdk-latest" : post_merge + t("08:00:00") + need_pgo},
"vm_name:graalpython_core" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalpython_enterprise" : {"linux:amd64:jdk-latest" : daily + t("08:00:00"),
"job_type:checkup" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")}
},
Expand All @@ -316,15 +314,11 @@
for bench in ["micro", "meso", "macro"]
} + {
[bench]: bench_task(bench) + platform_spec(no_jobs) + bench_variants({
"vm_name:graalvm_ce_default" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalvm_ee_default" : {"linux:amd64:jdk-latest" : post_merge + t("08:00:00") + need_pgo},
"vm_name:graalpython_core" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalpython_core_panama" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalpython_enterprise" : {"linux:amd64:jdk-latest" : daily + t("08:00:00"),
"job_type:checkup" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")}
},
"vm_name:graalpython_enterprise_multi" : {"linux:amd64:jdk-latest" : weekly + t("08:00:00")},
"vm_name:graalpython_enterprise_panama" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:cpython" : {"linux:amd64:jdk-latest" : monthly + t("04:00:00")},
"vm_name:pypy" : {"linux:amd64:jdk-latest" : on_demand + t("04:00:00")},
}),
Expand All @@ -333,41 +327,22 @@
// "small" benchmarks have their argument set such that they run in a resonable
// time in the interpreter and they are used for interpreter benchmarking
[bench]: bench_task(bench) + platform_spec(no_jobs) + bench_variants({
"vm_name:graalvm_ce_default_interpreter" : {"linux:amd64:jdk-latest" : on_demand + t("02:00:00")},
"vm_name:graalvm_ee_default_interpreter" : {"linux:amd64:jdk-latest" : daily + t("02:00:00") + need_pgo},
"vm_name:graalvm_ee_default_interpreter_uncached" : {"linux:amd64:jdk-latest" : daily + t("02:00:00") + need_pgo},
"vm_name:graalpython_core_interpreter" : {"linux:amd64:jdk-latest" : on_demand + t("02:00:00")},
"vm_name:graalpython_core_native_interpreter" : {"linux:amd64:jdk-latest" : on_demand + t("02:00:00")},
"vm_name:graalpython_enterprise_interpreter" : {"linux:amd64:jdk-latest" : weekly + t("02:00:00")},
"vm_name:graalpython_core_interpreter_multi" : {"linux:amd64:jdk-latest" : on_demand + t("02:00:00")},
"vm_name:graalpython_core_native_interpreter_multi" : {"linux:amd64:jdk-latest" : on_demand + t("02:00:00")},
"vm_name:cpython" : {"linux:amd64:jdk-latest" : weekly + t("02:00:00")},
}),
for bench in ["micro_small", "meso_small"]
} + {
// benchmarks executed via Java embedding driver
[bench]: bench_task(bench) + platform_spec(no_jobs) + bench_variants({
"vm_name:java_embedding_core_interpreter_multi_shared" : {"linux:amd64:jdk-latest" : weekly + t("02:00:00")},
"vm_name:java_embedding_enterprise_interpreter_multi_shared" : {"linux:amd64:jdk-latest" : weekly + t("02:00:00")},
}),
for bench in ["java_embedding_meso"]
} + {
[bench]: bench_task(bench) + platform_spec(no_jobs) + bench_variants({
"vm_name:graalpython_core" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalpython_enterprise" : {"linux:amd64:jdk-latest" : daily + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ce_default" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ee_default" : {"linux:amd64:jdk-latest" : daily + t("05:00:00") + forks_warmup + need_pgo},
"vm_name:graalpython_core_multi_tier" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalpython_enterprise_multi_tier" : {"linux:amd64:jdk-latest" : weekly + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ce_default_multi_tier" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ee_default_multi_tier" : {"linux:amd64:jdk-latest" : weekly + t("05:00:00") + forks_warmup + need_pgo},
"vm_name:graalpython_core_3threads" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalpython_enterprise_3threads" : {"linux:amd64:jdk-latest" : weekly + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ce_default_3threads" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ee_default_3threads" : {"linux:amd64:jdk-latest" : weekly + t("05:00:00") + forks_warmup + need_pgo},
"vm_name:graalpython_core_multi_tier_3threads" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalpython_enterprise_multi_tier_3threads" : {"linux:amd64:jdk-latest" : weekly + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ce_default_multi_tier_3threads" : {"linux:amd64:jdk-latest" : on_demand + t("05:00:00") + forks_warmup},
"vm_name:graalvm_ee_default_multi_tier_3threads" : {"linux:amd64:jdk-latest" : weekly + t("05:00:00") + forks_warmup + need_pgo},
"vm_name:pypy" : {"linux:amd64:jdk-latest" : on_demand + t("01:00:00")},
}),
for bench in ["warmup"]
Expand All @@ -381,23 +356,19 @@
} + {
// interop benchmarks only for graalpython, weekly is enough
[bench]: bench_task(bench) + platform_spec(no_jobs) + bench_variants({
"vm_name:java_jmh_core" : {"linux:amd64:jdk-latest" : daily + t("04:00:00")},
"vm_name:java_jmh_enterprise" : {"linux:amd64:jdk-latest" : daily + t("04:00:00")},
}),
for bench in ["jmh"]
} + {
// benchmarks with many forks for weekly performance reports
[bench + "-forks"]: bench_task(bench) + platform_spec(no_jobs) + bench_variants({
"vm_name:graalvm_ce_default" : {"linux:amd64:jdk-latest" : on_demand + t("10:00:00") + forks_meso},
"vm_name:graalvm_ee_default" : {"linux:amd64:jdk-latest" : weekly + t("10:00:00") + forks_meso + need_pgo},
}),
for bench in ["meso"]
} + {
// benchmarks with community benchmark suites for external numbers
[bench]: bench_task(bench, PY_BENCHMARKS) + platform_spec(no_jobs) + raw_results + bench_variants({
"vm_name:graalpython_core" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalpython_enterprise" : {"linux:amd64:jdk-latest" : weekly + t("08:00:00")},
"vm_name:graalvm_ce_default" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalvm_ee_default" : {"linux:amd64:jdk-latest" : weekly + t("08:00:00") + need_pgo},
"vm_name:cpython_launcher" : {"linux:amd64:jdk-latest" : monthly + t("08:00:00")},
"vm_name:pypy_launcher" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
Expand All @@ -406,11 +377,7 @@
} + {
// benchmarks with community benchmark suites for external numbers
[bench]: bench_task(bench, PY_BENCHMARKS) + platform_spec(no_jobs) + raw_results + bench_variants({
"vm_name:graalpython_core" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalpython_core_panama" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalpython_enterprise" : {"linux:amd64:jdk-latest" : weekly + t("08:00:00")},
"vm_name:graalpython_enterprise_panama" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalvm_ce_default" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
"vm_name:graalvm_ee_default" : {"linux:amd64:jdk-latest" : weekly + t("08:00:00") + need_pgo},
"vm_name:cpython_launcher" : {"linux:amd64:jdk-latest" : monthly + t("08:00:00")},
"vm_name:pypy_launcher" : {"linux:amd64:jdk-latest" : on_demand + t("08:00:00")},
Expand Down
66 changes: 2 additions & 64 deletions ci/constants.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,12 @@
NOTIFY_GROUPS:: ["tim.felgentreff@oracle.com"],

local ENV = {
graalpy_svm_ce: ["--env", "native-ce"],
graalpy_svm_ee: ["--env", "native-ee"],
graalpy_jvm_ce: ["--env", "jvm-ce-libgraal"],
graalpy_jvm_ee: ["--env", "jvm-ee-libgraal"],
libgraal_ce: ["--env", "../../graal/vm/mx.vm/libgraal"],
libgraal_ee: ["--env", "../../graal-enterprise/vm-enterprise/mx.vm-enterprise/libgraal-enterprise"],
},

local DY = {
ce: "/vm",
ee: "/vm-enterprise,/graalpython-enterprise",
},

Expand All @@ -66,12 +62,10 @@
native_interpreter_manual: "native-interpreter-manual",
interpreter_multi: "interpreter-multi",
native_interpreter_multi: "native-interpreter-multi",
default_multi_tier: "default-multi-tier",
native: "native",
native_manual: "native-manual",
native_multi: "native-multi",
launcher: "launcher",
panama: "panama",
},

local JAVA_EMBEDDING_VM_CONFIG = {
Expand All @@ -81,33 +75,16 @@

// the host VMs
local JVM_VM = {
graaljdk_ce: {
dy: ["--dynamicimports", DY.ce],
env: ENV.graalpy_jvm_ce,
edition: 'ce',
},
graaljdk_ee: {
dy: ["--dynamicimports", DY.ee],
env: ENV.graalpy_jvm_ee,
edition: 'ee',
},
graal_native_image_ce: {
dy: ["--dynamicimports", DY.ce],
env: ENV.graalpy_svm_ce,
edition: 'ce',
},
graal_native_image_ee: {
dy: ["--dynamicimports", DY.ee],
env: ENV.graalpy_svm_ee,
edition: 'ee',
},
server_libgraal_ce: {
jvm: 'server',
jvm_config: 'graal-core-libgraal',
dy: ["--dynamicimports", DY.ce],
env: ENV.libgraal_ce,
edition: 'ce',
},
server_libgraal_ee: {
jvm: 'server',
jvm_config: 'graal-enterprise-libgraal',
Expand Down Expand Up @@ -165,10 +142,6 @@
python_vm: PYVM.graalpython,
python_vm_config: PYVM_CONFIG.native_interpreter_multi,
},
graalpython_multi_tier: {
python_vm: PYVM.graalpython,
python_vm_config: PYVM_CONFIG.default_multi_tier,
},
graalpython_native: {
python_vm: PYVM.graalpython,
python_vm_config: PYVM_CONFIG.native,
Expand All @@ -181,10 +154,6 @@
python_vm: PYVM.graalpython,
python_vm_config: PYVM_CONFIG.native_multi,
},
graalpython_panama: {
python_vm: PYVM.graalpython,
python_vm_config: PYVM_CONFIG.panama,
},
java_embedding_multi_shared: {
python_vm: PYVM.graalpython,
python_vm_config: JAVA_EMBEDDING_VM_CONFIG.java_embedding_multi_shared,
Expand Down Expand Up @@ -213,56 +182,25 @@

local VM = {
// graalpy jvm standalones
graalpython_core: PYTHON_VM.graalpython + JVM_VM.graaljdk_ce,
graalpython_core_manual: PYTHON_VM.graalpython_manual + JVM_VM.graaljdk_ce,
graalpython_core_interpreter: PYTHON_VM.graalpython_interpreter + JVM_VM.graaljdk_ce,
graalpython_core_interpreter_manual: PYTHON_VM.graalpython_interpreter_manual + JVM_VM.graaljdk_ce,
graalpython_core_multi: PYTHON_VM.graalpython_multi + JVM_VM.graaljdk_ce,
graalpython_core_interpreter_multi: PYTHON_VM.graalpython_interpreter_multi + JVM_VM.graaljdk_ce,
graalpython_core_multi_tier: PYTHON_VM.graalpython_multi_tier + JVM_VM.graaljdk_ce,
graalpython_enterprise: PYTHON_VM.graalpython + JVM_VM.graaljdk_ee,
graalpython_enterprise_manual: PYTHON_VM.graalpython_manual + JVM_VM.graaljdk_ee,
graalpython_enterprise_multi: PYTHON_VM.graalpython_multi + JVM_VM.graaljdk_ee,
graalpython_enterprise_multi_tier: PYTHON_VM.graalpython_multi_tier + JVM_VM.graaljdk_ee,
graalpython_enterprise_interpreter: PYTHON_VM.graalpython_interpreter + JVM_VM.graaljdk_ee,
graalpython_enterprise_interpreter_manual: PYTHON_VM.graalpython_interpreter_manual + JVM_VM.graaljdk_ee,
graalpython_core_native: PYTHON_VM.graalpython_native + JVM_VM.graaljdk_ce,
graalpython_core_native_manual: PYTHON_VM.graalpython_native_manual + JVM_VM.graaljdk_ce,
graalpython_core_native_interpreter: PYTHON_VM.graalpython_native_interpreter + JVM_VM.graaljdk_ce,
graalpython_core_native_interpreter_manual: PYTHON_VM.graalpython_native_interpreter_manual + JVM_VM.graaljdk_ce,
graalpython_core_native_multi: PYTHON_VM.graalpython_native_multi + JVM_VM.graaljdk_ce,
graalpython_core_native_interpreter_multi: PYTHON_VM.graalpython_native_interpreter_multi + JVM_VM.graaljdk_ce,
graalpython_enterprise_native: PYTHON_VM.graalpython_native + JVM_VM.graaljdk_ee,
graalpython_enterprise_native_manual: PYTHON_VM.graalpython_native_manual + JVM_VM.graaljdk_ee,
graalpython_enterprise_native_multi: PYTHON_VM.graalpython_native_multi + JVM_VM.graaljdk_ee,
graalpython_core_panama: PYTHON_VM.graalpython_panama + JVM_VM.graaljdk_ce,
graalpython_enterprise_panama: PYTHON_VM.graalpython_panama + JVM_VM.graaljdk_ee,

// graalpy native standalones
graalvm_ce_default: PYTHON_VM.graalpython + JVM_VM.graal_native_image_ce,
graalvm_ce_default_interpreter: PYTHON_VM.graalpython_interpreter + JVM_VM.graal_native_image_ce,
graalvm_ee_default: PYTHON_VM.graalpython + JVM_VM.graal_native_image_ee,
graalvm_ee_default_manual: PYTHON_VM.graalpython_manual + JVM_VM.graal_native_image_ee,
graalvm_ee_default_interpreter: PYTHON_VM.graalpython_interpreter + JVM_VM.graal_native_image_ee,
graalvm_ee_default_interpreter_uncached: PYTHON_VM.graalpython_interpreter_uncached + JVM_VM.graal_native_image_ee,
graalvm_ee_default_interpreter_manual: PYTHON_VM.graalpython_interpreter_manual + JVM_VM.graal_native_image_ee,
graalvm_ce_default_multi_tier: PYTHON_VM.graalpython_multi_tier + JVM_VM.graal_native_image_ce,
graalvm_ee_default_multi_tier: PYTHON_VM.graalpython_multi_tier + JVM_VM.graal_native_image_ee,

// only 3 compiler threads
graalpython_core_3threads: PYTHON_VM.graalpython + JVM_VM.graaljdk_ce + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},
graalpython_enterprise_3threads: PYTHON_VM.graalpython + JVM_VM.graaljdk_ee + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},
graalvm_ce_default_3threads: PYTHON_VM.graalpython + JVM_VM.graal_native_image_ce + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},
graalvm_ee_default_3threads: PYTHON_VM.graalpython + JVM_VM.graal_native_image_ee + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},
graalpython_core_multi_tier_3threads: PYTHON_VM.graalpython_multi_tier + JVM_VM.graaljdk_ce + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},
graalpython_enterprise_multi_tier_3threads: PYTHON_VM.graalpython_multi_tier + JVM_VM.graaljdk_ee + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},
graalvm_ce_default_multi_tier_3threads: PYTHON_VM.graalpython_multi_tier + JVM_VM.graal_native_image_ce + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},
graalvm_ee_default_multi_tier_3threads: PYTHON_VM.graalpython_multi_tier + JVM_VM.graal_native_image_ee + {python_vm_config: super.python_vm_config + "-3-compiler-threads"},

// Java embedding
java_embedding_core_multi_shared: PYTHON_VM.java_embedding_multi_shared + JVM_VM.server_libgraal_ce,
java_embedding_core_interpreter_multi_shared: PYTHON_VM.java_embedding_interpreter_multi_shared + JVM_VM.server_libgraal_ce,
java_jmh_core: JVM_VM.server_libgraal_ce,
java_embedding_enterprise_multi_shared: PYTHON_VM.java_embedding_multi_shared + JVM_VM.server_libgraal_ee,
java_embedding_enterprise_interpreter_multi_shared: PYTHON_VM.java_embedding_interpreter_multi_shared + JVM_VM.server_libgraal_ee,
java_jmh_enterprise: JVM_VM.server_libgraal_ee,

// basline vms
Expand Down
5 changes: 2 additions & 3 deletions docs/contributor/CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,10 +294,9 @@ This is intended for focused reproducer runs on a branch.

Note that there may be a little confusion about the configuration names of benchmarks.

#### GraalVM Community Edition and Oracle GraalVM configurations
#### Oracle GraalVM configurations

We have benchmarks for GraalVM Community Edition and Oracle GraalVM.
For historical reasons, these are sometimes referred to in some config files as *CE* and *EE*; *core* and *enterprise*; *graalvm_ce* and *graalvm_ee*; or *graalpython_core* and *graalpython_enterprise*, respectively.
Benchmark CI jobs use Oracle GraalVM configurations. For historical reasons, these are sometimes referred to in config files as *EE*, *enterprise*, *graalvm_ee*, or *graalpython_enterprise*.

### Different GraalVM Python configurations

Expand Down
Loading
Loading