Skip to content

feat: add evaluation script for gemma model #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Ki-Seki wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: add evaluation script for gemma model #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3e33a26
feat: add evaluation script for gemma model
Ki-Seki Mar 18, 2026
1bd2a43
fix: export environment variables in eval script
Ki-Seki Mar 18, 2026
79d86cf
Refactor code structure for improved readability and maintainability
Ki-Seki Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions results/260318-gemma-eval/*eval.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash
Comment thread
Ki-Seki marked this conversation as resolved.

Comment thread
Ki-Seki marked this conversation as resolved.
set -x
Comment thread
Ki-Seki marked this conversation as resolved.

export API_KEY=xxx
export API_BASE=xxx
export MODEL=Sculpt-AI/2603171-gemma

hf download $MODEL

python -m gimbench.ppl.gim_sft --ref_model_device cpu --golden_truth_only --first_n 100
python -m gimbench.ppl.gim_sft --model_type vllm-offline --model_name $MODEL --output_type cfg --ref_model_device cpu --first_n 100
python -m gimbench.match.gim_regex --model_type vllm-offline --model_name $MODEL --output_type cfg --ref_model_device cpu --first_n 100

python -m "gimbench.mcqa.gpqa_diamond" --model_type vllm-offline --model_name "$MODEL" --auto_budget --num_proc 40 --first_n 198
python -m "gimbench.mcqa.medmcqa" --model_type vllm-offline --model_name "$MODEL" --auto_budget --num_proc 40 --first_n 1000
python -m "gimbench.mcqa.mmlu_pro" --model_type vllm-offline --model_name "$MODEL" --auto_budget --num_proc 40 --first_n 1000
python -m "gimbench.mcqa.qasc" --model_type vllm-offline --model_name "$MODEL" --auto_budget --num_proc 40 --first_n 920

python -m "gimbench.mcqa.gpqa_diamond" --model_type vllm-offline --model_name "google/gemma-3-270m-it" --no_gimkit --num_proc 40 --first_n 198
python -m "gimbench.mcqa.medmcqa" --model_type vllm-offline --model_name "google/gemma-3-270m-it" --no_gimkit --num_proc 40 --first_n 1000
python -m "gimbench.mcqa.mmlu_pro" --model_type vllm-offline --model_name "google/gemma-3-270m-it" --no_gimkit --num_proc 40 --first_n 1000
python -m "gimbench.mcqa.qasc" --model_type vllm-offline --model_name "google/gemma-3-270m-it" --no_gimkit --num_proc 40 --first_n 920

python -m gimbench.cv.cv_parse --model_type vllm-offline --model_name $MODEL --output_type cfg --api_key $API_KEY --base_url $API_BASE

shutdown -h +3
Comment thread
Ki-Seki marked this conversation as resolved.
Loading
Loading