Skip to content

Update the benchmark to launch one instance per (dataset, synthesizer) pair #611

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
R-Palazzo wants to merge 7 commits into
base: issue-604-2-private-bucket
Choose a base branch
from
+205 −621
Open

Update the benchmark to launch one instance per (dataset, synthesizer) pair #611

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
fba2cf7
def 605
R-Palazzo May 25, 2026
e42ee3f
unit tests
R-Palazzo May 25, 2026
e4126fe
test on rel-bench dataset
R-Palazzo May 27, 2026
bd41677
pip install from the current branch
R-Palazzo May 27, 2026
764c106
fix script.py: always launch 1 instance per job in CLI
R-Palazzo May 28, 2026
5cb84b0
undo changes for testing
R-Palazzo May 28, 2026
86d11d6
undo workflow change
R-Palazzo May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions .github/workflows/_run_benchmark.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,6 @@ on:
synthesizers:
required: false
type: string
num_instances:
required: false
type: string
output_destination:
required: false
type: string
Expand Down Expand Up @@ -110,12 +107,11 @@ jobs:
add_arg --modality "${{ inputs.modality }}"
add_list_arg --datasets "${{ inputs.datasets }}"
add_list_arg --synthesizers "${{ inputs.synthesizers }}"
add_arg --num-instances "${{ inputs.num_instances }}"
add_arg --output-destination "${{ inputs.output_destination }}"
add_arg --timeout "${{ inputs.timeout }}"

printf 'Running command:'
printf ' %q' "${CMD[@]}"
printf '\n'

"${CMD[@]}"
"${CMD[@]}"
6 changes: 0 additions & 6 deletions .github/workflows/run_benchmark_manual.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,6 @@ on:
description: "Space-separated synthesizers, e.g. 'CTGANSynthesizer TVAESynthesizer'. If not provided, all synthesizers for the modality will be included."
required: false
type: string
num_instances:
description: "Number of instances"
required: false
default: "1"
type: string
output_destination:
description: "Required output destination"
required: true
Expand All @@ -39,7 +34,6 @@ jobs:
modality: ${{ inputs.modality }}
datasets: ${{ inputs.datasets }}
synthesizers: ${{ inputs.synthesizers }}
num_instances: ${{ inputs.num_instances }}
output_destination: ${{ inputs.output_destination }}
timeout: ${{ inputs.timeout }}
secrets:
Expand Down
180 changes: 0 additions & 180 deletions sdgym/_benchmark_launcher/benchmark_multi_table.yaml
View file
Open in desktop

This file was deleted.

131 changes: 0 additions & 131 deletions sdgym/_benchmark_launcher/benchmark_single_table.yaml
View file
Open in desktop

This file was deleted.

Loading
Loading