Problem Description
A job is a single run of a dataset using a particular synthesizer and then evaluating the performance. Right now, we run jobs in parallel on a single instance, this can create large computational needs and can crash during benchmark execution. To alleviate any issues that can be caused by dataset/synthesizer needs, we want to update the benchmark config to always run only one job on an instance.
Note that this approach is not meant for resource optimization.
Expected behavior
For each job:
- launch an instance
- run the benchmark
- upload the results
- terminate the instance
This change can be done on the configuration level by the benchmark launcher. What's important here is that we want to have ease in adding new datasets and synthesizers to the configuration. Therefore, I would propose that we can write a script to create the config file.
Problem Description
A job is a single run of a dataset using a particular synthesizer and then evaluating the performance. Right now, we run jobs in parallel on a single instance, this can create large computational needs and can crash during benchmark execution. To alleviate any issues that can be caused by dataset/synthesizer needs, we want to update the benchmark config to always run only one job on an instance.
Note that this approach is not meant for resource optimization.
Expected behavior
For each job:
This change can be done on the configuration level by the benchmark launcher. What's important here is that we want to have ease in adding new datasets and synthesizers to the configuration. Therefore, I would propose that we can write a script to create the config file.