Feature request: re-submit jobs using information stored in `dstat --full`

[amstilp]

I'm using dsub on the All Of Us researcher workbench, and generally it has been great! I'm running a number of analyses at the same time on preemptible instances, which cuts down on cloud compute costs. Most of my jobs are submitted using a tasks file to run one job per chromosome (23 jobs in total). Because the instances are preemptible, some of them fail with the "worker was terminated" error. Many of the jobs take hours to run, so using dsub --wait is not practical. I'm looking for an easy way to resubmit these pre-empted jobs.

After reading through the documentation, I saw that the --skip parameter is an option, but my current tasks file has the output from each task going to the same directory with --recursive-output. Because some of them have succeeded and therefore have output existing in the --recursive-output directory, including the --skip argument when calling the dsub command again skips all tasks. I will probably change this for future jobs so I can use the --skip argument.

It also looks like most of the parameters for a job are saved in the output of dstat --full. Is there a way to use the output from dstat --full to resubmit a job without manually reconstructing the dsub command myself from that output? I'm thinking something like dsub --retry <job_id>, where it retries all failed jobs with that job id?

Apologies if this already exists and I missed it in the documentation!

[wnojopra]

Hi @amstilp !

dsub has a retries features that is built for precisely this use case. We even mention retrying specifically with --preemptible. Have you given this a try?

[amstilp]

I did see the --retries flag. From the docs, it looks like you need to run with --wait to use --retries. Because my jobs take hours to run and I’m submitting from the All of Us Researcher Workbench via a Jupyter notebook, it’s not practical to use --wait. If you don’t need that flag to use the retries feature, then I’d happily try it!

[wnojopra]

Ah, got it. Yes, we need --wait for --retries because its the dsub client that is orchestrating the retry attempts. But as you pointed out this is not reliable when running on a Workbench Jupyter notebook.

One thing I'd recommend trying is to run dsub in a dsub job. Because the dsub job that gets kicked off can reliably run to completion, what if we use that job to run dsub for your 23 tasks, but each with --retries enabled.

To do this you'll need to put dsub in a Docker image. In your AoU Workbench VM, kick off a dsub job that uses this image. This dsub job then kicks off your original job with the 23 tasks.

[amstilp]

Oh, huh, that’s an interesting solution. I’ll look into it, thanks.