Toil-work directory rerunning the same jobs iteratively

[nicolasmalexandre]

Hello,

I have been running the following using a bed file that has around 379,000 intervals. I've been setting my own toil directories because I only have scratch space in a mounted directory "~/Desktop" which I run the following commands in. Therefore, /data is in ~/Desktop.

docker run --rm -it
--cpus=50
--memory=200g
-v "$PWD":/data
-w /data
-e TMPDIR=/data/tmp
-e PARALLEL="--jobs 50"
quay.io/comparative-genomics-toolkit/cactus:v3.0.1

mkdir -p /data/toil-work /data/toil-coord /data/tmp

cactus-hal2maf
/data/FormGenomes/Cactus/Halper/RERConverge/Liftover_Step4/jobstore
/data/FormGenomes/Cactus/363/363-avian-2020.hal
/data/FormGenomes/Cactus/Halper/RERConverge/Liftover_Step4/Merged.Consensus.maf.gz
--refGenome Gallus_gallus_wlh
--bedRanges /data/FormGenomes/Cactus/Halper/RERConverge/Liftover_Step4/Merged.Consensus.3col.bed
--noAncestors
--workDir /data/toil-work
--coordinationDir /data/toil-coord
--onlyOrthologs
--filterGapCausingDupes
--outType single
--chunkSize 500000
--maxMemory 200G
--defaultCores 50
--maxDisk 1100G
--defaultDisk 1100G
--cleanWorkDir onSuccess

I'm not sure why but I seem to be getting the same maf.gz files getting generated one after the other in new toil-work subdirectories, its been running continuously for a few days

-rw-r--r-- 1 nicolasalexandre nicolasalexandre 121K Feb 21 04:37 /home/nicolasalexandre/Desktop/toil-work/toilwf-e827e8a6c7955ce3beec31734a65b3d1/0cb4/job/tmptbkrbgbs/Merged.Consensus_chunk_2239.single.maf.gz

-rw-r--r-- 1 root root 121K Feb 23 22:58 /home/nicolasalexandre/Desktop/toil-work/toilwf-e827e8a6c7955ce3beec31734a65b3d1/5be4/job/tmpaam04jdt/Merged.Consensus_chunk_2239.single.maf.gz

The commands seem to be the same as well:

head ./5be4/job/tmpaam04jdt/taf_cmds.txt
set -eo pipefail && gzip -dc Merged.Consensus_chunk_0.maf.gz | taffy view | /usr/bin/time -vp taffy norm -a 363-avian-2020.hal -k -d 2> 0.tn.time | taffy view | taffy sort -n genome.list | taffy view -m | mafDuplicateFilter -m - -k | bgzip > Merged.Consensus_chunk_0.single.maf.gz

head ./0cb4/job/tmptbkrbgbs/taf_cmds.txt
set -eo pipefail && gzip -dc Merged.Consensus_chunk_0.maf.gz | taffy view | /usr/bin/time -vp taffy norm -a 363-avian-2020.hal -k -d 2> 0.tn.time | taffy view | taffy sort -n genome.list | taffy view -m | mafDuplicateFilter -m - -k | bgzip > Merged.Consensus_chunk_0.single.maf.gz

The job has been running continuously for 4 days and seems to just be repeating processes.

[glennhickey]

You have multiple copies because it's failing (probably running out of memory or something), then retrying (beacuse I think by default --retryCount is 1 on single-machine), while leaving the previous run's output in the workDir because you're using --cleanWorkDir onSuccess for some reason.

Having multiple copies in your workDir isn't an issue in and of itself but is a symptom that it has crashed and probably will again (unless you reran with --restart after fixing something). That said I don't really recommend --cleanWorkDir onSuccess as it will use up disk unnecessarily.

The cactus-hal2maf logs are hard to read, but the underlying error/crash should be in it somewhere. You'll need to figure out what that is (likely running out of memory) in order to fix.

If you want to be able to have a shot of resuming after a failure, say by rerunning with more memory in your container, try increasing the --batchCount -- this should split the work into multiple toil jobs that will allow you to --restart only the ones that fail...

[glennhickey]

One other thing: The more cores you use per batch, the more memory is needed, especially the taffy postprocessing. So you can you --batchParallelTaf to ratchet that down a bit (I'd suggest using about half your total cores, so in this case about 25). But if it's hal2maf that's running out of memory then you would need a lower maxCores....

[nicolasmalexandre]

Thank you!

I ran the following and I'll keep an eye on the log file and report whether it worked:

docker run --rm -it
--cpus=50
--memory=200g
-v "$PWD":/data
-w /data
-e TMPDIR=/data/tmp
quay.io/comparative-genomics-toolkit/cactus:v3.0.1



cactus-hal2maf
/data/FormGenomes/Cactus/Halper/RERConverge/Liftover_Step4/jobstore
/data/FormGenomes/Cactus/363/363-avian-2020.hal
/data/FormGenomes/Cactus/Halper/RERConverge/Liftover_Step4/Merged.Consensus.maf.gz
--refGenome Gallus_gallus_wlh
--bedRanges /data/FormGenomes/Cactus/Halper/RERConverge/Liftover_Step4/Merged.Consensus.3col.bed
--noAncestors
--workDir /data/tmp
--onlyOrthologs
--filterGapCausingDupes
--outType single
--batchCount 200
--batchParallelTaf 25
--maxMemory 200G

[nicolasmalexandre]

Seems to be rerunning the same jobs over again by making a new tmp directory and making the same size single.maf.gz files in the new tmp directory. The only log file I can find is in the working directory under the tmp job folder labelled worker_log.txt (attached). Memory does not seem to be an issue as using is generally under 100Gb and I have 500Gb memory available. Essentially I'm getting an out.single.Merged.Consensus.maf.gz (287 Mb) that dissappears in the tmp directory before a new set of chunks are generated.

However looking at the progress bar, its ticking up from 3->4->5 after each of these iterations:
Workflow Progress 2%|██ | 5/203 (0 failures) [1h 07:24<1d 20h 29:03, 0.00 jobs/s]

worker_log.txt

When I run the above on only 100 bed intervals, it completes in very little time. The full set is ~379K intervals.

Any help understanding this is extremely helpful!