Skip to content

feat(config): SLURM job-array submission for per-sample fan-out stages#503

Open
tpall wants to merge 1 commit intoWrightonLabCSU:devfrom
tpall:pr472-focused
Open

feat(config): SLURM job-array submission for per-sample fan-out stages#503
tpall wants to merge 1 commit intoWrightonLabCSU:devfrom
tpall:pr472-focused

Conversation

@tpall
Copy link
Copy Markdown

@tpall tpall commented May 8, 2026

Summary

Adds optional SLURM job-array submission for the heavy per-sample fan-out stages of ANNOTATE (CALL, DB_SEARCH, QC:COLLECT_*). With many input fastas, this collapses N individual sbatch calls into one job array per stage, sparing fair-share priority on shared clusters.

Supersedes #472 — this PR is the same author's earlier work narrowed to only the array-support change with @madeline-scyphers' three review comments addressed in the design from the start. Other improvements that were bundled into #472 (Channel→channel cleanup, rRNA/tRNA distill fix, fasta gzip support, …) will come up as separate focused PRs.

What changed

f3ef6eb9 feat(config): SLURM job-array submission for per-sample fan-out stages

+19 lines, 3 files:

  • nextflow.config — new params.array_size = 0 (default disabled, local-executor safe).

  • nextflow_schema.json — schema entry for array_size so the param validates.

  • conf/modules.config — single withName block, gated on params.slurm:

    withName: 'DRAM:ANNOTATE:CALL:.*|DRAM:ANNOTATE:DB_SEARCH:.*|DRAM:ANNOTATE:QC:COLLECT.*' {
        array = params.slurm ? params.array_size : 0
    }

How this addresses @madeline-scyphers' review comments on #472

Comment Fix in this PR
"should only be used with executors that support it" — local executor doesn't support arrays array = params.slurm ? params.array_size : 0 — without --slurm the directive evaluates to 0, a no-op. Default array_size=0 further protects fresh users.
"jobs under DRAM:ANNOTATE:QC:COLLECT.* could also have a job array" Selector regex broadened to COLLECT.* so any current or future QC:COLLECT_* subworkflow picks it up.
"this code here in base.config for the job array should probably be in modules.config" Block lives entirely in modules.configbase.config is untouched by this PR.

Relationship to feature/condense-job-submission-using-collate

I noticed there's a feature/condense-job-submission-using-collate branch that takes a different (more invasive) approach to the same problem — concatenating per-sample fastas and running them as one task per group, reducing sbatch count by batching inputs rather than arraying tasks. That work has been dormant since 2025-07-22 (~10 months). The two approaches are orthogonal — array submits N tasks as a single sbatch; collation runs N inputs as a single task. They could coexist if the collation work resumes.

This PR is small enough (+19 lines) that it shouldn't conflict with future collation work; if collation does land, the array block in modules.config can be deleted in one commit.

Caveat

There's an open Nextflow issue (nextflow-io/nextflow#6108) — intermittent ConcurrentModificationException when arrays combine with Singularity. It's recoverable via -resume. Default array_size = 0 keeps users out of the race unless they opt in.

Test plan

  • nextflow inspect main.nf parses cleanly.
  • JSON schema valid.
  • HPC SLURM run with --slurm --array_size 20 --queue_size 200: array jobs appear in squeue and complete normally.
  • Local-executor run: confirm no array submission attempted (params default to 0).

🤖 Generated with Claude Code

Adds two params and a withName directive so multi-sample runs on SLURM
collapse N sbatches into one job array per stage, sparing fair-share
priority on shared clusters:

  params.array_size  (default 0 — disabled; local-executor safe)
  params.queue_size  (already existed; default 10)

  process {
      withName: 'DRAM:ANNOTATE:CALL:.*|DRAM:ANNOTATE:DB_SEARCH:.*|DRAM:ANNOTATE:QC:COLLECT.*' {
          array = params.slurm ? params.array_size : 0
      }
  }

The directive is gated on params.slurm so the local / standard executor
always sees array = 0 (a no-op) — only --slurm runs honour the user's
array_size override. The selector covers every per-sample fan-out stage
(CALL, DB_SEARCH, and any current or future QC:COLLECT_* subworkflow).

Same Nextflow #6108 caveat applies — intermittent ConcurrentModification-
Exception when arrays combine with Singularity, recoverable via -resume.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: To Sort

Development

Successfully merging this pull request may close these issues.

1 participant