Snakemake Module for esm_runscripts

A reusable Snakemake module for running Earth System Model simulations using esm_runscripts. This module provides ready-to-use rules with automatic resource extraction, making it easy to integrate ESM simulations into Snakemake workflows.

Features

• Self-contained: No need to install esm-tools package (only esm_runscripts binary required at runtime)
• Automatic resource extraction: Reads resource requirements from ESM configuration
• Phase-aware: Supports all esm_runscripts phases (prepcompute, compute, tidy, post)
• Flexible: Easy to customize and extend
• Reusable: Import into any Snakemake workflow

Installation

Clone this repository:

git clone https://github.com/esm-tools/snakemake-module-esm-runscripts.git

Or reference it directly in your workflow using a specific commit/tag.

Usage

Basic Usage

In your workflow's Snakefile:

module esm_compute:
    snakefile: "path/to/snakemake-module-esm-runscripts/workflow/Snakefile"
    config: {
        "runscript": "configs/my_experiment.yaml",
        "expid": "my_exp_001",
        "phase": "compute"
    }

use rule compute from esm_compute

Using All Phases

# Define module
module esm_sim:
    snakefile: "path/to/snakemake-module-esm-runscripts/workflow/Snakefile"
    config: {
        "runscript": "configs/experiment.yaml",
        "expid": "PI-CTRL"
    }

# Use all phases in sequence
rule all:
    input:
        "PI-CTRL_post.done"

use rule prepcompute from esm_sim
use rule compute from esm_sim
use rule tidy from esm_sim
use rule post from esm_sim

Customizing Rules

module esm_compute:
    snakefile: "path/to/snakemake-module-esm-runscripts/workflow/Snakefile"
    config: {
        "runscript": "configs/experiment.yaml",
        "expid": "{expid}"  # Use wildcards
    }

use rule compute from esm_compute with:
    input:
        runscript="configs/{expid}.yaml",
        model_ready="results/{expid}/model_installed.done"
    output:
        "results/{expid}/simulation_complete.done"
    log:
        "logs/{expid}_compute.log"

Multiple Experiments in Parallel

EXPERIMENTS = ["exp001", "exp002", "exp003"]

module esm_compute:
    snakefile: "snakemake-module-esm-runscripts/workflow/Snakefile"
    config: {
        "runscript": "configs/{expid}.yaml",
        "expid": "{expid}"
    }

rule all:
    input:
        expand("results/{expid}_done.txt", expid=EXPERIMENTS)

use rule compute from esm_compute as compute_experiment with:
    input:
        runscript="configs/{expid}.yaml"
    output:
        "results/{expid}_done.txt"

Configuration

The module accepts the following configuration parameters:

Parameter	Type	Required	Default	Description
runscript	str	Yes	-	Path to ESM runscript YAML file
expid	str	Yes	-	Experiment ID
phase	str	No	"compute"	Phase to execute (prepcompute/compute/tidy/post)
base_dir	str	No	null	Base directory for experiment
reuse_config	bool	No	false	Reuse existing finished_config.yaml
modify_config	str	No	null	Path to config modification file
current_date	str	No	null	Override start date (YYYY-MM-DD)
extra	str	No	""	Additional esm_runscripts flags
output	str	No	"{expid}_{phase}.done"	Output file pattern
log	str	No	"logs/{expid}_{phase}.log"	Log file pattern
wrapper	str	No	local path	Wrapper location

How It Works

1. Resource Extraction: The module's get_resources.py script runs esm_runscripts --check to generate configuration and extracts resource requirements (nodes, tasks, memory, runtime, partition, account)

2. Resource Declaration: Resources are automatically declared in the rule's resources: directive, allowing Snakemake to properly schedule jobs with SLURM

3. Execution: The esm_runscripts wrapper:

   • Finds the generated .run script
   • Extracts executable content (strips SLURM directives)
   • Executes within Snakemake's allocated resources

Requirements

• Snakemake >= 7.0
• esm_runscripts command available in PATH (part of esm-tools)
• Python >= 3.10
• PyYAML or ruamel.yaml

Rules Provided

prepcompute

Executes the preparation phase before computation.

compute

Executes the main model simulation.

tidy

Executes post-run cleanup and organization.

post

Executes post-processing and analysis.

Architecture

This module follows Snakemake best practices:

snakemake-module-esm-runscripts/
├── workflow/
│   ├── Snakefile              # Main module entry point
│   ├── rules/                 # Phase-specific rules
│   │   ├── prepcompute.smk
│   │   ├── compute.smk
│   │   ├── tidy.smk
│   │   └── post.smk
│   └── scripts/
│       └── get_resources.py   # Standalone resource extraction
├── config/
│   ├── config.yaml            # Default configuration
│   └── schema.yaml            # Configuration validation
└── README.md

Advantages Over Direct Wrapper Usage

Aspect	Direct Wrapper	This Module
Resource extraction	Manual get_resources() call	Automatic
Dependencies	Requires esm-tools installed	Self-contained
Reusability	Copy-paste rules	Import module
Maintenance	Update each workflow	Update module once
Configuration	Hardcoded paths	Configurable

Examples

See the examples/ directory (if present) for complete workflow examples.

Contributing

Contributions welcome! Please:

1. Test with your ESM setup
2. Add examples for new use cases
3. Update documentation
4. Submit pull requests

License

MIT License - see LICENSE for details.

Authors

• Paul Gierz - paul.gierz@awi.de

Related Projects

• esm-tools: Core ESM workflow tools
• snakemake-wrapper-esm-runscripts: Underlying wrapper
• esm-tools documentation: Complete reference

Citation

If you use this module in your research, please cite:

@software{snakemake_module_esm_runscripts,
  title = {Snakemake Module for esm\_runscripts},
  author = {Gierz, Paul},
  year = {2025},
  url = {https://github.com/esm-tools/snakemake-module-esm-runscripts}
}