Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ A clear and concise description of what you expected to happen.
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
- OS: [e.g. Linux Fedora 41, Windows 10...]
- OS: [e.g. Linux Fedora 43, Windows 11...]
- Version [e.g. v0.1.1]

**Additional context**
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/template-check-version.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,6 @@ jobs:
run: |
gh issue --repo ${{ github.repository }} \
create --title "Template update" \
--body "A new version of the template has been published. Please update this project using \`copier update --skip-answered --defaults\` and review changes. If you need help, refer to the \`clio\` documentation or reach out to the calliope project team."
--body "A new version of the template has been published. Please update this project using \`copier update --skip-answered --defaults\` and review changes. If you need help, refer to the Modelblocks documentation or reach out to the project team."
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
44 changes: 0 additions & 44 deletions .github/workflows/template-pr-ci.yml

This file was deleted.

3 changes: 2 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ repos:
# Repo quality
- id: check-added-large-files
args: [--enforce-all]
exclude: ^pixi\.lock$
- id: forbid-submodules
- id: check-case-conflict
- id: check-illegal-windows-names
Expand All @@ -37,4 +38,4 @@ repos:
- id: codespell

ci: # https://pre-commit.ci/
autoupdate_schedule: monthly
autoupdate_schedule: quarterly
2 changes: 1 addition & 1 deletion AUTHORS
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@ This is the list of contributors for copyright purposes.

This does not necessarily list everyone who has contributed to this software's
code or documentation. For a full contributor list, see:
<https://github.com/calliope-project/data-module-template/graphs/contributors>
<https://github.com/modelblocks-org/data-module-template/graphs/contributors>

Ivan Ruiz Manuel, TU Delft <i.ruizmanuel@tudelft.nl>
4 changes: 2 additions & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
# https://citation-file-format.github.io/
cff-version: 1.2.0
message: If you use this software or data produced by it, please cite it using the metadata from this file.
title: clio - data module template
repository: https://github.com/calliope-project/data-module-template
title: Modelblocks - data module template
repository: https://github.com/modelblocks-org/data-module-template
license: Apache-2.0
authors:
- given-names: Ivan
Expand Down
74 changes: 39 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,48 @@
# Data module template
# Modelblocks data module template

A template for modular data workflows using [`snakemake`](https://snakemake.readthedocs.io/en/stable/), part of the [`clio`](https://clio.readthedocs.io/) toolset.
A template for modular data workflows using [`snakemake`](https://snakemake.readthedocs.io/en/stable/) workflows, part of the [Modelblocks](https://www.modelblocks.org/) toolset.

> [!TIP]
> Looking for general information on Modelblocks?
> Check the [Modelblocks](https://www.modelblocks.org/) website and our [documentation and guidelines](https://modelblocks.readthedocs.io/en).

## Resources

To familiarise yourself with `clio` data modules:
To familiarise yourself with Modelblocks data modules:

- Check the auto-generated minimal example. You can find it in `tests/integration/Snakefile`.
- Read about the `clio` approach in [our documentation](https://clio.readthedocs.io/).
- Read about `snakemake` modularisation in [their documentation](https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#modules).

## Features

- Standardised layout compliant with the [snakemake workflow catalogue's](https://snakemake.github.io/snakemake-workflow-catalog/#) listing requirements, enabling them to be automatically included in their listings once published. Read more about those requirements [here](https://snakemake.github.io/snakemake-workflow-catalog/docs/catalog.html#standardized-usage-workflows).
- Standardised input-output structure across modules:
- `resources/`: files needed for the module's processes.
- `user/`: files that should be provided by users. Document them well!
- `automatic/`: files that the module downloads or prepares in intermediate steps.
- `results/`: files generated by the module's algorithms that are relevant to the user.
- Pre-made integration setup for your module.
- Continuous Integration (CI) settings, ready for [pre-commit.ci](https://pre-commit.ci/).
- GitHub actions to automate chores during pull requests and releases.
- Premade `pytest` setup.

> [!IMPORTANT]
>
> A few things to be aware of.
>
> - **Modules do not work like regular snakemake workflows**
> - The primary way to test them should be external (calling `module:`, passing resources, and requesting results). Check the pre-made example in `tests/integration` for more info.
> - Internal access (e.g., calling the `all:` rule) may not work, as the module may not have the necessary `resources/` to execute properly.
> - **Please be sure to maintain the following files to ensure Modelblocks compatibility**
> - `INTERFACE.yaml`: a simple description of the module's input/output structure.
> - `config/config.yaml`: a basic functioning example of how to configure this module.
> - `workflow/internal/config.schema.yaml`: the module's configuration schema, used by `snakemake` for [validation](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html).
> - `AUTHORS` / `CITATION.cff` / `LICENSE`: licensing and attribution of this module's code and methods.


## How to use this template

We recommend using [`pixi`](https://pixi.sh/) as your package manager. Once installed, do the following:
We require [`pixi`](https://pixi.sh/) as a package manager. Once installed, do the following:

1. Install the templater tool [`copier`](https://copier.readthedocs.io/en/stable/).

Expand All @@ -25,16 +55,17 @@ We recommend using [`pixi`](https://pixi.sh/) as your package manager. Once inst
We recommend you use the module name as the directory name.

```shell
copier copy https://github.com/calliope-project/data-module-template.git ./path/to/<module_name>
copier copy https://github.com/modelblocks-org/data-module-template.git ./<module_name>
```

If your terminal does not have access to `copier` then you may need to update your `PATH` variable to include `~/.pixi/bin`.
> [!TIP]
> If your terminal does not have access to `copier` then you may need to update your `PATH` variable to include `~/.pixi/bin`.

3. Answer some questions so can we pre-fill licensing, citation files, etc...
4. Initialise the `pixi` project environment of your new module.

```shell
cd ./path/to/<module_name> # navigate to the new project
cd ./<module_name> # navigate to the new project
pixi install --all # install the project environment
```

Expand All @@ -44,30 +75,3 @@ We recommend using [`pixi`](https://pixi.sh/) as your package manager. Once inst
cd tests/integration # go to the integration test...
pixi run snakemake --use-conda # run it!
```

## Features

- Standardised layout compliant with the [snakemake workflow catalogue's](https://snakemake.github.io/snakemake-workflow-catalog/#) listing requirements, enabling them to be automatically included in their listings once published. Read more about those requirements [here](https://snakemake.github.io/snakemake-workflow-catalog/docs/catalog.html#standardized-usage-workflows).
- Standardised input/output structure across modules:
- `resources/`: files needed for the module's processes.
- `user/`: files that should be provided by users. Document them well!
- `automatic/`: files that the module downloads or prepares in intermediate steps.
- `results/`: files generated by the module's algorithms that are relevant to the user.
- Pre-made integration setup for your module.
- Continuous Integration (CI) settings, ready for [pre-commit.ci](https://pre-commit.ci/).
- GitHub actions to automate chores during pull requests and releases.
- Premade `pytest` setup.

> [!IMPORTANT]
>
> A few things to be aware of.
>
> - **Modules do not work like regular snakemake workflows**
> - The primary way to test them should be external (calling `module:`, passing resources, and requesting results). Check the pre-made example in `tests/integration` for more info.
> - Internal access (e.g., calling the `all:` rule) may not work, as the module may not have the necessary `resources/` to execute properly.
> - **Please be sure to maintain the following files to ensure `clio` compatibility**
> - These are:
> - `INTERFACE.yaml`: a simple description of the module's input/output structure.
> - `config/config.yaml`: a basic functioning example of how to configure this module.
> - `workflow/internal/config.schema.yaml`: the module's configuration schema, used by `snakemake` for [validation](https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html).
> - `AUTHORS` / `CITATION.cff` / `LICENSE`: licensing and attribution of this module's code and methods.
8 changes: 4 additions & 4 deletions copier.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ _exclude:
_skip_if_exists:
# Respect user changes, except for a few standard files.
- config/**
- docs/**
- resources/**
- figures/**
- results/**
- tests/**
- workflow/**
Expand All @@ -32,7 +32,7 @@ _skip_if_exists:
- INTERFACE.yaml
# Special cases: these files should not be user-modified, even though they are in user-modifiable directories
- "!config/README.md"
- "!tests/clio_test.py"
- "!tests/integration_test.py"
_templates_suffix: .jinja
_answers_file: .copier-answers.yml

Expand Down Expand Up @@ -67,12 +67,12 @@ github_org:
type: str
help: >-
Please provide the name of the github account or organisation where
this module will be created (e.g., 'calliope-project' in 'github.com/calliope-project').
this module will be created (e.g., 'modelblocks-org' in 'github.com/modelblocks-org').
validator: >-
{% if not (github_org | regex_search('^[a-zA-Z0-9][a-zA-Z0-9-]*$')) %}
"Only lowercase letters, digits, hyphens and underscores are valid."
{% endif %}
default: "calliope-project"
default: "modelblocks-org"
license:
type: str
help: "Please choose a license."
Expand Down
1 change: 1 addition & 0 deletions mypy.ini
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
[mypy]
disable_error_code = import-untyped
exclude = (^|/)\.(snakemake|pixi)(/|$)
exclude_gitignore = True
Loading
Loading