Skip to content

[Feature] Allow cache_directory in resource_dict #945

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jan-janssen merged 5 commits into from
Mar 3, 2026
Merged

[Feature] Allow cache_directory in resource_dict #945

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
192e425
[Feature] Allow cache_directory in resource_dict
jan-janssen Mar 2, 2026
9ba4724
Format black
pyiron-runner Mar 2, 2026
6c083f7
fix test
jan-janssen Mar 2, 2026
e2a5a34
Merge branch 'cache_dir_validation' of github.com:pyiron/executorlib …
jan-janssen Mar 2, 2026
7833ac9
Add cache_dir in docs
jan-janssen Mar 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/trouble_shooting.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ The resource dictionary parameter `resource_dict` can contain one or more of the
* `cache_key` (str): Rather than using the internal hashing of executorlib the user can provide an external `cache_key`
to identify tasks on the file system. The initial file name is going to be `cache_key` + `_i.h5` and the final file
name is going to be `cache_key` + `_o.h5`.
* `cache_directory` (str): The directory to store cache files.
* `num_nodes` (int): number of compute nodes used for the evaluation of the Python function.
* `exclusive` (bool): boolean flag to reserve exclusive access to selected compute nodes - do not allow other tasks to
use the same compute node.
Expand Down
5 changes: 5 additions & 0 deletions src/executorlib/executor/flux.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ class FluxJobExecutor(BaseExecutor):
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down Expand Up @@ -146,6 +147,7 @@ def __init__(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python
function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes
Expand Down Expand Up @@ -275,6 +277,7 @@ class FluxClusterExecutor(BaseExecutor):
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down Expand Up @@ -370,6 +373,7 @@ def __init__(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python
function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes
Expand Down Expand Up @@ -530,6 +534,7 @@ def create_flux_executor(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down
5 changes: 5 additions & 0 deletions src/executorlib/executor/single.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ class SingleNodeExecutor(BaseExecutor):
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down Expand Up @@ -137,6 +138,7 @@ def __init__(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python
function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes
Expand Down Expand Up @@ -251,6 +253,7 @@ class TestClusterExecutor(BaseExecutor):
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down Expand Up @@ -340,6 +343,7 @@ def __init__(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python
function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes
Expand Down Expand Up @@ -469,6 +473,7 @@ def create_single_node_executor(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down
5 changes: 5 additions & 0 deletions src/executorlib/executor/slurm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ class SlurmClusterExecutor(BaseExecutor):
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down Expand Up @@ -141,6 +142,7 @@ def __init__(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python
function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes
Expand Down Expand Up @@ -285,6 +287,7 @@ class SlurmJobExecutor(BaseExecutor):
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down Expand Up @@ -380,6 +383,7 @@ def __init__(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python
function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes
Expand Down Expand Up @@ -508,6 +512,7 @@ def create_slurm_executor(
* cwd (str): current working directory where the parallel python task is executed
* cache_key (str): Rather than using the internal hashing of executorlib the user can
provide an external cache_key to identify tasks on the file system.
* cache_directory (str): The directory to store cache files.
* num_nodes (int): number of compute nodes used for the evaluation of the Python function.
* exclusive (bool): boolean flag to reserve exclusive access to selected compute nodes -
do not allow other tasks to use the same compute node.
Expand Down
1 change: 1 addition & 0 deletions src/executorlib/standalone/validate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ class ResourceDictValidation(BaseModel):
gpus_per_core: Optional[int] = None
cwd: Optional[str] = None
cache_key: Optional[str] = None
cache_directory: Optional[str] = None
num_nodes: Optional[int] = None
exclusive: Optional[bool] = None
error_log_file: Optional[str] = None
Expand Down
1 change: 1 addition & 0 deletions tests/unit/standalone/test_validate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ def test_get_accepted_keys(self):
"gpus_per_core",
"cwd",
"cache_key",
"cache_directory",
"num_nodes",
"exclusive",
"error_log_file",
Expand Down
Loading