combine inference from multiple ranks in evaluation

[iluise]

Description

implement automatic merging of multiple rank files. The code assigns a new incremental "global' sample index across ranks, because the sample index is repeated across files.

Three supported modes:

run_id:
    label: xxx
    channels: xxxx
    rank: "all" or [0,2,3,] (list ) or 0 (int)

• "all" automatic detection and merging of all ranks.
• list [0,1,2,3] useful e.g. for yearly predictions when one wants to load only one month of data
• int: e.g. 0 for backward compatibility.

Issue Number

Closes #541

Is this PR a draft? Mark it as draft.

Checklist before asking for review

• I have performed a self-review of my code
• My changes comply with basic sanity checks:
  • I have fixed formatting issues with ./scripts/actions.sh lint
  • I have run unit tests with ./scripts/actions.sh unit-test
  • I have documented my code and I have updated the docstrings.
  • I have added unit tests, if relevant
• I have tried my changes with data and code:
  • I have run the integration tests with ./scripts/actions.sh integration-test
  • (bigger changes) I have run a full training and I have written in the comment the run_id(s): launch-slurm.py --time 60
  • (bigger changes and experiments) I have shared a hegdedoc in the github issue with all the configurations and runs for this experiments
• I have informed and aligned with people impacted by my change:
  • for config changes: the MatterMost channels and/or a design doc
  • for changes of dependencies: the MatterMost software development channel