Recipe failing due to non existent directory.

[h2oreactor]

The recipe is failing when performing a copy[1], this was due to non existent source path.

cp: cannot stat
'/workspace/nemo_experiments/megatron_gpt/XXYYZZ/*':
No such file or directory

[1]

gpu-recipes/src/helm-charts/a3mega/nemo-training/templates/nemo-launcher-job.yaml

Line 476 in 5be5988

cp -r /workspace/nemo_experiments/megatron_gpt/$JOB_IDENTIFIER/* /gcs/nemo-experiments/$JOB_IDENTIFIER/

[Abhishekbhagwat]

@dasoto, could you please help to take a look ?

[dasoto]

If the folder does not exist, it means the training process fail. Please check if the training process start by checking the logs of the pods.