MPI/SRUN usage on a single cluster node

[Fantasy98]

Background: DRL using 4CFDs-Environments

• Using a computation node with 4GPUs and 64CPUS

• Using the smartsim configuration:

  smartsim:
    n_dbs: 1
    network_interface: "lo"
    run_command: "mpirun"
    launcher: "local"
  

Encountered issue:

• All of the CFD are run at GPU devices device:0, which leads to low-efficiency usage of the computation resources.

• The rank_file (i.e., .env000.txt) for launching mpirun is:

  rank 0=alvis4-05 slot=1        
  

Clearly, there is no binding of GPU devices.

Suggestions:

• Modify the usage of local / slurm configuration of SmartFlow to adopt this usage. We may consider any of the paths:

1. Make srun able to use for single cluster node
2. Incorporate the GPU-related arguments in rank_file.

I am currently working on the option#2, and I shall keep you posted by this issue. @soaringxmc

[soaringxmc]

Let's fix it together!

[Fantasy98]

Hi, I figured out the problem was in the lack of environment variable in the run_time configuration.

Modifications

Incorporate environment variable in Runtime:

if launcher == 'mpirun':
                    run_args = {
                        'rankfile': self.launch_config.rankfiles[i],
                        'report-bindings': None
                    }
                    env_args={}
                    ## YW added here: 
                    if n_gpus[i] !=0:
                        env_args["CUDA_VISIBLE_DEVICES"]=f"{idle_gpu}"
                        idle_gpu +=1
                    
                elif launcher == 'srun':
                    env_args={}
                    run_args = {
                        # 'mpi': 'pmix_v3',
                        'nodelist': ','.join(self.launch_config.hosts_per_exe[i]),
                        'distribution': 'block:block:block,Pack',
                        'cpu-bind': 'cores,verbose',
                        'exclusive': None,
                        'gres': 'gpu:0',
                    }
                
                run_settings = self.exp.create_run_settings(
                    exe=exe[i],
                    exe_args=exe_args[i],
                    run_command=launcher,
                    run_args=run_args,
                    #YW Add here: Update the Env variable to recognize the GPU device
                    env_vars=env_args,
                )

Add extra parameters in the environment configuration.

environment:
  gpus_per_cfd :    1  

This should be passed to run time argument accordingly.

Experiment: Multi-GPU usage is supported now.

It was able to work on 4 GPUs on local cluster. I am submitting the job to the cluster node for test.

I will keep you posted! @soaringxmc

[Fantasy98]

Lastest update: It is able to work on a 4-GPU computation node.

[soaringxmc]

Great news! Feel free to open a pull request whenever it's convenient for you. Then I could merge it to the main branch recently with your assistance.

[Fantasy98]

Great news! Feel free to open a pull request whenever it's convenient for you. Then I could merge it to the main branch recently with your assistance.

Sure thing. I would wait untill we finalize the results. Please let me know if other users encountered the same issue, then I will proceed with merging.

[soaringxmc]

Hi, @Fantasy98

I have merged your CUDA_VISIBLE_DEVICES into the newest commit. The most recent commit can support a single gpu node or multiple gpu nodes. You could try it on multiple gpu nodes if you need to. If there is any suggestions, feel free to do a PR. I tried not to bother you in the past couple of days, so I just merged your modifications without asking you to do a PR first.

We might consider doing another release after this commit is fully tested on cpus and gpus.

The new commit is in smartflow dev branch @LIUJUNLE97

[LIUJUNLE97]

Hi, @soaringxmc I have seen the latest update. I will soon make new env for the smartflow and do new simulations based on the new envs. Cheers.

[Fantasy98]

Hi, @soaringxmc I have seen the latest update. I will soon make new env for the smartflow and do new simulations based on the new envs. Cheers.

@LIUJUNLE97 I have added you to the SOD2D implementation Repo. Please find branch yw_alvis_dev for instructions for installation!

[Fantasy98]

Hi, @Fantasy98

I have merged your CUDA_VISIBLE_DEVICES into the newest commit. The most recent commit can support a single gpu node or multiple gpu nodes. You could try it on multiple gpu nodes if you need to. If there is any suggestions, feel free to do a PR. I tried not to bother you in the past couple of days, so I just merged your modifications without asking you to do a PR first.

We might consider doing another release after this commit is fully tested on cpus and gpus.

The new commit is in smartflow dev branch @LIUJUNLE97

@soaringxmc Thank you so much for putting the effort for integration. I will test the AFC case by using the new restart file.

[LIUJUNLE97]

Hi, @soaringxmc I have seen the latest update. I will soon make new env for the smartflow and do new simulations based on the new envs. Cheers.

@LIUJUNLE97 I have added you to the SOD2D implementation Repo. Please find branch yw_alvis_dev for instructions for installation!

Thanks. I will soon check the installation guidelines at yw_alvis_dev. Does it mean installing smartflow on Alvis coupled with SOD2D or CaLES?

[soaringxmc]

To test the new commit of smartflow, just keep using the most recent cales. After that, use sod2d.

[LIUJUNLE97]

@soaringxmc hi, Mao, just a quick check. The latest CaLES is the one with v2.0.0-smartflow tag shown in the figure?

[soaringxmc]

@LIUJUNLE97
Replied in a new discussion.