Launching SmartRedis on the cluster

[Fantasy98]

Hi!
I got an issue with lanunching the framework on a cluster.

Encountered issue:

When using a single GPU node on the cluster JUWELS, I encoutered an issue with using the slurm launcher

  # Setting up "slurm" runtime...
  �[91m
   !! Failed: Less than 2 hosts found in environment! !! 
  �[0m
  # Trying to setup LOCAL runtime instead...
  # Success!
  # Starting the Orchestrator...
  # Success!
  
  # Use this command to shutdown database if not terminated correctly:
  # $(smart dbcli) -h 127.0.0.1 -p 6557 shutdown
  
  # Configuration of runtime environment:
  #   Scheduler: local
  #   Hosts:     ['jwb0129']

This returns error regarding runtime, as the launcher is switched to local:

  File "/p/scratch/deepwing/yuningw/04-Reproduce-SOD2D/examples/juwels_gpu_cyl_rl/ywsmf/runtime.py", line 230, in launch_models
    raise ValueError('srun launcher only supported for SLURM scheduler!')
ValueError: srun launcher only supported for SLURM scheduler!

Background:

In config.yml:

      smartsim:
        n_dbs: 1
        network_interface: "ib0"
        run_command: "srun"
        launcher: "slurm"

For sbatch script:

      #SBATCH -t 01:00:00
      #SBATCH -p develbooster
      #SBATCH --exclusive
      #SBATCH -N 1
      #SBATCH --ntasks-per-node=1
      #SBATCH --cpus-per-task=48

My questions:

1. How to solve this issue?
2. Why requiring 2 hosts?
3. Any tricks for SBATCH?
4. Any room for improvement?
   Please let me know your thoughts on this, thanks a lot!

[Fantasy98]

I solve this by using one more node. The smartredis uses two nodes as worker and lord. It is solved now, thanks!

[soaringxmc]

Good!
If you want to use a single node on a cluster, you should treat the single node as a local workstation, i.e., local + mpirun
smartsim:
n_dbs: 1
network_interface: "lo"
run_command: "mpirun"
launcher: "local"

[Fantasy98]

Hi,
Now I encoutered another issue when using the major partition which does not have internet connection:

Default@20-52-54:ERROR: Redis IO error when executing command: failed to connect to Redis (10.13.23.59:6419): Connection refused
Default@20-52-55:ERROR: Redis IO error when executing command: failed to connect to Redis (10.13.23.59:6419): Connection refused
Default@20-52-56:ERROR: Redis IO error when executing command: failed to connect to Redis (10.13.23.59:6419): Connection refused
Default@20-52-57:ERROR: Redis IO error when executing command: failed to connect to Redis (10.13.23.59:6419): Connection refused
Default@20-52-58:ERROR: Redis IO error when executing command: failed to connect to Redis (10.13.23.59:6419): Connection refused
Default@20-52-59:ERROR: Redis IO error when executing command: failed to connect to Redis (10.13.23.59:6419): Connection refused
SmartRedis Library@20-53-00:ERROR: Exception at "/project/src/cpp/redis.cpp", line 671: Redis IO error when executing command: failed to connect to Redis (10.13.23.59:6419): Connection refused
20:53:00 jwb0065.juwels SmartSim[3720282:MainThread] ERROR 
20:53:00 jwb0065.juwels SmartSim[3720282:MainThread] ERROR Error while shutting down DB node. Return code: 1, err: Could not connect to Redis at 10.13.23.59:6419: Connection refused

20:53:00 jwb0065.juwels SmartSim[3720282:MainThread] INFO Stopping model orchestrator_0 with job name orchestrator_0-D9OG82L6OV6J
20:53:00 jwb0065.juwels SmartSim[3720282:MainThread] ERROR An error occurred while calling scancel: scancel: error: Kill job error on job step id 11457097.0: Invalid job id specified

20:53:00 jwb0065.juwels SmartSim[3720282:MainThread] WARNING Unable to cancel job step orchestrator_0-D9OG82L6OV6J
scancel: error: Kill job error on job step id 11457097.0: Invalid job id specified

Later the framework broke down due to TimeOut of DB connection:

  Traceback (most recent call last):
    File "/p/scratch/deepwing/yuningw/04-Reproduce-SOD2D/examples/juwels_gpu_cyl_rl/./ywsmf/main.py", line 62, in <module>
      main()
    File "/p/scratch/deepwing/yuningw/04-Reproduce-SOD2D/examples/juwels_gpu_cyl_rl/./ywsmf/main.py", line 28, in main
      run = wandb.init(
            ^^^^^^^^^^^
    File "/p/project1/deepwing/yuningw/envs/env-smartfow/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 1664, in init
      wandb._sentry.reraise(e)
    File "/p/project1/deepwing/yuningw/envs/env-smartfow/lib/python3.11/site-packages/wandb/analytics/sentry.py", line 156, in reraise
      raise exc.with_traceback(sys.exc_info()[2])
    File "/p/project1/deepwing/yuningw/envs/env-smartfow/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 1650, in init
      return wi.init(run_settings, run_config, run_printer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/p/project1/deepwing/yuningw/envs/env-smartfow/lib/python3.11/site-packages/wandb/sdk/wandb_init.py", line 1065, in init
      raise CommError(
  wandb.errors.errors.CommError: Run initialization has timed out after 90.0 sec. Please try increasing the timeout with the `init_timeout` setting: `wandb.init(settings=wandb.Settings(init_timeout=120))`.

Have you ever encountered similar issue?