Exchange slowdown

[lterrac]

We are experiencing slowdowns in the exchange operation used in HiCR. This happens for both MPI and LPF backends.

I have a minimal working example to start troubleshooting the issue.

The part of the example where the slowdown happens is in the createChannels(). In such functions, for N instances we are creating N channels, to organize the instances into a channel ring. To have an idea of the slowdown:

• 3 instances: 4 seconds
• 6 instances: 21 seconds
• 7 instances: 38 seconds

We are currently running in a single node setup with multiple ranks. Timings are similar for lpfThread and mpiThread.

Setup and compile:

git submodule update --init
meson setup build -Dengines=mpi,lpf -DbuildExamples=true -DbuildTests=true
meson compile -C build

Run examples:

mpirun -n 3 --oversubscribe env LPF_ENGINE=zero build/examples/telephoneGame/lpfThread
mpirun -n 3 --oversubscribe build/examples/telephoneGame/mpiThread

[lterrac]

@anyzelman

[anyzelman]

I quickly looked over the example and the number of channels created is linear, not quadratic. Assuming (to be checked) that the number of calls to lpf_register_global thus is linear, there should not be a quadratic run-time overhead according to the LPF performance semantics-- yet this is what the timings indicate. TBC and then delved deep on soon.

[lterrac]

@anyzelman yes, the number of channels is linear. Build instruction are either in the first message of this PR, or alternatively at the root of the channel creator repo link

[anyzelman]

Gathering comments so far-- the issue:

• disappears when the run is over P separate nodes, instead of having P processes on a single node;
• has been tracked down to a MPI collective operation (I suppose this one: https://github.com/Algebraic-Programming/LPF/blob/master/src/MPI/ibverbs.cpp#L512);
• occurs with OpenMPI (various versions), has not been tested or observed with other MPIs;
• is not on a critical path that requires resolution (but we nonetheless retain it in case it becomes more relevant or urgent later).

Given the information so far, couple of questions @lterrac :

• it is correct this happens when multiple processes are co-located, correct? (These are not multiple threads, these are multiple proceses?)
• I notice oversubscribe as an argument to mpirun: this leads me to suspect that perhaps the three processes are assigned to the same core? (If using slurm, are you sure you have multiple cores and that there are P slots that each go to their own core? Because if the answer was yes, then oversubscribe should not be necessary -- which leads me to believe that maybe these processes are indeed going to the same core)

[lterrac]

1. yes: multiple ranks, same machine
2. I forgot to take it out of the command. I used it when I was testing the channel creator frontend with huge number of ranks (btw, super slow but works 😄). But neither Kiril nor Sergio were using it with the flag.

P.s. I do not recall seeing the scenario you are describing for 2) (i.e., N ranks mapped to the same core). When I was running the tests on Sergio's workstation I always saw full utilization of the machine cores first. But maybe I am mistaken, I can do a quick run tomorrow