Understand why CMS sees a speedup in DY+4jets but not DY+3 jets

[valassi]

This is a followup to the meeting with CMS last week and the meeting with CMS yesterday https://indico.cern.ch/event/1373473/

@choij1589 presented results where Drell Yan plus 4 jets shows a speedup in cudacpp vs fortran, but DY+3 does not

We should understand why CMS sees a speedup in DY+4jets but not DY+3 jets

---

[valassi]

Hi @choij1589 one question here: which one is your Fortran here?

• (1) Is it 'old/production Fortran' that you create with output madevent?
• (2) Or is it 'new/development Fortran' that you create with output madevent_simd and where you then put backend=fortran in the runcard?

Apologies if I should know the answer, I think I asked you (and I think you said it is option 1?).

My point here is the following: in any case it would be useful to compare the speed of (and also the physics results of) cuda and cpp to both fortran options above. Do you think you could try that too Jin at some point please? (On a simple case, eg 2jets, not 3 or 4).

Specifically about speed, the new fortran is probably going to be slower than the old fortran, because

• In cudacpp, the subprocess handling is different, so you will end up with many more P* subdirectories. For instance, old case is qq_xxx, new case will separate uux_xxx, uu_xxx, uc_xxx, ucx_xxx etc... This means there will be more overhead recombining these, it would be nice to know how much this is.
• In cudacpp, the vector API of fortran will be used (maybe this is also used in the 'old fortran' you have, but I am not sure). This should not be a slowdown in itself, but it is different, so it would be nice to keep this in mind.
• (And then there might be subtle changes in compiler flags etc...)

Bottom line: having a quick measurement of both fortran modes in a CMS environment could be useful.

[choij1589]

Hi @valassi, it's actually output madevent_gpu with cudacpp_backend = FORTRAN, so (2) I believe. I just ran a quick test in my local machine with DY+2j, indeed it takes much longer for compilation and integration with many more subprocesses divided into parton level. Here are some results:

1. output madevent
   8 subprocess directories
   xsec: 2188 $\pm$ 4.23 pb
   timing: 4m 59s

2. output madevent_gpu
   285 subprocess directories
   xsec: 2206 $\pm$ 4.657 pb
   timing: 44m 17s

One question: It is much faster especially for the compilation, can CUDA and CPP also benefit from merging the subprocesses?

[valassi]

Hi @choij1589 thanks a lot! Very very interesting.

About the subprocesses... I am afraid that this is not possible yet in cudacpp, and it would take a lot of work to implement that. I created #951 as a reminder. We would need to add many arrays(nprocesses) throughout the code to keep track of all subprocesses. Doable, but it's work. I am afraid that for the moment we will probably need to stay like this for a while.

About timings, 4m vs 44m, do these include build times, or are these generation times only once gridpacks are created?

About cross sections, I will copy this on #944. The results look strange. I think errors are underestimated...

[choij1589]

Hi @valassi , the timings I quoted are gridpack production time which incliude build times. For the event generation after untarring the gridpack, Here are some results with generating 5000 events with single thread (which will 285 subprocess directories most struggle...it takes most of the time in INFO: need to improve 332 channels):

1. output madevent
   2m 50s
2. output madevent_gpu
   9m 16s

[valassi]

Hi @choij1589 I made quite some progress in adding timing profiles for DY+012jets. This is work in progress in PR #946 using other work in progress from PR #948.

Some interesting results are here https://github.com/valassi/madgraph4gpu/tree/65bee817df09606311b4c6bdb66231e18903ba66/epochX/cudacpp/tlau/fromgridpacks/pp_dy012j.mad

The overall totals are

pp_dy012j.mad/fortran/output.txt
[GridPackCmd.launch] OVERALL TOTAL    61.5920 seconds
[madevent COUNTERS]  PROGRAM TOTAL    57.3602
[madevent COUNTERS]  Fortran Overhead 43.4883
[madevent COUNTERS]  Fortran MEs      13.8719
--------------------------------------------------------------------------------
pp_dy012j.mad/cppnone/output.txt
[GridPackCmd.launch] OVERALL TOTAL    73.9099 seconds
[madevent COUNTERS]  PROGRAM TOTAL    69.5728
[madevent COUNTERS]  Fortran Overhead 56.3117
[madevent COUNTERS]  CudaCpp MEs      13.2617
--------------------------------------------------------------------------------
pp_dy012j.mad/cppsse4/output.txt
[GridPackCmd.launch] OVERALL TOTAL    61.4144 seconds
[madevent COUNTERS]  PROGRAM TOTAL    57.1732
[madevent COUNTERS]  Fortran Overhead 50.157
[madevent COUNTERS]  CudaCpp MEs      7.0158
--------------------------------------------------------------------------------
pp_dy012j.mad/cppavx2/output.txt
[GridPackCmd.launch] OVERALL TOTAL    54.6737 seconds
[madevent COUNTERS]  PROGRAM TOTAL    50.4001
[madevent COUNTERS]  Fortran Overhead 46.8922
[madevent COUNTERS]  CudaCpp MEs      3.508
--------------------------------------------------------------------------------
pp_dy012j.mad/cpp512y/output.txt
[GridPackCmd.launch] OVERALL TOTAL    54.2811 seconds
[madevent COUNTERS]  PROGRAM TOTAL    50.0135
[madevent COUNTERS]  Fortran Overhead 46.6647
[madevent COUNTERS]  CudaCpp MEs      3.3489
--------------------------------------------------------------------------------
pp_dy012j.mad/cpp512z/output.txt
[GridPackCmd.launch] OVERALL TOTAL    52.6996 seconds
[madevent COUNTERS]  PROGRAM TOTAL    48.2535
[madevent COUNTERS]  Fortran Overhead 45.865
[madevent COUNTERS]  CudaCpp MEs      2.3889
--------------------------------------------------------------------------------

This shows that, at least in DY up to 2 jets, the ME is much faster than the fortran overhead inside each madevent execution (random to momenta translation in phase space sampling, pdfs, etc). The combine events step here is negligible (the difference between "GridPackCmd.launch" and "madevent COUNTERS TOTAL").

Eventually I will repeat this for 3 and 4 jets.

Note that there is also the multicore variability that may be relevant. I am not really sure here why the python elapsed time is larger than the sum of all madevent times (normally there should be several threads), Ineed to cross check this

PS NB THE ABOVE ARE TIMES FOR GENERATING EVENTS ON PRE-BUILT GRIDPACKS. This means that none of build times or phase space optimizations are included. But my understanding is that this is what we (and CMS) should want to optimize: the build and phase space optimization is done once, the generate events is done on bilions of events.

[valassi]

Here are some first results for DY+3jets. This is itgold91, a powerful CPU node with no GPU. Quite surprisingly (for me), the scalar overhead is again inside each madevent executable.

Even more strangely (but I had seen something similar in the past), even if the ME calculation on C++ nicely goes faster with vectorization as expected, the 'Fortran overhead' has two weird characteristics

• 1. It is much slower in 'cppnone' that in 'fortran' MEs
• 2. It also decreases from 'cppnone' to 'cpp512z' MEs
     This is strange because I would normally expect this to be a scalar overhead, instead it seems to have a parallel component that exploits vectorization? This should be better understood, I opened Reduce madevent 'Fortran overhead' by restricting cudacpp helicity calculation (which scales with SIMD) to only 16 events #958.

[avalassi@itgold91 gcc11/usr] /data/avalassi/GPU2024/madgraph4gpuX/epochX/cudacpp/tlau/fromgridpacks> ./parseGridpackLogs.sh pp_dy3j.mad
pp_dy3j.mad/fortran/output.txt
[GridPackCmd.launch] OVERALL TOTAL    442.6794 seconds
[madevent COUNTERS]  PROGRAM TOTAL    438.456
[madevent COUNTERS]  Fortran Overhead 131.711
[madevent COUNTERS]  Fortran MEs      306.744
--------------------------------------------------------------------------------
pp_dy3j.mad/cppnone/output.txt
[GridPackCmd.launch] OVERALL TOTAL    734.9419 seconds
[madevent COUNTERS]  PROGRAM TOTAL    730.685
[madevent COUNTERS]  Fortran Overhead 424.426
[madevent COUNTERS]  CudaCpp MEs      306.259
--------------------------------------------------------------------------------
pp_dy3j.mad/cppsse4/output.txt
[GridPackCmd.launch] OVERALL TOTAL    437.2589 seconds
[madevent COUNTERS]  PROGRAM TOTAL    433.014
[madevent COUNTERS]  Fortran Overhead 279.465
[madevent COUNTERS]  CudaCpp MEs      153.548
--------------------------------------------------------------------------------
pp_dy3j.mad/cppavx2/output.txt
[GridPackCmd.launch] OVERALL TOTAL    260.5085 seconds
[madevent COUNTERS]  PROGRAM TOTAL    256.272
[madevent COUNTERS]  Fortran Overhead 193.151
[madevent COUNTERS]  CudaCpp MEs      63.1221
--------------------------------------------------------------------------------
pp_dy3j.mad/cpp512y/output.txt
[GridPackCmd.launch] OVERALL TOTAL    253.3131 seconds
[madevent COUNTERS]  PROGRAM TOTAL    249.01
[madevent COUNTERS]  Fortran Overhead 189.606
[madevent COUNTERS]  CudaCpp MEs      59.4043
--------------------------------------------------------------------------------
pp_dy3j.mad/cpp512z/output.txt
[GridPackCmd.launch] OVERALL TOTAL    206.2917 seconds
[madevent COUNTERS]  PROGRAM TOTAL    202.013
[madevent COUNTERS]  Fortran Overhead 166.504
[madevent COUNTERS]  CudaCpp MEs      35.5097
--------------------------------------------------------------------------------

[valassi]

PS I wonder, specifically, if there is one part of the code that is affected by C++ -march flags for vectorization.