openmp is not working on travis-ci

[TotoGaz]

When activating openmp the following travis-ci's unit tests fail in timeout.

• Ubuntu18.04_gcc8_release
• Ubuntu18.04_gcc8_debug
• Pangea2_gcc8.3.0_release (when the unit tests were activated.)

The failing test is testKrylovSolvers which seems to provide the proper results but which is incredibly slow.

log for Ubuntu18.04_gcc8_debug

        Start  81: testKrylovSolvers

81: Test command: /usr/bin/mpirun "-n" "2" "--allow-run-as-root" "/tmp/build/tests/testKrylovSolvers" "-x" "2"
81: Test timeout computed to be: 1500
81: --------------------------------------------------------------------------
81: [[39152,1],1]: A high-performance Open MPI point-to-point messaging module
81: was unable to find any relevant network interfaces:
81: 
81: Module: OpenFabrics (openib)
81:   Host: 8965c534eb7c
81: 
81: Another transport will be used instead, although this may result in
81: lower performance.
81: 
81: NOTE: You can disable this warning by setting the MCA parameter
81: btl_base_warn_component_unused to 0.
81: --------------------------------------------------------------------------
81: Max threads: 2
81: [==========] Running 12 tests from 4 test suites.
81: [----------] Global test environment set-up.
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
81: [==========] Running 12 tests from 4 test suites.
81: [----------] Global test environment set-up.
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
81: [8965c534eb7c:06635] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
81: [8965c534eb7c:06635] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
81: [       OK ] Trilinos/KrylovSolverTest/0.CG (11155 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Trilinos/KrylovSolverTest/0.CG (11152 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (20370 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (20370 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (186291 ms)
81: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (186291 ms)
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (217813 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverTest/0.CG
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (217816 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverTest/0.CG
81: [       OK ] Hypre/KrylovSolverTest/0.CG (17177 ms)
81: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Hypre/KrylovSolverTest/0.CG (17178 ms)
81: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (28200 ms)
81: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (28198 ms)
81: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
81: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
81: [       OK ] Hypre/KrylovSolverTest/0.GMRES (337348 ms)
81: [----------] 3 tests from Hypre/KrylovSolverTest/0 (382725 ms total)
81: 
81: [       OK ] Hypre/KrylovSolverTest/0.GMRES (337348 ms)
81: [----------] 3 tests from Hypre/KrylovSolverTest/0 (382724 ms total)
81: 
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (21988 ms)
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (21984 ms)
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (23074 ms)
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (23076 ms)
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (369169 ms)
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (369167 ms)
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (414227 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (414231 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
81: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (20332 ms)
81: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (20332 ms)
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
81: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (63424 ms)
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
81: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (63422 ms)
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
 81/123 Test  #81: testKrylovSolvers ....................................***Timeout 1500.10 sec

[TotoGaz]

A working log for the same job is

Working test log for Ubuntu18.04_gcc8_debug (no openmp)

        Start  80: testKrylovSolvers

80: Test command: /usr/bin/mpirun "-n" "2" "--allow-run-as-root" "/tmp/build/tests/testKrylovSolvers" "-x" "2"
80: Test timeout computed to be: 1500
80: --------------------------------------------------------------------------
80: [[17404,1],0]: A high-performance Open MPI point-to-point messaging module
80: was unable to find any relevant network interfaces:
80: 
80: Module: OpenFabrics (openib)
80:   Host: 23c09a095a52
80: 
80: Another transport will be used instead, although this may result in
80: lower performance.
80: 
80: NOTE: You can disable this warning by setting the MCA parameter
80: btl_base_warn_component_unused to 0.
80: --------------------------------------------------------------------------
80: [==========] Running 12 tests from 4 test suites.
80: [----------] Global test environment set-up.
80: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
80: [==========] Running 12 tests from 4 test suites.
80: [----------] Global test environment set-up.
80: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
80: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
80: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
80: [       OK ] Trilinos/KrylovSolverTest/0.CG (37 ms)
80: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
80: [       OK ] Trilinos/KrylovSolverTest/0.CG (33 ms)
80: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
80: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (44 ms)
80: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
80: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (43 ms)
80: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
80: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (347 ms)
80: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (428 ms total)
80: 
80: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
80: [ RUN      ] Hypre/KrylovSolverTest/0.CG
80: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (346 ms)
80: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (425 ms total)
80: 
80: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
80: [ RUN      ] Hypre/KrylovSolverTest/0.CG
80: [       OK ] Hypre/KrylovSolverTest/0.CG (33 ms)
80: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
80: [       OK ] Hypre/KrylovSolverTest/0.CG (31 ms)
80: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
80: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (51 ms)
80: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
80: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (51 ms)
80: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
80: [       OK ] Hypre/KrylovSolverTest/0.GMRES (423 ms)
80: [----------] 3 tests from Hypre/KrylovSolverTest/0 (507 ms total)
80: 
80: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
80: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
80: [       OK ] Hypre/KrylovSolverTest/0.GMRES (423 ms)
80: [----------] 3 tests from Hypre/KrylovSolverTest/0 (506 ms total)
80: 
80: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
80: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
80: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (81 ms)
80: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
80: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (81 ms)
80: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
80: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (124 ms)
80: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
80: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (124 ms)
80: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
80: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (730 ms)
80: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (935 ms total)
80: 
80: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
80: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
80: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (730 ms)
80: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (935 ms total)
80: 
80: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
80: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
80: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (93 ms)
80: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
80: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (93 ms)
80: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
80: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (154 ms)
80: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
80: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (154 ms)
80: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
80: [       OK ] Hypre/KrylovSolverBlockTest/0.GMRES (905 ms)
80: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0 (1152 ms total)
80: 
80: [----------] Global test environment tear-down
80: [==========] 12 tests from 4 test suites ran. (3022 ms total)
80: [       OK ] Hypre/KrylovSolverBlockTest/0.GMRES (905 ms)
80: [  PASSED  ] 12 tests.
80: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0 (1152 ms total)
80: 
80: [----------] Global test environment tear-down
80: [==========] 12 tests from 4 test suites ran. (3022 ms total)
80: [  PASSED  ] 12 tests.
80: Umpire            HOST high water mark:  718.7 KB
80: [23c09a095a52:06545] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
80: [23c09a095a52:06545] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
 80/122 Test  #80: testKrylovSolvers ....................................   Passed    3.24 sec

I cannot reproduce the log in the initial issue, but my process is still only ten times faster. The normal process is ~100 times faster (and often more).

And this is the Centos7.7_clang9.0.0_release target e.g.

Log for Centos7.7_clang9.0.0_release, openmp activated

       Start  81: testKrylovSolvers
81: Test command: /usr/lib64/openmpi/bin/mpirun "-n" "2" "--allow-run-as-root" "/tmp/build/tests/testKrylovSolvers" "-x" "2"
81: Test timeout computed to be: 1500
81: --------------------------------------------------------------------------
81: [[49735,1],0]: A high-performance Open MPI point-to-point messaging module
81: was unable to find any relevant network interfaces:
81: 
81: Module: OpenFabrics (openib)
81:   Host: 4a0f3d75b50b
81: 
81: Another transport will be used instead, although this may result in
81: lower performance.
81: --------------------------------------------------------------------------
81: [==========] Running 12 tests from 4 test suites.
81: [----------] Global test environment set-up.
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
81: Max threads: 2
81: [==========] Running 12 tests from 4 test suites.
81: [----------] Global test environment set-up.
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
81: [       OK ] Trilinos/KrylovSolverTest/0.CG (60 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Trilinos/KrylovSolverTest/0.CG (62 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (71 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (73 ms)
81: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (682 ms)
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (816 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverTest/0.CG
81: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (697 ms)
81: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (831 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverTest/0.CG
81: [       OK ] Hypre/KrylovSolverTest/0.CG (150 ms)
81: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Hypre/KrylovSolverTest/0.CG (145 ms)
81: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
81: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (82 ms)
81: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
81: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (85 ms)
81: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
81: [       OK ] Hypre/KrylovSolverTest/0.GMRES (923 ms)
81: [----------] 3 tests from Hypre/KrylovSolverTest/0 (1158 ms total)
81: 
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
81: [       OK ] Hypre/KrylovSolverTest/0.GMRES (923 ms)
81: [----------] 3 tests from Hypre/KrylovSolverTest/0 (1155 ms total)
81: 
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (274 ms)
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (271 ms)
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (253 ms)
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (253 ms)
81: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (1782 ms)
81: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (1784 ms)
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (2314 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
81: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (2310 ms total)
81: 
81: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
81: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (253 ms)
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
81: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (249 ms)
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
81: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (397 ms)
81: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (397 ms)
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
81: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
81: [4a0f3d75b50b:04932] 1 more process has sent help message help-mpi-btl-base.txt / btl:no-nics
81: [4a0f3d75b50b:04932] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
81: [       OK ] Hypre/KrylovSolverBlockTest/0.GMRES (2291 ms)
81: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0 (2941 ms total)
81: 
81: [----------] Global test environment tear-down
81: [       OK ] Hypre/KrylovSolverBlockTest/0.GMRES (2290 ms)
81: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0 (2937 ms total)
81: 
81: [----------] Global test environment tear-down
81: [==========] 12 tests from 4 test suites ran. (7236 ms total)
81: [  PASSED  ] 12 tests.
81: [==========] 12 tests from 4 test suites ran. (7239 ms total)
81: [  PASSED  ] 12 tests.
81: Umpire            HOST high water mark:  718.7 KB
 81/123 Test  #81: testKrylovSolvers ....................................   Passed    7.58 sec

[TotoGaz]

Actually, test testLAOperations is also running really slow, but under the 1500 second limit.

The standard test without openmp lasts

78/123 Test  #78: testLAOperations .....................................   Passed    3.19 sec

while with openmp we have

79/124 Test  #79: testLAOperations .....................................   Passed  224.98 sec

[TotoGaz]

Basic profilings let me think that ~85% of the time is spent in gomp_barrier_wait_end and gomp_team_barrier_wait_end. Still need to find out more precise information.

[TotoGaz]

I wanted to try with more recent compiler version.

Because of some build issue with current trilinos 12-18-1, I upgraded to version 13-0-1 (all other things being equal) the bug was still there (of course, all LAI seem impacted so 🤷 ).

Then I moved to ubuntu 20.10 with gcc 9.3.0 and the bug does not appear anymore.
While on ubuntu 18.04 with gcc 8.4.0, things are going wrong.

We shall try on std HPC cluster env to get further confirmation about that.

[klevzoff]

When working on #1086 I had some issues with OpenMP performance and I followed advice given by Kokkos to set the following:

OMP_PROC_BIND=spread
OMP_PLACES=threads

I don't know if this has anything to do with the issue at hand (and in particular in docker containers), but might be worth trying.

[TotoGaz]

On quartz with gcc 8.1.0

Debug:

        Start  88: testLAOperations
 88/135 Test  #88: testLAOperations .....................................   Passed    5.33 sec
        Start  89: testArrayLAOperations
 89/135 Test  #89: testArrayLAOperations ................................   Passed    3.19 sec
        Start  90: testKrylovSolvers
 90/135 Test  #90: testKrylovSolvers ....................................   Passed   17.65 sec

Release:

        Start  88: testLAOperations
 88/135 Test  #88: testLAOperations .....................................   Passed    4.06 sec
        Start  89: testArrayLAOperations
 89/135 Test  #89: testArrayLAOperations ................................   Passed    0.86 sec
        Start  90: testKrylovSolvers
 90/135 Test  #90: testKrylovSolvers ....................................   Passed   16.53 sec

and

> ctest -V -R testKrylovSolvers
UpdateCTestConfiguration  from :/usr/WS2/corbett5/GEOSX/build-quartz-gcc@8.1.0-release/DartConfiguration.tcl
Parse Config file:/usr/WS2/corbett5/GEOSX/build-quartz-gcc@8.1.0-release/DartConfiguration.tcl
UpdateCTestConfiguration  from :/usr/WS2/corbett5/GEOSX/build-quartz-gcc@8.1.0-release/DartConfiguration.tcl
Parse Config file:/usr/WS2/corbett5/GEOSX/build-quartz-gcc@8.1.0-release/DartConfiguration.tcl
Test project /usr/WS2/corbett5/GEOSX/build-quartz-gcc@8.1.0-release
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 90
    Start 90: testKrylovSolvers
90: Test command: /usr/bin/srun "-n" "2" "/usr/WS2/corbett5/GEOSX/build-quartz-gcc@8.1.0-release/tests/testKrylovSolvers" "-x" "2"
90: Test timeout computed to be: 1500
90: Max threads: 18
90: [==========] Running 12 tests from 4 test suites.
90: [----------] Global test environment set-up.
90: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
90: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
90: MKL max threads: 18
90: [==========] Running 12 tests from 4 test suites.
90: [----------] Global test environment set-up.
90: [----------] 3 tests from Trilinos/KrylovSolverTest/0, where TypeParam = geosx::TrilinosInterface
90: [ RUN      ] Trilinos/KrylovSolverTest/0.CG
90: [       OK ] Trilinos/KrylovSolverTest/0.CG (192 ms)
90: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
90: [       OK ] Trilinos/KrylovSolverTest/0.CG (194 ms)
90: [ RUN      ] Trilinos/KrylovSolverTest/0.BiCGSTAB
90: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (189 ms)
90: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
90: [       OK ] Trilinos/KrylovSolverTest/0.BiCGSTAB (189 ms)
90: [ RUN      ] Trilinos/KrylovSolverTest/0.GMRES
90: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (6575 ms)
90: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (6958 ms total)
90: 
90: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
90: [ RUN      ] Hypre/KrylovSolverTest/0.CG
90: [       OK ] Trilinos/KrylovSolverTest/0.GMRES (6575 ms)
90: [----------] 3 tests from Trilinos/KrylovSolverTest/0 (6956 ms total)
90: 
90: [----------] 3 tests from Hypre/KrylovSolverTest/0, where TypeParam = geosx::HypreInterface
90: [ RUN      ] Hypre/KrylovSolverTest/0.CG
90: [       OK ] Hypre/KrylovSolverTest/0.CG (26 ms)
90: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
90: [       OK ] Hypre/KrylovSolverTest/0.CG (26 ms)
90: [ RUN      ] Hypre/KrylovSolverTest/0.BiCGSTAB
90: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (40 ms)
90: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
90: [       OK ] Hypre/KrylovSolverTest/0.BiCGSTAB (40 ms)
90: [ RUN      ] Hypre/KrylovSolverTest/0.GMRES
90: [       OK ] Hypre/KrylovSolverTest/0.GMRES (350 ms)
90: [----------] 3 tests from Hypre/KrylovSolverTest/0 (416 ms total)
90: 
90: [       OK ] Hypre/KrylovSolverTest/0.GMRES (350 ms)
90: [----------] 3 tests from Hypre/KrylovSolverTest/0 (416 ms total)
90: 
90: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
90: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
90: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0, where TypeParam = geosx::TrilinosInterface
90: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.CG
90: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (203 ms)
90: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
90: [       OK ] Trilinos/KrylovSolverBlockTest/0.CG (203 ms)
90: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB
90: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (243 ms)
90: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
90: [       OK ] Trilinos/KrylovSolverBlockTest/0.BiCGSTAB (243 ms)
90: [ RUN      ] Trilinos/KrylovSolverBlockTest/0.GMRES
90: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (6920 ms)
90: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (7366 ms total)
90: 
90: [       OK ] Trilinos/KrylovSolverBlockTest/0.GMRES (6920 ms)
90: [----------] 3 tests from Trilinos/KrylovSolverBlockTest/0 (7366 ms total)
90: 
90: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
90: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
90: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0, where TypeParam = geosx::HypreInterface
90: [ RUN      ] Hypre/KrylovSolverBlockTest/0.CG
90: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (75 ms)
90: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
90: [       OK ] Hypre/KrylovSolverBlockTest/0.CG (75 ms)
90: [ RUN      ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB
90: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (129 ms)
90: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
90: [       OK ] Hypre/KrylovSolverBlockTest/0.BiCGSTAB (129 ms)
90: [ RUN      ] Hypre/KrylovSolverBlockTest/0.GMRES
90: [       OK ] Hypre/KrylovSolverBlockTest/0.GMRES (749 ms)
90: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0 (953 ms total)
90: 
90: [----------] Global test environment tear-down
90: [==========] 12 tests from 4 test suites ran. (15691 ms total)
90: [  PASSED  ] 12 tests.
90: [       OK ] Hypre/KrylovSolverBlockTest/0.GMRES (750 ms)
90: [----------] 3 tests from Hypre/KrylovSolverBlockTest/0 (954 ms total)
90: 
90: [----------] Global test environment tear-down
90: [==========] 12 tests from 4 test suites ran. (15695 ms total)
90: [  PASSED  ] 12 tests.
90: Umpire            HOST sum across ranks:  718.7 KB
90: Umpire            HOST         rank max:  359.3 KB
1/1 Test #90: testKrylovSolvers ................   Passed   16.56 sec
The following tests passed:
	testKrylovSolvers
100% tests passed, 0 tests failed out of 1
Total Test time (real) =  16.76 sec

but with OMP_NUM_THREADS=1 it takes 3 seconds.

Thx @corbett5 for running the cases.

[TotoGaz]

Closed as "won't fix". It's maybe a bug in the compiler?