lock in PetscMatrix::add has a lot of contention

[lindsayad]

This was a 4 million dof problem split among 30 threads. I think we need to do a lot more batching. 73% of run-time!

(pprof) list PetscMatrix::add
Total: 1916.43s
ROUTINE ======================== libMesh::PetscMatrix::add in /data/lindad/projects/cpu-moose/libmesh/src/numerics/petsc_matrix.C
     2.03s   1404.48s (flat, cum) 73.29% of Total
         .          .    987:
         .          .    988:template <typename T>
         .          .    989:void PetscMatrix<T>::add (const numeric_index_type i,
         .          .    990:                          const numeric_index_type j,
         .          .    991:                          const T value)
      60ms       60ms    992:{
         .          .    993:  libmesh_assert (this->initialized());
         .          .    994:
      10ms       10ms    995:  PetscInt i_val=i, j_val=j;
         .          .    996:
         .          .    997:  PetscScalar petsc_value = static_cast<PetscScalar>(value);
         .    850.22s    998:  std::scoped_lock lock(this->_petsc_matrix_mutex);
     1.84s      7.74s    999:  LibmeshPetscCall(MatSetValues(this->_mat, 1, &i_val, 1, &j_val,
         .          .   1000:                                &petsc_value, ADD_VALUES));
     120ms    546.45s   1001:}

[roystgnr]

Wait - is MOOSE using the 1×1 add(), not the n×n? Adding a dense submatrix with one call (or multiple dense submatrices with each call, in cases where the coupling is there) might be as helpful as batching multiple adds together.