Updating to PartitionedArrays v0.5

[JordiManyer]

We have been detached from the development of PartitionedArrays for a while, currently stuck at v0.3. This is also related to #137 .

This PR is an update to PartitionedArrays v0.5, with a substantial rework of the re-assembly process... This comes with several upsides, such as

• split format for distributed vectors and matrices
• re-partitioning of distributed data
• sub-assembled and dis-assembled matrices
• optimisations

Change Log

• FESpace free dof ids are now given by a permuted instance of LocalIndicesWithVariableBlockSize. We were already doing this (so our numbering strategy does not change), but it's now explicit. This has two advantages:

  • To find the index partitions for the assembled system, we just un-permute the indices. I.e we get the underlying LocalIndicesWithVariableBlockSize.
  • Finding owners for arbitrary gids does now require minimal communications (just a scan operation). This is quite handy for assembly, since expanding the ghost ids in the index partition can be done using PArrays's find_owner and union_ghost methods.

• With PA 0.5, new assembly strategies are available. Also, we've been wanting to fix the confusing names we currently have. We now have the following assembly strategies:

  • Assembled : We integrate on owned cells and collect contributions for owned and ghost row ids, then exchange ghost contributions and assemble. Current SubAssembledRows.
  • SubAssembled : We integrate on owned cells and collect contributions for owned and ghost row ids, but no exchange. The resulting matrix is sub-assembled, i.e each local matrix contains the contribution for it's owned and ghost rows/cols resulting from integrating over the owned part of the domain.
  • LocallyAssembled : This assumes all contributions can be found by integrating on the local portion of the domain. We integrate on local cells (owned and ghost), and keep contributions for owned row ids. No exchange is needed. Current FullyAssembledRows. Comes at your own risk.

• The indices in the matrices and vectors are now LocalIndicesWithVariableBlockSize, so that the owner of a dof can be locally deduced by it's gid. This will save us some communication.

To-Do

• New assembly stretegy equivalent to Assembled, but for which extra ghost rows are not allocated. This strategy will re-assemble using COO matrices (instead of directly inserting in the matrix). This can save binary searches, which can be quite costly. The re-assembly caches will be stored by the user (this will be an advanced api).
• Allow for DoF reordering such that we have owned then ghosts. The serial code is already prepared from this PR. We only need a kwarg within the distributed FESpace constructors. This will allow for the use of FESpace-allocated vectors with the system matrix for SubAssembled() assembly strategy.
• PartitionedArrays will relax the conditions on the ghost layouts for it's matrix-vector product. See Sparse-matrix vector product A*x for general ghosts in x PartitionedArrays/PartitionedArrays.jl#127.

Assembly cache reuse strategy

The most involved part of this PR is the assembly cache re-use. The issue is that so far we had been using the output PSparseMatrix as the cache, by allocating extra ghost rows that were only used in the sub-assembly process. This has quite a lot of downsides (see #137), so we are pivoting towards an external assembly cache model.

However, having external caches does not go well with the current Gridap API. Here is a possible solution:

• assemble_matrix! will not be supported, except if an Assembler is provided (see next point).
• The Assembler will hold the caches. To be able to reuse an assembler for multiple matrices, we will use a Dict and the matrix object-id to hold multiple caches (tied to a single matrix each). Cache re-use will be activated by boolean variable reuse that is held by the Assembler.
• The FEOperators will keep an instance of their assembler, allowing re-use by default for nonlinear and transient FEOperators.

[codecov]

Codecov Report

❌ Patch coverage is 66.48822% with 313 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.38%. Comparing base (8268f41) to head (e491eae).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
src/Partitions.jl	42.54%	208 Missing ⚠️
src/PArraysExtras.jl	71.95%	53 Missing ⚠️
src/Assembly.jl	87.44%	27 Missing ⚠️
src/BlockPartitionedArrays.jl	33.33%	14 Missing ⚠️
src/FESpaces.jl	87.01%	10 Missing ⚠️
src/GridapExtras.jl	96.87%	1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (8268f41) and HEAD (e491eae). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (8268f41)	HEAD (e491eae)
mpi-fespaces	2	1
mpi-physics	2	1

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #157      +/-   ##
==========================================
- Coverage   84.72%   79.38%   -5.34%     
==========================================
  Files          16       19       +3     
  Lines        4249     4255       +6     
==========================================
- Hits         3600     3378     -222     
- Misses        649      877     +228     

Flag	Coverage Δ
mpi-adaptivity	41.11% <43.25%> (-4.53%)	⬇️
mpi-fespaces	33.70% <54.60%> (-6.40%)	⬇️
mpi-geometry	23.22% <12.75%> (-0.09%)	⬇️
mpi-misc	44.92% <56.31%> (-5.54%)	⬇️
mpi-physics	29.41% <49.61%> (-6.38%)	⬇️
mpi-transient	20.16% <42.58%> (-4.79%)	⬇️
seq-adaptivity	35.67% <35.88%> (-3.89%)	⬇️
seq-fespaces	36.84% <54.38%> (-5.35%)	⬇️
seq-geometry	23.23% <12.75%> (-0.11%)	⬇️
seq-misc	29.76% <52.25%> (-5.52%)	⬇️
seq-physics	30.29% <50.91%> (-5.38%)	⬇️
seq-transient	19.26% <41.44%> (-4.93%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[JordiManyer]

@amartinhuertas I think this starts being ready. There are still a couple things I want to do, but they are optional. All tests we have in master pass, I believe. So (no rush) I think it is time to start reviewing this.