Fix solver options for Real space

[connorjward]

The matrix type of a Real function space is now 'python' instead of 'nest' and this was failing to hit a case when we built our default solver options.

Fixes #5140

Note that we now appear to have two different mat types for matrix-free things: "matfree" and "python". Do we want to try and deprecate one? @leo-collins WDYT?

[leo-collins]

It would simplify things to use "python" as that's what PETSc calls them. But then "matfree" is clearer to users. I don't really mind either way.

[pbrubeck]

Note that we now appear to have two different mat types for matrix-free things: "matfree" and "python". Do we want to try and deprecate one? @leo-collins WDYT?

We always intercept -mat_type matfree to construct a MatPython wrapping a firedrake.matrix_free.operators.ImplicitMatrixContext. The MatPython created for the Real space is an entirely different beast, pyop2.types.mat._DatMatPayload. Overloading -mat_type python might not be a great idea because we might want to potentially introduce other MatPython classes (e.g. matfree_cuda, I know that the plan for GPU is a context manager, but suppose we also want finer control over which bits of a solver run on a given device from command line options).

If we change the interface we are just going to make things very ambiguous, how does python indicate the fact that we are not storing the matrix entries? But also, we should not change or deprecate matfree for the sake of not breaking users' codes.

Where are we letting -mat_type python happen?

[pbrubeck]

Whoever populated this dictionary with {"mat_type": A.type} within the adjoint should have set {"mat_type": "matfree" if A.type == "python" else A.type}.

Or more correclty, the matrix type should come from _SNESContext.mat_type

[pbrubeck]

I think the logic here could be responsible, but not 100% sure

firedrake/firedrake/matrix.py

Lines 22 to 27 in 2018173

	def _get_mat_type(petscmat: PETSc.Mat) -> str:
	"""Maps PETSc matrix types to Firedrake notation"""
	mat_type = petscmat.getType()
	if mat_type.startswith("seq") or mat_type.startswith("mpi"):
	return mat_type[3:]
	return mat_type

[connorjward]

Yeah that's what I found. I think we should:

1. Intercept mat_type=python
2. Extract the Python context from the mat
3. Use that to return matfree, denserow etc

[connorjward]

Maybe the logic failure here is that by having mat_type=python PETSc is handing responsibility over to us. Their solvers will never process this. We should never deal in mat_type=python and instead have mat_type=matfree, mat_type=denserow etc.

[pbrubeck]

This might be unrelated, but MUMPS supports MatNest when all blocks are aij. We shouldn't default to jacobi in that case.

[connorjward]

Yeah that's definitely unrelated to this. But we should open an issue for it at least.