SubGridZ

[FlorianDeconinck]

Outside of the main fortran call there's a re-grid using a SubGridZ routine which looks very similar to the ported SubGridZ in pyFV3

Original code: FV_StateMod.f90:2057

            call fv_subgrid_z(isd, ied, jsd, jed, isc, iec, jsc, jec, FV_Atm(1)%npz, & ...

Parent: GEOS-ESM/SMT-Nebulae#55

---

• Port fv_subgrid_z to ndsl
• Expand the bridge to include the ported fv_subgrid_z

[gmao-ckung]

Overall, the version of fv_subgrid_z.F90 from FV3GFS_Fortran and GEOS are almost identical. There are differences codebases depending on the use of macros; however, the code path used to develop PyFV3 from FV3GFS_Fortran and the "typical" codepath used in GEOS are also almost identical.

Updated fv_subgridz.py to consider computation on varying levels of k as the original PyFV3 version iterated the entire range of k.

Translate tests are run using the following settings.

• PACE_FLOAT_PRECISION=32 (environment variable)
• PACE_CONSTANTS="GEOS" (environment variable)
• GEOS built in Debug mode
• Constants from constants.py listed under from ndsl.constants import(... in fv_subgridz.py were cast using Float based off of from ndsl.dsl.typing import Float
• For some reason, MPI is needed to run the translate tests: mpirun -mca orte_abort_on_non_zero_status 1 -np 6 --oversubscribe python -m pytest -v -s --data_path=../test_data/8.1.3/c12_6ranks_standard/dycore --backend=numpy -m parallel --which_modules=FVSubgridZ --threshold_overrides_file=./tests/savepoint/translate/overrides/standard.yaml ./tests/savepoint

The number of non-matching solutions are minimal (maximum of 0.43% not matching for pt); however, having another dataset to verify will ensure more confidence that fv_subgridz.py is working properly.

Variable	Rank 0	Rank 1	Rank 2	Rank 3	Rank 4	Rank 5
ua (64800 entries)	X (15 failed / 0.02%)				X (3 failed / 0.0%)
va (64800 entries)	X (15 failed / 0.02%)				X (3 failed / 0.0%)
w (64800 entries)	X (5 failed / 0.01%)				X (2 failed / 0.0%)
pt (64800 entries	X (279 failed / 0.43%)	X (268 failed / 0.41%)	X (272 failed / 0.42%)	X (266 failed / 0.41%)	X (288 failed / 0.44%)	X (268 failed / 0.41%)
qvapor (64800 entries)	X (1 failed / 0.0%)
qliquid
qice
qrain
qsnow
qgraupel
qcld
qo3mr
u_dt (41472 entries)	X (15 failed / 0.04%)				X (3 failed / 0.01%)
v_dt (41472 entries)	X (13 failed / 0.03%)				X (2 failed / 0.00%)
t_dt	X (279 failed / 0.43%)	X (268 failed / 0.41%)	X (272 failed / 0.42%)	X (266 failed / 0.41%)	X (288 failed / 0.44%)	X (268 failed / 0.41%)
w_dt	X (5 failed / 0.01%)				X (2 failed / 0.0%)

Tracers are still handled directly instead of in a loop.

Note that certain tracers in fv_subgridz.py were commented (ex: q0_sgs_tke and qsgs_tke) since the equivalent wasn't found in GEOS.

The updated code is pushed into GEOS-ESM's PyFV3 repo under the branch feature/fv_subgrid_z

[FlorianDeconinck]

MPI is needed to run the translate tests

For some reason the FVSubgridZ translate was done as a parallel test to mirror the larger dycore and acoustic ones. There is no real reason for that, it should eventually be refactored into a non-parallel test