Coarse resolution grid for fast testing

[kdraeder]

This issue stems from cime #4933, which is about developing a large ensemble test
motivated by DART applications.

Because of the large ensemble, the testing will be more managable
if it uses a coarse resolution grid. An ne3 grid is available for CAM and CTSM,
and now a ~10-degree resolution is available in MOM6 (MOM_interface #311).
These have been combined into a new CESM grid and used in ERI and MCC tests,
which also use a new testmod tailored to DART needs.
I'm open to suggestions for a shorter testmod name,
but @billsacks and I feel that it will be helpful to have DART in it.
This grid (especially the MOM6 grid) limits the tasks/instance to 12
(6 for MOM, 6 for the other components).

The CESM version I'm using is copied from what @alperaltuntas used
for developing the coarse resolution MOM6 grid (cesm3_0_alpha08d):
/glade/work/raeder/Models/cesm3_0_alpha08d_mar13

An MCC test for a small ensemble passes all test stages
(/glade/work/raeder/Exp/CESM+DART_testing/MCC_cG.ne3pg3_10deg.B_DART.lowres)
but ensembles which require more than 1 node mostly fail
with an error in cmeps/cesm/driver/ensemble_driver.F90.
This seems to arise from smaller ensembles fitting into a single (develop queue) node,
where the exact number of processors needed is assigned to them,
while larger ensembles need multiple (cpu/main) nodes
and more processors are assigned to the job than are needed.
For example, 40 instances request 12 x 40 = 480 processors.
This means that 4 nodes x 128 = 512 processors are assigned.
This difference causes an error:
PetCount ( 512) - Async IOtasks ( 0) must be evenly divisable by number of members ( 40).

When the check for this error is commented out, the job goes farther,
but hangs just before the time stepping in CAM.
One final data point is that a 32 instance ensemble fits exactly into 3 nodes
and it does not fail in ensemble_driver.F90, but it also hangs later.
This should maybe be a separate issue, or at least a separate PR.

These may not be problems or issues for standard-resolution grids which require 1 or more whole nodes per instance.

The hanging happens between the reading of
fndep_clm_hist_b.e21.BWHIST.f09_g17.CMIP6-historical-WACCM.ensmean_1849-2015_monthly_0.9x1.25_c180926.nc
in CAM and the start of the time stepping.
All of the processors on one of the cpu nodes were used at 100%, but nothing was happening in the $RUNDIR.
This could be related to @alperaltuntas comment in MOM_interface #314:
"Instead of modifying RUN_STARTDATE (which causes slow stream reads)"

I can't assign people or labels, so please choose some.

[kdraeder]

I've run some more tests with attempts to generate more clues.
In a couple of the cases one of the members stalled very early in the run
(only 13 lines are written to the atm_####.log file).
while the others proceeded until they hung just before the time stepping (see previous message).

I set CREATE_ESMF_PET_FILES=TRUE, so there's lots of PET output to search.

I added debug lines to ensemble_driver.F90 around line 443,
call NUOPC_DriverGetComp(ensemble_driver, drvrinst, comp=driver, rc=rc)
which was flagged in some tests as the call which generated some problems.

The output is in

/glade/derecho/scratch/raeder/MCC_cG_C15.ne3pg3_10deg.B_DART.pio_debug3_post_tweaks/MCC_cG_C15.ne3pg3_ne3pg3_t232.BHISTC_LT_DART.derecho_intel.allactive-DART_BHIST_lowres--mom-tx10deg.20260323_133246_d8p3bs/run:
   cesm.log.5510079.desched1.260323-135238
   PET*

In this case all of the instances hung at the same place;
between the reading of the fnddep_clm_hist_b file and the time stepping.

Here's a screen shot of a compute node long after all of the writing to $RUNDIR is stalled.

[kdraeder]

@hkershaw-brown suggested that this might be a memory problem.
I've been worried about one instance being divided between 2 nodes, which seems more vulnerable.
In the 15 instance test each instance gets the 12 PEs requested,
and then 5 more (which don't seem to be responsible for anything).
The result is that instance 8 is given 8 PEs on node 1 and 9 on node 2.

A 3 instance test uses only the 3 x 12 PETs requested with no extras, on 1 node.
In standard resolution tests (ne30) it clearly works to have an instance divided among nodes
when it (and the other instances) use whole nodes, but maybe not when most instances
are contiguous on a node, but some are distributed.

The 5 "extra" PETs don't seem to be accounted for in the test of whether NINST evenly divides
the total PEs assigned to the job (Nnodes * 128), so that may be a clue that this
15 instance, 2 node, MCC test was not anticipated.

[kdraeder]

A most excellent data point:
A 14 instance test with MAX_TASKS_PER_NODE=119 (7 x 17) passed all test phases.
No instances are divided among 2 nodes.
So this coarse resolution, large ensemble test can work using these testmods.

It's open for discussion whether any code changes are warranted to enable an instance
to share 2 nodes with other instances.
I've been looking, but I haven't found where that would need to happen.
@billsacks @jedwards4b @ekluzek @cacraigucar Do you have any thoughts?
Should anyone else weigh in?

[jedwards4b]

It should work with each instance over two nodes - just change the pelayout.

[kdraeder]

Are you saying each of the 14 instances should span 2 nodes?
I haven't seen a way to set a root_pe for each instance, only for components.

[jedwards4b]

You don't need to set a rootpe for each instance, you set the parameters for one instance and the number of instances and cime takes care of determining root_pe for each instance.

[kdraeder]

My 15 instance test has:

   MAX_TASKS_PER_NODE = 128 (default)
   NTASKS = 6 for each component (limited by MOM coarse resolution)
   NTHRDS = 1 ...
   PSTRID = 1 ...
   ROOTPE = 0 ... except MOM = 6 (total of 12 tasks requested per instance)
   MULTI_DRIVER = TRUE
   PIO_ASYNC_INTERFACE = FALSE

Did I miss any relevant parameters?
When I let cime choose the layout it ends up with TOTALPEs = 180 (15 x 12 tasks),
but each instance is given 17 PETs, leading to PETcount = 255 + 1 (non-compute PET).
This divides instance 8 across 2 nodes and the job never finishes, as in
/glade/work/raeder/Exp/CESM+DART_testing/MCC_cG_C15.ne3pg3_10deg.B_DART.PET_list_post_tweaks
The same thing would happen for a 14 instance, default parameters test.

When I do the 14 instance test but limit MAX_TASKS_PER_NODE to 119 (7 x 17 PETs),
cime still gives it 17 PETs/instance but puts 7 whole instances on each node.
This test finishes correctly. TOTALPES ends up being 168 (14 x 12 tasks/inst).
.../MCC_cG_C14.ne3pg3_10deg.B_DART.2x119tasks_post_tweaks

I haven't figured out why the split instance case can't finish,
but carefully setting NINST and MAX_TASKS_PER_NODE is a solution, if not the best one.
It's fine if the best strategy in the short term for this new/unusual test is
"set MAX_TASKS_PER_NODE in a testmod and NINST in the test name so that instances are not divided".
But if this is pointing out something that needs to be (eventually) fixed, then this issue should stay open.

[jedwards4b]

I guess that I didn't understand the question - you said that you wanted each instance to span 2 nodes, but I see that the ocn for this case is limited to 6 tasks and the atm is limited to 54 tasks. I suspect that you are running into a problem when you try to use a partial node in the test. It will work best if MAX_TASKS_PER_NODE is a multiple of the number of tasks in an instance and the total number of tasks is a multiple of MAX_TASKS_PER_NODE.

[kdraeder]

I think that we've cleared up the misunderstandings and I should proceed with a PR that implements
a testmod and advice for using it, but no changes to the cime layout code.

I'm curious about where you see that atm is using 54 tasks.
My understanding was that each atm instance was using 6, and there 14 instances, so 84 tasks total for atm.
Those are shared with lnd, ice, and rof.

Also, if you have any pointers about why each instance is given 17 PETs instead of 12.
Does it use 12 x NINST to figure that it needs 2 nodes, then divides 2*128 by NINST?

Thanks!

[jedwards4b]

what I meant to say is that the maximum number of tasks for the atm is 3x3x6=54 tasks.

…

On Wed, Mar 25, 2026 at 12:48 PM kdraeder ***@***.***> wrote: *kdraeder* left a comment (ESCOMP/CMEPS#641) <#641 (comment)> I think that we've cleared up the misunderstandings and I should proceed with a PR that implements a testmod and advice for using it, but no changes to the cime layout code. I'm curious about where you see that atm is using 54 tasks. My understanding was that each atm instance was using 6, and there 14 instances, so 84 tasks total for atm. Those are shared with lnd, ice, and rof. Also, if you have any pointers about why each instance is given 17 PETs instead of 12. Does it use 12 x NINST to figure that it needs 2 nodes, then divides 2*128 by NINST? Thanks! — Reply to this email directly, view it on GitHub <#641?email_source=notifications&email_token=ABOXUGCI6LK5SNCWKHR36XT4SQSWBA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMJSHA4TINRTHEZKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-4128946392>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOXUGE3H4JA63FJ2VEUVQ34SQSWBAVCNFSM6AAAAACWYNMRJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCMRYHE2DMMZZGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Jim Edwards STAND UP FOR SCIENCE CESM Software Engineer National Center for Atmospheric Research Boulder, CO

[kdraeder]

I don't see where the 3x3 comes from.
It's a BHIST test, so I've also requested 6 more tasks/ocn_instance, for a total of 12 tasks/instance.
It runs on 2 nodes. I don't see a 3 anywhere in the PE layout.

Cime is assigning extra PETs to the job, but those seem to be unused because I didn't ask to use them.

[jedwards4b]

The maximum number of tasks for the se dycore is given by nene6
Since you have a ne3 grid that's 336. Note this is the maximum, not necessarily what you are using.

[kdraeder]

As described in the first comment, a multi-instance test,
which fits onto a single node (develop queue), passes the MCC test.
If each instance is large enough that it makes sense to put it on dedicated node(s),
then that test is also highly likely to pass, at least for small ensembles.

The middle-sized tests, in which several instances can fit on 1 node
but there are too many to all fit on 1 node, fails with an error message
as described above. So it seems that this check needs modification
or removal. I have not been able to find the information needed to modify it.

When the error check is removed, most tests fail fail by stalling before the time stepping starts.
This is apparently caused by 1 (or a few) instances being laid out across 2 nodes.
This can be prevented by limiting the number of instances on each node by setting
MAX_NTASKS_PER_NODE to be a multiple of the PEs assigned by cime andor ESMF to each instance.
This may be different than the number of tasks requested using NTASKS and ROOTPE.
There's an algorithm to calculate this in a new

/glade/work/raeder/Models/cesm3_0_alpha08d_mar13/cime_config/testmods_dirs/allactive/DART_BHIST_lowres/README_layout.

This testmod is central to the coarse-resolution large-ensemble testing,
so MAX_NTASKS_PER_NODE should be set there, instead of in CMEPS.

The PR I'll submit (to remove the check) relies on changes in other components as described in cime #4933,
so testing needs to use code like cesm3_0_alpha08d_mar13.
This includes a revised MCC test, which can handle ensembles larger than 3.

The change to CMEPS is simple enough that it can probably be handled
without extensive additional testing, but any commonly run tests would be helpful.

I've made the CMEPS change in a branch from commit 8d3ff3a,
which is the commit which works with the rest of the CESM I copied from Alper Altuntas,
which enables the coarse-resolution MOM6.
What's the standard way to put a change like this into a PR?

[jedwards4b]

PR's for each component should be made to the main branch of that component with notes about any inter-dependencies in the initial comment.