[Draft] OpenACC port of halo exchanges #1355

Draft

abishekg7 wants to merge 34 commits intoMPAS-Dev:developfrom

abishekg7:framework/acc_halo_exch

Commits on Aug 13, 2025

Consolidating OpenACC device-host memory transfers
abishekg7
committed
Fixing bug associated with rho_zz_2 not being copied out at the end of dynamics
abishekg7
committed
Moving some OpenACC data movements to subroutines
abishekg7
committed
Removing acc data xfer timers for device variables using create/delete
abishekg7
committed
Using acc declare create for rho_zz_int and corresponding cleanup
abishekg7
committed
Removing atm_advance_scalars_mono ACC_data_xfer timers around create/delete
abishekg7
committed
Simplifying OpenACC data transfers around the call to mpas_reconstruct_2d
abishekg7
committed

Commits on Aug 14, 2025

Need to copyout u_2 and w_2 at the end of dynamics
abishekg7
committed

Commits on Aug 22, 2025

Fixes to produce correct results with CURVATURE
abishekg7
committed

Commits on Oct 3, 2025

Adding option to enable GPU execution of mpas_reconstruct_2d
abishekg7
committed

Commits on Oct 13, 2025

fixes needed with intel compiler
abishekg7
committed
Add data movement for some fields under the mpas_halo_groups

G. Dylan Dickerson
authored and
abishekg7
committed
Add a data region and acc kernels to the 2D packing code

G. Dylan Dickerson
authored and
abishekg7
committed
Add the update directives that should have been part of the last commit

G. Dylan Dickerson
authored and
abishekg7
committed
Comment out data present region, see if this causes an error

G. Dylan Dickerson
authored and
abishekg7
committed
Expand the data managed on the GPU for the halo exchange

G. Dylan Dickerson
authored and
abishekg7
committed
Remove the OpenACC management of recvBuf

G. Dylan Dickerson
authored and
abishekg7
committed
Add update host(sendBuf) back, address answer diff

G. Dylan Dickerson
authored and
abishekg7
committed
Expand to other packing kernels, only update sendBuf after packing finishes

G. Dylan Dickerson
authored and
abishekg7
committed
Change to simple integers to access the buffers and the field arrays

G. Dylan Dickerson
authored and
abishekg7
committed
Add kernels to unpacking loops and use a data present region to try to force GPUDirect MPI

G. Dylan Dickerson
authored and
abishekg7
committed
Change from data copyin regions to enter/exit directives for the r?arr variables

G. Dylan Dickerson
authored and
abishekg7
committed
Re-enable update host for sendBuf, add update device recvBuf

G. Dylan Dickerson
authored and
abishekg7
committed
Remove update directives, use acc host_data use_device(...) near MPI calls instead

G. Dylan Dickerson
authored and
abishekg7
committed
checkpoints: acc pack + cuda aware mpi working
abishekg7
committed
seems to be working
abishekg7
committed
Optimized packing and unpacking loops. Adding timers and other cleanup
abishekg7
committed
Working savepoint
abishekg7
committed
u_2 and w_2 need to be copied out after dynamics + cleanup
abishekg7
committed
using attach in a directive instead of the acc_attach library call
abishekg7
committed
Using attach clause in parallel region will also auto detach at end of region
abishekg7
committed
Reverting the indexing in loops and comment cleanup
abishekg7
committed
New namelist option to switch on or off GPU-Aware MPI
abishekg7
committed
Adding a dependency to mpas_timer.o in mpas_halo.o
abishekg7
committed