Incorrect Handling of NetCDF File Formats in IOStreams and IO Modules

[grnydawn]

During debugging of a PIO error in the Polaris tests, I identified two potential bugs in the Omega IOStream and IO module that could explain the issue.

Bug 1: Hard-coded NetCDF format ignores omega.yml
In IOStream.cpp, there are two calls like:

IO::openFile(InFileID, InFileName, Mode, IO::FmtDefault, ExistAction);

IO::FmtDefault is a compile-time constant defined in IO.h (currently netcdf4c).
As a result, the NetCDF format specified in omega.yml is ignored, and files are always opened as netcdf4c.

Bug 2: Global NetCDF format variable is shadowed
A global default format is defined in IO.cpp:

FileFmt DefaultFileFmt = FmtDefault; // FmtDefault = netcdf4c

This global variable is supposed to be updated in IO::init() using the value from omega.yml.
However, in IO::init() a local variable with the same name is declared:

FileFmt DefaultFileFmt = FileFmtFromString(InFileFmt);

This shadows the global variable, so the global DefaultFileFmt is never updated.

Still having errors even after fixing Bug 1 & 2
Once I fixed both issues:
• Things work if defaultfmt = netcdf4c (same as the current hard-coded behavior)
• But, using netcdf4p triggers many errors, including:

• Collective mode not supported with an unlimited time dimension
• Attempts to define variables while in NetCDF data mode

This suggests additional assumptions in the IO layer about NetCDF format semantics.

Observation: Mixed NetCDF formats in Polaris tests

I also noticed multiple NetCDF formats being generated or copied during Polaris testing:

$ ncdump -k output.nc        → netCDF-4
$ ncdump -k init.nc          → cdf5
$ ncdump -k mesh.nc          → netCDF-4
$ ncdump -k OmegaPlanarMesh.nc → 64-bit offset

Given the current IOStream bugs, this mix of formats may not be handled consistently and could be a root cause of the PIO errors.
Suggestion: Standardizing on a single NetCDF format for Polaris tests might significantly reduce current and future IO/PIO issues.

[cbegeman]

@grnydawn Can I go ahead and open a Polaris issue to keep track of the multiple netCDF formats there? Or would you prefer to open that yourself or use another strategy?

[rljacob]

Note that we decided a while ago that all input files should be in cdf3 format. https://e3sm.atlassian.net/wiki/spaces/EIDMG/pages/769130507/Picking+a+netcdf+type+for+all+input+files That's for the main model components but if you're testing pieces of OMEGA, then cdf3 is what to use.

[xylar]

@rljacob, I think you mean NetCDF3 CDF5 format, right?

[cbegeman]

FYI: the netcdf file formats from a Polaris test /lcrc/group/e3sm/ac.cbegeman/scratch/MPAS-Ocean-output/chrys/polaris-update-omega-20260122/ocean/planar/manufactured_solution/del4/forward/200km_300s/:

manufactured_solution.cfg:

format = NETCDF3_64BIT_DATA
engine = netcdf4

ncinfo mesh.nc
<class 'netCDF4.Dataset'>
root group (NETCDF3_64BIT_DATA data model, file format NETCDF3):

ncinfo init.nc 
<class 'netCDF4.Dataset'>
root group (NETCDF3_64BIT_DATA data model, file format NETCDF3):

ncinfo output.nc 
<class 'netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):

[grnydawn]

@grnydawn Can I go ahead and open a Polaris issue to keep track of the multiple netCDF formats there? Or would you prefer to open that yourself or use another strategy?

@cbegeman , In my opinion, we may need separate issues for Omega and Polaris to change their default NetCDF file formats. I will create an Omega issue to change its NetCDF format to CDF5.

[xylar]

@cbegeman , In my opinion, we may need separate issues for Omega and Polaris to change their default NetCDF file formats. I will create an Omega issue to change its NetCDF format to CDF5.

In my view, @cbegeman's message above shows that no changes are needed in Polaris. NETCDF3_64BIT_DATA is the same thing as NetCDF3 CDF5 format. The data are already in the desired format and only Omega needs to be fixed.

[rljacob]

@rljacob, I think you mean NetCDF3 CDF5 format, right?

Yes sorry. Re-reading I guess I mean CDF5 or CDF2. We were concerned about general support for CDF5 outside of Pnetcdf but that's probably less of an issue now.

[philipwjones]

Since we use PIO format types, we need to choose one of:

/** Parallel Netcdf  (parallel) */
PIO_IOTYPE_PNETCDF = 1,

/** Netcdf3 Classic format (serial) */
PIO_IOTYPE_NETCDF = 2,

/**  NetCDF4 (HDF5) compressed format (serial) */
PIO_IOTYPE_NETCDF4C = 3,

/** NetCDF4 (HDF5) parallel */
PIO_IOTYPE_NETCDF4P = 4,

/** NetCDF4 (HDF5) parallel NCZarr */
PIO_IOTYPE_NETCDF4P_NCZARR = 5,

/** ADIOS parallel */
PIO_IOTYPE_ADIOS = 6,

/** ADIOS parallel with compression */
PIO_IOTYPE_ADIOSC = 7,

/** HDF5 parallel */
PIO_IOTYPE_HDF5 = 8

In an old PR still waiting, @amametjanov had selected PNETCDF. Is another preferred?

[cbegeman]

Just noting that I see the following build error on pm-cpu. It would be good to check whether this is resolved with fixing this issue.

/pscratch/sd/c/cbegeman/polaris-repos/fix-sphere-transport-edge-normal/e3sm_submodules/Omega/components/omega/src/base/IO.cpp: In function ‘void OMEGA::IO::openFile(int&, const std::string&, Mode, FileFmt, IfExists)’:
/pscratch/sd/c/cbegeman/polaris-repos/fix-sphere-transport-edge-normal/e3sm_submodules/Omega/components/omega/src/base/IO.cpp:254:27: warning: comparison between ‘enum OMEGA::IO::FileFmt’ and ‘enum PIO_IOTYPE’ [-Wenum-compare]
  254 |    int IsCDF5 = (InFormat == PIO_IOTYPE_PNETCDF) ? PIO_64BIT_DATA : 0;
      |                  ~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~