Provide the ability to delete (remove) HDF5 files at the end of integration tests

[bieryAtFnal]

Description

The primary change covered by this Pull Request is the addition of common utility that will remove HDF5 files at the end of each regression/integration test based on directives from within each integtest and from a command-line option that can be specified when each integtest is run.

• The ability to remove HDF5 files at the end of each integtest was requested several months(?) ago in discussions about changes to the integrationtest infrastructure that people would like to see. I believe that the goal is to make it easier to run integtests on computers that have limited disk space for storing the output of the tests.
• In this description and in the code itself, I have chosen to refer to the files that can be removed at the end of each integtest as "HDF5" files. I considered calling them "data" files, but the files that get removed include raw data files, TP stream files, and TRMon files. All of those files contain information in the HDF5 format and all of them have the a ".hdf5" filename extension, but I wasn't comfortable referring to the information in all of them as "data". So, I went with the choice that seems to be the most accurate: HDF5 files. Of course, if we ever choose to store raw data, TP stream data, etc. in a format other than HDF5, my choice may fall out of favor.

Some additional changes that were made:

• the names of several fields in the CreateConfigResult data class were modified to try to make their meaning more clear
• the "--remove-hdf5-files" command-line option was added
• a "utility_functions.py" file was added and the pre-existing "basic_checks" function was copied from a dedicated file into this "utility" file, with the goal of getting rid of that dedicated "basic_checks" file soon

As part of this work, corresponding changes were made to the integration tests in various repositories. Those changes are covered by the following PRs:

• Modified the integtests to support user-requested deletion of HDF5 files at the end of each integtest. daqsystemtest#320
• Modified the integtests to support user-requested deletion of HDF5 files at the end of each integtest. dfmodules#490
• Modified the integtest to support user-requested deletion of HDF5 files at the end of the integtest hsilibs#86
• Modified the integtest to use the new utility_functions module from the integrationtest repo instead of the basic_checks module. listrev#124
• Modified the integtest to support user-requested deletion of HDF5 files at the end of the integtest snbmodules#34
• Modified the integtest to support user-requested deletion of HDF5 files at the end of the integtest. trigger#433

Here is a suggested set of steps for testing these changes:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest
dbt-create -n NFD_DEV_260526_A9 ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDDevTest_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b kbiery/remove_integtest_hdf5_files
git clone https://github.com/DUNE-DAQ/dfmodules.git -b kbiery/remove_integtest_hdf5_files
git clone https://github.com/DUNE-DAQ/hsilibs.git -b kbiery/remove_integtest_hdf5_files
git clone https://github.com/DUNE-DAQ/listrev.git -b kbiery/remove_integtest_hdf5_files
git clone https://github.com/DUNE-DAQ/snbmodules.git -b kbiery/remove_integtest_hdf5_files
git clone https://github.com/DUNE-DAQ/trigger.git -b kbiery/remove_integtest_hdf5_files
cd ..

cd pythoncode
git clone https://github.com/DUNE-DAQ/integrationtest.git -b kbiery/hdf5_removal_utility
cd ..

. ./env.sh
dbt-build -j 12
dbt-workarea-env

dunedaq_integtest_bundle.sh --verbosity 2 -r local --random 5

For reference, here is a sample pytest "test" that provides the hooks for removing (deleting) HDF5 files at the end of an integtest:

import integrationtest.utility_functions as utility_functions

def test_cleanup(run_dunerc):
    utility_functions.remove_hdf5_files_if_requested(run_dunerc, this_test_requests_hdf5_file_removal=False)

The flag that is specified in the remove_hdf5_files_if_requested() call is what the integrationtest developer feels is the best option. In any case, the presence of this function call in the integtest enables the ability for end users to request that the HDF5 files get removed by specifying the '--remove-hdf5-files' pytest option when the test is run.

Type of change

• New feature or enhancement (non-breaking change which adds functionality)

Testing checklist

• Minimal system quicktest passes (pytest -s minimal_system_quick_test.py)
• Full set of integration tests pass (dunedaq_integtest_bundle.sh)

[bieryAtFnal]

Sorry for the last-minute change. I realized that it might be helpful to request that HDF5 files be kept on disk in certain situations, despite what the integtest writer specified. (That's in addition to the ability to request that HDF5 files be deleted in certain situations.)
To support this, I modified the --request-hdf5-removal command-line option to be a string instead of a boolean. And modified the code so that values of True/true or False/false will cause the infrastructure to always remove HDF5 files or always keep them.