IOManager: I/O Abstraction, documentation, and the IOManagerDF class

[rburghol]

@PaulDudaRESPEC @timcera

• UCI and hdf5/pandas stuff:
  • develop-InMem - Branch with IO into pandas dataframe direct from uci, with minimal (or no) write-backs during execution?
    • Goals:
      • Be able to send hsp2 all needed inputs from CSV(s), omitting the UCI
      • Eliminate the requirement of the hdf5 store for all data, allowing file storage instead?
    • Code:
    • Tasks:
      • Code the class IOManagerDF as child of IOManager
      • Insure all IOManager classes are compatible with the get_timeseries() function (a standalone function in the hsp2.hsp2.utilities file)
        • read_ts: The get_timeseries() function lives outside of the IOManager class but expects an IOManager interface compatible object containing a read_ts method, thus, by sub-classing read_ts on the IOManagerDF we can make them seamlessly integrate and eliminate the redundant function in HSP2UtilitiesInMem.py
          • _get_in_memory() must read the pandas df (should also behave exactly same as hdf5 which already caches in the object?)
          • Falls back on reading from the hdf5, but this should instead look at the self._input, which is the object passed in as io_combined at class creation.
      • Merge duplicate code into HSP2UtilitiesInMem.py, hsp2/main.py, hsp2io/io.py, and hsp2tools/readUCI.py
      • Run test scripts
      • PR
    • Questions:
      • Test script/UCI:
      • Should readUCIinMem.py replace readUCI.py entirely? (looks like it does)
  • Redundant/unused hdf5 class: There are classes named HDF5 defined in two places:
    • src/hsp2/hsp2io/hdf.py
    • and in src/hsp2/hsp2tools/HDF5.py
    • NOTE: the HDF5 file does not appear to be loaded anywhere, see: fgrep -iR "import from hsp2tools" src/*|grep hdf -i

Code development

• Diff of develop-InMem branch (Paul's prototype): develop...develop-InMem

Caching

• IO class (read hdf) allows caching of timeseries, so that there is no need to reload a series that has already been loaded from disk.
• KO class does not currently have the ability to store data in memory, save_timeseries pushes the data to disk, and then, that seems to incur large overhead.

Documentation

• Protocols are defined in src/hsp2/hsp2io/protocols.py
  • Defines what things should look like, ex: write_ts, read_ts
  • But they don't actually do anything
• Classes actually implement the interfaces
  • Ex: src/hsp2/hsp2io/io.py
  • Defines the code that implements the function, such as write_ts()

    HSPsquared/src/hsp2/hsp2io/io.py

    Line 58 in 6f40cc3

    def write_ts(self,

[aufdenkampe]

@rburghol, thanks for recording these notes in an issue!

I have one clarification/correction, however. The implementation of the file interface is actually done by file type in modules such as src/hsp2/hsp2io/hdf.py. So if you wanted to implement IO for a new file type (i.e. Parquet), you would want to copy the pattern in hdf.py.

The io.py module only implements the IOManager() class that orchestrates different IO protocols for the IO interface you pass into it.

To see an example of how that is used, look at the Create an instance of the HSP2 IO Manager section in the examples/1_Intro_to_HSP2.ipynb notebook. NOTE that this notebook appears to have gotten broken with some of the repo reorganization that's happened in the last year.

This PR describes it all reasonably well too:

• IO Abstraction #68

@ptomasula, anything else to add?

[rburghol]

Thanks @aufdenkampe -- I was flying by the seat of my pants!

[aufdenkampe]

@rburghol, with PR #184, I fixed the Example 1 notebook, so you should be able to use it to explore the hsp2io.io.IOManager() class.

[rburghol]

Thanks @aufdenkampe