Random thoughts for further developments

[stevehadd]

Things to demonstrate

• use of frameworks
  • pytorch lighting
  • keras 3
  • tensorflow (precip rediagnosis)
  • xgboost (XBT dataset)
• experiment tracking - mlflow
• vis - consider met office etch framework
• zoo - integrate with private hugging face server?

feature ideas:

• create a local site archive, with some download scripts, so people can create a site archive on their laptop.
  • also have a "build your own site archive" tutorial
  • some sort of archive generator from netcdf or zarr files would be useful
  • also generate from an intake catalog

Ideas for UX improvement

• can we change how site archive are imported. it would be nice if the import was always "site_archive" and then your local one was found and imported. That would make the notebooks and other code more portable as you wouldn't have to change the import statements.
• create a template notebook fpr pyearthtools tutorials to give a more consistent feel. We might like to use NOAA tutorial notebook template as it has a nice strcuture from a teaching perspective.
• GPU - have a wrapper that wraps GPU operation for inference, maybe redundant with model
• iterator range should be defined outside the pipeline, not a part of it, so you can define multiple iterators from one pipeline, which would be especially sueful to have train, validate and test iterators from 1 pipeline definition
• how can one get the output from part way through a pipeline?

development questions / tutorial ideas

• how do we use a pipeline to calculate scores for a whole test set, rather than just 1 prediction? How should the iterator be used to do this efficiently in parallel?