Major refactor#53
Open
sahiljhawar wants to merge 132 commits into
Open
Conversation
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
… make sure mat and pickle files have the same data (fixes #45)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…e_single_file method
- Add `MonthlyFileStrategy` with write and load dispatch methods for h5, nc, cdf, and mat - Implement generic append/load/merge/write flow - Preserve custom monthly NetCDF writer and dependencies - Harden CDF metadata writing and append loading for empty attributes/variables
…rd part of the MonthlyFileStrategy
- Change defaults of \`save_strategy\` and \`data_standard\` to "gfz" across recipes - Update docs navigation entries to use "GFZ" terminology - Remove example scripts for ESA and PROBAV and add \`el_paso.recipes.esa.__init__.py\` export for \`process_ngrm_electron_fluxes\` - Update tests to expect "gfz" and add a dataset test for missing-file datetime handling - Add \`original_requested_name\` handling in \`DataSet\` and adjust GFZ strategy docstring and minor logging cleanup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request makes significant improvements to the data standards and saving strategies within the
el_pasopackage, focusing on modernization, code simplification, and clearer documentation. The main changes include replacing the legacyDataOrgStandardwith the newGFZStandard, updating the data standard base class for better extensibility and type safety, refactoring the consistency checking logic, and cleaning up unused or outdated documentation and configuration.Key changes include:
Data Standards Modernization
DataOrgStandardand replaced it with the new, more extensibleGFZStandard, updating all imports, documentation, and references accordingly. (el_paso/data_standards/data_org_standard.pyremoved,el_paso/data_standards/__init__.py,docs/API_reference/data_standards/gfz.md,docs/API_reference/overview.md, [1] [2] [3]DataStandardbase class to use generics, improved type safety, and added a newVariableInfostructure for variable metadata. The standardization logic is now more generic and easier to extend. (el_paso/data_standard.py, el_paso/data_standard.pyL6-R109)Consistency Checking Refactor
el_paso/data_standard.py, el_paso/data_standard.pyL58-R164)Saving Strategies and Documentation Updates
MonthlyFileStrategyinstead ofMonthlyH5Strategy,GFZStrategyinstead ofDataOrgStrategy). (docs/API_reference/saving_strategies/gfz.md,docs/API_reference/saving_strategies/monthly.md,docs/API_reference/saving_strategies/data_org.mdrenamed, [1] [2]docs/API_reference/saving_strategies/monthly_netcdf.md, docs/API_reference/saving_strategies/monthly_netcdf.mdL1-L11)Package Initialization and API Cleanup
el_paso/__init__.pyfile by removing unnecessary imports, adding new ones forutilsandtyping, and updating the__all__list to reflect the new public API. (el_paso/__init__.py, [1] [2] [3]Developer Tooling
pyright-prettypre-commit hook from the configuration. (.pre-commit-config.yaml, .pre-commit-config.yamlL23-L28)These changes modernize the codebase, improve maintainability, and provide a clearer and more extensible framework for data standards and saving strategies.