Migrate to diagnostic-centric test data #475
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* origin/main: ci: Update uv in CI chore: change to pin for fastprogress fix: Bump a yanked version of deprecated fix: Add IPython to depedencies temporarily
* origin/main: fix: use uv not pip docs: Changelog ci: Ignore dependabot PRs Bump version: 0.8.0 → 0.8.1 docs: Clean up missing changelogs chore(deps): bump the github-actions group with 5 updates chore: Add dependabot
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| # Wrap the runner to convert exceptions to pytest.skip | ||
| class PytestTestCaseRunner: | ||
| def run(self, diagnostic, test_case_name="default", output_dir=None): |
…mplicit (fall through) returns' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
* origin/main: chore: pin xarray for ilamb3 due to issue fix: handle a regression in xarray fix: Clean up some type hints from updating mypy docs: fix typo chore(deps-dev): bump the python-dependencies group with 31 updates
Contributor
Author
|
This is getting pretty big and have already had a few iterations of cleaning, but is still backwards compatible so I'll get this merged and then add it to the test suite |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a diagnostic-centric test data management system to replace the monolithic test data approach. Each diagnostic now defines its own test data requirements via TestDataSpecification, enabling selective data fetching and independent testing.
Key Changes
- New ESGF data fetching infrastructure with support for CMIP6 and obs4MIPs datasets
- CLI commands (
ref test-cases fetch/list/run) for managing test cases TestCaseandTestDataSpecificationclasses for defining reproducible test scenarios- Test harness with
TestCaseRunnerand pytest fixtures for diagnostic testing
Reviewed changes
Copilot reviewed 29 out of 31 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
packages/climate-ref-core/src/climate_ref_core/testing.py |
Core testing infrastructure with TestCase and TestDataSpecification classes |
packages/climate-ref-core/src/climate_ref_core/esgf/ |
New ESGF module with CMIP6Request, Obs4MIPsRequest, and ESGFFetcher |
packages/climate-ref-core/src/climate_ref_core/exceptions.py |
New test case exceptions (NoTestDataSpecError, TestCaseNotFoundError, DatasetResolutionError) |
packages/climate-ref/src/climate_ref/testing.py |
TestCaseRunner implementation and catalog path utilities |
packages/climate-ref/src/climate_ref/cli/test_cases.py |
New CLI commands for fetching, listing, and running test cases |
packages/climate-ref/src/climate_ref/datasets/__init__.py |
Refactored imports to support new dataset adapters |
conftest.py |
New pytest fixtures (run_test_case, esgf_test_data_dir, esgf_solve_catalog) |
docs/how-to-guides/testing-diagnostics.md |
Comprehensive documentation for the new testing infrastructure |
packages/climate-ref-example/src/climate_ref_example/example.py |
Example diagnostic with test_data_spec |
packages/climate-ref-core/pyproject.toml |
Added intake-esgf and pyarrow dependencies |
uv.lock |
Updated lock file with new dependencies |
pyproject.toml |
Added pytest warning filters for test domain classes |
tests/test-data/.gitignore |
Added esgf-data directory to gitignore |
| Various test files | Comprehensive unit tests for new ESGF and testing infrastructure |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
packages/climate-ref-core/src/climate_ref_core/esgf/obs4mips.py
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is the first step in reworking how we do sample data.
Diagnostics are now responsible for defining the test datasets they need from ESGF.
These will be able to be selectively fetched/cached as needed so any diagnostic author doesn't require the full set of data (the CI still needs the full set).
This PR adds the data structures for defining test cases, and the CLI to run specific tests.
The next PR will migrate the sample data requests to this new format.
#472
Checklist
Please confirm that this pull request has done the following:
changelog/