Support datetime expressions in weather-sp output templates#536
Support datetime expressions in weather-sp output templates#536zeel2104 wants to merge 2 commits into
Conversation
|
@zeel2104, can you please fix |
|
Thanks for pointing it out! I fixed the lint issue by wrapping the long datetime output template in the test file and pushed the update. |
| # limitations under the License. | ||
|
|
||
| from dataclasses import dataclass | ||
| import ast |
There was a problem hiding this comment.
Instead of using a complex AST-based parser, can we implement a simple native wrapper for date and time that preserves the raw GRIB date/time field formats by default?
There was a problem hiding this comment.
With this approach, if a pipeline is initiated with an output-template using double quotes or without quotes, it can cause the pipeline to fail. We should handle these scenarios as well.
| NETCDF_FILE_ENDINGS = ('.nc', '.cd') | ||
|
|
||
|
|
||
| class _DateTimeNamespace: |
There was a problem hiding this comment.
we can simplify the codebase by removing this class and helper methods, as we can remove the AST based approach!
| dims.append(name) | ||
| return dims | ||
|
|
||
| def formatted_output_path(self, splits: t.Dict[str, str]) -> str: |
There was a problem hiding this comment.
Please update formatted_output_path to use native Python string formatting.
Summary
Fixes #535.
Adds support for richer Python-style expressions in
weather-spoutput templates, including datetime formatting expressions like{datetime.strptime(f"{time}", "%H%M").strftime("%H:%M:%S")}.This allows an output template such as:
gs://test-output/splits/test-file_{date}-{datetime.strptime(f"{time}", "%H%M").strftime("%H:%M:%S")}.gribto produce:
gs://test-output/splits/test-file_20200101-12:00:00.gribChanges
OutFileInfo.time, are included.GribSplitterV2to use raw split-dimension temp filenames forgrib_copy, then apply the richer Python formatter during upload.Testing
Manual validation passed:
['date', 'time']gs://test-output/splits/test-file_20200101-12:00:00.gribFull pytest was not run locally because this Windows Python 3.14 environment is missing
pygrib/ecCodes support.