Skip to content

Support datetime expressions in weather-sp output templates#536

Open
zeel2104 wants to merge 2 commits into
google:mainfrom
zeel2104:fix-weather-sp-datetime-template-535
Open

Support datetime expressions in weather-sp output templates#536
zeel2104 wants to merge 2 commits into
google:mainfrom
zeel2104:fix-weather-sp-datetime-template-535

Conversation

@zeel2104
Copy link
Copy Markdown

@zeel2104 zeel2104 commented May 2, 2026

Summary

Fixes #535.

Adds support for richer Python-style expressions in weather-sp output templates, including datetime formatting expressions like {datetime.strptime(f"{time}", "%H%M").strftime("%H:%M:%S")}.

This allows an output template such as:

gs://test-output/splits/test-file_{date}-{datetime.strptime(f"{time}", "%H%M").strftime("%H:%M:%S")}.grib

to produce:

gs://test-output/splits/test-file_20200101-12:00:00.grib

Changes

  • Added expression-aware output template parsing/evaluation in OutFileInfo.
  • Updated split-dimension detection so variables used inside expressions, such as time, are included.
  • Adjusted GribSplitterV2 to use raw split-dimension temp filenames for grib_copy, then apply the richer Python formatter during upload.
  • Added tests for datetime expression formatting and split-dimension extraction.

Testing

Manual validation passed:

['date', 'time']

gs://test-output/splits/test-file_20200101-12:00:00.grib

Full pytest was not run locally because this Windows Python 3.14 environment is missing pygrib/ecCodes support.

@mahrsee1997 mahrsee1997 self-requested a review May 3, 2026 08:03
@mahrsee1997
Copy link
Copy Markdown
Collaborator

@zeel2104, can you please fix lint issue, thanks!

@mahrsee1997 mahrsee1997 requested a review from Piyush-Ingale May 6, 2026 08:10
@zeel2104
Copy link
Copy Markdown
Author

@mahrsee1997

Thanks for pointing it out! I fixed the lint issue by wrapping the long datetime output template in the test file and pushed the update.

# limitations under the License.

from dataclasses import dataclass
import ast
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of using a complex AST-based parser, can we implement a simple native wrapper for date and time that preserves the raw GRIB date/time field formats by default?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this approach, if a pipeline is initiated with an output-template using double quotes or without quotes, it can cause the pipeline to fail. We should handle these scenarios as well.

NETCDF_FILE_ENDINGS = ('.nc', '.cd')


class _DateTimeNamespace:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can simplify the codebase by removing this class and helper methods, as we can remove the AST based approach!

dims.append(name)
return dims

def formatted_output_path(self, splits: t.Dict[str, str]) -> str:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please update formatted_output_path to use native Python string formatting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[weather-sp] : better formatting of date & time in output template

3 participants