Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion projects/python-weather-diagnostics-toolkit/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ artifacts. It focuses on:
- Magnus-formula dewpoint diagnostics and round-trip humidity checks
- geopotential-height conversion
- relative-vorticity, horizontal-advection, and moisture-flux diagnostics
- dry pressure-coordinate temperature-tendency decomposition
- station-to-grid interpolation and precipitation accumulation conversion
- anomaly, composite, and grid-point correlation helpers
- cosine-latitude regional means
Expand All @@ -38,6 +39,7 @@ python-weather-diagnostics-toolkit/
| +-- diagnostic-analysis.md
| +-- station-precipitation-workflows.md
| +-- climate-statistical-diagnostics.md
| +-- focused-case-abstraction.md
| +-- methodology.md
| +-- reproducibility.md
| +-- reviewer-path.md
Expand Down Expand Up @@ -112,6 +114,8 @@ Dynamic layer:
- estimates latitude/longitude grid spacing from spherical Earth geometry
- computes relative vorticity as `dv/dx - du/dy`
- computes horizontal scalar advection as `-(u dS/dx + v dS/dy)`
- decomposes dry temperature tendency into zonal, meridional, vertical, and
adiabatic terms
- computes moisture flux divergence as `d(q u)/dx + d(q v)/dy`
- keeps finite-difference assumptions explicit for reviewer inspection

Expand All @@ -128,6 +132,7 @@ Statistical layer:
- computes anomalies, standardized anomalies, composites, and grid-point correlations
- constructs time-ordered forecast tables from regional features
- fits a deterministic ridge-regression baseline without random shuffling
- evaluates ridge alpha grids with the same time-ordered split
- reports RMSE, MAE, bias, and correlation as workflow diagnostics
- summarizes synthetic ensemble spread, quantiles, and threshold probabilities

Expand Down Expand Up @@ -208,7 +213,8 @@ The more detailed technical route is:
2. [`docs/diagnostic-analysis.md`](docs/diagnostic-analysis.md)
3. [`docs/station-precipitation-workflows.md`](docs/station-precipitation-workflows.md)
4. [`docs/climate-statistical-diagnostics.md`](docs/climate-statistical-diagnostics.md)
5. [`docs/source-to-public-mapping.md`](docs/source-to-public-mapping.md)
5. [`docs/focused-case-abstraction.md`](docs/focused-case-abstraction.md)
6. [`docs/source-to-public-mapping.md`](docs/source-to-public-mapping.md)

## Privacy-Safe Scope

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,14 @@ The public project preserves the reusable calculation ideas:
- geopotential-height conversion
- relative-vorticity calculation
- horizontal temperature-advection diagnostics
- dry pressure-coordinate temperature-tendency decomposition
- moisture flux divergence diagnostics
- station missing-value handling and lightweight station-to-grid interpolation
- accumulated precipitation conversion and event-total summaries
- anomaly, standardized-anomaly, composite, and correlation-field diagnostics
- cosine-latitude regional means
- time-ordered ridge-regression baseline evaluation
- ridge alpha-grid metric review
- synthetic ensemble summary mechanics

## Remaining Assumptions
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,13 @@ diagnostics:
geopotential_height_500hpa: true
vorticity_500hpa: true
temperature_advection_850hpa: true
temperature_tendency_terms_850hpa: true
moisture_flux_divergence_850hpa: true
station_interpolation: false
precipitation_event_total: false
climate_statistics: false
ridge_temperature_baseline: true
ridge_alpha_grid_review: true

baseline_model:
lead_steps: 24
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,43 @@ warming tendency. In a full thermodynamic budget, this is only one term. The
public mini-lab intentionally keeps the default implementation to horizontal
advection so that tests remain small and dependency-light.

The toolkit also exposes component terms:

```text
zonal_advection = -u dS/dx
meridional_advection = -v dS/dy
horizontal_advection = zonal_advection + meridional_advection
```

This makes it possible to identify whether an east-west or north-south flow
component dominates a synthetic tendency diagnosis.

## Dry Temperature-Tendency Terms

For pressure-level fields, the focused case-study abstraction uses a dry
pressure-coordinate decomposition:

```text
dT/dt ~= -u dT/dx - v dT/dy - omega dT/dp + kappa T omega / p
kappa = Rd / Cp
```

The public function reports:

```text
zonal_advection
meridional_advection
horizontal_advection
vertical_advection
adiabatic_compression
dry_dynamic_tendency
```

Inputs are temperature in kelvin, wind in m/s, pressure vertical velocity
`omega` in Pa/s, and pressure in hPa. Returned terms are in K/s. This dry
diagnostic excludes diabatic heating and analysis increments; those should be
handled as residual or externally supplied terms in a real budget study.

## Moisture Flux Divergence

For lower-tropospheric precipitation diagnostics, the toolkit includes a
Expand Down Expand Up @@ -199,6 +236,11 @@ minimize ||X beta - y||^2 + alpha ||beta||^2
Features are standardized using the training partition only. The intercept is
not penalized.

For compact model-selection reviews, `ridge_alpha_grid` evaluates multiple
alpha values with the same time-ordered split and returns a metric table sorted
by RMSE. This is a reproducible diagnostic surface, not nested cross-validation
or proof of forecast skill.

## Evaluation Metrics

The toolkit reports:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,30 @@ fluxes, and analysis increments. The public project keeps the default
calculation narrow so it remains reproducible and testable without heavy
external data.

## Temperature-Tendency Decomposition

The focused case-study abstraction needs a fuller tendency view than one
horizontal advection map. The dry pressure-coordinate decomposition separates:

- zonal temperature advection
- meridional temperature advection
- vertical temperature advection
- adiabatic compression or expansion

Interpretation pattern:

```text
dominant negative horizontal term + synoptic cold-air pathway -> cold-advection support
vertical/adiabatic terms of opposite sign -> partial compensation or reinforcement
```

Care points:

- signs depend on coordinate conventions and pressure-level ordering
- terms are K/s unless rescaled for display
- diabatic heating is not diagnosed by this compact helper
- a real case should compare term sums with observed or analyzed temperature change

## Moisture Transport And Heavy Precipitation

For heavy-precipitation case studies, the toolkit supports horizontal moisture
Expand Down Expand Up @@ -154,6 +178,10 @@ The current public implementation does not claim skill because it does not
bundle a real validation dataset or persistence comparison. It provides the
calculation structure needed for such a review.

The alpha-grid helper adds a small model-selection surface. It is useful for
checking whether the baseline is sensitive to regularization strength, while
preserving time order in the train/test split.

## Ensemble Plume Interpretation

The synthetic ensemble output demonstrates how to summarize multiple model
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Focused Case Abstraction

One local report was treated as the primary technical source for this
iteration. The public project does not preserve that report, its identity
fields, original figures, source data, or prose. Instead, the report's
technical structure was abstracted into reusable code and reviewer-safe
documentation.

## Extracted Technical Arc

The useful public arc is:

```text
data cleaning
-> gridded field normalization
-> synoptic circulation context
-> temperature-tendency term decomposition
-> physics-informed feature engineering
-> time-ordered ridge baseline
-> metric and residual review
```

This arc is stronger than a plotting-only example because it links dynamic
interpretation to a transparent modeling baseline.

## Public Code Added From The Arc

The focused report emphasized temperature-tendency diagnosis. The toolkit now
decomposes dry pressure-coordinate tendency terms:

```text
zonal_advection = -u dT/dx
meridional_advection = -v dT/dy
vertical_advection = -omega dT/dp
adiabatic_compression = kappa T omega / p
dry_dynamic_tendency = sum of the above terms
```

The report also used regularized linear modeling with explicit metric review.
The toolkit now includes `ridge_alpha_grid`, which evaluates a small alpha grid
with the same time-ordered split used by the baseline model.

## Interpretation Boundary

The public project should say:

- the calculation shows how to separate horizontal, vertical, and adiabatic
contributions under dry pressure-coordinate assumptions
- the ridge alpha grid is a reproducible model-selection surface for a compact
baseline
- synthetic examples demonstrate mechanics only

The public project should not say:

- this repository reproduces the original case study
- the bundled synthetic values describe a real weather event
- the ridge baseline proves operational forecast skill
- any original report figure or dataset is redistributed

## Review Questions

A reviewer can now ask:

- Are the temperature-tendency signs and units stated?
- Are horizontal and vertical terms separated instead of hidden inside one map?
- Is model selection done without random time leakage?
- Are metrics reported as diagnostics rather than forecast claims?
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ The dynamic calculations operate on regular latitude/longitude grids:
geopotential height = geopotential / g0
relative vorticity = dv/dx - du/dy
horizontal advection = -(u dS/dx + v dS/dy)
dry temperature tendency ~= -u dT/dx - v dT/dy - omega dT/dp + kappa T omega / p
```

Grid spacing is approximated from spherical Earth geometry:
Expand Down Expand Up @@ -111,6 +112,10 @@ the result reproducible. Metrics include RMSE, MAE, bias, and correlation.
The baseline is included for workflow demonstration only. It is not a claim of
forecast skill.

For focused model review, an alpha grid can be evaluated with the same
time-ordered split. This gives a compact view of regularization sensitivity
without random temporal leakage.

## 8. Synthetic Ensemble Summary

The synthetic Nino-style ensemble utility creates deterministic plume data with
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ Inspect:
- [`docs/diagnostic-analysis.md`](diagnostic-analysis.md)
- [`docs/station-precipitation-workflows.md`](station-precipitation-workflows.md)
- [`docs/climate-statistical-diagnostics.md`](climate-statistical-diagnostics.md)
- [`docs/focused-case-abstraction.md`](focused-case-abstraction.md)
- [`docs/data-policy.md`](data-policy.md)
- [`examples/synthetic-weather-diagnostics-report.md`](../examples/synthetic-weather-diagnostics-report.md)
- [`examples/sample_metadata.json`](../examples/sample_metadata.json)
Expand All @@ -36,6 +37,8 @@ Questions to answer:
described with equations or explicit numerical assumptions?
- Are station interpolation, precipitation accumulation conversion, and
climate-statistics helpers documented with missing-data behavior?
- Are temperature-tendency components and ridge alpha-grid evaluation separated
from original case-study prose and data?
- Are synthetic examples clearly labeled as synthetic?
- Are forecast-skill claims avoided unless real validation data are supplied?

Expand Down Expand Up @@ -88,14 +91,15 @@ For a deeper review, read the project in this order:
2. `src/python_weather_diagnostics_toolkit/thermodynamics.py` and
`docs/calculation-methods.md` for dewpoint formulas and round-trip checks.
3. `src/python_weather_diagnostics_toolkit/dynamics.py` for grid spacing,
vorticity, advection, and moisture flux divergence.
vorticity, advection, temperature-tendency terms, and moisture flux
divergence.
4. `src/python_weather_diagnostics_toolkit/precipitation.py` and
`src/python_weather_diagnostics_toolkit/interpolation.py` for station and
precipitation preparation helpers.
5. `src/python_weather_diagnostics_toolkit/climate.py` for anomaly,
composite, and correlation helpers.
6. `src/python_weather_diagnostics_toolkit/features.py` for cosine-latitude
regional means and time-ordered baseline modeling.
regional means, time-ordered baseline modeling, and alpha-grid review.
7. `src/python_weather_diagnostics_toolkit/ensemble.py` for deterministic
synthetic plume summaries.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ not a submitted report archive.
| 500 hPa height and vorticity maps | `dynamics.py`, `diagnostic-analysis.md`, tests | map-specific code converted to numerical fields |
| 850 hPa temperature advection | `dynamics.py`, `run_dynamics_summary.py` | calculation made dependency-light and synthetic-testable |
| moisture transport diagnostics | `dynamics.py`, `diagnostic-analysis.md` | water-vapor process reframed as flux-divergence calculation |
| focused cold-season case-study report | `focused-case-abstraction.md`, `dynamics.py`, `features.py` | report structure abstracted into tendency decomposition and alpha-grid review |
| station observation cleaning and interpolation | `precipitation.py`, `interpolation.py`, `station-precipitation-workflows.md` | sentinel handling and IDW interpolation made synthetic-testable |
| accumulated precipitation conversion | `precipitation.py`, `station-precipitation-workflows.md` | forecast accumulations converted without redistributing products |
| anomaly, composite, and correlation exercises | `climate.py`, `climate-statistical-diagnostics.md` | statistical methods separated from local datasets |
Expand Down Expand Up @@ -50,11 +51,13 @@ Preserved:
- humidity and dewpoint diagnostics
- geopotential-height conversion
- relative-vorticity and advection calculations
- dry temperature-tendency term decomposition
- moisture flux divergence
- station interpolation and precipitation accumulation conversion
- anomaly, composite, and correlation helpers
- regional feature engineering
- time-ordered baseline modeling
- ridge alpha-grid metric review
- ensemble summary interpretation

Not preserved:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@
"thermodynamic dewpoint checks",
"relative-vorticity diagnostics",
"temperature-advection diagnostics",
"dry temperature-tendency decomposition",
"station interpolation and precipitation accumulation conversion",
"climate anomaly, composite, and correlation diagnostics",
"time-ordered baseline evaluation",
"ridge alpha-grid metric review",
"synthetic ensemble summaries"
]
}
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,18 @@ warmer or colder air across the toy domain. In real analysis, this inference
would need unit checks, map-domain checks, temporal context, and source-data
quality control.

## Temperature-Tendency Terms

The focused case abstraction decomposes dry pressure-coordinate temperature
tendency into horizontal advection, vertical advection, and adiabatic
compression terms.

Interpretation:
This makes the tendency budget auditable: a reviewer can see whether the
diagnosis is driven by zonal flow, meridional flow, vertical motion, or dry
adiabatic effects. The synthetic examples do not include diabatic heating or a
real event.

## Baseline Prediction

The ridge-regression baseline uses region-mean features and a time-ordered
Expand All @@ -58,6 +70,11 @@ The baseline is useful as a workflow sanity check. It should not be described
as operational forecast skill without independent validation, comparison
baselines, and real-data provenance.

Alpha-grid review:
The ridge alpha-grid helper evaluates candidate regularization strengths with
the same time-ordered split. It is a compact model-selection diagnostic, not a
substitute for independent forecast validation.

## Station And Precipitation Workflow

The synthetic station workflow replaces a missing-value sentinel, interpolates
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,16 @@
from .aliases import get_data_array, standardize_coordinates
from .climate import anomaly, composite_mean, pearson_correlation_field, standardized_anomaly
from .dynamics import (
DRY_ADIABATIC_KAPPA,
geopotential_to_height,
horizontal_advection,
horizontal_advection_components,
moisture_flux_divergence,
relative_vorticity,
temperature_tendency_terms,
)
from .ensemble import ensemble_summary, make_synthetic_nino_ensemble
from .features import area_mean, regression_metrics, ridge_regression_fit_predict
from .features import area_mean, regression_metrics, ridge_alpha_grid, ridge_regression_fit_predict
from .interpolation import idw_interpolate_station_to_grid
from .precipitation import (
cumulative_to_increment,
Expand All @@ -27,11 +30,13 @@
"composite_mean",
"cumulative_to_increment",
"cumulative_to_rate",
"DRY_ADIABATIC_KAPPA",
"ensemble_summary",
"event_total",
"geopotential_to_height",
"get_data_array",
"horizontal_advection",
"horizontal_advection_components",
"idw_interpolate_station_to_grid",
"increment_to_rate",
"magnus_dewpoint_celsius",
Expand All @@ -42,8 +47,10 @@
"regression_metrics",
"relative_humidity_from_dewpoint",
"relative_vorticity",
"ridge_alpha_grid",
"ridge_regression_fit_predict",
"standardize_coordinates",
"standardized_anomaly",
"temperature_tendency_terms",
"threshold_exceedance",
]
Loading