Handle missing forecasts before summarisation

[seabbs-bot]

Summary

Provide utilities for handling missing forecasts before summarising scores, so that downstream analyses (pairwise comparisons, model confidence sets, ensemble importance) operate on a consistent and well-defined set of scores.

Motivation

When models submit forecasts for different subsets of targets or dates, summarisation and comparison functions can produce misleading results.
For example, a model that only forecasts easy targets will appear to perform better than one that forecasts everything.

Several downstream features depend on principled handling of missingness:

• get_pairwise_comparisons() already restricts to overlapping forecasts per pair, but this is pairwise rather than global
• Model confidence sets (Model confidence sets #1055) need a consistent comparison set across all models
• Ensemble importance (Model importance metrics: LOMO, Shapley values, and PAR for ensemble contribution #1121) needs complete forecast sets for ensemble construction

Kim et al.'s modelimportance package handles this in model_importance_summary() with three strategies: drop, worst-case imputation, and average imputation.
These are general-purpose strategies that belong in scoringutils rather than being reimplemented in each downstream feature in my opinion.

Possible approaches

• Filter to common targets: restrict to the intersection of targets covered by all (or a minimum number of) models. Simple and transparent.
• Impute missing scores: fill in missing scores with worst-case, average, or custom values. More flexible but requires assumptions.
• Flag and report: identify and report missingness without modifying data, leaving the choice to the user.

These are not mutually exclusive. A utility that detects and reports missingness, plus helpers for filtering or imputing, would cover most use cases.

Connection to existing functionality

• get_pairwise_comparisons() handles missingness per pair via overlapping forecast sets
• summarise_scores() silently summarises whatever is present
• validate_forecast() checks structure but not completeness

See also #1055 (model confidence sets) and #1121 (ensemble importance).

This was opened by a bot. Please ping @seabbs for any questions.

[seabbs]

I think this is quite worth having. Happy to take on a feature PR for it. One question is if we want this to be a function i.e. impute, documentation, a series of functions, a function that takes a function (with a few options) etc. I am thinking we want it to be a function with a series of options and then specify how to write your own (very standard data munging) as that should really be quite easy for people once they are aware they might want to do it.

[nikosbosse]

I like it. Johannes once made the point that pairwise comparisons are essentially a form of imputation. I'm not sure this is the best form of imputation, but it would be cool to add that bit of information to docs somewhere (and maybe even implement the same imputation as a standalone imputation).

[seabbs]

One thing to consider is do we want this imputation layer on the score level or on the forecast level. I assume we would want a variable to track imputed vs non-imputed forecasts if we add this?

I assumed that we would want this on the score level but perhaps we want it on the forecast level? I can't imagine we want both but I suppose that is also an option?

[seabbs-bot]

Design proposal

Two user-facing functions, both taking a strategy function for extensibility, plus a set of strategy factories.

API

# Filter — drop targets/rows based on a strategy
filter_missing_scores(
  scores,
  strategy = filter_to_intersection(),
  compare = "model"
)

# Impute — fill in missing score rows based on a strategy
impute_missing_scores(
  scores,
  strategy,
  compare = "model"
)

Both always operate on scores objects (after score()).

Filter strategies

filter_to_intersection(min_coverage = 1)
# min_coverage = 1 → strict intersection (targets covered by ALL models)
# min_coverage = 0.5 → targets covered by at least half of models

No .imputed column — filtered rows are simply dropped. Emits cli_inform() with count of dropped targets.

Impute strategies

impute_worst_score()       # worst (max) observed score across models for that target
impute_mean_score()        # mean observed score across models for that target
impute_na_score()          # NA_real_ (users need na.rm = TRUE in summarise_scores())
impute_model_score(model)  # use a specific model's scores (e.g. a baseline model)

Custom strategies are supported — any function with signature function(scores, missing_rows, metrics) returning a data.table of filled rows.

impute_missing_scores() always adds an .imputed logical column (TRUE for new rows, FALSE for existing). .imputed is added to get_protected_columns() so it doesn't contaminate get_forecast_unit().

How missing detection works (shared internal)

1. get_forecast_unit(scores) → forecast unit columns
2. target_cols <- setdiff(forecast_unit, compare)
3. Complete grid = unique(scores[, target_cols]) cross-joined with unique(scores[[compare]]) — uses observed target combinations, not per-column expand.grid, to avoid inventing phantom targets
4. Anti-join against actual scores → missing rows

Diagnostics

No new diagnostic function. get_forecast_counts() already builds the expand grid and shows zero counts — it serves this role. Documentation should point users there.

Interaction with existing functions

• summarise_scores() — works unchanged. .imputed is protected so excluded from grouping. With impute_na_score(), users need na.rm = TRUE.
• get_pairwise_comparisons() — after filter_missing_scores(), the inner merge in compare_forecasts() always returns the full set. After impute_missing_scores(), imputed values participate in comparisons.
• get_forecast_counts() — serves as the diagnostic for understanding missingness before choosing a strategy.

Files

Action	File
Create	R/filter-missing-scores.R
Create	R/impute-missing-scores.R
Create	tests/testthat/test-filter-missing-scores.R
Create	tests/testthat/test-impute-missing-scores.R
Modify	R/get-protected-columns.R — add .imputed
Modify	R/z-globalVariables.R — add .imputed

Example usage

# Filter to targets all models forecasted
scores |>
  filter_missing_scores(strategy = filter_to_intersection()) |>
  summarise_scores(by = "model")

# Impute with baseline model's scores
scores |>
  impute_missing_scores(strategy = impute_model_score("EuroCOVIDhub-baseline")) |>
  get_pairwise_comparisons(compare = "model")

# Impute with worst observed score
scores |>
  impute_missing_scores(strategy = impute_worst_score()) |>
  summarise_scores(by = "model")

This was opened by a bot. Please ping @seabbs for any questions.

[seabbs-bot]

Updated design

Adding filter_to_model() as a filter strategy and a worked integration example.

Filter strategies (updated)

filter_to_intersection(min_coverage = 1)
# min_coverage = 1 → targets covered by ALL models
# min_coverage = 0.5 → targets covered by at least half of models

filter_to_model(model)
# Keep only targets that the specified model forecasted
# Useful for evaluating all models on a baseline's target set

Integration pattern

A natural two-step pipeline: restrict to a baseline model's targets, then fill any remaining gaps with that model's scores.

scores |>
  filter_missing_scores(strategy = filter_to_model("EuroCOVIDhub-baseline")) |>
  impute_missing_scores(strategy = impute_model_score("EuroCOVIDhub-baseline"))

This should be a documented example and integration test.

Everything else from the previous design comment remains unchanged.

This was opened by a bot. Please ping @seabbs for any questions.

[seabbs-bot]

Updated design (v3)

Collapsing filter_to_model() into filter_to_intersection() via a models argument.

Filter strategies (updated)

filter_to_intersection(min_coverage = 1, models = NULL)
# models = NULL → intersection across all models (default)
# models = "baseline" → targets that specific model covers
# models = c("model_a", "model_b") → intersection of those models' targets
# min_coverage = 0.5 → targets covered by at least half of the specified models

This replaces the previously proposed filter_to_model() — it's just filter_to_intersection(models = "baseline").

Updated integration pattern

scores |>
  filter_missing_scores(
    strategy = filter_to_intersection(models = "EuroCOVIDhub-baseline")
  ) |>
  impute_missing_scores(strategy = impute_model_score("EuroCOVIDhub-baseline"))

Everything else from the original design comment remains unchanged.

This was opened by a bot. Please ping @seabbs for any questions.