feat(evaluation): Add Pareto-Optimal Evaluation

[ankitlade12]

Description

This pull request introduces multi-objective evaluation capabilities to utilsforecast. It adds a robust ParetoFrontier class directly within evaluation.py, providing a model-agnostic and dataframe-agnostic way to identify the best-performing models across conflicting metrics (e.g., minimizing RMSE while minimizing MAE, or minimizing latency while maximizing accuracy).

Since utilsforecast acts as the foundational evaluation layer for Nixtla's ecosystem, integrating Pareto selection natively enables downstream libraries (like mlforecast and statsforecast) to leverage model multi-objective benchmarking out-of-the-box using the standard output of evaluate().

Key Changes

• Added ParetoFrontier class in evaluation.py: Includes is_dominated mathematically validated bounding logic and exposed find_non_dominated routines.
• Built-in 2D Plotting: Implemented plot_pareto_2d() to visually inspect the trade-off frontier. Matplotlib is lazily imported and handled gracefully with an explicit warnings.warn if missing, addressing reviewer feedback to avoid raw print statements.
• Dataframe Agnosticism (AnyDFType): Ensures pandas and polars arrays are passed cleanly through the mathematical logic, addressing previous maintainer concerns surrounding hard pandas dependencies.
• Exposed in __init__.py: Integrated evaluate and ParetoFrontier into __all__ for easy top-level access (from utilsforecast import ParetoFrontier).

Example Usage

from utilsforecast.evaluation import evaluate, ParetoFrontier
from utilsforecast.losses import mae, rmse

# 1. Run standard evaluation
performance_df = evaluate(cv_results, metrics=[mae, rmse], agg_fn="mean")

# 2. Get Pareto Optimal Models
pareto_optimal_df = ParetoFrontier.find_non_dominated(performance_df)

# 3. Visualize Trade-offs
ax = ParetoFrontier.plot_pareto_2d(performance_df, metric_x='rmse', metric_y='mae')

[CLAassistant]

All committers have signed the CLA.

[nasaul]

The core Pareto logic looks algorithmically sound, but the PR needs some changes and add tests for ParetoFrontier

[nasaul]

Thanks for the updates — the core Pareto dominance algorithm is correct and the narwhals-based approach in find_non_dominated is the right direction. There are a few bugs and design issues that need to be addressed before merging.

Bugs

1. plot_pareto_2d breaks polars (hard pd.DataFrame dependency)

In the "metric" column branch, a plain pd.DataFrame is constructed internally, and then pandas-only methods are called on the result of find_non_dominated:

pareto_sorted = pareto_df.sort_values(metric_x)  # pandas only
ax.scatter(pareto_df[metric_x], ...)              # pandas-only indexing
for _, row in plot_df.iterrows():                 # pandas only

The method signature accepts AnyDFType but breaks silently for polars input. Either convert to pandas explicitly at the start of the plotting method, or use narwhals throughout.

2. plot_pareto_2d mutates a caller-provided DataFrame

plot_df["model"] = plot_df.index.astype(str)

This modifies the passed-in DataFrame in place when it's pandas. Use .copy() or .assign() instead.

Design Issues

3. __init__.py unexpectedly exports evaluate at package level

evaluate was not previously exported from utilsforecast.__init__. Adding it here is an unintended API surface change and causes eager import of all of evaluation.py on every import utilsforecast. Only ParetoFrontier (if desired) should be added.

4. Confusing metrics parameter in find_non_dominated

In the "metric" column branch (evaluate() output format), when metrics is passed it is actually used as model names, not metric names:

else:
    models = metrics  # misleading: metrics is used as model names here

A user will naturally pass metric names like ["rmse", "mae"] and get unexpected behavior. Please rename the parameter (e.g., model_subset) or document this clearly, and consider whether the evaluate() format branch even needs this override.

Minor

• evaluation.py module-level __all__ still only has ['evaluate'] — add ParetoFrontier.
• Missing two blank lines before the ParetoFrontier class definition (PEP 8).
• Test has a stray # wait: debug comment that should be removed.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[nasaul]

Hello @ankitlade12 thanks for the update on this contribution. I've cleaned up a little bit the implementation and added a tutorial. Let me know if this tutorial is what you had in mind for the ParetoFrontier functionality. If your okay, we can merge the PR.

[ankitlade12]

Hi @nasaul — thanks a lot for taking the time to clean this up and put together the tutorial!

The refactor is a clear improvement over what I had:

• Moving ParetoFrontier into its own model_selection.py module is the right call — it keeps evaluation.py focused and gives the functionality a more natural home.
• The vectorized is_dominated (broadcasting over the full (n, m) matrix) is much cleaner and faster than my per-model loop.
• Using nw.from_dict(..., backend=nw.get_native_namespace(...)) in plot_pareto_2d is a nicer way to keep polars input as polars all the way through — better than my workaround.
• Adding id_col / cutoff_col parameters to find_non_dominated is a good catch for users who use non-default column names in evaluate().
• The two extra tests (test_plot_pareto_2d_no_model_column, test_plot_pareto_2d_polars) cover cases I had missed.

The tutorial is exactly the kind of walkthrough I had in mind — it motivates the multi-metric problem clearly, shows the standard evaluate() → ParetoFrontier flow, and the CV section is a nice bonus.

All good on my end — happy to merge whenever you are. Thanks again!

[nasaul]

Thanks for this contribution @ankitlade12!