Skip to content

Add CI performance benchmarks with regression detection #908

@brendancol

Description

@brendancol

Problem

The benchmarks/ directory exists with ASV configuration and benchmark files for 13 modules, but:

  1. Benchmarks are never run in CI — there is no performance regression detection.
  2. Benchmark results may be stale.
  3. No coverage for recently added features (dask viewshed, cost_distance dask, emerging_hotspots).
  4. The CI matrix (.github/workflows/test.yml) runs pytest on 3 OS × 3 Python versions but has no performance tracking.

For a library whose primary value proposition is performance (numba JIT, GPU, dask), not detecting regressions is a critical gap.

Proposed Fix

  1. Add an ASV-based benchmark CI job on a fixed Ubuntu runner (consistent hardware for comparison).
  2. Create/update benchmarks for the top 5 performance-sensitive functions: slope, proximity, zonal.stats, cost_distance, focal_stats.
  3. Each benchmark should test at least numpy and dask backends at representative sizes (e.g., 2000×2000 for numpy, 10000×10000 for dask).
  4. Store benchmark results as CI artifacts and compare against the base branch.
  5. Add a performance label to PRs that touch numba/CUDA kernels for automatic benchmark runs.
  6. Fail CI on >20% regression (configurable threshold).

Impact

A library that bills itself as "fast" and "scalable" needs to prove it on every commit. Without regression detection, a well-intentioned refactoring could introduce a 10× slowdown in a numba kernel and nobody would notice until users complain.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions