Skip to content

Add benchmarking infrastructure #727

@chmjkb

Description

@chmjkb

Build automated benchmarking infrastructure to measure and track model performance across devices and configurations.

Currently, benchmarking models consumes a lot of time:

  • Manual testing on each device
  • No standardized benchmark suite
  • Results scattered across different runs/people
  • No comparison between model configurations (quantization, delegates)

Goals

  • Automated benchmark suite
  • Standardized metrics across all models
  • Historical tracking to catch performance regressions

Metadata

Metadata

Assignees

No one assigned

    Labels

    CoredocumentationImprovements or additions to documentationperformanceRelated to all issues and tasks focused on improving performance

    Type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions