Contributing to Eval Hub

Thank you for your interest in contributing to Eval Hub! This document provides guidelines for contributing to the project.

Code of Conduct
Getting Started
Development Setup
How to Contribute
Development Workflow
Code Standards
Testing
Pull Request Process
Issue Reporting
Documentation
Community

Code of Conduct

This project and everyone participating in it is governed by our Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to the project maintainers.

Getting Started

Eval Hub is an API REST server that serves as a routing and orchestration layer for evaluation backends. It supports flexible deployment options from local development to production Kubernetes/OpenShift clusters. Before contributing, familiarize yourself with:

Architecture: Read the README.md for project overview
API Documentation: Check API.md for endpoint specifications
Deployment Options: Understand local development, Podman, and Kubernetes/OpenShift deployment models

Prerequisites

Required for All Development:

Go 1.25.0+
Make for build automation
Git

Optional for Container Testing:

Podman (for containerization testing)

Optional for Cluster Integration Testing:

Access to a Kubernetes/OpenShift cluster
kubectl or oc CLI tools

Development Setup

Fork and Clone

git clone https://github.com/your-username/eval-hub.git
cd eval-hub

Install Dependencies

# Download and tidy Go dependencies
make install-deps

Configure Environment

cp .env.example .env
# Edit .env with your local configuration
# Or edit config/config.yaml directly

Install Pre-commit Hooks
```
pre-commit install
```

Verify Setup

# Run tests to verify everything works
make test

# Start the development server (default port 8080)
make start-service

# Or use a custom port
PORT=3000 make start-service

How to Contribute

We welcome contributions in various forms:

Types of Contributions

Bug Fixes: Fix issues in existing functionality
Features: Add new evaluation backends, API endpoints, or capabilities
Documentation: Improve README, API docs, or add examples
Testing: Add test coverage or improve test infrastructure
Performance: Optimize existing code or reduce resource usage
DevOps: Improve CI/CD, deployment, or monitoring

Contribution Areas

Backend Executors: Add support for new evaluation frameworks
API Endpoints: Extend the REST API with new functionality
Deployment Integration: Improve local, Podman, or Kubernetes deployment and orchestration
MLFlow Integration: Enhance experiment tracking capabilities
Monitoring: Add metrics, logging, or health checks
Documentation: User guides, API documentation, examples

Development Workflow

1. Create an Issue

Before starting work, create an issue to discuss:

Bug Reports: Describe the problem with reproduction steps
Feature Requests: Explain the use case and proposed solution
Architectural Changes: See special requirements below
Questions: Ask for clarification or guidance

Architectural Changes

Definition: Changes that affect system design, component interactions, or technology choices, including:

New backend executors or evaluation frameworks
API endpoint additions or modifications
Database schema changes
Deployment architecture updates
New dependencies or technology stack changes
Performance or security architectural decisions

Required Process:

Create Issue: Use kind/architecture label
Discussion: Allow community input and maintainer feedback in the issue
Approval: Maintainers add status/accepted label after discussion
Implementation: Only proceed with implementation after approval
Closure: Issues without approval will be closed with explanation

Note: Implementation PRs for architectural changes will only be accepted if the corresponding issue has status/accepted label.

2. Branch Strategy

# Create a feature branch from main
git checkout main
git pull origin main
git checkout -b feature/your-feature-name

# Or for bug fixes
git checkout -b fix/issue-description

3. Development Process

Write Tests First: For new features, write tests before implementation
Implement Changes: Write code following our standards
Test Locally: Run full test suite and verify functionality
Document Changes: Update relevant documentation

4. Commit Guidelines

Use conventional commits:

# Format: type(scope): description
git commit -m "feat(api): add collection-based evaluation endpoint"
git commit -m "fix(executor): handle timeout errors in NeMo evaluator"
git commit -m "docs(readme): update deployment instructions"
git commit -m "test(integration): add MLFlow integration tests"

Types: feat, fix, docs, test, refactor, perf, ci, chore

PRs targeting main will fail CI if any commit message does not follow this format.

If you have pre-commit installed, commit messages are also checked locally:

pre-commit install --hook-type commit-msg

Code Standards

Code Quality Tools

We use automated tools to maintain code quality:

# Format code
make fmt

# Lint code
make lint

# Vet code
make vet

# Run all quality checks
pre-commit run --all-files

Go Standards

Go Version: Support 1.25.0+
Code Style: Follow standard Go conventions (enforced by gofmt)
Error Handling: Always check and handle errors explicitly
Documentation: Use godoc-style comments for exported types and functions
Import Grouping: Standard library, then external packages, then internal packages

Code Organization

Packages: Keep packages focused and cohesive
Dependencies: Add new dependencies carefully
Error Handling: Return errors explicitly; use error wrapping with fmt.Errorf and %w
Logging: Use structured logging with zap (wrapped in slog interface)
Configuration: Use Viper for configuration management

Example Code Structure

// Package handlers provides HTTP request handlers for evaluation operations.
package handlers

import (
	"encoding/json"

	"github.com/your-org/eval-hub/internal/executioncontext"
)

// EvaluationRequest represents an evaluation request.
type EvaluationRequest struct {
	Model          string   `json:"model"`
	Benchmarks     []string `json:"benchmarks"`
	ExperimentName string   `json:"experiment_name,omitempty"`
}

// HandleCreateEvaluation processes an evaluation request.
// Returns evaluation results or an error.
func (h *Handlers) HandleCreateEvaluation(ctx *executioncontext.ExecutionContext, w http.ResponseWriter, r *http.Request) {
	var req EvaluationRequest
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		ctx.Logger.Error("Failed to decode request", "error", err)
		http.Error(w, "Invalid request", http.StatusBadRequest)
		return
	}

	ctx.Logger.Info("Processing evaluation", "model", req.Model)
	// Implementation here
}

Testing

Test Categories

Unit Tests: Test individual functions and packages (in internal/)
FVT (Functional Verification Tests): BDD-style tests using godog (in tests/features/)
Integration Tests: Test component interactions

Running Tests

# Run all tests (unit + FVT)
make test-all

# Run only unit tests
make test

# Run only FVT tests
make test-fvt

# Generate FVT HTML report (requires Node dev deps)
npm install
make fvt-report

# Run tests with coverage
make test-coverage

# Run specific unit test
go test -v ./internal/handlers -run TestHandleName

# Run specific FVT test
go test -v ./tests/features -run TestFeatureName

Test Requirements

New Features: Must include unit and integration tests
Bug Fixes: Must include regression tests
Coverage: Maintain >80% test coverage
Performance: Include performance tests for critical paths

Test Structure

package handlers

import (
	"net/http"
	"net/http/httptest"
	"testing"

	"github.com/your-org/eval-hub/internal/executioncontext"
)

func TestHandleCreateEvaluation_Success(t *testing.T) {
	// Arrange
	req := httptest.NewRequest(http.MethodPost, "/api/v1/evaluations/jobs", nil)
	w := httptest.NewRecorder()

	// Act
	handler.HandleCreateEvaluation(ctx, w, req)

	// Assert
	if w.Code != http.StatusOK {
		t.Errorf("expected status %d, got %d", http.StatusOK, w.Code)
	}
}

func TestHandleCreateEvaluation_Timeout(t *testing.T) {
	// Test timeout handling
}

OpenShift Deployment Testing

EvalHub can be deployed on OpenShift via the TrustyAI operator, which is included in OpenDataHub.

Prerequisites

Access to an OpenShift cluster
Cluster admin permissions or sufficient RBAC permissions
A container registry account (e.g., quay.io) for hosting your custom EvalHub image

Deployment Steps

Install OpenDataHub from OperatorHub

Install OpenDataHub 3.3 (recommended) from the OpenShift OperatorHub:
- Navigate to Operators → OperatorHub in the OpenShift console
- Search for "Open Data Hub"
- Install version 3.3 (or latest stable version)

Create a DataScienceCluster

Create a DataScienceCluster with the TrustyAI component enabled (enabled by default):

apiVersion: datasciencecluster.opendatahub.io/v1
kind: DataScienceCluster
metadata:
  name: default-dsc
spec:
  components:
    trustyai:
      managementState: Managed

Build and Push Your EvalHub Image

Build your custom EvalHub image and push it to a container registry:

# Build the image
podman build -t quay.io/<your-username>/eval-hub:latest .

# Push to registry
podman push quay.io/<your-username>/eval-hub:latest

Update Manifests with Custom Image

In your fork of the TrustyAI operator, update the params.env file in your manifests to reference your custom EvalHub image:
```
evalHubImage=quay.io/<your-username>/eval-hub:latest
```

Configure Custom Image Reference

You have two options to use your custom image:

Option A: Using devFlags

Update your DataScienceCluster to reference your custom manifests:

apiVersion: datasciencecluster.opendatahub.io/v1
kind: DataScienceCluster
metadata:
  name: default-dsc
spec:
  components:
    trustyai:
      devFlags:
        manifests:
          - contextDir: config
            sourcePath: ""
            uri: "https://github.com/<your-org>/trustyai-service-operator/tarball/<your-branch>"
      managementState: Managed

Option B: Mount manifests directly

Update the manifest files with your custom image reference and mount them to the operator. See the OpenDataHub Component Development Guide for details on mounting manifests.

Deploy an EvalHub Custom Resource

Create an EvalHub CR to deploy your instance:

apiVersion: trustyai.opendatahub.io/v1alpha1
kind: EvalHub
metadata:
  name: evalhub-instance
  namespace: <your-namespace>
spec:
  # Add your EvalHub configuration here

Additional Resources

For more detailed information on deployment and development workflows:

Pull Request Process

Before Submitting

Rebase on Main: Ensure your branch is up-to-date

git checkout main
git pull origin main
git checkout your-branch
git rebase main

Run Full Test Suite
```
pytest
pre-commit run --all-files
```
Update Documentation: Include relevant documentation updates

PR Template

When creating a pull request, include:

Description

Brief summary of changes
Link to related issue(s)

Type of Change

Bug fix (non-breaking change)
New feature (non-breaking change)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Testing

Unit tests pass
Integration tests pass
New tests added for new functionality

Checklist

Code follows project style guidelines
Self-review of code completed
Documentation updated
No new warnings introduced

Review Process

Automated Checks: CI must pass (tests, linting, type checking)
OWNERS Assignment: TBD - Project maintainers will be assigned as reviewers
Code Review: Component experts and maintainer approval required
Testing: Reviewers may test functionality manually
Documentation: Ensure documentation is clear and complete

Issue Reporting

We use a structured labeling system with kind/* prefixes to categorize issues.

Bug Reports

When reporting bugs, include:

**Description**: Clear description of the issue

**To Reproduce**: Steps to reproduce the behavior
1. Go to '...'
2. Click on '....'
3. See error

**Expected Behavior**: What you expected to happen

**Environment**:
- OS: [e.g. Ubuntu 22.04]
- Go Version: [e.g. 1.25.0]
- eval-hub Version: [e.g. 0.1.1]
- Kubernetes Version: [e.g. 1.28]

**Additional Context**: Any additional information

Feature Requests

For feature requests, include:

**Problem Statement**: What problem does this solve?

**Proposed Solution**: Describe your proposed solution

**Alternatives**: Any alternative solutions considered

**Use Case**: Real-world scenario where this would be useful

**Implementation Notes**: Technical considerations or constraints

Documentation

Types of Documentation

API Documentation: OpenAPI specs and endpoint documentation
User Guides: How-to guides for common tasks
Developer Docs: Architecture and implementation details
Deployment Guides: Kubernetes/OpenShift deployment instructions

Documentation Standards

Clarity: Write for your intended audience
Examples: Include practical examples
Accuracy: Keep documentation in sync with code
Structure: Use consistent formatting and organization

Building Documentation

# The OpenAPI specification is maintained in docs/openapi.yaml
# Update the spec as you add or modify endpoints

# To view the API documentation locally, you can use any OpenAPI viewer
# or serve it through the running service at /api/v1/openapi

Community

Communication Channels

Issues: GitHub Issues for bug reports and feature requests
Discussions: GitHub Discussions for general questions
Pull Requests: GitHub PRs for code contributions

Getting Help

Check Existing Issues: Search for similar problems
Read Documentation: Review README and API docs
Ask Questions: Create a GitHub Discussion
Join Community: Engage with other contributors

Recognition

Contributors are recognized in:

Release Notes: Major contributions highlighted
Contributors: GitHub automatically tracks contributors
Acknowledgments: Special recognition for significant contributions

License

By contributing to Eval Hub, you agree that your contributions will be licensed under the Apache License 2.0.

Thank you for contributing to Eval Hub! Your efforts help improve ML evaluation capabilities for the entire community.

FilesExpand file tree

CONTRIBUTING.md

Latest commit

History

CONTRIBUTING.md

File metadata and controls

Contributing to Eval Hub

Table of Contents

Code of Conduct

Getting Started

Prerequisites

Development Setup

How to Contribute

Types of Contributions

Contribution Areas

Development Workflow

1. Create an Issue

Architectural Changes

2. Branch Strategy

3. Development Process

4. Commit Guidelines

Code Standards

Code Quality Tools

Go Standards

Code Organization

Example Code Structure

Testing

Test Categories

Running Tests

Test Requirements

Test Structure

OpenShift Deployment Testing

Prerequisites

Deployment Steps

Additional Resources

Pull Request Process

Before Submitting

PR Template

Review Process

Issue Reporting

Bug Reports

Feature Requests

Documentation

Types of Documentation

Documentation Standards

Building Documentation

Community

Communication Channels

Getting Help

Recognition

License