Skip to content

developer-reference.md

Latest commit

 

History

History
256 lines (199 loc) · 10.5 KB

developer-reference.md

File metadata and controls

256 lines (199 loc) · 10.5 KB

Developer Reference

This document is intended for CX Framework and BenchKit developers. It collects structural and operational details that are too implementation-focused for the top-level README.

Project Structure

benchkit/
|- programs/
|  `- <code>/
|     |- build.sh
|     |- run.sh
|     |- estimate.sh
|     `- list.csv
|- benchpark-bridge/
|  |- config/
|  `- scripts/
|- result_server/
|  |- routes/
|  |- templates/
|  |- utils/
|  |- tests/
|  |- app.py
|  |- app_dev.py
|  `- create_admin.py
|- scripts/
|  |- result_server/
|  `- estimation/
|- config/
|- docs/
|  |- cx/
|  `- guides/
`- .gitlab-ci.yml

Key Areas

  • programs/<code>/ App-specific build, run, and estimation entry points.
  • benchpark-bridge/ BenchPark integration and conversion support.
  • result_server/ Flask-based result portal, ingest API, authentication, admin pages, and tests.
  • scripts/ Shared CI helpers, result shaping, estimation helpers, and portal upload scripts.
  • config/ System, queue, and hardware metadata.
  • docs/ Specifications and operational guides.

Contributor Responsibility Boundaries

The current repository workflow uses the following responsibility split. These are working roles for repository changes and reviews, not a full portal-side approval workflow. A future "requester" or "applicant" role belongs to the not-yet-implemented request/approval workflow.

Role Owns Usually edits
App maintainer App-specific build, run, result emission, app-side estimation declarations, and app-local test cases. programs/<code>/build.sh, programs/<code>/run.sh, programs/<code>/estimate.sh, programs/<code>/list.csv
Site maintainer System registration, queue/scheduler settings, runner setup assumptions, and portal display metadata for systems. config/system.csv, config/queue.csv, config/system_info.csv, add-site.md
Estimation package maintainer Estimation algorithm, required inputs, metadata, applicability, fallback, and package-specific assumptions. scripts/estimation/packages/, scripts/estimation/section_packages/, add-estimation-package.md
BenchKit common maintainer Shared shell helpers, Result/Estimate JSON handoff, portal implementation, common CI checks, and repository-wide contracts. scripts/bk_functions.sh, scripts/result.sh, scripts/estimation/common.sh, scripts/result_server/, result_server/, .github/, .gitlab-ci.yml
Admin / reviewer / approver Review, manual CI judgment, PR acceptance, and portal admin operations. In the current workflow these are the same operational role. GitHub PRs, GitLab manual CI, portal admin pages

Scaffolding or code generation may be added later as convenience tooling, but it is not required for contributors. The supported baseline is that contributors can add apps, sites, and estimation packages by following the guides and existing examples.

Result Portal

Overview

result_server/ provides:

  • ingest APIs for results, estimates, profiler archives, and estimation inputs
  • public and confidential result views
  • detailed result and estimate pages
  • usage reporting
  • TOTP-based authentication
  • admin pages for user management

Main Route Groups

  • result_server/routes/results.py Result-related blueprint registration.
  • result_server/routes/results_list_routes.py Public and confidential result list pages.
  • result_server/routes/results_detail_routes.py Result detail, compare, downloads, and related views.
  • result_server/routes/results_usage_routes.py Usage reporting pages.
  • result_server/routes/estimated.py Estimated-result blueprint registration.
  • result_server/routes/estimated_list_routes.py Estimated-result list pages.
  • result_server/routes/estimated_detail_routes.py Estimated-result detail and downloads.
  • result_server/routes/api.py Ingest and query APIs.
  • result_server/routes/auth.py Login, setup, logout, and TOTP flow.
  • result_server/routes/admin.py Admin-only user management.

Main Templates

  • result_server/templates/_results_base.html Shared shell for portal pages.
  • result_server/templates/_table_base.html Shared table page base.
  • result_server/templates/results.html Result list page.
  • result_server/templates/result_detail.html Result detail page.
  • result_server/templates/result_compare.html Result comparison page.
  • result_server/templates/estimated_results.html Estimated-result list page.
  • result_server/templates/estimated_detail.html Estimated-result detail page.
  • result_server/templates/usage_report.html Usage report page.
  • result_server/templates/systemlist.html System list page.
  • result_server/templates/auth_login.html Login page.
  • result_server/templates/auth_setup.html TOTP setup page.
  • result_server/templates/admin_users.html Admin user management page.

CI Pipeline Structure

1. Main Pipeline

  • Reads programs/<code>/list.csv, config/system.csv, and config/queue.csv
  • Generates .gitlab-ci.generated.yml with scripts/matrix_generate.sh
  • Supports both cross-build and native execution modes
  • Enables or disables jobs based on list.csv

2. Benchmark Execution Pipelines

Cross mode

  • build
  • run
  • send_results

Native mode

  • build_run
  • send_results

Common Notes

  • build.sh and run.sh are the primary application entry points.
  • run.sh receives system, nodes, numproc_node, and nthreads.
  • scripts/bk_functions.sh provides shared emit helpers such as result, section, and overlap output.
  • record_timestamp.sh, collect_timing.sh, and result.sh shape timing and result JSON data before upload.

3. Result Transfer and Storage

  • scripts/result_server/send_results.sh Uploads result JSON and profiler archives.
  • scripts/result_server/send_estimate.sh Uploads estimated-result JSON.
  • scripts/result_server/fetch_result_by_uuid.sh Fetches uploaded result data by UUID.

4. Estimation Pipeline

  • App-specific estimation logic lives in programs/<code>/estimate.sh.
  • Shared helpers live under scripts/estimation/.
  • Re-estimation uses result UUIDs as the main input contract.

5. BenchPark Integration

  • BenchPark-specific conversion and bridge logic lives under benchpark-bridge/.
  • BenchPark CI can adapt BenchPark-native outputs into the ingest schema used by the result server.

Configuration Files

  • config/system.csv System execution configuration.
  • config/queue.csv Queue configuration.
  • config/system_info.csv Hardware and display metadata for the portal.
  • programs/<code>/list.csv App-specific execution matrix.

CI Execution Control

BenchKit uses GitHub for source hosting and GitLab CI for benchmark execution.

The current policy keeps pull requests lightweight. Heavy GitLab benchmark CI is not started automatically for pull requests or protected-branch synchronization; maintainers start it explicitly through GitHub Actions when needed.

See CI Execution Control / CI実行制御 for the active workflow policy, manual GitLab CI inputs, protected-branch synchronization behavior, and legacy commit-message controls.

System-Specific Execution Environments

Execution environments are controlled by:

  • config/system.csv
  • config/queue.csv
  • per-app list.csv
  • app-specific build.sh and run.sh

Each system can define queue group, build mode, run mode, node count, and related scheduler settings.

Runtime Requirements

Typical requirements include:

  • Bash and standard shell tooling
  • GitLab CI runner support
  • site-specific scheduler/runtime support
  • Python 3.12 or later for result shaping, estimation support, and portal components
  • Flask-related Python packages for result_server
  • optional profiler tools depending on system support

For local portal work, see the route, template, and utility layout under result_server/.

Result Portal Local Test Workflow

For the lightweight result_server verification path:

  • install dependencies with python -m pip install -r requirements-result-server.txt
  • run the portal test suite with python result_server/tests/run_result_server_tests.py
  • CI coverage for portal-only changes is provided by .github/workflows/result-server-tests.yml

For production portal deployments:

  • Set FLASK_SECRET_KEY to a strong secret and run result_server/app.py, not app_dev.py.
  • app.py binds to 127.0.0.1:8800 by default; set RESULT_SERVER_HOST and RESULT_SERVER_PORT explicitly when the deployment requires a different bind address.
  • Set runner-scoped ingest keys with RESULT_SERVER_KEYS=runner-a:<RUNNER_A_KEY>,runner-b:<RUNNER_B_KEY>.
  • FLASK_SECRET_KEY and each ingest key must be at least 32 characters and must not use known insecure examples such as dev-api-key, changeme, or secret; production startup refuses these values.
  • The legacy RESULT_SERVER_KEY variable is still accepted as runner default for compatibility, but should be rotated to RESULT_SERVER_KEYS.
  • See docs/deploy/key-management.md for generation and rotation guidance.
  • REDIS_URL must point to a monitored Redis instance; production authentication refuses login when Redis is unavailable.
  • Login verification, API ingest/query, and admin write endpoints use Redis-backed rate limits by default; set RESULT_SERVER_MAX_UPLOAD_MB and RESULT_SERVER_MAX_ARCHIVE_MEMBER_MB when deployment-specific upload limits are needed.
  • Repeated login failures are tracked per email for audit context only; source-scoped Redis rate limits enforce login traffic control without hard-locking a target account.
  • Admin-managed affiliations are only rejected when they contain unsafe path/control characters or the comma delimiter used by the form; set RESULT_SERVER_ALLOWED_AFFILIATIONS only when a deployment wants to enforce a fixed comma-separated allowlist.
  • Security-relevant auth, API, and admin actions emit structured benchkit.audit events; see docs/cx/AUDIT_LOG_SPEC.md.
  • app_dev.py is localhost-only, uses ephemeral development secrets when none are provided, and enables the Werkzeug debugger only with RESULT_SERVER_DEV_DEBUG=1.

Result Quality Visibility

BenchKit keeps result-quality scoring inside the portal. Normal pull requests should not be blocked on quality scoring beyond producing valid result JSON with a FOM value.

Portal quality visibility currently lives in:

  • result list quality badges
  • result detail quality rows
  • /results/usage current-state quality summaries

Treat missing source_info, fom_breakdown, or artifact references as internal improvement candidates, not upload-time or pull-request gates.