RFC-0054: HUD Integration for Out-of-Tree CI Results

[jewelkm89]

Summary

This RFC defines the HUD-side ingestion and display layer for Out-of-Tree (OOT) CI results, building on RFC-0050 (Cross-Repository CI Relay for PyTorch Out-of-Tree Backends).

Data Flow

flowchart LR
    subgraph Downstream["Downstream CI (OOT Backend)"]
        DS["Run tests\n+ upload artifacts"]
    end

    subgraph ART["Artifact Storage (org-managed)"]
        STORE[("Logs, test reports,\nJUnit XML")]
    end

    subgraph Relay["Relay Server"]
        RH["Result Handler\n• OIDC verify\n• Allowlist check\n• Rate limit"]
    end

    subgraph HUD["HUD"]
        API["/api/oot/results\n• Auth check\n• Payload caps (2MB)"]
    end

    subgraph Storage["Storage"]
        DDB[("DynamoDB\ntorchci-oot-workflow-job\n(in_progress + completed)")]
        STR["DynamoDB Stream"]
        REP["clickhouse-replicator-dynamo"]
        CH[("ClickHouse\ndefault.oot_workflow_job\n(SharedReplacingMergeTree)")]
    end

    subgraph Frontend["HUD Frontend"]
        P1["/oot — Global Summary"]
        P2["/oot/org/repo — Per-Backend"]
        P3["/pr/N — OOT Section"]
    end

    DS -->|"Upload artifacts"| STORE
    DS -->|"① POST in_progress\n② POST completed\n+ artifact_url\n(OIDC token)"| RH
    RH -->|"X-OOT-Relay-Token\n{trusted, untrusted}"| API
    API -->|"UpdateItem"| DDB
    DDB --> STR --> REP --> CH
    CH -->|"Query results +\nartifact_url"| P1 & P2 & P3
    P2 & P3 -.->|"User clicks\nexternal link"| STORE

Loading

Key points:

• Artifact URLs are included in the completed callback payload and flow through the Result Handler → HUD API → DynamoDB → ClickHouse
• HUD pages read artifact_url from ClickHouse and render it as an external link — no direct connection between HUD and downstream storage
• Both in_progress and completed records are replicated to ClickHouse; SharedReplacingMergeTree handles deduplication — when a completed record arrives for the same dynamoKey, it replaces the in_progress row

What this RFC covers

• Write path: Downstream CI → Result Handler → HUD API → DynamoDB → ClickHouse
  • in_progress callbacks → DynamoDB via UpdateItem → replicated to ClickHouse (shows "running" indicators)
  • completed callbacks → DynamoDB via UpdateItem (merges into existing record) → replicated to ClickHouse (replaces in_progress row via SharedReplacingMergeTree)
  • Artifact URLs flow through the callback payload, not sent directly to HUD
• Read path: Three new HUD views:
  • /oot — Global OOT CI summary (cross-repo health overview, repos sorted by pass rate)
  • /oot/[org]/[repo] — Per-backend dashboard (matrix view: PRs × jobs, failure drill-down, external artifact links)
  • /pr/[number] — Collapsible "Out-of-Tree Backends" section in existing PR pages
• Storage schemas: DynamoDB table and ClickHouse table designs
• DB protection: Rate limiting (per-repo at relay), payload caps (2MB at HUD API)
• Security: OIDC authentication, trusted/untrusted payload split, dedicated X-OOT-Relay-Token header, error handling strategy, signed callback token proposal, 3-state machine for status transitions
• Sample payloads: Two-stage wire format examples (downstream → relay, relay → HUD) with full field definitions
• Implementation plan: 6-phase rollout with task-level breakdown:
  1. Storage Layer — DynamoDB + ClickHouse + replicator mapping
  2. HUD API Endpoint — types, extraction, UpdateItem write logic
  3. Relay Integration — handler → HUD forwarding, rate limiting, reusable GHA action
  4. HUD Frontend Pages — 3 views + saved ClickHouse queries
  5. End-to-End Validation — real downstream repo testing
  6. Security Hardening — callback token, state machine (future)

Reference implementation

A reference implementation is available at pytorch/test-infra#8069, which includes the API endpoint, ClickHouse schema, replicator mapping, saved ClickHouse queries, and all three frontend pages.

HUD Mockup design

The draft OOT HUD UI mockup can be seen in OOT HUD Mockup link

Status

Ready for review

[meta-cla]

Hi @jewelkm89!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[ZainRizvi]

A few quick high level comments based on the PR description (I really appreciate the diagram btw!):

• I see some payload validation happening in HUD. Why not have that in Relay Server instead? In HUD, auth check should likely be limited to authenticating the relay server's header value and a maaaaybe payload cap to ensure we don't overburden the database. Everything else probably will find a better home in Relay Server, where the logic can be centralized. And the payload cap is likely worth adding to the relay server as well (HUD's check is more of a sanity check as the db guardian.)

• Why does the Storage box show that only completed jobs will be replicated to ClickHouse? I think the default behavior would be to replicate both in progress and completed, and that would offer more value as well. Current practice for in-tree CI is we update the job status in clickhouse (in progress -> completed) and we should do the same for OOT too. That's why we're storing the data in dynamo, else we could have stored it in S3

Security note: We should give the relay tool it's own unique header to send to hud and only that header should be allowed to make these relay api calls. The X-Hud-Internal-Bot is a more widely used key and so far it has only been used for readonly operations. Let's not give it read-write access since the key is at higher risk of leaking.

[ZainRizvi]

Nice work — the architecture diagrams are really clear and the RFC builds on RFC-0050 well. A few things to sort out, mostly around aligning with the L2 PR's actual wire format and a couple of bugs in the extraction logic.

[ZainRizvi]

There are a couple of bugs here:

• started_at is set to now() on every callback, including completed. That's the time HUD received the POST, not when the job started. On a completed callback, started_at ≈ completed_at, so the ClickHouse duration_seconds alias will always be ≈ 0. Neither started_at nor completed_at are available in the github context, so the action should capture its own wall-clock time and include it in the payload — in_progress sends started_at, completed sends completed_at. Then HUD stores what the action reports instead of using now().

• queue_time is set from trusted.ci_metrics?.queue_time, which is null on completed callbacks (the relay only sends it on in_progress). This needs to skip setting queue_time when it's null rather than writing null over a valid value. Same applies to execution_time on in_progress callbacks.

Also, the DynamoDB write needs to use UpdateItem with SET expressions instead of PutItem. PutItem does a full item replace, so even with the above fixes, the completed callback would clobber queue_time and started_at from the in_progress record. UpdateItem lets each callback write only its own fields.

[subinz1]

All three fixed:

• started_at/completed_at now use downstream-reported timestamps (wf.started_at, wf.completed_at) instead of new Date().toISOString(). Note: these fields need to be added to the L2 PR's callback action (action.yml), which currently doesn't send wall-clock timestamps.
• queue_time/execution_time are only set when the relay provides a non-null value. in_progress sets queue_time; completed sets execution_time. Neither overwrites the other.
• Changed from PutItem to UpdateItem with SET expressions throughout — Hop 3, data flow table, code samples, and implementation plan.

[ZainRizvi]

This key needs to include the job name and attempt number. The callback action runs inside a specific job, and if a downstream repo uses it in multiple jobs within the same workflow (build, test-cpu, test-gpu), all those jobs share the same github.workflow name and would write to the same dynamoKey — last writer wins, the rest are silently lost. And without the attempt number, job retries would overwrite the original run.

Let's add github.job and github.run_attempt to the key: {repo}/{delivery_id}/{workflow_name}/{job_name}/{attempt}. That gives each job its own row, preserves retry history, and makes the /oot/[org]/[repo] matrix view ("columns = OOT jobs") actually useful.

[subinz1]

Thanks you for the thorough review @ZainRizvi, Changed from {repo}/{delivery_id}/{workflow_name} to {repo}/{delivery_id}/{workflow_name}/{job_name}/{run_attempt}. Added job_name and run_attempt to both DynamoDB and ClickHouse schemas. The L2 callback action needs to be updated to include github.job and github.run_attempt in the payload.

[ZainRizvi]

These samples don't match the actual wire format. They show a flat structure with top-level status, head_sha, pr_number, etc. But the L2 PR's action.yml sends a nested structure — delivery_id and payload from the dispatch envelope, with a workflow sibling containing status/conclusion/name/url. Your own TypeScript code (RelayPayload, extractDynamoRecord) parses the nested format correctly, so the code is right — but a downstream developer implementing from these samples would produce payloads the system can't parse. These need to match the actual {delivery_id, payload, workflow} structure from the action.

[subinz1]

Replaced all flat-structure samples with the actual nested wire format, shown at two stages:

Stage 1 (downstream → relay): {event_type, delivery_id, payload, workflow} — matching the callback action's output from action.yml
Stage 2 (relay → HUD): {trusted: {verified_repo, ci_metrics}, untrusted: {callback_payload: ...}} — matching forward_to_hud in hud.py
Also added a note about fields (job_name, run_attempt, started_at, completed_at) that need to be added to the L2 action.

[ZainRizvi]

in_progress records need to be replicated to ClickHouse too, not just completed. Current practice for in-tree CI is we update job status in ClickHouse (in_progress → completed) and OOT should follow the same pattern. SharedReplacingMergeTree already supports upserts. Without this, the /pr/N page wouldn't be able to show running jobs.

[subinz1]

Fixed — both in_progress and completed records now replicate to ClickHouse. The replicator forwards all DynamoDB stream events without filtering. SharedReplacingMergeTree handles the upsert — completed replaces in_progress for the same dynamoKey after merges.

[ZainRizvi]

Schema validation and field extraction should live in the Relay Server, not HUD. The relay already does OIDC, allowlist checks, and rate limiting — adding schema validation there keeps HUD simple (authenticate the relay, write what it's told) and avoids duplicating validation if other consumers of relay data appear later. HUD should still enforce a payload size cap as a safety net.

[subinz1]

We have moved — schema validation is now at the relay (Hop 1, step 5). The HUD API endpoint is simplified to: validate X-OOT-Relay-Token header, enforce 2MB payload cap, extract/flatten the record, and UpdateItem write. HUD trusts that the relay has already validated the structure.

[ZainRizvi]

Let's use a dedicated auth header for the relay instead of X-Hud-Internal-Bot. That header is shared across multiple systems (DrCI, trymerge) and has only been used for read-only operations. Since this is a write path, giving the relay its own key limits blast radius if any single one leaks.

[subinz1]

Definitely @ZainRizvi,
we have changed to X-OOT-Relay-Token — a dedicated header scoped only to the OOT relay write path. Added an explanation in the RFC that this isolates the write path so key rotation or revocation doesn't affect other internal HUD consumers (DrCI, trymerge).

[subinz1]

• I see some payload validation happening in HUD. Why not have that in Relay Server instead?

Hello @ZainRizvi, Agreed — moved schema validation to the relay (Hop 1, step 5). The relay now validates delivery_id, workflow.status, and conclusion presence before forwarding. HUD is simplified to: auth (X-OOT-Relay-Token) + payload cap (2MB) + UpdateItem write. This aligns with the L2 PR's result_handler.py which already validates these fields at the relay level

[subinz1]

• Why does the Storage box show that only completed jobs will be replicated to ClickHouse? I think the default behavior would be to replicate both in progress and completed, and that would offer more value as well. Current practice for in-tree CI is we update the job status in clickhouse (in progress -> completed) and we should do the same for OOT too. That's why we're storing the data in dynamo, else we could have stored it in S3

Fixed — removed all "completed only" language. Both in_progress and completed records now replicate to ClickHouse. SharedReplacingMergeTree handles the deduplication — when the completed record arrives for the same dynamoKey, it replaces the in_progress row after merges. This matches in-tree CI behavior and allows the /pr/N page to show running jobs.

Security note: We should give the relay tool it's own unique header to send to hud and only that header should be allowed to make these relay api calls. The X-Hud-Internal-Bot is a more widely used key and so far it has only been used for readonly operations. Let's not give it read-write access since the key is at higher risk of leaking.

Changed everywhere — diagrams, data flow table, authentication flow table, code samples, and implementation plan. The relay now uses a dedicated X-OOT-Relay-Token header instead of the shared X-Hud-Internal-Bot. This isolates the OOT write path so rotating/revoking the token doesn't affect DrCI/trymerge.

[KarhouTam]

Hey, @jewelkm89. Thanks for this RFC, really a great work! According to the previous comments and suggestions on CRCR L2 implementation PR, some inconsistencies existed between the current implementation and this RFC. I've left some comments. PTAL!

cc @fffrog @can-gaa-hou

[KarhouTam]

Why No record → completed should be accepted?

[subinz1]

Hello @KarhouTam,
During the initial design, we have decided that if the in_progress couldn't make it to the DynamoDB due to outages, the downstream repo should still be allowed to insert the completed record. Please refer comment.

[subinz1]

After further review and aligning with the L2 implementation, we've updated the RFC to reject No record → completed. With check_run_id-based state tracking and the DISPATCHED prerequisite, accepting a completed without prior in_progress would bypass the state integrity check. The strict DISPATCHED → IN_PROGRESS → COMPLETED flow provides better replay and fabrication protection.

[KarhouTam]

Currently, when an in_progress callback is lost, the system returns a 400 error, which requires the on-call operator to manually rerun the job. While this behavior was designed to reflect real-world network instability, we have decided after thoughtful internal discussion that it would be more efficient to ensure a completed callback is always preceded by an in_progress callback.

cc @fffrog @can-gaa-hou

[subinz1]

Thanks for confirming @KarhouTam. This aligns with the update we made to the RFC — No record → completed is now rejected, and the strict DISPATCHED → IN_PROGRESS → COMPLETED flow is enforced. If in_progress is lost, the operator reruns the job rather than accepting a completed without state integrity. The RFC reflects this as the agreed-upon behavior.

[KarhouTam]

In my state machine implementation, the Redis key now includes check_run_id instead of run_attempt. Since both values change with reruns, I believe check_run_id is a better fit for representing a unique job than {job_name}:{run_attempt}. Accordingly, the Redis state key has been updated to oot:state:{delivery_id}:{repo}:{check_run_id}. run_attempt still in the dict, but not used by relay server.

To improve HUD usability, I added run_id and job_name to the workflow dictionary, which keep consistent among all runs. This allows the HUD to use a grouping key like {job_name}:{run_id}, preventing a single downstream workflow from appearing as multiple rows due to reruns. Under this approach, every unique check_run_id is treated as a separate job.

[subinz1]

Great improvements @KarhouTam. Aligned the RFC with your changes:

• State machine key: Updated from run_id to check_run_id — oot:state:{delivery_id}:{repo}:{check_run_id}. Since check_run_id is GitHub-assigned and unique per job execution, it's a better uniqueness key than {job_name}:{run_attempt}.
• 3-state model: Adopted the DISPATCHED → IN_PROGRESS → COMPLETED flow with no shortcuts. DISPATCHED is repo-level (set by webhook handler), IN_PROGRESS/COMPLETED are job-level per check_run_id. Added your mermaid diagram.
• New fields: check_run_id, run_id, schema_version added to the workflow interface, DynamoDB/ClickHouse schemas, sample payloads, and the reference implementation.
• DynamoKey: Switched from {job_name}/{run_attempt} to {job_name}/{check_run_id} for per-execution uniqueness.
• HUD grouping: Frontend groups by job_name and keeps the latest run_attempt per PR, preventing reruns from appearing as duplicate rows.

run_attempt is still in the dict and stored for informational display, but not used for keying.

[KarhouTam]

The current state machine differs slightly from this RFC. Here are the state transitions:

According to previous comments in the L2 implementation PR, this state machine has three states:

• DISPATCHED (set only once at the webhook handler per repository; this state is repo-level and every job needs it to update its state).
• IN_PROGRESS (job-level, cannot be duplicated; reruns are treated as separate jobs).
• COMPLETED (job-level, cannot be duplicated; reruns are treated as separate jobs).

Note that the direction graph below is for a single check run, reruns have different check_run_id and are treated as separate jobs, so they won't violate the state machine since they won't have a prior IN_PROGRESS or COMPLETED record.

stateDiagram-v2
    direction LR
    [*] --> DISPATCHED: webhook sends
    DISPATCHED --> IN_PROGRESS: first callback
    IN_PROGRESS --> COMPLETED: completion
    IN_PROGRESS --> IN_PROGRESS: ❌ duplicate
    DISPATCHED --> COMPLETED: ❌ skip IN_PROGRESS
    COMPLETED --> COMPLETED: ❌ duplicate
    COMPLETED --> IN_PROGRESS: ❌ wrong direction
    [*] --> IN_PROGRESS: ❌ no dispatch
    [*] --> COMPLETED: ❌ no dispatch

Loading

State types:

• DISPATCHED: Repo-level state (check_run_id=DISPATCH_CHECK_RUN_ID) — one per repository.
• IN_PROGRESS / COMPLETED: Job-level states — independent per job.

Valid flows:

• Main: DISPATCHED → IN_PROGRESS → COMPLETED (linear)
• Reruns: IN_PROGRESS → IN_PROGRESS or COMPLETED → IN_PROGRESS (update timestamps)

Rejected:

• Skipping IN_PROGRESS, duplicate COMPLETED, or missing DISPATCHED state.
• Replay attack: duplicate IN_PROGRESS (same check_run_id).

[can-gaa-hou]

For the retry scenario, queue_time may be an invalid value. I suppose we should not update this value in this case.

[subinz1]

Good catch @can-gaa-hou. On retries (run_attempt > 1), queue_time = dispatch_timestamp → rerun_in_progress_timestamp could be hours or days old — meaningless as a CI metric.

The fix belongs on the relay side: the relay should set queue_time = null when run_attempt > 1. On the HUD side, we already only write queue_time when the relay provides a non-null value, so no HUD changes are needed.

Updated the RFC to document this behavior.

[KarhouTam]

More inconsistencies found.

[KarhouTam]

downstream_repo_level is included for HUD convenience now.

Values: L1 ~ L4 (string)

[subinz1]

Thanks @KarhouTam. Good call — updated the RFC and reference implementation to source downstream_repo_level from trusted (relay-determined from the allowlist) instead of from the untrusted callback payload. Changes:

RFC:

• Added downstream_repo_level to the trusted fields description and RelayPayload interface
• extractDynamoRecord now reads it from trusted.downstream_repo_level
• Both sample payloads (in_progress and completed) now include "downstream_repo_level": "L2" in the trusted section

Reference implementation (subinz1/test-infra#1):

• Added downstream_repo_level to RelayTrusted and OotWorkflowJobRecord interfaces
• extractDynamoRecord extracts it from trusted.downstream_repo_level

The DynamoDB and ClickHouse schemas already had the column from a prior commit — only the ingestion/extraction path was missing.

[KarhouTam]

workflow now is like:

workflow: dict = {
    "schema_version": ...,
    "status": ...,
    "conclusion": ...,
    "name": ...,
    "url": ...,
    "run_attempt": ...,
    "job_name": ...,
    "check_run_id": ...,
    "run_id": ...,
    "started_at": ...,
    "completed_at": ...,
    "test-results": ...
}

[subinz1]

Thanks for the updated field list @KarhouTam. Updated the RFC and reference implementation to align:

• Added schema_version, check_run_id, and run_id to the workflow interface, DynamoDB/ClickHouse schemas, queries, and sample payloads
• Switched dynamoKey from run_attempt to check_run_id for per-execution uniqueness
• Fixed test-results key to use the hyphenated form matching the L2 action (was test_results — would have silently failed in JS)

One question: should test-results stay hyphenated, or would it be cleaner to normalize to test_results in the action? Hyphenated keys require bracket notation in TS (wf["test-results"]). Happy either way — just want to confirm the convention.

[KarhouTam]

Good one! Actually, in the latest action.yml, we changed it to workflow["test_results"] already.

https://github.com/KarhouTam/test-infra/blob/c2833bd9eee6f7299311695597ad1e57ad974c7b/.github/actions/cross-repo-ci-relay-callback/action.yml#L121-L126

Sorry for the misleading comment.

[subinz1]

Thanks @KarhouTam. We've updated both the RFC and reference implementation to use test_results (underscore) — it matches the L2 action's Python code (workflow["test_results"] = ...) and is more convenient for TypeScript (dot notation vs bracket notation). Also aligned test_results to be summary counts only (total/passed/failed/skipped), with detailed test results going via artifact_url as the action describes.

[jewelkm89]

Hello @albanD, @huydhn and @atalman
Request your review for the RFC as well.
Associated PR is pytorch/test-infra#8069