Refactor token usage record/metrics construction into shared token-persistence helpers

[Copilot]

HTTP and WebSocket token trackers were independently constructing the same token usage metrics + JSONL record shape, creating duplicate maintenance points for schema or field updates. This PR centralizes that logic so both paths use one implementation.

• Shared token usage helpers

  • Added buildTokenUsageRecord(normalized, opts) in containers/api-proxy/token-persistence.js to produce the canonical token-usage record shape (_schema, timing, model/path/status, token fields, response_bytes).
  • Added incrementTokenMetrics(metricsRef, provider, normalized) to centralize input_tokens_total / output_tokens_total increments.

• HTTP tracker now delegates shared logic

  • containers/api-proxy/token-tracker-http.js now calls:
    • incrementTokenMetrics(...) instead of inline metric increments.
    • buildTokenUsageRecord(...) instead of inline record assembly.
  • Existing HTTP-specific enrichments (x_initiator, billing) remain applied after shared record construction.

• WebSocket tracker now delegates shared logic

  • containers/api-proxy/token-tracker-ws.js now uses the same shared helpers for metrics and record construction, while preserving WS-specific values (status: 101, streaming: true, responseBytes: totalBytes - headerBytes).

• Schema-focused test coverage

  • Extended containers/api-proxy/token-tracker.schema.test.js with helper-level assertions:
    • shared record builder outputs schema-compatible records,
    • metrics helper is a safe no-op when no metrics sink is present.

// token-persistence.js
function buildTokenUsageRecord(normalized, { requestId, provider, model, reqPath, status, streaming, duration, responseBytes }) {
  return {
    _schema: TOKEN_USAGE_SCHEMA,
    timestamp: new Date().toISOString(),
    request_id: requestId,
    provider,
    model: model || 'unknown',
    path: reqPath,
    status,
    streaming,
    input_tokens: normalized.input_tokens,
    output_tokens: normalized.output_tokens,
    cache_read_tokens: normalized.cache_read_tokens,
    cache_write_tokens: normalized.cache_write_tokens,
    duration_ms: duration,
    response_bytes: responseBytes,
  };
}

[github-actions]

Claude Engine Smoke Test Results

✅ GitHub API — 2 PR entries confirmed
✅ GitHub Check — playwright_check passed (verified GitHub homepage load)
✅ File Verify — smoke-test-claude-26260067091.txt exists

**Result: PASS

💥 [THE END] — Illustrated by Smoke Claude

[Copilot]

Pull request overview

Centralizes token usage JSONL record construction and token metric increments into shared helpers within the API proxy, so both HTTP and WebSocket token trackers produce a consistent schema and reduce duplicated logic.

Changes:

• Added buildTokenUsageRecord() and incrementTokenMetrics() helpers in token-persistence.js.
• Updated HTTP and WebSocket token trackers to delegate record/metrics construction to the shared helpers.
• Extended schema tests to assert helper output is schema-compatible and metrics helper is safe when no metrics sink is provided.

Show a summary per file

File	Description
containers/api-proxy/token-persistence.js	Adds shared helpers for canonical token usage record construction and metric increments.
containers/api-proxy/token-tracker-http.js	Replaces inline record/metric logic with shared helper calls while preserving HTTP-specific enrichments.
containers/api-proxy/token-tracker-ws.js	Replaces inline record/metric logic with shared helper calls while preserving WS-specific values (101/streaming/byte accounting).
containers/api-proxy/token-tracker.schema.test.js	Adds helper-level assertions for schema compatibility and no-op behavior without metrics.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 4/4 changed files
• Comments generated: 0

[github-actions]

🔬 Smoke Test Results

Test	Status
GitHub MCP connectivity	✅ PR list returned (latest: #3561 "docs(spec): add port information to /reflect endpoint sections")
GitHub.com HTTP connectivity	⚠️ Pre-step data not substituted (template vars unexpanded)
File write/read	⚠️ Pre-step data not substituted (template vars unexpanded)

Overall: PARTIAL — MCP connectivity confirmed; pre-computed step outputs were not injected into the workflow prompt.

PR author: @Copilot | Assignees: @lpcox, @Copilot

📰 BREAKING: Report filed by Smoke Copilot

[github-actions]

🔥 Smoke Test: Copilot BYOK (Offline) Mode

Test	Result
GitHub MCP connectivity	✅ (PR #3561 fetched via MCP)
GitHub.com HTTP	⚠️ N/A (pre-step template not expanded)
File write/read	⚠️ N/A (pre-step template not expanded)
BYOK inference (this response)	✅

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

Overall: PASS (core BYOK path functional)

PR author: @Copilot · Assignees: @lpcox, @Copilot

🔑 BYOK report filed by Smoke Copilot BYOK

[github-actions]

Chroot Smoke Test Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌
Node.js	v24.15.0	v22.22.3	❌
Go	go1.22.12	go1.22.12	✅

Overall: ❌ Not all tests passed — Python and Node.js versions differ between host and chroot.

Tested by Smoke Chroot

[github-actions]

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	all passed	✅ PASS
Node.js	execa	✅	all passed	✅ PASS
Node.js	p-limit	✅	all passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #3566 · ● 6.9M · ◷

[github-actions]

Smoke Test Results — FAIL

Check	Result
Redis PING	❌ Connection refused/timeout
PostgreSQL pg_isready	❌ No response
PostgreSQL SELECT 1	❌ Connection failed

host.docker.internal resolves to 172.17.0.1 but none of the service ports (6379, 5432) are reachable from this runner. Service containers may not be running or network routing is blocked.

Overall: FAIL

🔌 Service connectivity validated by Smoke Services

[github-actions]

Gemini Engine Smoke Test Results

• GitHub MCP Testing: ✅
  • Reduce test-coverage-improver token burn by narrowing file access and preloading target sources #3521: Reduce test-coverage-improver token burn
  • [Test Coverage] Improve coverage for main-action and preflight uncovered branches #3515: [Test Coverage] Improve coverage
• GitHub.com Connectivity: ❌ (HTTP 000, SSL error 35)
• File Writing Testing: ✅
• Bash Tool Testing: ✅

Overall Status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

[github-actions]

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Result	Details
S1: Module Loading	✅	otel.js loads successfully; exports: startRequestSpan, setTokenAttributes, endSpan, endSpanError, shutdown, isEnabled + internals
S2: Test Suite	✅	32/32 tests passed in otel.test.js — span creation, token attrs, OTLP export, graceful degradation
S3: Env Var Forwarding	✅	api-proxy-service.ts forwards OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, GITHUB_AW_OTEL_TRACE_ID, GITHUB_AW_OTEL_PARENT_SPAN_ID
S4: Token Tracker Integration	✅	onUsage callback exists in both token-tracker-http.js (line 237) and token-tracker-ws.js (line 198) as the OTEL hook point
S5: OTEL Diagnostics	✅	FileSpanExporter and ProxyAwareOtlpExporter both functional; no span export errors

All 5 scenarios pass. OTEL tracing integration is working correctly.

📡 OTel tracing validated by Smoke OTel Tracing

[github-actions]

Smoke Codex
✅ PR: Refactor token usage record/metrics construction into shared token-persistence helpers
✅ PR: Refactor Docker Compose teardown into a shared helper
❌ safeinputs-gh: unavailable
✅ Playwright/file/bash/build
❌ Tavily/discussion tools unavailable
Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex