fix(api-proxy): generalize deprecated header stripping for any provider/header

[lpcox]

Problem

PR #3657 added retry-on-400 logic for deprecated anthropic-beta headers, but it was too narrowly scoped:

1. Only triggered for provider === 'anthropic' — missed the copilot provider path
2. Only handled the anthropic-beta header specifically — would not adapt to other headers being rejected
3. Only buffered 400s when anthropic-beta was present in the request

This caused Copilot CLI v1.0.48/v1.0.51 workflows to fail when sending context-1m-2025-08-07 through the copilot provider (run 26347219614).

Solution

Generalized the retry-on-400 logic into a header-agnostic system:

How it works

1. On any 400 from anthropic/copilot providers, buffer the response body
2. Parse the error for the pattern: Unexpected value(s) \` for the `` header`
3. Learn the (header, value) pair — store in Map<headerName, Set<rejectedValues>>
4. Strip the value and retry once — if stripping leaves the header empty, remove it entirely
5. Proactively strip on subsequent requests — no retry round-trip needed after first encounter

Key design decisions

• General pattern matching — DEPRECATED_HEADER_PATTERN captures both the rejected value AND the header name, so it works for any header Anthropic (or the Copilot gateway) might reject in the future
• Per-header value sets — deprecatedHeaderValues is a Map<string, Set<string>> with a 200-value cap per header
• Provider-agnostic learning — values learned from one provider are applied across all providers on subsequent requests
• No hardcoded header names — the system discovers which headers have deprecated values purely from upstream error responses

Testing

5 test cases covering:

1. Reactive retry via anthropic provider (existing)
2. Proactive stripping for anthropic provider (existing)
3. Reactive retry via copilot provider (new)
4. Cross-provider proactive stripping (new)
5. Arbitrary header name — proves the system works for any header, not just anthropic-beta (new)

All 47 proxy tests pass.

Related

• Closes API proxy: auto-strip deprecated anthropic-beta header values on 400 rejection #3656
• Supersedes the narrow fix in Retry Anthropic requests after deprecated anthropic-beta header rejection #3657
• Failing run: https://github.com/github/gh-aw/actions/runs/26347219614/job/77559127053

[github-actions]

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	96.36%	96.43%	📈 +0.07%
Statements	96.21%	96.28%	📈 +0.07%
Functions	97.96%	97.96%	➡️ +0.00%
Branches	90.35%	90.39%	📈 +0.04%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
src/config-writer.ts	83.0% → 85.6% (+2.54%)	83.0% → 85.6% (+2.54%)

---

Coverage comparison generated by scripts/ci/compare-coverage.ts

[Copilot]

Pull request overview

Extends the API proxy’s “deprecated anthropic-beta header” retry/learning logic to also apply when requests are routed via the Copilot provider (port 10002), ensuring 400 rejections from the Copilot gateway→Anthropic path trigger stripping + a single retry and that learned values are proactively removed on subsequent Copilot requests.

Changes:

• Apply proactive cached stripping of learned deprecated anthropic-beta values for provider === 'copilot' when the header is present.
• Apply reactive “buffer 400 → detect deprecated value → strip → retry once” logic for provider === 'copilot'.
• Improve observability by logging the actual provider (copilot vs always anthropic) when stripping occurs, and add targeted regression tests for both reactive and proactive Copilot paths.

Show a summary per file

File	Description
containers/api-proxy/proxy-request.js	Extends deprecated anthropic-beta stripping/learning logic to cover Copilot provider requests and fixes provider attribution in logs.
containers/api-proxy/server.proxy.test.js	Adds regression tests covering Copilot-provider retry-on-400 stripping and proactive cached stripping after learning.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 2/2 changed files
• Comments generated: 0

[github-actions]

Smoke Test Results

✅ GitHub API (recent-prs.json): 2 PR entries confirmed
✅ GitHub check (playwright_check): PASS
✅ File verify (smoke-test-claude-26375928252.txt): Passed at 2026-05-24 23:36:51 UTC

Overall: PASS

💥 [THE END] — Illustrated by Smoke Claude

[github-actions]

Smoke Test Results

✅ MCP: PR #3670 retrieved: [Test Coverage] Add comprehensive tests for config-assembly validator
❌ File: Test file not found
❓ HTTP: Status not provided

Status: FAIL (file verification failed)

@lpcox

📰 BREAKING: Report filed by Smoke Copilot

[github-actions]

Copilot BYOK Smoke Test Results

PR: #3689 - fix(api-proxy): generalize deprecated header stripping (@lpcox)

✅ GitHub MCP connectivity (fetched PR #3689)
❌ File write/read test (file not found at expected path)
✅ BYOK inference (responding via api-proxy → api.githubcopilot.com)

Note: Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

Status: PARTIAL PASS (2/3 core tests passed, 1 infra test failed)

🔑 BYOK report filed by Smoke Copilot BYOK

[github-actions]

Smoke test FAIL: connectivity and MCP issues detected.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

[github-actions]

Smoke Test Codex: FAIL
PRs: [Test Coverage] Add comprehensive tests for config-assembly validator; [docs] Document Anthropic deprecated beta header retry handling
✅ GitHub PR review, Playwright, file/bash, discussion lookup, build
❌ safeinputs-gh unavailable; Tavily search unavailable
Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

[github-actions]

API Proxy OTEL Tracing Smoke Test Results

Summary

All OTEL integration scenarios validated successfully.

Results

✅ Scenario 1: Module Loading

• Status: PASS
• otel.js module exists and loads successfully
• Exports expected functions: isEnabled(), startRequestSpan(), setTokenAttributes(), endSpan(), endSpanError(), shutdown()

✅ Scenario 2: Test Suite

• Status: PASS
• OTEL test suite exists: containers/api-proxy/otel.test.js
• 35 OTEL tests passed covering:
  • Module initialization and exports
  • OTLP header parsing
  • Span creation with GenAI attributes
  • Parent trace context propagation from GITHUB_AW_OTEL_* env vars
  • Token usage attributes (gen_ai.usage.*)
  • Status handling (OK/ERROR)
  • OTLP/HTTP export with proxy-aware agent
  • File fallback exporter
• Overall test suite: 753 passed, 10 failed (failures unrelated to OTEL)

✅ Scenario 3: Env Var Forwarding

• Status: PASS
• src/services/api-proxy-service.ts lines 128-140 forward all OTEL env vars:
  • OTEL_EXPORTER_OTLP_ENDPOINT (activates OTLP export)
  • OTEL_EXPORTER_OTLP_HEADERS (auth headers)
  • OTEL_SERVICE_NAME (default: awf-api-proxy)
  • GITHUB_AW_OTEL_TRACE_ID (parent trace context)
  • GITHUB_AW_OTEL_PARENT_SPAN_ID (parent span context)

✅ Scenario 4: Token Tracker Integration

• Status: PASS
• token-tracker-http.js line 62 defines onUsage callback parameter
• token-tracker-http.js line 237-239 invokes onUsage(normalized, model) after extracting usage
• Integration point allows OTEL module to attach GenAI semantic conventions to spans

✅ Scenario 5: OTEL Diagnostics

• Status: PASS (no runtime spans expected without live endpoint)
• OTEL module uses graceful degradation: file fallback when OTEL_EXPORTER_OTLP_ENDPOINT unset
• Tests confirm proxy-aware OTLP export routes through Squid when endpoint configured

Architecture Notes

• Span creation: startRequestSpan() creates one CLIENT span per proxied LLM request
• Parent context: Spans link to workflow trace via GITHUB_AW_OTEL_TRACE_ID / GITHUB_AW_OTEL_PARENT_SPAN_ID
• Token attributes: GenAI semantic conventions applied via setTokenAttributes() after response parsing
• Export path: OTLP/HTTP → Squid proxy → collector (respects domain whitelist)
• Fallback: When OTLP endpoint unset, spans → /var/log/api-proxy/otel.jsonl

Conclusion

All scenarios validated. OTEL integration is production-ready.

Warning

Firewall blocked 3 domains

The following domains were blocked by the firewall during workflow execution:

• 127.0.0.1
• api.example.com
• api.openai.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "127.0.0.1"
    - "api.example.com"
    - "api.openai.com"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing

[github-actions]

Chroot Runtime Version Test Results

Runtime	Host Version	Chroot Version	Match?
Python	3.12.13	3.12.3	❌ NO
Node.js	v24.15.0	v22.22.3	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Overall Result: ❌ Tests failed - version mismatches detected

The chroot environment does not have matching runtime versions for Python and Node.js. This could indicate:

• The agent container image needs updating
• Different base images between host and agent container
• Bind mounts may not be exposing the expected runtime paths

Tested by Smoke Chroot

[github-actions]

Smoke Test: Services Connectivity — ❌ FAIL

• ❌ Redis: Connection timeout (no response from host.docker.internal:6379)
• ❌ PostgreSQL pg_isready: Connection timeout (no response from host.docker.internal:5432)
• ❌ PostgreSQL SELECT 1: Not tested (pg_isready failed)

Result: FAIL — AWF sandbox cannot reach GitHub Actions service containers via host.docker.internal

🔌 Service connectivity validated by Smoke Services

[github-actions]

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	All passed	✅ PASS
Node.js	execa	✅	All passed	✅ PASS
Node.js	p-limit	✅	All passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

All build and test operations completed successfully across all 18 projects in 8 ecosystems. The firewall successfully allowed necessary package manager traffic (npm, cargo, Maven, go mod, bun, deno, dotnet, CMake) while maintaining network isolation.

Generated by Build Test Suite for issue #3689 · ● 12.5M · ◷