fix: upgrade test-coverage-improver sandbox to AWF v0.25.51 by Copilot · Pull Request #3583 · github/gh-aw-firewall

Copilot · 2026-05-22T14:26:11Z

The Test Coverage Improver agentic workflow was failing immediately before the Copilot agent was ever invoked. The install_awf_binary.sh step tried to download checksums.txt from release v0.25.29, which doesn't exist as a binary release, returning 404 and aborting the job.

Changes

test-coverage-improver.md: Updated sandbox.agent.version from v0.25.29 → v0.25.51 (latest release with full binary assets)
All *.lock.yml files: Recompiled via gh aw compile + post-processing script (postprocess-smoke-workflows.ts) as required after any version bump — updates container image digests and replaces the install-from-release step with a local build step in affected workflows

The failure signature was: curl: (22) The requested URL returned error: 404 on releases/download/v0.25.29/checksums.txt, with GH_AW_EFFECTIVE_TOKENS: (none) confirming the AI agent was never reached.

The workflow was failing because sandbox.agent.version v0.25.29 did not have binary artifacts in releases (checksums.txt returned 404). Updated to v0.25.51 (latest release with all required assets), then recompiled all workflows with `gh aw compile` and ran the required post-processing script.

Copilot

Pull request overview

This pull request updates the agentic workflow sandbox/runtime configuration to prevent the Test Coverage Improver workflow from failing before the Copilot agent runs, and regenerates the compiled/locked workflows with the newer gh-aw compiler output.

Changes:

Bumped test-coverage-improver sandbox AWF agent version from v0.25.29 to v0.25.51.
Recompiled multiple *.lock.yml workflows (compiler metadata updated to v0.74.8) and updated generated execution/env wiring accordingly.
Updated the maintenance workflow and the actions lockfile content as part of the recompilation.

Show a summary per file

File	Description
.github/workflows/update-release-notes.lock.yml	Regenerated lock workflow with gh-aw v0.74.8 and updated generated execution/env wiring.
.github/workflows/test-coverage-improver.md	Bumps sandbox AWF agent version to `v0.25.51`.
.github/workflows/smoke-claude.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata.
.github/workflows/security-review.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/issue-monster.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/issue-duplication-detector.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/firewall-issue-dispatcher.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/export-audit.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/dependency-security-monitor.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and additional checkout handling.
.github/workflows/copilot-token-usage-analyzer.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/copilot-token-optimizer.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/cli-flag-consistency-checker.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/claude-token-usage-analyzer.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/ci-doctor.lock.yml	Regenerated lock workflow output with updated gh-aw v0.74.8 metadata and env wiring tweaks.
.github/workflows/agentics-maintenance.yml	Regenerated maintenance workflow to use updated gh-aw action versions.
.github/aw/actions-lock.json	Updates the action pin lockfile contents alongside regenerated workflows.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

.github/aw/actions-lock.json:88

.github/aw/actions-lock.json no longer contains entries for github/gh-aw-actions/setup@v0.74.8 / github/gh-aw-actions/setup-cli@v0.74.8, but the regenerated workflows reference those actions. If this lock file is meant to provide immutable action pins, please add the missing entries (repo/version/commit SHA) so compilation can emit SHA-pinned uses: references and avoid mutable tag execution.

      "repo": "docker/setup-buildx-action",
      "version": "v4.0.0",
      "sha": "4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd"
    },
    "github/gh-aw/actions/setup@v0.72.1": {
      "repo": "github/gh-aw/actions/setup",
      "version": "v0.72.1",
      "sha": "489dbab88cc78e35506b5ccbf08a4037166824ac"
    },
    "softprops/action-gh-release@v3.0.0": {
      "repo": "softprops/action-gh-release",
      "version": "v3.0.0",
      "sha": "b4309332981a82ec1c5618f44dd2e27cc8bfbfda"
    }
  },

Files reviewed: 39/39 changed files
Comments generated: 3

    steps:
      - name: Setup Scripts
        id: setup
-        uses: github/gh-aw-actions/setup@d3abfe96a194bce3a523ed2093ddedd5704cdf62 # v0.74.4
+        uses: github/gh-aw-actions/setup@v0.74.8
        with:
          destination: ${{ runner.temp }}/gh-aw/actions


    steps:
      - name: Setup Scripts
-        uses: github/gh-aw-actions/setup@d3abfe96a194bce3a523ed2093ddedd5704cdf62 # v0.74.4
+        uses: github/gh-aw-actions/setup@v0.74.8



      - name: Install gh-aw
-        uses: github/gh-aw-actions/setup-cli@d3abfe96a194bce3a523ed2093ddedd5704cdf62 # v0.74.4
+        uses: github/gh-aw-actions/setup-cli@v0.74.8


lpcox · 2026-05-22T14:57:59Z

@copilot address review feedback

Copilot · 2026-05-22T15:04:51Z

@copilot address review feedback

Addressed. I pinned github/gh-aw-actions/setup and setup-cli to immutable commit SHA efa55847f72aadb03490d955263ff911bf758700 (with # v0.74.8 annotation) across the affected workflows and added corresponding v0.74.8 entries in .github/aw/actions-lock.json.

Included in commit: 84e1ff3.

github-actions · 2026-05-22T15:16:31Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	95.98%	96.05%	📈 +0.07%
Statements	95.81%	95.87%	📈 +0.06%
Functions	98.02%	98.02%	➡️ +0.00%
Branches	89.44%	89.48%	📈 +0.04%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/config-writer.ts`	83.0% → 85.6% (+2.54%)	83.0% → 85.6% (+2.54%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot · 2026-05-22T15:25:21Z

@copilot fix this failing ci check https://github.com/github/gh-aw-firewall/actions/runs/26295530841/job/77408819527?pr=3583

Fixed in commit 5e4f5bd.

I investigated run 26295530841 / job 77408819527 and updated the Smoke Claude prompt to explicitly require direct safe-output tool calls (add_comment/add_labels/noop), forbid safeoutputs ... CLI usage, and forbid empty args for add_comment/add_labels (use noop when not ready). I also added a test assertion for this guidance in scripts/ci/smoke-claude-workflow.test.ts.

lpcox · 2026-05-22T15:41:18Z

@copilot fix this failing ci check https://github.com/github/gh-aw-firewall/actions/runs/26296541420/job/77410889228?pr=3583

Copilot · 2026-05-22T15:47:09Z

@copilot fix this failing ci check https://github.com/github/gh-aw-firewall/actions/runs/26296541420/job/77410889228?pr=3583

Fixed in commit e4962a3.

I traced run 26296541420/job 77410889228 and updated the Smoke Claude prompt to require safeoutputs CLI usage with real arguments and explicitly forbid pipe-to-stdin JSON payloads (use file redirection from /tmp/gh-aw/agent/*.json instead). This avoids the empty-arguments add_comment failure that consumed extra turns and caused max_turns failure. I also updated scripts/ci/smoke-claude-workflow.test.ts to assert the new guidance.

github-actions · 2026-05-22T15:56:59Z

Smoke Test Results

✅ All tests PASS

✅ GitHub API: 2 recent PR entries verified
✅ GitHub check: playwright_check=✅ PASS
✅ File verify: smoke-test-claude-26297617582.txt exists

Claude engine validation complete.

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-05-22T15:57:07Z

Smoke Test Results

PR #3584: Remove unused export from ProbeResult interface

✅ GitHub MCP: Connected successfully
❌ GitHub.com HTTP: File test failed (file not found)
❌ File Write/Read: /tmp/smoke-test-file.txt not present

Overall Status: ❌ FAIL

cc @Copilot @lpcox

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-05-22T15:57:23Z

Smoke Test: Copilot BYOK (Offline) — PARTIAL FAIL

Tests:

✅ GitHub MCP: Retrieved PR Remove unused export from ProbeResult interface #3584 "Remove unused export from ProbeResult interface"
❓ GitHub.com connectivity: No HTTP code provided in pre-step
❌ File write/read: /tmp/smoke-test-file.txt not found
✅ BYOK inference: Running in offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

Current PR: #3583 "fix: upgrade test-coverage-improver sandbox to AWF v0.25.51" by @Copilot, assigned to @lpcox, @Copilot

Overall: FAIL — file test failed, missing pre-step HTTP data

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-05-22T15:58:38Z

Smoke Test

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-05-22T15:59:19Z

Smoke Test: Services Connectivity - FAIL ❌

❌ Redis (host.docker.internal:6379): Connection timeout
❌ PostgreSQL (host.docker.internal:5432): No response
❌ PostgreSQL query: Not attempted (connection failed)

Result: FAIL - No service containers are reachable from AWF sandbox

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-05-22T15:59:57Z

Smoke Test Results

GitHub MCP: ❌
GitHub.com Connectivity: ❌
File Writing: ✅
Bash Tool: ✅

Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

github-actions · 2026-05-22T16:03:05Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	1/1 passed	✅ PASS
Node.js	execa	✅	1/1 passed	✅ PASS
Node.js	p-limit	✅	1/1 passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

All build and test operations completed successfully across all 18 projects.

Generated by Build Test Suite for issue #3583 · ● 15.7M · ◷

Initial plan

1e04434

Copilot AI assigned Copilot and lpcox May 22, 2026

Copilot started work on behalf of lpcox May 22, 2026 14:30 View session

Copilot AI linked an issue May 22, 2026 that may be closed by this pull request

[aw] Test Coverage Improver failed #3577

Closed

lpcox marked this pull request as ready for review May 22, 2026 14:43

Copilot AI review requested due to automatic review settings May 22, 2026 14:43

Copilot AI changed the title ~~[WIP] Debug workflow failure for test coverage improver~~ fix: upgrade test-coverage-improver sandbox to AWF v0.25.51 May 22, 2026

Copilot finished work on behalf of lpcox May 22, 2026 14:44

Copilot AI requested a review from lpcox May 22, 2026 14:44

Copilot started reviewing on behalf of lpcox May 22, 2026 14:45 View session