Manual Test Plan: CI Docker Build Fix Verification

[github-actions]

Manual Test Plan: CI Docker Build Fix Verification

Issue: Docker image artifact save failing with "reference does not exist" error
Fix Date: 2026-01-12
Test Target: .github/workflows/docker-build.yml (Save Docker Image as Artifact step)
Test Priority: HIGH (blocks PR builds and supply chain verification)

---

Test Objective

Verify that the CI Docker build fix resolves the "reference does not exist" error and enables successful PR builds with artifact generation and supply chain verification.

---

Prerequisites

• Changes merged to a feature branch or development
• Ability to create test PRs against the target branch
• Access to GitHub Actions logs for the test PR
• Understanding of expected workflow behavior

---

Test Scenarios

Scenario 1: Standard PR Build (Happy Path)

Objective: Verify normal PR build succeeds with image artifact save

Steps:

1. Create a test PR with a minor change (e.g., update README.md)
2. Wait for docker-build.yml workflow to trigger
3. Monitor the workflow execution in GitHub Actions

Expected Results:

• ✅ build-and-push job completes successfully
• ✅ "Save Docker Image as Artifact" step completes without errors
• ✅ Step output shows: "🔍 Detected image tag: ghcr.io/wikid82/charon:pr-XXX"
• ✅ Step output shows: "✅ Artifact created: /tmp/charon-pr-image.tar"
• ✅ "Upload Image Artifact" step succeeds
• ✅ Artifact pr-image-XXX appears in workflow artifacts
• ✅ verify-supply-chain-pr job starts and uses the artifact
• ✅ Supply chain verification completes successfully

Pass Criteria: All checks pass, no "reference does not exist" errors

---

Scenario 2: Metadata Tag Validation

Objective: Verify defensive validation catches missing or invalid tags

Steps:

1. Review the "Save Docker Image as Artifact" step logs
2. Check for validation output

Expected Results:

• ✅ Step logs show: "🔍 Detected image tag: ghcr.io/wikid82/charon:pr-XXX"
• ✅ No error messages about missing tags
• ✅ Image inspection succeeds (no "not found locally" errors)

Pass Criteria: Validation steps execute and pass cleanly

---

Scenario 3: Supply Chain Verification Integration

Objective: Verify downstream job receives and processes the artifact correctly

Steps:

1. Wait for verify-supply-chain-pr job to start
2. Check "Download Image Artifact" step
3. Check "Load Docker Image" step
4. Check "Verify Loaded Image" step

Expected Results:

• ✅ Artifact downloads successfully
• ✅ Image loads without errors
• ✅ Verification step confirms image exists: "✅ Image verified: ghcr.io/wikid82/charon:pr-XXX"
• ✅ SBOM generation step uses correct image reference
• ✅ Vulnerability scanning completes
• ✅ PR comment appears with supply chain verification results

Pass Criteria: Full supply chain verification pipeline executes end-to-end

---

Scenario 4: Error Handling (Edge Case)

Objective: Verify defensive validation catches actual errors (if possible to trigger)

Note: This scenario is difficult to test without artificially breaking the build. Monitor for this in production if a natural failure occurs.

Expected Behavior (if error occurs):

• Step fails fast with clear diagnostics
• Error message shows exact issue (missing tag, image not found, etc.)
• Available images are listed for debugging
• Workflow fails with actionable error message

Pass Criteria: If error occurs, diagnostics are clear and actionable

---

Regression Testing

Check Previous Failure Cases

Steps:

1. Review previous failed PR builds (before fix)
2. Note the exact error messages
3. Confirm those errors no longer occur

Expected Results:

• ✅ No "reference does not exist" errors
• ✅ No "image not found" errors during save
• ✅ No manual tag reconstruction mismatches

Pass Criteria: Previous failure patterns are eliminated

---

Performance Validation

Objective: Ensure fix does not introduce performance degradation

Metrics to Monitor:

• Build time (build-and-push job duration)
• Artifact save time
• Artifact upload time
• Total PR workflow duration

Expected Results:

• Build time: ~10-15 minutes (no significant change)
• Artifact save: <30 seconds
• Artifact upload: <1 minute
• Total workflow: <20 minutes for PR builds

Pass Criteria: No significant performance regression (±10% acceptable variance)

---

Rollback Plan

If Tests Fail:

1. Immediate Action:

   • Revert commit fixing the artifact save step
   • Notify team of rollback
   • Create new issue with failure details

2. Investigation:

   • Capture full workflow logs
   • Check docker images output from failing run
   • Verify metadata action output format
   • Check for platform-specific issues (amd64 vs arm64)

3. Recovery:

   • Develop alternative fix approach
   • Test in isolated branch
   • Reapply fix after validation

---

Test Log Template

Test Execution Date: [YYYY-MM-DD]
Test PR Number: #XXX
Workflow Run: [Link to GitHub Actions run]
Tester: [Name]

Scenario 1: Standard PR Build

• Status: [ ] PASS / [ ] FAIL
• Notes:

Scenario 2: Metadata Tag Validation

• Status: [ ] PASS / [ ] FAIL
• Notes:

Scenario 3: Supply Chain Verification Integration

• Status: [ ] PASS / [ ] FAIL
• Notes:

Scenario 4: Error Handling

• Status: [ ] PASS / [ ] FAIL / [ ] N/A
• Notes:

Regression Testing

• Status: [ ] PASS / [ ] FAIL
• Notes:

Performance Validation

• Status: [ ] PASS / [ ] FAIL
• Build time: X minutes
• Artifact save: X seconds
• Total workflow: X minutes
• Notes:

---

Sign-Off

Test Result: [ ] PASS / [ ] FAIL
Tested By: _____________________
Date: _____________________
Approved By: _____________________
Date: _____________________

---

References

• Original issue: See current_spec.md for root cause analysis
• Workflow file: .github/workflows/docker-build.yml
• Related fix: Lines 135-167 (Save Docker Image as Artifact step)
• CHANGELOG entry: See "Fixed" section under "Unreleased"

---

Auto-created from manual-test-ci-docker-fix-20260112.md