-
Notifications
You must be signed in to change notification settings - Fork 315
Closed
Labels
cookieIssue Monster Loves Cookies!Issue Monster Loves Cookies!
Description
Overview
Health check for 178 agentic workflows as of 2026-03-30T12:08Z. Overall score: 72/100 (↑1 from 71).
This run saw 3 recoveries and 1 escalation since yesterday's report.
Summary
| Category | Count | Change |
|---|---|---|
| Healthy | 173 | ↑3 (3 recovered) |
| Warning | 2 | — (Smoke Gemini, WHM self) |
| Critical | 2 | ↑1 (Smoke Codex escalated P2→P1) |
| Missing lock files | 0 | — |
✅ Recovered This Run
| Workflow | Issue | Consecutive Successes |
|---|---|---|
| PR Triage Agent | #23151 (closed) | 5 (#301–#305) |
| Metrics Collector | #23446 (closed) | 1 (#109 after 3 failures) |
| Lockfile Statistics | #23397 (closed) | 2 (#210 was transient) |
Critical Issues 🚨
Smoke Codex — P1 (Escalated from P2)
- Issue: [aw] Smoke Codex failed #23431 (updated)
- Status: Failing on ALL run types — 4 consecutive schedule failures AND all PR runs
- Error:
"This user's access to this model has been temporarily restricted"— OpenAI model access issue has broadened from schedule-only to all contexts - Last success: Run Add Command field to status --json output #2538 (2026-03-28T01:09Z)
- Impact: Every PR generates a failure event; Codex smoke validation completely blocked
- Action needed: Investigate Codex API credential/quota status
Smoke Update Cross-Repo PR — P1 (Ongoing)
- Issue: [aw] Smoke Update Cross-Repo PR failed #23193 (updated)
- Status: Schedule runs [Custom Engine Test] Test Pull Request - Custom Engine Safe Output #438, [Custom Engine Test] Test Issue Created by Custom Engine #442 failed; PR runs succeeding
- Root cause:
push_repo_memory → Post Setup Scriptsbug — git branch checkout removesactions/setup - Fix:
git checkout HEADrestore inpush_repo_memoryjob template incompiler_yaml.go
Smoke Create Cross-Repo PR — P1 (Ongoing)
- Issue: [WHM] Smoke Create Cross-Repo PR - Persistent Schedule Failures (Systemic Bug) #23447 (updated)
- Status: 8+ consecutive schedule failures since 2026-03-25; same root cause as above
- Fix: Same as Smoke Update Cross-Repo PR
Warnings ⚠️
Smoke Gemini (#23399 — open)
- Schedule run Fix multi commit PRs #655 failed (2026-03-30T01:01Z); PR runs succeeding
- Error: exit code 41 (engine terminated unexpectedly)
- Pattern: intermittent schedule failures, PR runs healthy → P2, monitor
Workflow Health Manager (self)
- Runs Safe outputs: push-to-branch for updating a pull request #282–[UPDATED] Update Issue Test - Processed by Codex #285 (4 consecutive) failed in
safe_outputsjob - This run ([UPDATED] Update Issue Test - Processed by Codex #286) expected to succeed
- Root cause: safe_outputs infrastructure issue in previous runs
- Note: Run [UPDATED] Update Issue Test - Processed by Codex #285 was yesterday (2026-03-29T11:56Z); current run is operating normally
Systemic Issues
Push_Repo_Memory Bug
- Affected: Smoke Update Cross-Repo PR, Smoke Create Cross-Repo PR
- Pattern: Schedule-triggered workflows that use
push_repo_memoryhave their workspace corrupted when the memory branch is checked out - Fix location:
pkg/workflow/compiler_yaml.go— addgit checkout HEADafter push_repo_memory step - Priority: P1 — blocking 2 smoke tests
Codex API Access Restriction
- Affected: Smoke Codex
- Pattern:
"access to this model has been temporarily restricted"— now affecting ALL run types - Suspected cause: API quota exhaustion or subscription change
- Priority: P1 — blocking all Codex CI validation
Health Trends
- Overall score: 72/100 (↑1 from 71 yesterday, ↑2 from 70 two days ago)
- 7-day success rate (estimated): ~96% across all workflows
- Workflows needing attention: 4 (Smoke Codex, Smoke Update Cross-Repo PR, Smoke Create Cross-Repo PR, Smoke Gemini)
Actions Taken This Run
| Action | Details |
|---|---|
| Closed #23151 | PR Triage Agent — 5 consecutive successes |
| Closed #23446 | Metrics Collector — recovered |
| Closed #23397 | Lockfile Stats — transient failure |
| Updated #23193 | Smoke Update Cross-Repo PR — status update |
| Updated #23447 | Smoke Create Cross-Repo PR — status update |
| Updated #23431 | Smoke Codex — escalated to P1 |
References:
- §23743866195 — This WHM run
- §23731204453 — PR Triage run don't run
add-reactionon PRs of forked repos #305 (success) - §23723528722 — Smoke Update [Custom Engine Test] Test Issue Created by Custom Engine #442 (failure)
Note
🔒 Integrity filter blocked 2 items
The following items were blocked because they don't meet the GitHub integrity level.
- #19099
search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved". - #18407
search_issues: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
To allow these resources, lower min-integrity in your GitHub frontmatter:
tools:
github:
min-integrity: approved # merged | approved | unapproved | noneGenerated by Workflow Health Manager - Meta-Orchestrator · ◷
- expires on Mar 31, 2026, 12:14 PM UTC
Reactions are currently unavailable
Metadata
Metadata
Labels
cookieIssue Monster Loves Cookies!Issue Monster Loves Cookies!
Type
Fields
Give feedbackNo fields configured for issues without a type.