[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-05-23

[github-actions[bot]]

Summary

NLP clustering analysis of 1,000 copilot agent PRs created between 2026-05-05 and 2026-05-23. TF-IDF features (unigrams + bigrams) were clustered with K-means; k=9 was selected by silhouette score on a k-sweep from 3–9.

• Total tasks analyzed: 1,000
• Clusters identified: 9
• Overall merge rate: 80.7%
• Avg files changed / PR: 31.2
• Avg LOC delta / PR: +380 / −249

Key Findings

1. Largest theme is workflow/MCP plumbing. Cluster 3 (Workflow / MCP / shared agent plumbing (catch-all)) absorbs 435 PRs (44%) — it's the catch-all bucket for shared workflow/MCP/agent changes and dominates daily volume.
2. SPDD compliance work is the highest-yield surface. Cluster 6 (SPDD spec safeguards & compliance) merges at 95% (n=20) — small, focused spec/safeguards work with consistently low rejection.
3. AWF/firewall/gateway version bumps are the riskiest. Cluster 7 (AWF / firewall / gateway version bumps) merges at only 63% — these touch infra defaults where rollback is common (firewall versions, gateway images, golden fixtures).
4. AWF/version-bump PRs are also the heaviest by code volume. Cluster 7 averages 105 files / PR with +1067/−952 LOC — most of that is regenerated golden fixtures.
5. "WIP: Failing GitHub Actions" is its own recoverable bucket. Cluster 1 (n=27) is a clean, recognizable signature of the auto-fix workflow targeting broken CI jobs. Merge rate 74% means roughly 1-in-4 attempts gets abandoned/closed without landing — a useful supervised signal for improving that prompt.

Cluster Overview

C	Theme	PRs	Merge%	Avg Files	LOC Δ	Commits	Top Terms
3	Workflow / MCP / shared agent plumbing (catch-all)	435	79%	30.5	+425/-297	4.4	workflow, fix, shared, workflows, add
2	Safe-outputs, branch & issue handling	156	89%	18.2	+252/-108	5.0	safe, fix, output, safe output, branch
0	Docs & CLI surface	118	80%	13.7	+189/-103	3.6	docs, cli, package, pkg, help
5	Experiments, prompts & token telemetry	93	87%	20.1	+269/-147	3.9	experiment, prompt, token, workflow, agent
7	AWF / firewall / gateway version bumps	65	63%	104.5	+1067/-952	3.8	awf, bump, gateway, version, golden
4	Pull-request / reviewer flows	48	83%	21.5	+317/-75	6.1	request, pull request, pull, reviewer, review
8	Model providers (Copilot/Codex/BYOK)	38	82%	50.7	+203/-96	3.4	model, copilot, multiplier, alias, inventory
1	WIP: Failing GitHub Actions auto-fix	27	74%	49.7	+255/-58	3.1	failing github, fix failing, actions job, actions, github actions
6	SPDD spec safeguards & compliance	20	95%	28.1	+496/-312	4.0	spdd, spec, safeguards, hash, sync

Merge Rate by Cluster

Cluster Complexity (Avg Files Changed)

Cluster Details

Cluster 3: Workflow / MCP / shared agent plumbing (catch-all) — 435 PRs, 79% merged

• Top terms: workflow, fix, shared, workflows, add, mcp, files, agent, run, validation

• Avg files changed: 30.5

• Avg LOC: +425 / −297

• Avg commits per PR: 4.4

• Avg reviews / comments: 1.5 reviews, 2.6 comments

• Sample PRs:

• #34124 — fix: skip unlock job when activation was skipped ✅ merged

• #34123 — Consolidate workflow FieldLocation onto console ErrorPosition ✅ merged

• #34120 — Increase audit workflow repo-memory patch budget to prevent push_repo_memory failures ✅ merged

Cluster 2: Safe-outputs, branch & issue handling — 156 PRs, 89% merged

• Top terms: safe, fix, output, safe output, branch, issue, test, outputs, field, safe outputs

• Avg files changed: 18.2

• Avg LOC: +252 / −108

• Avg commits per PR: 5.0

• Avg reviews / comments: 1.8 reviews, 3.7 comments

• Sample PRs:

• #34144 — Refactor workflow helper hotspots from semantic clustering audit ✅ merged

• #34117 — Guard OTLP attribute merge against allocation-size overflow ✅ merged

• #34066 — safe-outputs: resolve base branch from origin/HEAD and harden full patch base selection ✅ merged

Cluster 0: Docs & CLI surface — 118 PRs, 80% merged

• Top terms: docs, cli, package, pkg, help, add, sites, logger, updated, rename

• Avg files changed: 13.7

• Avg LOC: +189 / −103

• Avg commits per PR: 3.6

• Avg reviews / comments: 1.5 reviews, 1.5 comments

• Sample PRs:

• #34122 — Validate logger migration completeness across targeted packages

• #34121 — docs(reference): add non-Copilot engine examples to targeted reference pages

• #34018 — Clarify status/list pattern semantics and add CLI help example validation ✅ merged

Cluster 5: Experiments, prompts & token telemetry — 93 PRs, 87% merged

• Top terms: experiment, prompt, token, workflow, agent, inline, report, usage, variant, sub

• Avg files changed: 20.1

• Avg LOC: +269 / −147

• Avg commits per PR: 3.9

• Avg reviews / comments: 1.6 reviews, 1.6 comments

• Sample PRs:

• #34063 — Optimize ab-testing-advisor prompt with inline sub-agents ✅ merged

• #34037 — fix: hypothesis always "(not specified)" in daily experiment report ✅ merged

• #34036 — Surface OTel token usage from agent-stdio.log when proxy usage logs are missing ✅ merged

Cluster 7: AWF / firewall / gateway version bumps — 65 PRs, 63% merged

• Top terms: awf, bump, gateway, version, golden, firewall, default, awf config, config, mcp

• Avg files changed: 104.5

• Avg LOC: +1067 / −952

• Avg commits per PR: 3.8

• Avg reviews / comments: 1.5 reviews, 4.9 comments

• Sample PRs:

• #34129 — Fix Codex smoke workflow by preserving OPENAI_API_KEY in AWF container env ✅ merged

• #34114 — Bump gh-aw-firewall to v0.25.52 and sync embedded AWF schema ✅ merged

• #34081 — Bump default MCP Gateway image to gh-aw-mcpg v0.3.18 ✅ merged

Cluster 4: Pull-request / reviewer flows — 48 PRs, 83% merged

• Top terms: request, pull request, pull, reviewer, review, workflows, command, request reviewer, create pull, reaction

• Avg files changed: 21.5

• Avg LOC: +317 / −75

• Avg commits per PR: 6.1

• Avg reviews / comments: 1.8 reviews, 5.3 comments

• Sample PRs:

• #34139 — fix: exclude merged upstream commits from diffSize in push_to_pull_request_branch incremental mode ✅ merged

• #34138 — fix: reject create_pull_request/push_to_pull_request_branch when branch equals base_branch after detection ✅ merged

• #34133 — Create REQUEST_CHANGES review for create_pull_request threat-warning mode ✅ merged

Cluster 8: Model providers (Copilot/Codex/BYOK) — 38 PRs, 82% merged

• Top terms: model, copilot, multiplier, alias, inventory, provider, model inventory, engine, byok, codex

• Avg files changed: 50.7

• Avg LOC: +203 / −96

• Avg commits per PR: 3.4

• Avg reviews / comments: 1.4 reviews, 1.7 comments

• Sample PRs:

• #34149 — Use Copilot BYOK platform default model instead of hard-coded Claude fallback ✅ merged

• #34148 — Pin Matt Pocock reviewer to supported Copilot model ✅ merged

• #34136 — Add @app/copilot-swe-agent as a copilot bot alias ✅ merged

Cluster 1: WIP: Failing GitHub Actions auto-fix — 27 PRs, 74% merged

• Top terms: failing github, fix failing, actions job, actions, github actions, job, wip fix, failing, wip, github

• Avg files changed: 49.7

• Avg LOC: +255 / −58

• Avg commits per PR: 3.1

• Avg reviews / comments: 0.9 reviews, 0.3 comments

• Sample PRs:

• #34119 — [WIP] Fix failing GitHub Actions job agent

• #33673 — [WIP] Fix failing GitHub Actions job js-typecheck

• #33672 — [WIP] Fix failing GitHub Actions job Lint Gate

Cluster 6: SPDD spec safeguards & compliance — 20 PRs, 95% merged

• Top terms: spdd, spec, safeguards, hash, sync, compliance, frontmatter hash, daily, added, norms

• Avg files changed: 28.1

• Avg LOC: +496 / −312

• Avg commits per PR: 4.0

• Avg reviews / comments: 1.4 reviews, 0.8 comments

• Sample PRs:

• #33591 — Close daily SPDD spec gaps across MCP Scripts, Effective Tokens, Forecast, Frontmatter Hash, and Fuzzy Schedule ✅ merged

• #33365 — SPDD: close safeguards/consistency gaps across 5 specs and add targeted compliance vectors ✅ merged

• #33096 — docs(spdd): strengthen safeguards, OTel norms, and compliance tests across 5 spec files ✅ merged

Recent PRs by Cluster (latest 50)

PR	Title	Cluster	Theme	State	Files	Δ
#34149	Use Copilot BYOK platform default model instead of hard-coded Claude fallba	8	Model providers (Copilot/Codex	✅ merged	155	+441/-448
#34148	Pin Matt Pocock reviewer to supported Copilot model	8	Model providers (Copilot/Codex	✅ merged	2	+18/-17
#34144	Refactor workflow helper hotspots from semantic clustering audit	2	Safe-outputs, branch & issue h	✅ merged	14	+436/-349
#34139	fix: exclude merged upstream commits from diffSize in push_to_pull_request_	4	Pull-request / reviewer flows	✅ merged	2	+464/-3
#34138	fix: reject create_pull_request/push_to_pull_request_branch when branch equ	4	Pull-request / reviewer flows	✅ merged	4	+120/-17
#34136	Add @app/copilot-swe-agent as a copilot bot alias	8	Model providers (Copilot/Codex	✅ merged	4	+69/-32
#34133	Create REQUEST_CHANGES review for create_pull_request threat-warning mode	4	Pull-request / reviewer flows	✅ merged	6	+91/-9
#34129	Fix Codex smoke workflow by preserving OPENAI_API_KEY in AWF container en	7	AWF / firewall / gateway versi	✅ merged	17	+96/-96
#34124	fix: skip unlock job when activation was skipped	3	Workflow / MCP / shared agent	✅ merged	4	+22/-15
#34123	Consolidate workflow FieldLocation onto console ErrorPosition	3	Workflow / MCP / shared agent	✅ merged	3	+16/-9
#34122	Validate logger migration completeness across targeted packages	0	Docs & CLI surface	❌ closed	0	+0/-0
#34121	docs(reference): add non-Copilot engine examples to targeted reference page	0	Docs & CLI surface	❌ closed	9	+112/-2
#34120	Increase audit workflow repo-memory patch budget to prevent push_repo_memor	3	Workflow / MCP / shared agent	✅ merged	2	+17/-16
#34119	[WIP] Fix failing GitHub Actions job agent	1	WIP: Failing GitHub Actions au	❌ closed	4	+13/-3
#34117	Guard OTLP attribute merge against allocation-size overflow	2	Safe-outputs, branch & issue h	✅ merged	21	+78/-24
#34114	Bump gh-aw-firewall to v0.25.52 and sync embedded AWF schema	7	AWF / firewall / gateway versi	✅ merged	246	+2786/-2755
#34090	fix: set GH_AW_WORKFLOW_SOURCE_URL for local workflows in failure issues	3	Workflow / MCP / shared agent	✅ merged	236	+1551/-71
#34088	Render sandbox.firewall models.json in AWF step summaries	8	Model providers (Copilot/Codex	✅ merged	2	+345/-2
#34081	Bump default MCP Gateway image to gh-aw-mcpg v0.3.18	7	AWF / firewall / gateway versi	✅ merged	245	+1114/-1109
#34079	Remove deprecated model state; retain full multiplier history	8	Model providers (Copilot/Codex	✅ merged	6	+16/-94
#34078	Pin Matt Pocock Skills Reviewer to an explicit Copilot model	8	Model providers (Copilot/Codex	❌ closed	4	+28/-17
#34068	Remove shared/apm.md; point to microsoft/apm canonical source	3	Workflow / MCP / shared agent	✅ merged	6	+34/-377
#34066	safe-outputs: resolve base branch from origin/HEAD and harden full patch ba	2	Safe-outputs, branch & issue h	✅ merged	6	+163/-28
#34064	fix(check_membership): skip roles check for allowlisted bots to eliminate s	2	Safe-outputs, branch & issue h	✅ merged	2	+140/-52
#34063	Optimize ab-testing-advisor prompt with inline sub-agents	5	Experiments, prompts & token t	✅ merged	4	+108/-51
#34062	feat: add .sentrux/rules.toml with architectural quality gates	3	Workflow / MCP / shared agent	✅ merged	4	+45/-21
#34061	fix: infer MCP tool-call status from level/error when status field is absen	3	Workflow / MCP / shared agent	✅ merged	3	+106/-3
#34060	feat(failure-handler): add cascade detection when ≥10 [aw] failures fire wi	3	Workflow / MCP / shared agent	✅ merged	5	+578/-21
#34038	Add SEC-004 exemption for generate_safe_outputs_tools.cjs false positive	2	Safe-outputs, branch & issue h	✅ merged	3	+87/-37
#34037	fix: hypothesis always "(not specified)" in daily experiment report	5	Experiments, prompts & token t	✅ merged	5	+269/-35
#34036	Surface OTel token usage from agent-stdio.log when proxy usage logs are mis	5	Experiments, prompts & token t	✅ merged	2	+120/-3
#34035	Improve daily-experiment-report readability with progressive disclosure, qu	5	Experiments, prompts & token t	✅ merged	3	+142/-40
#34033	Add tone-style A/B experiment to Typist workflow	5	Experiments, prompts & token t	✅ merged	4	+275/-51
#34031	Switch Developer Documentation Consolidator cadence from daily to weekly	3	Workflow / MCP / shared agent	✅ merged	3	+89/-40
#34027	Add A/B experiment wiring for smoke-pi sub-agent decomposition	5	Experiments, prompts & token t	✅ merged	2	+167/-11
#34026	feat: expand copilot as a bot alias to all GitHub Copilot bot identities	8	Model providers (Copilot/Codex	✅ merged	6	+163/-5
#34025	Add success check-run publishing to reviewer workflows	4	Pull-request / reviewer flows	✅ merged	13	+70/-30
#34020	[ab-advisor] Add sub_agent_strategy A/B experiment to smoke-temporary-id wo	5	Experiments, prompts & token t	✅ merged	4	+227/-94
#34019	fix: update deprecated CopilotBYOKDefaultModel to claude-sonnet-4-5-2025092	3	Workflow / MCP / shared agent	✅ merged	155	+460/-439
#34018	Clarify status/list pattern semantics and add CLI help example validati	0	Docs & CLI surface	✅ merged	7	+239/-3
#34011	Format {files} placeholders as inline code in JS template rendering	2	Safe-outputs, branch & issue h	✅ merged	7	+103/-6
#34008	Make add/add-wizard and update manifest-aware for aw.yml package in	3	Workflow / MCP / shared agent	✅ merged	12	+742/-6
#34007	Support object-form runs-on in custom jobs schema	3	Workflow / MCP / shared agent	✅ merged	7	+372/-52
#34005	Update blog workflow links to githubnext/agentics to fix broken `add-wiza	3	Workflow / MCP / shared agent	✅ merged	18	+106/-101
#34002	Add maintenance compile PR mode and configurable GitHub token secret	3	Workflow / MCP / shared agent	✅ merged	16	+910/-31
#34001	[WIP] Recompile workflows to sync lock files	3	Workflow / MCP / shared agent	❌ closed	0	+0/-0
#33987	[WIP] Add request_review protected-files mode for create_pull_request	4	Pull-request / reviewer flows	✅ merged	6	+71/-12
#33954	Add request_review protected-files mode for create_pull_request	4	Pull-request / reviewer flows	✅ merged	6	+188/-53
#33947	Refactor dispatch/call workflow duplication with shared input, tool, and re	3	Workflow / MCP / shared agent	✅ merged	10	+364/-254
#33946	Migrate remaining pkg/* logging callsites off log.* linter pattern to `pk	0	Docs & CLI surface	✅ merged	9	+60/-59

Recommendations

1. Carve cluster 3 (workflow plumbing) into sub-themes. At 435 PRs / 44% of volume, the residual cluster is too broad to act on. A second-pass run on just this cluster (with min_df lowered and max_df raised) would surface internal structure — likely "logger migration", "validation refactor", "shared workflow helpers", and "MCP plumbing" as distinct sub-buckets.
2. Treat AWF/firewall bumps (cluster 7) as a quality-control surface. 63% merge rate with the heaviest LOC footprint suggests these PRs over-regenerate fixtures relative to the actual config delta. Worth investigating whether the bump prompt should split "version constant change" from "regenerate goldens" into separate PRs.
3. Mine cluster 6 (SPDD/spec) for prompt patterns to replicate. 95% merge rate at 20 PRs is the cleanest signal in the dataset — small scope, consistent terminology ("safeguards", "hash", "compliance"). Whatever pattern that workflow uses to scope itself is worth lifting into riskier clusters.
4. Improve the WIP-failing-actions prompt (cluster 1). 74% merge rate vs. 81% overall, and these are explicitly auto-fix attempts. A targeted review of the 7 closed-not-merged PRs in this cluster would highlight what kinds of CI failures the current prompt can't handle.
5. Cluster 8 (model/provider config) deserves a guardrail. Repeated PRs like "pin model X to value Y" suggest model-pinning churn — could likely be replaced with a single config knob rather than per-workflow edits.

Methodology

• Source: 1,000 PRs from /tmp/gh-aw/prompt-cache/pr-full-data/ (full PR JSON with body, reviews, commits, files).
• Prompt text: title (3× weight) + cleaned body. Cleaning removed firewall-block warnings, HTML comments, fenced code, inline code, URLs, and markdown formatting.
• Features: TF-IDF, max 2,000 features, 1–2 grams, min_df=5, max_df=0.6, alpha-only tokens.
• Clustering: K-means (n_init=20, random_state=42); k selected by silhouette over k∈[3, 9].
• Notes: Silhouette scores are low (~0.027) — clusters reflect lexical themes, not strictly separated topics. Cluster 3 is the residual / catch-all and absorbs PRs that don't match a strong specific theme.

References:

• Workflow run §26330085695

Generated by 📊 Copilot Agent Prompt Clustering Analysis · ● 10.6M · ◷

• expires on May 24, 2026, 10:37 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #34410.