Sync .github directory with azure-sdk-tools for PR 15062#48773
Sync .github directory with azure-sdk-tools for PR 15062#48773
Conversation
Rename 5 shared skills to use the azsdk-common- prefix so the sync-.github.yml pipeline can pattern-match and distribute them to all subscribed Azure SDK language repos. Renames: generate-sdk-locally → azsdk-common-generate-sdk-locally apiview-feedback-resolution → azsdk-common-apiview-feedback-resolution pipeline-troubleshooting → azsdk-common-pipeline-troubleshooting prepare-release-plan → azsdk-common-prepare-release-plan sdk-release → azsdk-common-sdk-release Updated: - SKILL.md name: fields in all 5 skills - All eval.yaml and trigger_tests.yaml skill/name references - .github/skills/README.md links and naming convention docs - sensei SKILL.md example reference - 8 instruction files in eng/common/instructions/azsdk-tools/ with Related Skill cross-references pointing to new skill paths - copilot/sdk-release.instructions.md with skill cross-reference Relates to #15049 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Delete 7 instruction files whose content is now fully covered by skills: - local-sdk-workflow.instructions.md → azsdk-common-generate-sdk-locally - create-release-plan.instructions.md → azsdk-common-prepare-release-plan - sdk-details-in-release-plan.instructions.md → azsdk-common-prepare-release-plan - verify-namespace-approval.instructions.md → azsdk-common-prepare-release-plan - check-package-readiness.instructions.md → azsdk-common-sdk-release - check-package-validation.instructions.md → azsdk-common-generate-sdk-locally - copilot/sdk-release.instructions.md → azsdk-common-sdk-release Skills updated with all unique details from instruction files: - generate-sdk-locally: added commit checkpoints, language selection, config file identification, SDK project path, metadata update tools - prepare-release-plan: added critical LLM instructions (NextSteps field), Release Planner Tool link, package name extraction rules per language, valid/invalid examples table, JSON structure, namespace approval flow - sdk-release: added readiness check details, interaction flow pattern, do-not-ask-for-PR/plan guard, case-sensitive language note typespec-to-sdk orchestrator updated to reference skills directly. Removed all related_skill frontmatter and blockquotes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous commit incorrectly rewrote 3 SKILL.md files from scratch. This restores the original content authored by the tools team, with only the name: field updated to use the azsdk-common- prefix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Syncs the .github/skills directory with the upstream azure-sdk-tools changes (PR 15062) by adding multiple “utility skill” definitions plus their task/eval suites, enabling standardized Copilot/MCP-assisted workflows (release, release planning, pipeline troubleshooting, local generation, APIView feedback).
Changes:
- Add 5 common workflow skills (
sdk-release,prepare-release-plan,pipeline-troubleshooting,generate-sdk-locally,apiview-feedback-resolution) with SKILL.md guidance and references/fixtures. - Add task YAMLs for each skill (happy-path, edge cases, negative/anti-trigger).
- Add evaluation configs (
eval.yaml,evals/*) to validate trigger behavior and basic response constraints.
Reviewed changes
Copilot reviewed 60 out of 60 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| .github/skills/azsdk-common-sdk-release/tasks/release-trigger-001.yaml | Task case for triggering release pipeline |
| .github/skills/azsdk-common-sdk-release/tasks/release-readiness-001.yaml | Task case for readiness-only flow |
| .github/skills/azsdk-common-sdk-release/tasks/release-negative-001.yaml | Negative task case (should not trigger) |
| .github/skills/azsdk-common-sdk-release/tasks/release-basic-001.yaml | Happy-path full release flow task |
| .github/skills/azsdk-common-sdk-release/SKILL.md | Skill definition and usage guidance |
| .github/skills/azsdk-common-sdk-release/evals/trigger_tests.yaml | Trigger/anti-trigger prompt set |
| .github/skills/azsdk-common-sdk-release/evals/tasks/basic-trigger.yaml | Basic trigger eval task |
| .github/skills/azsdk-common-sdk-release/evals/tasks/anti-trigger.yaml | Anti-trigger eval task |
| .github/skills/azsdk-common-sdk-release/evals/eval.yaml | Evals suite config |
| .github/skills/azsdk-common-sdk-release/eval.yaml | Evaluation suite config (non-evals) |
| .github/skills/azsdk-common-prepare-release-plan/tasks/should-not-trigger.yaml | Negative task case to avoid activating on “release package” requests |
| .github/skills/azsdk-common-prepare-release-plan/tasks/link-sdk-prs.yaml | Task case for linking SDK PRs to a release plan |
| .github/skills/azsdk-common-prepare-release-plan/tasks/edge-case.yaml | Edge-case validation task for package naming formats |
| .github/skills/azsdk-common-prepare-release-plan/tasks/basic-usage.yaml | Basic usage task for creating a release plan |
| .github/skills/azsdk-common-prepare-release-plan/SKILL.md | Skill definition and usage guidance |
| .github/skills/azsdk-common-prepare-release-plan/references/release-plan-details.md | Detailed reference steps and constraints |
| .github/skills/azsdk-common-prepare-release-plan/fixtures/tspconfig.yaml | Fixture TypeSpec config used for examples/tests |
| .github/skills/azsdk-common-prepare-release-plan/evals/trigger_tests.yaml | Trigger/anti-trigger prompt set |
| .github/skills/azsdk-common-prepare-release-plan/evals/tasks/basic-trigger.yaml | Basic trigger eval task |
| .github/skills/azsdk-common-prepare-release-plan/evals/tasks/anti-trigger.yaml | Anti-trigger eval task |
| .github/skills/azsdk-common-prepare-release-plan/evals/eval.yaml | Evals suite config |
| .github/skills/azsdk-common-prepare-release-plan/eval.yaml | Evaluation suite config (non-evals) |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/should-not-trigger.yaml | Negative task case (TypeSpec authoring shouldn’t trigger) |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/local-reproduction.yaml | Task case for local reproduction guidance |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/edge-case.yaml | Edge-case task for multi-language failures |
| .github/skills/azsdk-common-pipeline-troubleshooting/tasks/basic-usage.yaml | Basic usage task for pipeline failure analysis |
| .github/skills/azsdk-common-pipeline-troubleshooting/SKILL.md | Skill definition and usage guidance |
| .github/skills/azsdk-common-pipeline-troubleshooting/references/failure-patterns.md | Reference guide for common pipeline failure patterns |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/trigger_tests.yaml | Trigger/anti-trigger prompt set |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/tasks/basic-trigger.yaml | Basic trigger eval task |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/tasks/anti-trigger.yaml | Anti-trigger eval task |
| .github/skills/azsdk-common-pipeline-troubleshooting/evals/eval.yaml | Evals suite config |
| .github/skills/azsdk-common-pipeline-troubleshooting/eval.yaml | Evaluation suite config (non-evals) |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/rename-client.yaml | Task case for TypeSpec customization to rename client/model |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/hide-operation.yaml | Task case for hiding internal operations from SDK surface |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/full-workflow.yaml | Task case for full local generation/build/test workflow |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/edge-case.yaml | Edge-case task for build failure recovery |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/customization-workflow.yaml | Task case for customization drift/duplicate field conflicts |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/breaking-changes.yaml | Task case for resolving breaking changes after spec updates |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/basic-usage.yaml | Basic usage task for local generation (Python example) |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/anti-trigger.yaml | Anti-trigger task (pipeline-based generation) |
| .github/skills/azsdk-common-generate-sdk-locally/tasks/analyzer-errors.yaml | Task case for .NET analyzer-driven renames/customizations |
| .github/skills/azsdk-common-generate-sdk-locally/SKILL.md | Skill definition and usage guidance |
| .github/skills/azsdk-common-generate-sdk-locally/references/sdk-repos.md | Reference mapping of languages to SDK repos |
| .github/skills/azsdk-common-generate-sdk-locally/references/customization-workflow.md | Reference describing 2-phase customization workflow |
| .github/skills/azsdk-common-generate-sdk-locally/fixtures/tspconfig.yaml | Fixture TypeSpec config used for examples/tests |
| .github/skills/azsdk-common-generate-sdk-locally/eval.yaml | Evaluation suite config |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/should-not-trigger.yaml | Negative task case (TypeSpec project creation) |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/no-feedback.yaml | Task case for “no APIView comments found” |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/edge-case.yaml | Edge-case task for feedback requiring TypeSpec changes |
| .github/skills/azsdk-common-apiview-feedback-resolution/tasks/basic-usage.yaml | Basic usage task for retrieving APIView comments |
| .github/skills/azsdk-common-apiview-feedback-resolution/SKILL.md | Skill definition and usage guidance |
| .github/skills/azsdk-common-apiview-feedback-resolution/references/feedback-resolution-steps.md | Reference steps for categorizing/resolving feedback |
| .github/skills/azsdk-common-apiview-feedback-resolution/fixtures/apiview-comment.json | Fixture APIView comment payload |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/trigger_tests.yaml | Trigger/anti-trigger prompt set (incl. URL-only prompts) |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/url-trigger.yaml | Eval task for APIView URL-only trigger |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/basic-trigger.yaml | Basic trigger eval task |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/tasks/anti-trigger.yaml | Anti-trigger eval task |
| .github/skills/azsdk-common-apiview-feedback-resolution/evals/eval.yaml | Evals suite config |
| .github/skills/azsdk-common-apiview-feedback-resolution/eval.yaml | Evaluation suite config (non-evals) |
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: |
There was a problem hiding this comment.
This eval config isn’t valid YAML: the list item under graders: isn’t indented, and the regex_not_match patterns are also aligned as if they were a sibling key. YAML parsers will treat this as a top-level sequence / malformed mapping. Indent the grader item(s) under graders: and indent the regex_not_match list under that key.
| graders: | ||
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: |
There was a problem hiding this comment.
This eval config isn’t valid YAML: the list item under graders: isn’t indented, and the regex_not_match patterns are also aligned as if they were a sibling key. Indent the grader item(s) under graders: and indent the regex_not_match list under that key so the file can be parsed.
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
This eval config isn’t valid YAML: the grader list item is not indented under graders:, and the regex_not_match entries are not nested as a YAML list under that key. This will fail parsing in most YAML loaders. Please fix the indentation for the graders section.
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' | |
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' |
| - type: text | ||
| name: no_fatal_errors | ||
| config: | ||
| regex_not_match: | ||
| - (?i)fatal error | ||
| - (?i)unhandled exception | ||
| - '(?i)panic:' |
There was a problem hiding this comment.
This eval config isn’t valid YAML: the grader list item starts at column 0 instead of being indented under graders:, and the regex_not_match values are not nested as a list under that key. Please indent the graders section so the file parses correctly.
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' | |
| - type: text | |
| name: no_fatal_errors | |
| config: | |
| regex_not_match: | |
| - (?i)fatal error | |
| - (?i)unhandled exception | |
| - '(?i)panic:' |
| config: | ||
| regex_not_match: | ||
| - (?i)error occurred | ||
| - (?i)failed |
There was a problem hiding this comment.
The no_fatal_errors grader forbids the word failed (case-insensitive). That’s likely to cause false failures for this skill, since normal troubleshooting guidance often includes phrases like “build failed” or “tests failed”. Consider aligning this grader with the others (e.g., only block truly fatal/unhandled error patterns) or narrowing the regexes to avoid rejecting legitimate responses.
| - (?i)failed | |
| - (?i)fatal error | |
| - (?i)unhandled exception |
| - Check if tests need playback recording updates | ||
| - Verify test fixtures are present and valid |
There was a problem hiding this comment.
This guidance suggests updating/re-recording test playbacks as a generic remediation. For Azure SDK repos this is usually not an acceptable first-line fix (it can mask real regressions and introduces churn). Consider replacing this with diagnostics-first guidance (identify why playback diverged, validate sanitizers/matchers, update test logic) and only mention recording refresh as a last resort when service behavior legitimately changed.
| | ----------------------------- | ------------------------------ | ---------------------------------- | | ||
| | Missing types/models | TypeSpec compilation issue | Fix TypeSpec, regenerate | | ||
| | Breaking change detected | API surface changed | Add `@clientName` or revert change | | ||
| | Test playback failure | Recorded responses outdated | Re-record test sessions | |
There was a problem hiding this comment.
The failure-pattern table recommends “Re-record test sessions” as the fix for playback failures. This is generally not a good default recommendation (it can hide regressions and creates unnecessary churn). Prefer guidance to diagnose the mismatch first and reserve recording refresh for cases where the live service behavior legitimately changed.
Sync .github directory with azure-sdk-tools for PR Azure/azure-sdk-tools#15062 See eng/common workflow