Skip to content

Update eval-driven-dev skill#1092

Open
yiouli wants to merge 6 commits intogithub:stagedfrom
yiouli:staged
Open

Update eval-driven-dev skill#1092
yiouli wants to merge 6 commits intogithub:stagedfrom
yiouli:staged

Conversation

@yiouli
Copy link
Contributor

@yiouli yiouli commented Mar 19, 2026

Pull Request Checklist

  • I have read and followed the CONTRIBUTING.md guidelines.
  • I have read and followed the Guidance for submissions involving paid services.
  • My contribution adds a new instruction, prompt, agent, skill, or workflow file in the correct directory.
  • The file follows the required naming convention.
  • The content is clearly structured and follows the example format.
  • I have tested my instructions, prompt, agent, skill, or workflow with GitHub Copilot.
  • I have run npm start and verified that README.md is up to date.

Description

Update the SKILL description to carry out instrumentation and dataset generation better, also add a step to check for latest version of SKILL on Github when SKILL is invoked.


Type of Contribution

  • New instruction file.
  • New prompt file.
  • New agent file.
  • New plugin.
  • New skill file.
  • New agentic workflow.
  • Update to existing instruction, prompt, agent, plugin, skill, or workflow.
  • Other (please specify):

Additional Notes


By submitting this pull request, I confirm that my contribution abides by the Code of Conduct and will be licensed under the MIT License.

@yiouli yiouli requested a review from aaronpowell as a code owner March 19, 2026 22:15
Copilot AI review requested due to automatic review settings March 19, 2026 22:15
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the eval-driven-dev skill to improve eval-driven instrumentation guidance and adds a self-check mechanism intended to detect when the skill content is outdated.

Changes:

  • Added a resources/ directory containing version.json and a check_version.py helper script.
  • Extended skills/eval-driven-dev/SKILL.md with a “Version check” step and substantial guidance updates (hard gates, eval-boundary selection, run harness stage).
  • Updated docs/README.skills.md to list the new resources folder for the skill.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 6 comments.

File Description
skills/eval-driven-dev/resources/version.json Adds local version metadata intended for remote version comparison.
skills/eval-driven-dev/resources/check_version.py Adds a script intended to compare local vs remote skill version metadata.
skills/eval-driven-dev/SKILL.md Adds “version check first” guidance and expands the workflow (notably run harness + hard gates).
docs/README.skills.md Updates the skill’s references column to include resources.

yiouli and others added 5 commits March 19, 2026 16:01
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Contributor

@aaronpowell aaronpowell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the metadata field of the skill frontmatter we can store a version, so a separate "version file" wouldn't be needed.

But more broadly speaking, I'm a little apprehensive on this because it's trying to solve a tooling problem (tool doesn't see updates) with a custom solution.

I'm going to bring this up with the Copilot team on the topic of versioning, as I know it's something that has been discussed but I'm unsure what the status of it is.

@yiouli
Copy link
Contributor Author

yiouli commented Mar 20, 2026

In the metadata field of the skill frontmatter we can store a version, so a separate "version file" wouldn't be needed.

But more broadly speaking, I'm a little apprehensive on this because it's trying to solve a tooling problem (tool doesn't see updates) with a custom solution.

I'm going to bring this up with the Copilot team on the topic of versioning, as I know it's something that has been discussed but I'm unsure what the status of it is.

Yea I don't like the custom solution as well, it'd be great if copilot can solve this for all skill distribution so I don't have to do this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants