Add One-Shot Skills Reliability and Guardrails by bpulluta · Pull Request #397 · NatLabRockies/COMPASS

bpulluta · 2026-03-17T19:15:40Z

Summary

This PR hardens the one-shot extraction skills to make setup and iteration more repeatable, simpler to follow, and safer against runtime misconfiguration.

What Changed

Updated extraction-run/SKILL.md
- Added preflight checks before running.
- Added explicit pass/fail gates:
  - treat runs with zero extracted jurisdictions as failed extraction quality.
  - treat config exceptions in logs as failures even if process exits successfully.
Updated yaml-setup/SKILL.md
- Added non-negotiable runtime constraints.
- Added canonical required heuristic_keywords structure with all required lists.
- Added minimal run-config contract that prompts users to provide their own model name and client settings.
Updated web-scraper/SKILL.md
- Added required heuristic list expectations and explicit failure behavior when missing/empty.
- Normalized keyword naming guidance to canonical forms.
Updated schema-creation/SKILL.md
- Added quality gate requiring smoke runs to produce extracted rows, not just process completion.

Why

users can avoid common setup mistakes and get a working, repeatable baseline faster.

Testing Notes

Documentation/skill-only updates.
Guidance now reflects validated runtime constraints and triage behavior.

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

bpulluta · 2026-03-17T22:16:54Z

@copilot open a new pull request to apply changes based on the comments in this thread

Copilot · 2026-03-17T22:17:02Z

@bpulluta I've opened a new pull request, #399, to work on those changes. Once the pull request is ready, I'll request review from you.

…rmatting (#399) * Initial plan * Fix all review comments in skills documentation Co-authored-by: bpulluta <115118857+bpulluta@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: bpulluta <115118857+bpulluta@users.noreply.github.com>

ppinchuk

Thanks @bpulluta! I think this will be super useful for folks who are new to COMPASS and would like to set up a new tech with the help of AI.

My main question is what you envision the extraction-run skill being used for, and how it's different from the schema-creation skill?

.github/skills/extraction-run/SKILL.md

ppinchuk · 2026-03-18T18:01:12Z

.github/skills/extraction-run/SKILL.md

+- Schema exists and plugin config points to it.
+- You are onboarding a new technology (diesel generator, geothermal, CHP, hydrogen).


Aren't these the exact opposite? If a schema exists, you're no longer onboarding a new technology, right?

.github/skills/extraction-run/SKILL.md

.github/skills/web-scraper/SKILL.md

.github/skills/yaml-setup/SKILL.md

ppinchuk · 2026-03-18T19:19:20Z

.github/skills/plugin-config-setup/SKILL.md

+| Field | Type | Behavior |
+|---|---|---|
+| `schema` | string (path) | **Required.** Path to JSON schema file, relative to plugin YAML location. |
+| `data_type_short_desc` | string | Short description used in LLM prompts (e.g. `utility-scale <tech> ordinance`). |
+| `query_templates` | list | Search query templates; `{jurisdiction}` is replaced at runtime. |
+| `website_keywords` | dict | Keyword → score map for URL ranking during website crawl. |
+| `heuristic_keywords` | dict or `true` | Pre-LLM text filter. If `true`, LLM generates lists from schema. |
+| `collection_prompts` | list or `true` | Text collection prompt(s). If **`true`**, LLM auto-generates from schema. |
+| `text_extraction_prompts` | list or `true` | Text consolidation prompt(s). If **`true`**, LLM auto-generates from schema. |
+| `extraction_system_prompt` | string | Overrides default LLM system prompt for the extraction step. Use this to scope extraction tightly to the target technology. |
+| `cache_llm_generated_content` | bool | Cache LLM-generated `query_templates`, `website_keywords`, and `heuristic_keywords`. Set to `false` when iterating schema to see live changes. |


There is already a new keyword for this function. Would it be worthwhile to instead point to a function that the model should read to learn about the input parameters? Otherwise, this will become out of date, as it is right now

.github/skills/plugin-config-setup/SKILL.md

.github/skills/yaml-setup/SKILL.md

bpulluta added 2 commits March 17, 2026 12:11

Add COMPASS workflow skills

d9e868d

Added one-shot skills

54b8d29

bpulluta requested review from castelao and ppinchuk as code owners March 17, 2026 19:15

Copilot AI review requested due to automatic review settings March 17, 2026 19:15

Copilot started reviewing on behalf of bpulluta March 17, 2026 19:16 View session

This comment was marked as outdated.

Sign in to view

update one-shot SKILL.md structure and trigger contracts

a71447f

bpulluta requested a review from Copilot March 17, 2026 20:01

Copilot started reviewing on behalf of bpulluta March 17, 2026 20:01 View session

This comment was marked as resolved.

Sign in to view

This comment was marked as duplicate.

Sign in to view

Copilot AI mentioned this pull request Mar 17, 2026

[WIP] Add one-shot skills reliability and guardrails #398

Merged

This comment was marked as duplicate.

Sign in to view

Initial plan (#398)

74495a6

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

bpulluta requested a review from Copilot March 17, 2026 22:05

Copilot started reviewing on behalf of bpulluta March 17, 2026 22:07 View session

This comment was marked as resolved.

Sign in to view

Copilot AI mentioned this pull request Mar 17, 2026

Fix skills documentation: correct paths, caching behavior, and tab formatting #399

Merged

ppinchuk requested changes Mar 18, 2026

View reviewed changes

renamed skills and fixed minor comments

1b8571f

		- Schema exists and plugin config points to it.
		- You are onboarding a new technology (diesel generator, geothermal, CHP, hydrogen).

Conversation

bpulluta commented Mar 17, 2026

Summary

What Changed

Why

Testing Notes

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as duplicate.

This comment was marked as duplicate.

This comment was marked as resolved.

Uh oh!

bpulluta commented Mar 17, 2026

Uh oh!

Copilot AI commented Mar 17, 2026

Uh oh!

ppinchuk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ppinchuk Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ppinchuk Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants