feat: add Onboarding Docs Validator pipeline#1122
Draft
decko wants to merge 2 commits into
Draft
Conversation
Two-step Alcove workflow that validates pulp-docs onboarding guides weekly (Monday 6 AM UTC): - Doc Runner: parses Getting Started docs into executable steps, runs them in a dev container with command allowlist and safety checks, dispatches UXD/Docs Writer/Adversarial QE specialists per doc in parallel - Validation Reporter: deduplicates against existing Jira tickets using deterministic step-ID labels, creates structured bug tickets, produces weekly digest Target docs: cli-guide.md, api-access.md, install-bindings.md Artifacts: - .alcove/workflows/onboarding-validator.yml - .alcove/agents/doc-runner.yml - .alcove/agents/validation-reporter.yml - .alcove/security-profiles/docs-validator.yml - .alcove/security-profiles/docs-reporter.yml Assisted-by: Claude Code
Critical: - Add dev_container (ghcr.io/pulp/hosted-pulp-dev-env:main) to Doc Runner — without it, dev container shim is unreachable and all execution fails - Add credential detection for packages.redhat.com URLs and inline -u credentials with pipe characters High: - Remove invalid network: section from docs-validator profile — egress restriction must be enforced at runtime level (NetworkPolicy/Podman) - Add role-assertion preamble to Doc Runner prompt to override any conflicting CLAUDE.md injected from pulp-docs - Remove skill name references from specialist dispatch — skills are not available in Skiff --bare mode; inline prompts are sufficient - Replace shell heredoc output with python3 json.dump for reliable JSON generation (avoids quote/escape breakage) - Add pip uninstall and chmod to command allowlist (safe in ephemeral containers, avoids weekly false-failure tickets) - Change workflow depends to validate.Completed so Reporter runs even when Doc Runner crashes (guard clause handles empty results) - Add dry_run workflow input for testability - Add max_retries: 1 to both workflow steps for transient failures Medium: - Add enforcement_mode: monitor to both agents (matches existing pattern) - Add explicit repos: [] to Validation Reporter - Specify specialist output schema: always return findings array, severity is HIGHEST among findings, empty = severity: none - Add Jira MCP tool names to Reporter prompt - Serialize specialist dispatch per doc (3 concurrent, not 9) - Add brace-expansion skip rule to parsing contract Assisted-by: Claude Code
Contributor
Reviewer's GuideAdds a scheduled Alcove workflow that runs a Doc Runner agent to parse and safely execute onboarding docs in a dev container, then passes structured results to a Validation Reporter agent that deduplicates and manages Jira tickets using deterministic labels and produces a weekly digest, all under new least-privilege security profiles. Sequence diagram for the scheduled onboarding docs validation runsequenceDiagram
actor Cron
participant Workflow as Onboarding_Validator_Workflow
participant DocRunner as Agent_Doc_Runner
participant GitLab as GitLab_pulp-docs
participant Dev as Dev_Container_exec_API
participant UXD as UXD_Specialist
participant Docs as Docs_Writer_Specialist
participant QE as Adversarial_QE_Specialist
participant Reporter as Agent_Validation_Reporter
participant Jira as Jira_PULP
Cron->>Workflow: Trigger_cron_0_6_*_*_1
Workflow->>DocRunner: Start_step_validate(docs_list,dry_run)
DocRunner->>GitLab: Clone_repo(hosted-pulp/pulp-docs)
GitLab-->>DocRunner: Repo_contents
loop For_each_doc
DocRunner->>DocRunner: Parse_code_blocks_apply_safety_and_credentials_rules
loop For_each_safe_step
DocRunner->>Dev: POST_/exec(command,timeout_120s)
Dev-->>DocRunner: Command_result(stdout_stderr_status)
end
par Specialist_reviews_in_parallel
DocRunner->>UXD: Request_UXD_review(doc_content)
UXD-->>DocRunner: UXD_JSON_findings
DocRunner->>Docs: Request_Docs_Writer_review(doc_content)
Docs-->>DocRunner: Docs_Writer_JSON_findings
DocRunner->>QE: Request_Adversarial_QE_review(doc_content)
QE-->>DocRunner: QE_JSON_findings
end
end
DocRunner->>DocRunner: Aggregate_steps_and_reviews
DocRunner-->>Workflow: outputs(results,overall)
Workflow->>Reporter: Start_step_report(results,overall,dry_run)
alt results_empty_or_doc_with_zero_steps
Reporter->>Jira: jira_create_issue(alert_ticket)
Jira-->>Reporter: Alert_ticket_key
Reporter-->>Workflow: outputs(digest_ticket,tickets_created,tickets_commented,stale_tickets_flagged,summary)
else normal_processing
loop For_each_failed_step_or_critical_high_finding
Reporter->>Jira: jira_search(step_label)
Jira-->>Reporter: Matching_issues
alt ticket_exists
Reporter->>Jira: jira_add_comment(update_with_latest_results)
else no_ticket
Reporter->>Jira: jira_create_issue(step_bug_or_doc_epic)
end
end
loop For_each_doc_all_steps_pass
Reporter->>Jira: jira_search(doc_label_open_tickets)
Jira-->>Reporter: Open_doc_tickets
Reporter->>Jira: jira_add_comment(step_now_passes)
Reporter->>Jira: jira_set_labels(add_validation-passing)
end
Reporter->>Jira: jira_search(digest_ticket_open)
Jira-->>Reporter: Existing_digest_or_empty
alt digest_exists
Reporter->>Jira: jira_add_comment(weekly_summary)
else create_digest
Reporter->>Jira: jira_create_issue(weekly_digest_ticket)
end
Jira-->>Reporter: Digest_ticket_key
Reporter-->>Workflow: outputs(digest_ticket,tickets_created,tickets_commented,stale_tickets_flagged,summary)
end
Flow diagram for Doc Runner parsing, execution, and specialist reviewsflowchart TD
A_start["Start Doc_Runner"] --> B_clone
B_clone["Clone hosted-pulp/pulp-docs"] --> C_docs_loop
C_docs_loop{More_docs?} -->|yes| D_load_doc
C_docs_loop -->|no| Z_aggregate
D_load_doc["Load_next_doc"] --> E_parse_blocks
E_parse_blocks["Parse_fenced_code_blocks_and_assign_step_ids"] --> F_steps_loop
F_steps_loop{More_steps?} -->|yes| G_classify_step
F_steps_loop -->|no| P_specialists
G_classify_step["Classify_block_language_and_content"] --> H_placeholder_check
H_placeholder_check{Unresolved_placeholders_or_brace_expansion?} -->|yes| H1_mark_skip_placeholder
H_placeholder_check -->|no| I_credential_check
H1_mark_skip_placeholder["Mark_step_skip_reason_unresolved-placeholder_or_brace-expansion"] --> F_steps_loop
I_credential_check{Requires_credentials?} -->|yes| I1_mark_skip_creds
I_credential_check -->|no| J_allowlist_check
I1_mark_skip_creds["Mark_step_skip_reason_requires-credentials"] --> F_steps_loop
J_allowlist_check{Command_starts_with_allowlisted_prefix?} -->|no| J1_mark_not_allowlist
J_allowlist_check -->|yes| K_denylist_check
J1_mark_not_allowlist["Mark_step_skip_reason_not-in-allowlist"] --> F_steps_loop
K_denylist_check{Contains_denylisted_pattern?} -->|yes| K1_mark_unsafe
K_denylist_check -->|no| L_execute
K1_mark_unsafe["Mark_step_skip_reason_unsafe-command"] --> F_steps_loop
L_execute["Log_and_execute_via_dev_container_exec_API_with_timeout_120s"] --> M_timeout_or_result
M_timeout_or_result{Timeout_or_error?} -->|timeout| M1_mark_fail_timeout
M_timeout_or_result -->|error| M2_mark_fail_error
M_timeout_or_result -->|success| M3_record_pass
M1_mark_fail_timeout["Record_status_fail_reason_timeout"] --> F_steps_loop
M2_mark_fail_error["Record_status_fail_with_error_output"] --> F_steps_loop
M3_record_pass["Record_status_pass_and_actual_output_truncated_to_64KB"] --> F_steps_loop
P_specialists["Dispatch_specialist_reviews_in_parallel"] --> P1_UXD
P_specialists --> P2_DocsWriter
P_specialists --> P3_QE
P1_UXD["UXD_review_JSON_result_or_error_wrapper"] --> Q_collect
P2_DocsWriter["Docs_Writer_review_JSON_result_or_error_wrapper"] --> Q_collect
P3_QE["Adversarial_QE_review_JSON_result_or_error_wrapper"] --> Q_collect
Q_collect["Attach_reviews_to_doc_results"] --> C_docs_loop
Z_aggregate["Aggregate_all_doc_results_and_summaries"] --> AA_build_json
AA_build_json["Build_output_JSON_with_results_and_overall_summary"] --> AB_write_file
AB_write_file["python3_json.dump_to_/tmp/alcove-outputs.json"] --> AC_end
AC_end["End Doc_Runner"]
Flow diagram for Validation Reporter Jira deduplication and digestflowchart TD
A_start["Start Validation_Reporter"] --> B_guard_clause
B_guard_clause{Results_empty?} -->|yes| B1_create_alert_no_results
B_guard_clause -->|no| C_check_zero_steps
B1_create_alert_no_results["Create_Jira_alert_ticket_no_results"] --> Y_build_outputs
C_check_zero_steps["For_each_doc_if_zero_steps_create_alert_ticket"] --> D_main_loop
D_main_loop{More_items_to_process?} -->|yes| E_next_item
D_main_loop -->|no| S_stale_docs
E_next_item["Take_next_failed_step_or_critical_high_specialist_finding"] --> F_search_ticket
F_search_ticket["jira_search_project_PULP_labels_docs-validation_and_step_label"] --> G_ticket_exists
G_ticket_exists{Matching_ticket?} -->|yes| H_comment
G_ticket_exists -->|no| I_create_ticket
H_comment["jira_add_comment_with_latest_results_and_findings"] --> J_record_commented
I_create_ticket["jira_create_issue_Bug_or_Epic_with_step_and_doc_labels"] --> K_record_created
J_record_commented["Append_ticket_key_to_tickets_commented"] --> D_main_loop
K_record_created["Append_ticket_key_to_tickets_created"] --> D_main_loop
S_stale_docs["For_each_doc_where_all_steps_pass"] --> T_search_stale
T_search_stale["jira_search_open_tickets_with_doc_label"] --> U_have_stale
U_have_stale{Any_open_doc_tickets?} -->|yes| V_flag_stale
U_have_stale -->|no| W_digest
V_flag_stale["For_each_stale_ticket_add_pass_comment_and_add_validation-passing_label"] --> V1_record_stale
V1_record_stale["Append_ticket_keys_to_stale_tickets_flagged"] --> W_digest
W_digest["Create_or_update_weekly_digest_ticket"] --> W1_search_digest
W1_search_digest["jira_search_open_docs-validation-digest_ticket"] --> W2_have_digest
W2_have_digest{Digest_exists?} -->|yes| W3_add_comment
W2_have_digest -->|no| W4_create_digest
W3_add_comment["jira_add_comment_with_run_summary_and_links"] --> X_collect_digest_key
W4_create_digest["jira_create_issue_weekly_digest"] --> X_collect_digest_key
X_collect_digest_key["Store_digest_ticket_key"] --> Y_build_outputs
Y_build_outputs["Build_output_JSON_digest_ticket_created_commented_stale_summary"] --> Z_write
Z_write["python3_json.dump_to_/tmp/alcove-outputs.json"] --> AA_end
AA_end["End Validation_Reporter"]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- The command parsing contract allows executing bare
cd,curl,pulp, etc. from unlabeled code blocks, but the safety allowlist only permits specific prefixes (and omits some of those likecd), which means some parsed steps will always be skipped asnot-in-allowlist; consider aligning the parsing rules and allowlist so that any command you intend to validate can actually run. - The
dry_runinput is plumbed through the workflow but not mentioned or honored in either agent prompt; consider either wiring it into the behavior (e.g., skip Jira mutations or container execution) or removing it to avoid confusion. - In the Validation Reporter guard clause, a doc with zero parsed steps triggers an alert ticket but the instructions don’t say whether processing for that doc should stop or continue, which could lead to inconsistent behavior; consider explicitly specifying whether to skip further ticketing for that doc after raising the alert.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- The command parsing contract allows executing bare `cd`, `curl`, `pulp`, etc. from unlabeled code blocks, but the safety allowlist only permits specific prefixes (and omits some of those like `cd`), which means some parsed steps will always be skipped as `not-in-allowlist`; consider aligning the parsing rules and allowlist so that any command you intend to validate can actually run.
- The `dry_run` input is plumbed through the workflow but not mentioned or honored in either agent prompt; consider either wiring it into the behavior (e.g., skip Jira mutations or container execution) or removing it to avoid confusion.
- In the Validation Reporter guard clause, a doc with zero parsed steps triggers an alert ticket but the instructions don’t say whether processing for that doc should stop or continue, which could lead to inconsistent behavior; consider explicitly specifying whether to skip further ticketing for that doc after raising the alert.
## Individual Comments
### Comment 1
<location path=".alcove/agents/doc-runner.yml" line_range="28-37" />
<code_context>
+ | No language tag | If content starts with a known CLI tool (pip, pulp, curl, export, cd, cat, echo, ls, grep, jq), execute; otherwise skip with reason "ambiguous-block" |
</code_context>
<issue_to_address>
**issue (bug_risk):** CLI tools listed as executable in the parsing contract (e.g., `cd`) are missing from the ALLOWLIST and will always be skipped.
Because execution requires both being parsed as executable and passing the ALLOWLIST, `cd` (and any other CLI tools listed in the contract but not in the ALLOWLIST) will always be marked `skip` with reason `not-in-allowlist`. This contradicts the contract and can break multi-step flows that depend on directory changes. Please either add `cd` (and any other intended commands) to the ALLOWLIST or update the contract so they’re consistent.
</issue_to_address>
### Comment 2
<location path=".alcove/agents/validation-reporter.yml" line_range="17-22" />
<code_context>
+ The `results` input contains a JSON array of per-doc validation results.
+ The `overall` input contains a one-line summary string.
+
+ Use the Jira MCP tools for all Jira operations:
+ - jira_search (search for existing tickets)
+ - jira_create_issue (create new tickets)
+ - jira_add_comment (comment on existing tickets)
+ - jira_get_issues (read ticket details)
+ - jira_set_labels (update labels on tickets)
+
+ ## Guard Clause
</code_context>
<issue_to_address>
**issue (bug_risk):** The Jira tool function names in the prompt don’t match the operations exposed in the security profile.
This mismatch will likely cause runtime tool invocation failures, since the agent will call functions that are not defined in the `docs-reporter` security profile (`search_issues`, `create_issue`, `add_comment`, `read_issues`, `update_issue`). Please either rename the tools in the prompt to match these operations (e.g., use `search_issues` and `update_issue` for label changes) or update the MCP/tool definitions so the names the agent is instructed to use are actually available.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
Comment on lines
+28
to
+37
| | No language tag | If content starts with a known CLI tool (pip, pulp, curl, export, cd, cat, echo, ls, grep, jq), execute; otherwise skip with reason "ambiguous-block" | | ||
| | Contains unresolved placeholders (<component>, <version>, ${VAR}) | Skip with reason "unresolved-placeholder" | | ||
| | Contains brace expansion ({a,b,c}) | Skip with reason "brace-expansion" — requires bash, not portable | | ||
|
|
||
| Ignore inline code (backtick-wrapped text within paragraphs). | ||
|
|
||
| Assign each step a sequential ID: `<doc-basename>-<NN>` (e.g., `cli-guide-01`). | ||
|
|
||
| ## Credential Handling | ||
|
|
Contributor
There was a problem hiding this comment.
issue (bug_risk): CLI tools listed as executable in the parsing contract (e.g., cd) are missing from the ALLOWLIST and will always be skipped.
Because execution requires both being parsed as executable and passing the ALLOWLIST, cd (and any other CLI tools listed in the contract but not in the ALLOWLIST) will always be marked skip with reason not-in-allowlist. This contradicts the contract and can break multi-step flows that depend on directory changes. Please either add cd (and any other intended commands) to the ALLOWLIST or update the contract so they’re consistent.
Comment on lines
+17
to
+22
| Use the Jira MCP tools for all Jira operations: | ||
| - jira_search (search for existing tickets) | ||
| - jira_create_issue (create new tickets) | ||
| - jira_add_comment (comment on existing tickets) | ||
| - jira_get_issues (read ticket details) | ||
| - jira_set_labels (update labels on tickets) |
Contributor
There was a problem hiding this comment.
issue (bug_risk): The Jira tool function names in the prompt don’t match the operations exposed in the security profile.
This mismatch will likely cause runtime tool invocation failures, since the agent will call functions that are not defined in the docs-reporter security profile (search_issues, create_issue, add_comment, read_issues, update_issue). Please either rename the tools in the prompt to match these operations (e.g., use search_issues and update_issue for label changes) or update the MCP/tool definitions so the names the agent is instructed to use are actually available.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cli-guide.md,api-access.md,install-bindings.md) into executable steps, runs them in a dev container with command allowlist and safety checks, and dispatches UXD/Docs Writer/Adversarial QE specialist reviewers per docDesign highlights
step:<id>labels — exact and idempotent across runspython3 json.dumpinstead of shell heredoc for reliabilityArtifacts
.alcove/workflows/onboarding-validator.yml.alcove/agents/doc-runner.yml.alcove/agents/validation-reporter.yml.alcove/security-profiles/docs-validator.yml.alcove/security-profiles/docs-reporter.ymlReview process
Design spec reviewed by 3 SDLC specialists (UXD, Docs Writer, Adversarial QE). Implementation reviewed by 4 specialists (AI Pipeline, SRE, Test, AI Harness). All critical, high, and medium findings addressed in second commit.
Design spec:
agent-project/docs/superpowers/specs/2026-04-24-onboarding-docs-validator-design.mdTest plan
python3 -c "import yaml; yaml.safe_load(...)")docs-validationlabel🤖 Generated with Claude Code
Summary by Sourcery
Introduce an automated weekly Alcove workflow to validate pulp-docs onboarding guides and file or update Jira tickets based on execution and specialist review results.
New Features:
Enhancements: