Skip to content

feat(#1797): extract challenger pass into dedicated sub-agent#1812

Open
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/1797-challenger-sub-agent
Open

feat(#1797): extract challenger pass into dedicated sub-agent#1812
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/1797-challenger-sub-agent

Conversation

@fullsend-ai-coder
Copy link
Copy Markdown

The challenger pass (step 6e in the pr-review skill) previously ran inline within the orchestrator's LLM context. The same context that synthesized findings also challenged them, reducing false-positive detection effectiveness since the LLM had already formed opinions during synthesis.

This change extracts the challenger into a dedicated sub-agent with fresh context, as specified in the parent issue #1085. The challenger sub-agent receives only the raw merged findings and the PR diff — not the orchestrator's synthesis — preserving context isolation.

Changes:

  • Add sub-agents/challenger.md with adversarial review instructions,
    structured output format, and evidence-based removal requirements
  • Update SKILL.md step 6e to dispatch the challenger as a sub-agent
    after dimension sub-agents complete (sequential, not parallel)
  • Add challenger to the sub-agent roster table
  • Define graceful degradation: if the challenger fails, fall back to
    the pre-challenger finding set with an info-level disclosure

Note: make lint could not run (Go toolchain permission denied in sandbox). Changes are markdown-only; no Go code modified.


Closes #1797

Post-script verification

  • Branch is not main/master (agent/1797-challenger-sub-agent)
  • Secret scan passed (gitleaks — 972ce7f6654f0b7edaf699298901b10db69e4e4c..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

The challenger pass (step 6e in the pr-review skill) previously ran
inline within the orchestrator's LLM context. The same context that
synthesized findings also challenged them, reducing false-positive
detection effectiveness since the LLM had already formed opinions
during synthesis.

This change extracts the challenger into a dedicated sub-agent with
fresh context, as specified in the parent issue #1085. The challenger
sub-agent receives only the raw merged findings and the PR diff —
not the orchestrator's synthesis — preserving context isolation.

Changes:
- Add sub-agents/challenger.md with adversarial review instructions,
  structured output format, and evidence-based removal requirements
- Update SKILL.md step 6e to dispatch the challenger as a sub-agent
  after dimension sub-agents complete (sequential, not parallel)
- Add challenger to the sub-agent roster table
- Define graceful degradation: if the challenger fails, fall back to
  the pre-challenger finding set with an info-level disclosure

Note: make lint could not run (Go toolchain permission denied in
sandbox). Changes are markdown-only; no Go code modified.

Closes #1797

Signed-off-by: fullsend-code <fullsend-code@users.noreply.github.com>
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 2, 2026

Site preview

Preview: https://fc9756fd-site.fullsend-ai.workers.dev

Commit: 38bddcd6418016360fcdfbc1e526adf37cf74574

@fullsend-ai-review
Copy link
Copy Markdown

Review

Findings

Medium

  • [logic-error] internal/scaffold/fullsend-repo/skills/pr-review/SKILL.md:541 — The challenger sub-agent receives the full merged finding set from steps 6a–6d, including orchestrator-only PR-specific findings from step 6d (protected-path, scope-authorization, PR body injection). Step 6e.4 instructs the orchestrator to "Replace the merged finding set with the challenger's adjudicated_findings," but the challenger has no instruction exempting these orchestrator-only findings from removal or downgrade. If the challenger removes a protected-path finding, the orchestrator's invariant ("the presence of a protected-path finding at high or medium severity means the outcome MUST NOT be approve") would be violated. The post-review.sh script provides a backup safety net, but the review agent's own constraint would be broken.
    Remediation: Either (a) exclude step 6d findings from the context package sent to the challenger and re-merge them after consuming the challenger's output, or (b) add an explicit instruction in challenger.md that protected-path, scope-authorization, and PR-body-injection findings from orchestrator-only checks must never be removed or downgraded.

Low

  • [edge-case] internal/scaffold/fullsend-repo/skills/pr-review/sub-agents/challenger.md:48 — The challenger output schema defines adjudicated_findings with extra fields (challenger_action, challenger_reason) not present in the standard finding format validated by fullsend-check-output. Step 6e.4 says to replace the merged finding set with the challenger's output, but does not instruct the orchestrator to strip challenger-specific metadata fields before writing agent-result.json. If these fields propagate, schema validation (additionalProperties: false) would reject them.

  • [edge-case] internal/scaffold/fullsend-repo/skills/pr-review/SKILL.md:549 — Step 6e.5 specifies fallback when the challenger fails (timeout, error, empty response), but does not define how to handle a valid JSON response with an empty adjudicated_findings array. An empty array could mean "all findings legitimately removed" or indicate a malfunction. Consider adding a sanity check: if the challenger removes all findings when the input set contained critical/high findings, treat it as suspicious and fall back to the pre-challenger set.

Info

  • [sub-agent-failure] N/A — The style-conventions sub-agent did not return findings: model claude-sonnet-4-5@20250929 not available on vertex deployment.

  • [sub-agent-failure] N/A — The docs-currency sub-agent did not return findings: model claude-sonnet-4-5@20250929 not available on vertex deployment.

  • [sub-agent-failure] N/A — The intent-coherence sub-agent did not return findings: model claude-sonnet-4-5@20250929 not available on vertex deployment.

@fullsend-ai-review fullsend-ai-review Bot added the requires-manual-review Review requires human judgment label Jun 2, 2026
Comment on lines +17 to +20
**Do not own:** Generating new findings. You only challenge, downgrade,
or remove existing ones. If you discover a genuine issue not covered by
any finding, note it — but your primary job is quality control of the
existing set.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
**Do not own:** Generating new findings. You only challenge, downgrade,
or remove existing ones. If you discover a genuine issue not covered by
any finding, note it — but your primary job is quality control of the
existing set.
**Do not own:** Generating new findings.
You only challenge, downgrade,
or remove existing ones. If you discover a genuine issue not covered by
any finding, note it — but your primary job is quality control of the
existing set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

requires-manual-review Review requires human judgment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(review agent): challenger pass should use a dedicated sub-agent, not the orchestrator

1 participant