feat(ai-reviewer): pushback patterns for AI reviewer comments#74
Merged
Conversation
New references/ai-reviewer-pushback.md documenting how to evaluate, respond to, and resolve AI reviewer comments (Copilot, gemini-code-assist, CodeRabbit, Sourcery, etc.) without rubber-stamping wrong advice or ignoring valid feedback. Covers four common failure modes: - Field-name / API hallucination (real example: gemini-code-assist suggesting non-existent pnpm `ignoredBuilds:` field). - Stale knowledge of release status (claiming current versions are unreleased; recommending outdated minimum versions). - Pattern advice frozen at a past major (jQuery in vanilla DOM code, Vue 2 Options API in Vue 3, deprecated GitHub Actions inputs). - Inverting a security control to make a build pass. Includes a six-step pushback workflow (verify against primary sources, check empirical evidence on the PR, read the bot rationale, decide, reply on the thread, resolve), reply templates for both pushback and partial-agreement, gh-graphql snippets for thread reply and resolve, anti-patterns, and bot-specific quirks. Source: production review handling on netresearch/timetracker-ui#717 (gemini-code-assist incorrectly suggesting `ignoredBuilds` for pnpm 11 — pushed back with docs + green CI evidence rather than applying the wrong change). Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>
Contributor
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new guide, ai-reviewer-pushback.md, which provides strategies for evaluating and responding to automated AI code reviews, covering failure modes like hallucinations and stale knowledge. The main SKILL.md file was also updated to include this new reference and improve existing descriptions. Feedback from the review correctly identified an accidental deletion of existing documentation references in the main skill file and suggested technical refinements to the provided shell and GraphQL snippets to ensure better error handling and consistency.
There was a problem hiding this comment.
Pull request overview
Adds a new reference document to help maintainers handle incorrect or stale AI reviewer feedback in PR threads, and updates the skill’s summary/entry points to include this use case.
Changes:
- Added a new reference guide:
ai-reviewer-pushback.md(workflows, templates, GraphQL snippets, failure-mode catalog). - Updated
SKILL.mdfront-matter description and “When to Use” list to include AI reviewer scenarios. - Updated the
SKILL.mdReferences table to link the new guide (currently at the cost of dropping two existing reference links).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| skills/github-project/SKILL.md | Expands the skill description + “When to Use” bullets; updates the References table to include the new AI reviewer pushback doc. |
| skills/github-project/references/ai-reviewer-pushback.md | New reference describing common AI review failure modes, an evidence-based pushback workflow, and thread-reply/resolve GraphQL commands. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Restore upstream SKILL.md (When-to-Use compression and the
org-security-settings / tag-validation reference rows that were
accidentally overwritten when initial copy came from a stale main
worktree, per Copilot + gemini-code-assist comments).
- Bump skill metadata version 2.13.1 -> 2.14.0 for the new reference.
- Reference doc fixes (Copilot review):
* Rephrase 'Stale knowledge of release status' examples as
pattern-shapes with a 'these will go stale' caveat instead of
naming specific current versions that will rot.
* Rewrite 'CKEditor 4 plugin shapes in a CKE5 file' with the full
framework name on both sides for clarity.
* Add a parenthetical introducing Context7 as a docs-lookup MCP
server so the mention has context.
- Reference doc fixes (gemini-code-assist review):
* Add jq '?' suffix and '// \"\"' fallback in the thread-listing
snippet so empty 'comments' nodes don't error.
* Rewrite the resolveReviewThread snippet to use a GraphQL variable
for thread ID, matching the style of the reply snippet.
Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>
Signed-off-by: Sebastian Mendel <info@sebastianmendel.de>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new reference for handling AI reviewer comments (Copilot, gemini-code-assist, CodeRabbit, Sourcery, Codium / PR-Agent) without either rubber-stamping wrong advice or ignoring valid feedback.
File:
skills/github-project/references/ai-reviewer-pushback.md(+ link from SKILL.md table, description updated)What it covers
Four common failure modes documented with real examples:
ignoredBuilds:for pnpm 11. That field doesn't exist; the legacy name wasignoredBuiltDependencies(removed in pnpm 11), and the modern equivalent isallowBuilds: { pkg: false }.fail_on_errorinstead offail_level), CKE4 plugin shapes in CKE5.Plus:
Why now
This came up on netresearch/timetracker-ui#717, where gemini-code-assist marked a config change as a high-priority bug and recommended replacing a working
allowBuildsmap with a non-existentignoredBuildsarray. The right response was to push back with docs links and green CI evidence rather than apply the wrong change. The pattern is going to keep coming up across all our repos as more bot reviewers come online.Relationship to existing references
auto-merge-guide.md(which covers the Copilot-as-reviewer race condition — a different failure mode where the bot blocks a merge by not reviewing).Test plan
bash scripts/verify-harness.sh→ Level 3 COMPLETE | 0 errors, 0 warnings