Skip to content

Implement automated CI failure investigator#3861

Open
shralex wants to merge 1 commit into
mainfrom
shralex_investigate
Open

Implement automated CI failure investigator#3861
shralex wants to merge 1 commit into
mainfrom
shralex_investigate

Conversation

@shralex
Copy link
Copy Markdown
Collaborator

@shralex shralex commented May 9, 2026

This PR implements a Gemini-powered automated investigator to diagnose CI test failures in the repository. It enables the @gemini-cli /investigate command and runs automatically on pipeline failures to provide actionable diagnostic feedback.

Key Features:

  • Automated Failure Diagnostics: Formulates a systematic diagnostics report containing the specific test flavor/jobs that failed, exact traceback logs, and logical context of the failure.
  • Diagnostics Protocol: Guides the model to perform efficient diagnostics—starting with log parsing, searching git history for flakes, and analyzing recent code changes in the PR or main branch.
  • Targeted Fix Recommendations: Directs the model to generate high-confidence code diff suggestions when the root cause is identified.
  • Integrated Reporting: Automatically writes the final markdown report to .gemini/findings.md and posts it as a comment directly to the target PR or failure issue.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

  • I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have run end-to-end tests tests and provided workload links above if applicable.
  • I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 9, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions
Copy link
Copy Markdown

🤖 Hi @RissyRan, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

Copy link
Copy Markdown

@github-actions github-actions Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

## 📋 Review Summary

This Pull Request successfully implements an automated CI failure investigator powered by Gemini. The diagnostics protocol is well-defined and systematic, offering a great balance between efficiency and depth.

🔍 General Feedback

  • Regression Alert: There is a regression in gemini-dispatch.yml where additional_context parsing was removed for the /review command. I've suggested a fix to restore this and extend it to /investigate.
  • Workflow Robustness: The fallback logic for identifying the failed run ID could be made more specific to the current PR branch to avoid false positives in high-traffic periods.
  • Protocol Excellence: The systematic approach in gemini-investigate.toml (cheap actions first, grouping failures) is a highlight and should serve as a model for other automated commands.

const additionalContext = request.replace(/^@gemini-cli \/review/, '').trim();
core.setOutput('additional_context', additionalContext);
core.setOutput('additional_context', '');
} else if (request.startsWith("@gemini-cli /investigate")) {
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 This change removes the ability to provide additional context to the `/review` command, which appears to be an unintended regression. The previous implementation correctly parsed and passed the context.
Suggested change
} else if (request.startsWith("@gemini-cli /investigate")) {
} else if (request.startsWith("@gemini-cli /review")) {
core.setOutput('command', 'review');
const additionalContext = request.replace(/^@gemini-cli \/review/, '').trim();
core.setOutput('additional_context', additionalContext);
} else if (request.startsWith("@gemini-cli /investigate")) {
core.setOutput('command', 'investigate');
const additionalContext = request.replace(/^@gemini-cli \/investigate/, '').trim();
core.setOutput('additional_context', additionalContext);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its intended, to protect against context injection.


# Determine target run ID
if [ -z "$RUN_ID" ]; then
# Fallback to finding the latest failed run for this PR if triggered manually
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The fallback logic here will retrieve the latest failed run for the entire repository, which might not be the run associated with the current Pull Request if multiple PRs are running CI concurrently.

Consider filtering by the head branch or PR to ensure the investigator analyzes the relevant failure. If this is triggered via issue_comment on a PR, you can retrieve the head branch name and pass it to gh run list --branch <branch>.

workflow_call:
inputs:
additional_context:
type: 'string'
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 The `workflow_call` trigger defines `additional_context` as an input, but `failed_run_id` is missing from the `workflow_call` inputs definition. This means `github.event.inputs.failed_run_id` will likely be undefined when called from another workflow.
Suggested change
type: 'string'
workflow_call:
inputs:
additional_context:
type: 'string'
required: false
failed_run_id:
type: 'string'
required: false

"args": [
"run",
"-i",
"--rm",
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 It's good practice to ensure that the investigator has access to the most relevant tools. Since the prompt mentions searching git history and exploring files, ensure `mcpServers` configuration includes all necessary permissions if they are not already covered by the defaults or the explicitly listed tools.

The current list is good, but for "searching git history", you might eventually want tools that can run git log or git blame if the shell tool is too restricted (though here you've allowed cat, grep, etc., which is a good start).

Copy link
Copy Markdown
Collaborator

@RissyRan RissyRan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gemini-cli /investigate

@shralex shralex force-pushed the shralex_investigate branch from 7f80543 to a58ee37 Compare May 11, 2026 21:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants