feat: Update AI Code Review docs for GA (status checks, image, overall UX) #15679
Conversation
- Add GitHub Permissions section explaining required permissions (Pull Requests, Checks, Commit Statuses) - Add comprehensive GitHub Status Checks section covering: - Status check behavior (pass/neutral/fail states) - Visibility information for repository users - Instructions for integrating with branch protection rules - Add three new FAQ entries about status checks: - Why status checks fail - How to disable status checks - Update GitHub integration docs to reference AI Code Review in permissions table Fixes: https://linear.app/getsentry/issue/CW-11
|
Someone is attempting to deploy a commit to the Sentry Team on Vercel. A member of the Team first needs to authorize it. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
- Change 'regular PR comments' to 'review comments' - Fix status check name to 'Seer Code Review' (not 'Sentry AI Code Review') - Update recommendation to NOT make status check required - Update status check behavior to include all states: success, neutral, error, cancelled, timed_out - Clarify that success indicates review completed, not code quality - Rewrite FAQ about failing status checks to be more accurate
sfanahata
left a comment
sfanahata
left a comment
There was a problem hiding this comment.
Couple of nits, but looks good whenever you're ready.
- Remove Commit Statuses permission (AI Code Review only uses Checks) - Update status check behavior: remove Error state, clarify Neutral includes service errors - Update FAQ about status check to reflect neutral status instead of error - Apply sfanahata's suggestion for optional check wording
|
|
||
| ### Status Check Behavior | ||
|
|
||
| - **Success**: If AI Code Review completes successfully, the status check will show as successful (green checkmark). This happens regardless of whether issues were found - the check indicates the review completed, not that the code is error-free. |
There was a problem hiding this comment.
oh that's not the current behavior. If issues were found the check is failed.
We can update to do this instead tho
There was a problem hiding this comment.
@giovanni-guidini - did this UX get changed, or should I update the docs?
There was a problem hiding this comment.
sorry yeah the docs should be updated. we output "neutral" if a comment is found.
https://github.com/getsentry/seer/blob/main/src/seer/automation/codegen/pr_review_step.py#L531
sfanahata
left a comment
sfanahata
left a comment
There was a problem hiding this comment.
Looks good! Let me know what help you want with some image refreshes.
sfanahata
changed the title
| - **Why aren't `@sentry` commands working?** | ||
|
|
||
| AI Code Review will respond with a message to enable the `Show Generative AI Features` and `Enable AI Code Review` settings in your [organization settings](https://sentry.io/orgredirect/settings/:orgslug/), if you have not already enabled them. Once enabled, try `@sentry review` again to run AI Code Review. | ||
| AI Code Review will only run once `on command phrase` settings are turned on in [Seer settings](https://sentry.io/orgredirect/organizations/:orgslug/settings/seer/). Once enabled, try `@sentry review` again to manually run AI Code Review. |
There was a problem hiding this comment.
maybe reframe to "AI Code Review will respond to commands only once ...."
|
|
||
| ### Status Check Behavior | ||
|
|
||
| - **Success**: If AI Code Review completes successfully, the status check will show as successful (green checkmark). This happens regardless of whether issues were found - the check indicates the review completed, not that the code is error-free. |
There was a problem hiding this comment.
sorry yeah the docs should be updated. we output "neutral" if a comment is found.
https://github.com/getsentry/seer/blob/main/src/seer/automation/codegen/pr_review_step.py#L531
|
|
||
|  | ||
| - **Success**: If AI Code Review completes successfully, the status check will show as successful (green checkmark). This happens regardless of whether issues were found - the check indicates the review completed, not that the code is error-free. | ||
| - **Neutral**: If AI Code Review runs but has no specific pass/fail outcome, or if there's a service issue or timeout. We use neutral status for errors so users don't assume there's something wrong with their code when the issue is on our end. |
There was a problem hiding this comment.
Looking through https://github.com/getsentry/seer/blob/main/src/seer/automation/codegen/pr_review_step.py#L531
Neutral should be when we find any issues.
| - **Success**: If AI Code Review completes successfully, the status check will show as successful (green checkmark). This happens regardless of whether issues were found - the check indicates the review completed, not that the code is error-free. | ||
| - **Neutral**: If AI Code Review runs but has no specific pass/fail outcome, or if there's a service issue or timeout. We use neutral status for errors so users don't assume there's something wrong with their code when the issue is on our end. | ||
| - **Cancelled**: If the review was cancelled, typically because a new commit was pushed and the previous review was superseded. | ||
|
|
There was a problem hiding this comment.
Maybe we want to describe the errored and timedout states as well?
Because we were earlier saying that we have it all in neutral, but in fact we have erroed and timeout to communicate if something went wrong on our end
https://github.com/getsentry/seer/blob/main/src/seer/automation/codegen/pr_review_step.py#L93
![sentry[bot]](https://avatars.githubusercontent.com/u/1396951?s=60&v=4){kind=small}
| - **Why aren't `@sentry` commands working?** | ||
|
|
||
| AI Code Review will respond with a message to enable the `Show Generative AI Features` and `Enable AI Code Review` settings in your [organization settings](https://sentry.io/orgredirect/settings/:orgslug/), if you have not already enabled them. Once enabled, try `@sentry review` again to run AI Code Review. | ||
| AI Code Review will respond to commands only once `on command phrase` settings are turned on in [Seer settings](https://sentry.io/orgredirect/organizations/:orgslug/settings/seer/). Once enabled, try `@sentry review` again to manually run AI Code Review. |
There was a problem hiding this comment.
Bug: The documentation references a non-existent or incorrectly named setting, on command phrase, for enabling the @sentry review command, making the feature undiscoverable.
Severity: MEDIUM | Confidence: High
🔍 Detailed Analysis
The FAQ section at line 93 instructs users to enable a setting named on command phrase in Seer settings to use the @sentry review command. However, this setting is not mentioned anywhere else in the documentation, including the Seer-specific docs. This suggests the setting does not exist or is incorrectly named, which will prevent users from being able to find and enable the @sentry review command functionality.
💡 Suggested Fix
Verify the correct name and location of the setting that enables the @sentry review command. Update the FAQ to reference the correct setting name and location (e.g., 'Seer settings' or 'organization settings') so that users can successfully enable the feature.
🤖 Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: docs/product/ai-in-sentry/ai-code-review/index.mdx#L93
Potential issue: The FAQ section at line 93 instructs users to enable a setting named
`on command phrase` in Seer settings to use the `@sentry review` command. However, this
setting is not mentioned anywhere else in the documentation, including the Seer-specific
docs. This suggests the setting does not exist or is incorrectly named, which will
prevent users from being able to find and enable the `@sentry review` command
functionality.
Did we get this right? 👍 / 👎 to inform future reviews.
Reference ID: 7777918
4 participants
Main preview: https://sentry-docs-git-fork-rohan-at-sentry-docs-ai-code-review-cc3117.sentry.dev/product/ai-in-sentry/ai-code-review/
Fixes: https://linear.app/getsentry/issue/CW-11
DESCRIBE YOUR PR
Tell us what you're changing and why. If your PR resolves an issue, please link it so it closes automatically.
IS YOUR CHANGE URGENT?
Help us prioritize incoming PRs by letting us know when the change needs to go live.
SLA
Thanks in advance for your help!
PRE-MERGE CHECKLIST
Make sure you've checked the following before merging your changes:
LEGAL BOILERPLATE
Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.
EXTRA RESOURCES