Repo Gardening: Add AI-based check for user-facing PR changes by jeherve · Pull Request #46701 · Automattic/jetpack

jeherve · 2026-01-21T11:12:21Z

Proposed changes:

This PR adds a new checkIfDocsNeeded task to the repo-gardening action that uses AI to automatically detect user-facing changes in PRs. When detected with medium or high confidence, it:

Applies the [Status] UI Changes label to flag the PR for documentation review
Sends a Slack notification to the quality channel with the AI's reasoning (currently #jetpack-developer-ambassadors in the Jetpakc repo).

Since it's a new task, it will only be enabled in the Jetpack repo for now. In the future, it can be enabled in other repos too.

Bailout conditions:

The task skips processing when:

No openai_api_key input is provided
PR is from a fork
PR already has [Status] UI Changes label
PR title contains "revert"
PR diff is too small to analyze

Other information:

Have you written new tests for your changes, if applicable?
Have you checked the E2E test CI results, and verified that your changes do not break them?
Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with a script to run)?

Jetpack product discussion

N/A

Does this pull request change what data or activity we track or use?

No. The task sends PR diffs and descriptions to OpenAI for analysis but does not store or track any new data.

Testing instructions:

This is easier tested in a fork. Here are 2 examples.

PR, with the label added: About page: simplify credits display jeherve/jetpack#214 (comment)
- Matching Slack message: p1769008065095639-slack-CN2FSK7L4
Another PR, that doesn't make any UI changes: Refactor: Move review link functionality from Jetpack class to dedicated file jeherve/jetpack#213
- No label should be added.
- No message should be sent.
- The action logs explain why:

check-if-docs-needed: OpenAI response for PR #213: {
  "is_user_facing": false,
  "confidence": "high",
  "reason": "The PR is a refactor that reorganizes the code related to the 5-star review link without changing its functionality or appearance to users. Users will experience the review link the same way as before, with no visible changes."
}
check-if-docs-needed: PR #213 is not user-facing or low confidence. Not adding label. Reason: The PR is a refactor that reorganizes the code related to the 5-star review link without changing its functionality or appearance to users. Users will experience the review link the same way as before, with no visible changes.

Fixes MONOREP-325 Add a new checkIfDocsNeeded task that uses OpenAI to analyze PR diffs and descriptions to determine if changes are user-facing. If detected with medium/high confidence, the [Status] UI Changes label is applied. The task: - Fetches and cleans PR diffs (filtering lock files, binary files, minified code) - Sends to OpenAI for analysis - Applies [Status] UI Changes label for user-facing changes - Uses [Experiment] AI UI check marker label to prevent re-processing - Bails out for drafts, bots, reverts, and already-labeled PRs

github-actions · 2026-01-21T11:13:36Z

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

✅ Include a description of your PR changes.
✅ Add a "[Status]" label (In Progress, Needs Review, ...).
✅ Add testing instructions.
✅ Specify whether this PR includes any changes to data or privacy.
✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

Follow this PR Review Process:

Ensure all required checks appearing at the bottom of this PR are passing.
Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
You can use GitHub's Reviewers functionality to request a review.
When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

… task Add documentation and Slack notification support for the AI-based user-facing changes detection task: - Create task readme.md with usage and bailout conditions - Update main README.md to include task in the list - Send Slack notification to quality channel when PR is flagged

Copilot

Pull request overview

This PR adds an AI-powered task to automatically detect user-facing changes in closed PRs. The task uses OpenAI's API to analyze PR diffs and descriptions, applying a [Status] UI Changes label when user-facing changes are detected with medium or high confidence. A Slack notification is sent to the quality channel when configured.

Changes:

New checkIfDocsNeeded task that runs on pull_request_target closed events
New getDiff utility to fetch and cache PR diffs from GitHub API
Integration with existing OpenAI and Slack utilities for analysis and notifications

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
`src/tasks/check-if-docs-needed/index.js`	Main task implementation with AI analysis logic, label application, and Slack notifications
`src/tasks/check-if-docs-needed/readme.md`	Documentation explaining the task's purpose, behavior, and bailout conditions
`src/utils/get-diff.js`	Utility function to fetch PR diffs with caching, filtering, and size limits
`src/index.js`	Registers the new task to run on PR closed events with fork protection
`changelog/2026-01-20-18-55-37-299081`	Changelog entry documenting the new feature
`README.md`	Updated task list to include the new checkIfDocsNeeded task

…cter sanitization Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.

StefMattana · 2026-01-21T17:45:06Z

@jeherve could you replace #jetpack-developer-ambassadors with #jetpack-product-ambassadors for the Slack destination ping?

Replace slack_quality_channel with slack_product_ambassadors_channel for the check-if-docs-needed task to send notifications to the #jetpack-product-ambassadors channel instead.

anomiex · 2026-01-21T17:00:05Z

+	content = content.replace( /\\/g, '\\\\' );
+
+	return content.replace( /`/g, '\\`' );


The AI that wrote this doesn't know about using /[\\`]/g to escape both characters at once?

Also potentially of note is that a \ doesn't actually escape backticks inside backticks in (GitHub-flavored) markdown in the first place. You have to increase the number of framing backticks there too, as I did above.

Done in 947ebca

anomiex · 2026-01-21T17:15:54Z

+	// Replace sequences of three or more backticks (which could break markdown code blocks)
+	// with a safe placeholder. This prevents prompt injection via code block delimiters.
+	// This is the critical fix: triple backticks would prematurely close the code block.
+	content = content.replace( /```+/g, '[code-fence]' );


In actual GitHub-flavored Markdown, you could just ensure that the real fence has more backticks than the content. For example,

This makes a code block: ``` code block! ```

I have no idea what the AI might need though.

Done in 947ebca

anomiex · 2026-01-21T17:23:00Z

+
+	return `You are analyzing a GitHub Pull Request to determine if users will experience something DIFFERENT after this change is deployed.
+
+The key question is: "Will users notice any difference in behavior, appearance, or functionality?"


Is this the right question? We don't care about all user-facing changes, just ones that will need updates to the support documentation.

Updated in 38cc325 and 924d5dc

anomiex · 2026-01-21T17:26:23Z

+${ sanitizedTitle }
+
+Here is the PR description:
+${ sanitizedBody || '(No description provided)' }


You're not fencing this at all, and it might be long. How does the AI know that everything up to "Here is the code diff" isn't instruction? Particularly since it probably contains headings and all sorts of stuff.

Done in 8598950

anomiex · 2026-01-21T17:30:36Z

+		return;
+	}
+
+	// Skip if PR title contains "revert" (case-insensitive).


"Undo revert and fix feature with a bunch of additional UI changes"?

"New UI for post revert feature"?

Switched to a stricter approach (only look for revert at the beginning of the title) in ca0c4c1

anomiex · 2026-01-21T17:34:28Z

+	// Check if OpenAI API key is provided.
+	const apiKey = getInput( 'openai_api_key' );
+	if ( ! apiKey ) {
+		debug( `check-if-docs-needed: No OpenAI key is provided. Bail.` );


This seems like it would be useful to output to the log even when not in testing mode. "Why did this fail? There's no diagnostic output."

Same goes for many of the error messages below.

Done in f224ca5

anomiex · 2026-01-21T17:36:22Z

+		result = JSON.parse( response );
+	} catch ( error ) {
+		debug(
+			`check-if-docs-needed: Failed to parse OpenAI response for PR #${ number }: ${ error }`


It may be useful to know what OpenAI did respond with, not just the parse error.

Done in f224ca5

anomiex · 2026-01-21T17:37:55Z

+		return;
+	}
+
+	const isUserFacing = typeof result?.is_user_facing === 'boolean' ? result.is_user_facing : false;


If it's not boolean, that probably deserves diagnostic output too.

Done in f224ca5

anomiex · 2026-01-21T17:38:48Z

+	const confidence =
+		result?.confidence &&
+		typeof result.confidence === 'string' &&
+		[ 'low', 'medium', 'high' ].includes( result.confidence.trim().toLowerCase() )
+			? result.confidence.trim().toLowerCase()
+			: 'low';


Same for this. Instead of (just) assuming 'low', we should probably also log the unexpected data.

Done in f224ca5

anomiex · 2026-01-21T17:52:43Z

+	// Remove unwanted file blocks (e.g., lockfiles) before further processing/truncation.
+	diff = filterDiff( diff );
+
+	// Filter out lines longer than 500 characters (likely minified code).


This may result in broken diffs.

At a quick grep, this will also catch a fair bit of inline SVG code. And a few paragraphs in various markdown docs.

There's also the question as to why we're committing minified code in the first place.

58fb4b3 should solve that, although I don't know if the added complexity is worth it in this case ; I don't know that it's important for the LLM to get a valid diff since it doesn't have to apply it anywhere. I suppose it could become important in the future.

There's also the question as to why we're committing minified code in the first place.

Probably a larger discussion to have :)

I guess my thinking was that it might confuse the AI to have broken/empty diff chunks like

diff --git a/some/file.min.js b/some/file.min.js index aaaaaaaaaaaa..bbbbbbbbbb 100644 --- a/some/file.min.js +++ b/some/file.min.js @@ -1 +1 \ No newline at end of file diff --git a/some/other.file b/some/other.file ...

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

In GitHub-flavored Markdown, backslashes don't escape backticks inside code blocks. The proper approach is to use more backticks for the fence than appear in the content. - Use 4 backticks for code fences in the prompt - Only replace sequences of 4+ backticks in content (instead of 3+) - Remove unnecessary backslash and backtick escaping

The prompt previously asked about user-facing changes in general. Updated to specifically ask whether support documentation would need to be updated, which is the actual goal of this feature.

Wrap the PR title and description in code fences so the AI can clearly distinguish between instruction text and PR content. This prevents the AI from potentially interpreting PR content (which may contain markdown formatting) as additional instructions.

Only skip PRs where the title starts with "Revert" (the standard GitHub revert format), instead of skipping any PR with "revert" anywhere in the title. This prevents incorrectly skipping PRs like "Undo revert and fix feature" or "New UI for post revert feature".

- Include PR number in the missing API key message - Include the actual OpenAI response when JSON parsing fails - Log when is_user_facing is not a boolean with the actual value - Log when confidence is not a valid value with the actual value This makes it easier to diagnose issues when the AI returns unexpected responses.

The previous implementation filtered all lines longer than 500 characters, which could break diff structure by removing header lines. Now: - Preserve diff header lines (diff --, +++, ---, @@, etc.) regardless of length - Only filter actual content lines (additions, deletions, context) - Add logging when lines are filtered - Add comments explaining the tradeoff with SVG/markdown content

Update references from "quality channel/team" to "product ambassadors channel/team" to match the actual parameter name `slack_product_ambassadors_channel`.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

jeherve · 2026-01-22T10:12:36Z

@jeherve could you replace #jetpack-developer-ambassadors with #jetpack-product-ambassadors for the Slack destination ping?

@StefMattana Sure thing, done!

anomiex

Seems reasonable. Several comments inline, but nothing that's both too important and also is actionable IMO.

anomiex · 2026-01-23T18:20:46Z

+ *
+ * In GitHub-flavored Markdown, backslashes don't escape backticks inside code blocks.
+ * Instead, you use more backticks for the fence than appear in the content.
+ * We use 4 backticks for our fences, so we replace any sequence of 4+ backticks.


Optional: We could have this function add the fencing backticks itself after scanning the content to see how many are necessary, instead of arbitrarily limiting it.

Oh, I see what you mean. I like that. I'll keep it in mind if this becomes a problem ; I'm thinking the LLM will tolerate markup issues a bit more than a human would so this may never come up.

anomiex · 2026-01-23T18:23:51Z

+	return `You are analyzing a GitHub Pull Request to determine if it contains changes that would require updates to user-facing support documentation.

-The key question is: "Will users notice any difference in behavior, appearance, or functionality?"
+The key question is: "Would support documentation need to be updated to reflect these changes?"


It seems like we're basically asking the AI to guess what might be on the documentation site and what might not. I wonder how well that'll work out?

This may be completely infeasible, but could we point it at the documentation we're concerned with?

Things that seem like they'd make it infeasible include if some of the documentation is private for HEs, if the AI can't access the web, or if the AI can access the web but would get bogged down trying to access the whole documentation site.

Yeah, I think it's best to keep it general for now, and we can see how accurate it is. If it makes a lot of mistakes, we could consider giving it access to the support docs to start.

anomiex · 2026-01-23T18:28:55Z

-		debug( `check-if-docs-needed: PR #${ number } title contains "revert". Skipping.` );
+	// Skip if PR title starts with "Revert" (the standard GitHub revert format).
+	// This avoids false positives like "Undo revert and fix..." or "New UI for post revert feature".
+	if ( /^revert\b/i.test( title ) ) {


Should we cover a few more cases like

Suggested change

if ( /^revert\b/i.test( title ) ) {

if ( /^revert(|ed|ing)\b/i.test( title ) ) {

Looking at previous revert PRs in this repo, I think just testing for "Revert" is enough, that seems to be used every time.

anomiex · 2026-01-23T18:32:33Z

+	// We keep diff header lines (diff --, +++, ---, @@, etc.) regardless of length to avoid
+	// breaking the diff format. Only actual content lines (starting with +, -, or space) are filtered.
+	// Note: This may also filter legitimate long lines like inline SVG or long markdown paragraphs,
+	// but such lines are typically not useful for AI analysis of user-facing changes.


SVGs might be relevant to evaluating whether a screenshot needs updating. Although above we only ask the AI about changed text that appears in screenshots.

anomiex · 2026-01-23T18:35:48Z

+				line.startsWith( 'diff --git' ) ||
+				line.startsWith( '---' ) ||
+				line.startsWith( '+++' ) ||
+				line.startsWith( '@@' ) ||
+				line.startsWith( 'index ' ) ||
+				line.startsWith( 'new file' ) ||
+				line.startsWith( 'deleted file' ) ||
+				line.startsWith( 'Binary files' )


I wonder whether this would be better as a heuristic for what to keep:

Line does not start with , +, -, or \ No newline.

Line starts with --- or +++ (or even --- a/ or +++ b/).

I'm thinking the existing list is maybe clearer, this way we know exactly what we keep, no exceptions.

anomiex · 2026-01-23T18:41:40Z

          slack_editorial_channel: ${{ vars.SLACK_EDITORIAL_CHANNEL }}
          slack_he_triage_channel: ${{ vars.SLACK_HE_TRIAGE_CHANNEL }}
          slack_quality_channel: ${{ vars.SLACK_QUALITY_CHANNEL }}
+          slack_product_ambassadors_channel: ${{ vars.SLACK_PRODUCT_AMBASSADORS_CHANNEL }}


Thanks for documenting the new variable at PCYsg-xsv-p2 ! 🙂

Copilot AI review requested due to automatic review settings January 21, 2026 11:12

jeherve added the [Status] Needs Review This PR is ready for review. label Jan 21, 2026

jeherve self-assigned this Jan 21, 2026

Copilot started reviewing on behalf of jeherve January 21, 2026 11:12 View session

github-actions Bot added [Action] Repo Gardening Github Action: manage PR and issues in your Open Source project Actions GitHub actions used to automate some of the work around releases and repository management Docs labels Jan 21, 2026

jeherve added Enhancement [Status] In Progress [Pri] Low and removed [Status] Needs Review This PR is ready for review. labels Jan 21, 2026

github-advanced-security AI found potential problems Jan 21, 2026

View reviewed changes

Comment thread projects/github-actions/repo-gardening/src/tasks/check-if-docs-needed/index.js Fixed

jeherve force-pushed the add/ai-ui-changes-check branch from 22f63b4 to f62b42c Compare January 21, 2026 11:16

jeherve force-pushed the add/ai-ui-changes-check branch from f62b42c to 4bab131 Compare January 21, 2026 11:17

Update docs to clarify where the openai key is used

df81e29

Copilot AI reviewed Jan 21, 2026

View reviewed changes

Potential fix for code scanning alert no. 550: Incomplete multi-chara…

1b96389

…cter sanitization Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

Copilot AI review requested due to automatic review settings January 21, 2026 11:44

Copilot started reviewing on behalf of jeherve January 21, 2026 11:45 View session

jeherve and others added 7 commits January 21, 2026 12:46

Update projects/github-actions/repo-gardening/src/tasks/check-if-docs…

9b67b68

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update projects/github-actions/repo-gardening/src/tasks/check-if-docs…

6bff8dc

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update projects/github-actions/repo-gardening/src/tasks/check-if-docs…

f1c182c

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update projects/github-actions/repo-gardening/src/utils/get-diff.js

47c6bf8

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update projects/github-actions/repo-gardening/src/tasks/check-if-docs…

7eebc9c

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update projects/github-actions/repo-gardening/src/utils/get-diff.js

c784b92

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update projects/github-actions/repo-gardening/src/tasks/check-if-docs…

85257fa

…-needed/index.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot AI reviewed Jan 21, 2026

View reviewed changes

jeherve requested a review from a team January 21, 2026 15:22

jeherve added [Status] Needs Review This PR is ready for review. and removed [Status] In Progress labels Jan 21, 2026

anomiex requested review from StefMattana and bobmatyas January 21, 2026 17:25

Use product ambassadors channel for UI changes notifications

b3ddd0d

Replace slack_quality_channel with slack_product_ambassadors_channel for the check-if-docs-needed task to send notifications to the #jetpack-product-ambassadors channel instead.

Copilot AI review requested due to automatic review settings January 21, 2026 17:53

Copilot started reviewing on behalf of jeherve January 21, 2026 17:53 View session

anomiex reviewed Jan 21, 2026

View reviewed changes

Copilot AI reviewed Jan 21, 2026

View reviewed changes

jeherve added 9 commits January 22, 2026 10:37

Update AI prompt to focus on documentation needs

38cc325

The prompt previously asked about user-facing changes in general. Updated to specifically ask whether support documentation would need to be updated, which is the actual goal of this feature.

Fix terminology consistency in documentation

b0872d2

Update references from "quality channel/team" to "product ambassadors channel/team" to match the actual parameter name `slack_product_ambassadors_channel`.

Merge remote-tracking branch 'origin/trunk' into add/ai-ui-changes-check

22ba15c

Remove button mention from prompt

924d5dc

Copilot AI review requested due to automatic review settings January 22, 2026 09:57

Copilot started reviewing on behalf of jeherve January 22, 2026 09:57 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

Comment thread projects/github-actions/repo-gardening/src/tasks/check-if-docs-needed/index.js

anomiex approved these changes Jan 23, 2026

View reviewed changes

Merge remote-tracking branch 'origin/trunk' into add/ai-ui-changes-check

dd1f379

jeherve merged commit 04d65d0 into trunk Jan 29, 2026
69 checks passed

jeherve deleted the add/ai-ui-changes-check branch January 29, 2026 14:51

github-actions Bot added [Status] UI Changes Add this to PRs that change the UI so documentation can be updated. and removed [Status] Needs Review This PR is ready for review. labels Jan 29, 2026

		content = content.replace( /\\/g, '\\\\' );

		return content.replace( /`/g, '\\`' );


		return `You are analyzing a GitHub Pull Request to determine if users will experience something DIFFERENT after this change is deployed.

		The key question is: "Will users notice any difference in behavior, appearance, or functionality?"

	if ( /^revert\b/i.test( title ) ) {
	if ( /^revert(\|ed\|ing)\b/i.test( title ) ) {

Conversation

jeherve commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes:

Bailout conditions:

Other information:

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

Uh oh!

github-actions Bot commented Jan 21, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StefMattana commented Jan 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeherve commented Jan 21, 2026 •

edited

Loading