[CI] Add Buildkite step to validate changelog PR links by mrodm · Pull Request #19171 · elastic/integrations

mrodm · 2026-05-22T13:41:42Z

Proposed commit message

[CI] Add Buildkite step to validate changelog PR links

Add a Buildkite step and script to verify that newly added changelog.yml
entries in a PR point to the correct PR link. Issue links are accepted as-is.
The validation logic is extracted into testable shell functions with a full
test suite. The shell scripts test runner is also refactored into a thin
orchestrator with each script having its own dedicated test file.

WHAT:

Add a new Buildkite CI step that inspects changelog.yml files modified in a pull request and verifies that newly added link: entries point to the current PR URL. The check:

Detects any changelog.yml file changed at any directory depth (supporting both flat and nested package layouts).
Accepts issue links (/issues/<n>) without requiring them to match the PR URL.
Posts a Buildkite annotation when a mismatch is found.
Only runs on PR builds and only when at least one changelog.yml is part of the changed files (if_changed).

The validation logic is extracted into discrete, testable shell functions (github_repo_path, get_added_links, check_changelog_file) with a BASH_SOURCE main guard so the script can be sourced by tests without executing.

A dedicated test file (test_check_changelog_entries.sh) is added covering: SSH/HTTPS URL parsing, correct PR links, wrong PR links, issue link exemption, no new entries, deleted files, and multiple-error accumulation.

As part of this work, the existing run_dev_scripts_tests.sh is refactored into a thin orchestrator (run_shell_scripts_tests.sh) and the get_release_commit.sh tests are moved to their own self-contained file (dev/scripts/test_get_release_commit.sh), following the same pattern.

WHY:

Changelog entries with wrong PR links (e.g. copied from another PR) are a recurring source of noise in the release process. Automating this check at CI time catches the mistake before merge without requiring manual review of every changelog entry.

Author's Checklist

How to test this PR locally

Run the shell script test suite:

.buildkite/scripts/run_shell_scripts_tests.sh

To simulate the changelog link check against a PR branch manually (this is based on the remote origin and it must be updated):

BUILDKITE_PULL_REQUEST=<pr_number> \
BUILDKITE_PULL_REQUEST_BASE_BRANCH=main \
BUILDKITE_REPO=git@github.com:elastic/integrations.git \
.buildkite/scripts/check_changelog_entries.sh

Related issues

Relates Improve package backport tooling for routine multi-version support #19016

Screenshots

Buildkite annotation describing the failures:

GitHub comment with the failures:

Set pipeline as soft_fail:

GitHub comment after running successfully:

This PR was generated with the assistance of Claude (claude-sonnet-4-6).

Add a new script `.buildkite/scripts/check_changelog_entries.sh` that inspects any `changelog.yml` files modified in a pull request and verifies that newly added `link:` entries point to the current PR. Issue links (`/issues/<n>`) are accepted as-is. A matching Buildkite step is wired into `.buildkite/pipeline.yml`, gated on PR builds and triggered only when a `changelog.yml` file is changed (`if_changed`). Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Extract the validation logic of check_changelog_entries.sh into discrete, testable functions (github_repo_path, get_added_links, check_changelog_file) guarded by a BASH_SOURCE main guard so the script can be sourced by tests without executing. Fix the common.sh source path to be relative to the script's own directory. Add test_check_changelog_entries.sh covering URL parsing, correct PR links, wrong PR links, issue link exemption, no new entries, deleted files, and multiple-error accumulation. Hook the new test file into run_dev_scripts_tests.sh so it runs in the existing CI step. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Extract all tests for get_release_commit.sh from run_dev_scripts_tests.sh into a dedicated dev/scripts/test_get_release_commit.sh, following the same pattern established for test_check_changelog_entries.sh. The runner run_dev_scripts_tests.sh is now a thin orchestrator that delegates to each test file in sequence. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Rename run_dev_scripts_tests.sh to run_shell_scripts_tests.sh and update the Buildkite step label and key accordingly (check-shell-scripts). Update all references in common.sh and the test file comments. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

elastic-vault-github-plugin-prod · 2026-05-22T13:59:33Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

mrodm · 2026-05-22T14:50:48Z

/test

…iles Refactor notify_changelog_mismatch to accept a combined message and a fixed annotation context, then accumulate all failing changelog files into an array during the loop and call the function once after it with a bulleted list of every affected file instead of one notification per file. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add a depends_on on check-shell-scripts (allow_failure: false) to the check-changelog-pr-links step so it only runs once the shell unit tests have completed successfully. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add edit_gh_pr_comment_if_exists to common.sh next to add_or_edit_gh_pr_comment: it edits an existing PR comment identified by its marker, and does nothing if no matching comment is found. Use it in check_changelog_entries.sh to update the mismatch comment with a resolved message when all changelog links are correct, avoiding a spurious success comment on clean runs. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add delete_and_create_gh_pr_comment to common.sh: it deletes any existing comment with the given marker and posts a fresh one, moving the status to the bottom of the PR thread for visibility. Use it in check_changelog_entries.sh on the success path instead of edit_gh_pr_comment_if_exists, so the resolved message appears as a new comment rather than an in-place edit. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Replace add_or_edit_gh_pr_comment with delete_and_create_gh_pr_comment in notify_changelog_mismatch so that repeated failures always post a fresh comment at the bottom of the PR thread instead of editing in place. Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

- Fix skip-label check using wrong label string ('skip-changelog' instead of 'changelog-link-check:skip'); extract into should_skip_changelog_check() with tests - Fix unquoted word-splitting in changelog file loop; use while/read - Add --repo flag to gh pr comment in delete_and_create_gh_pr_comment() - Add check_changelog_entries.sh and test file to non_package_patterns - Guard dev/scripts copy in backport_branch.sh with git ls-tree existence check Made with Claude

Made with Claude

elastic-vault-github-plugin-prod · 2026-05-25T14:04:12Z

Changelog link mismatch — expected https://github.com/elastic/integrations/pull/19171 in the following file(s):

packages/elastic_package_registry/changelog.yml

View Buildkite build
@mrodm

mrodm · 2026-05-25T14:47:42Z

Moved tests from ".buildkite/scripts/run_dev_scripts_tests.sh"

mrodm · 2026-05-25T15:10:03Z

+        allow_failure: false
+    if: |
+      build.env('BUILDKITE_PULL_REQUEST') != "false" &&
+      !(build.env("GITHUB_PR_LABELS") =~ /changelog-link-check:skip/)


What about this label ? Any other suggestion?

There is already a skip-changelog but I think it could be misunderstood the expected actions.

mrodm · 2026-05-25T15:12:01Z

+    while IFS= read -r link; do
+        [[ -z "${link}" ]] && continue
+        if [[ "${link}" =~ /issues/[0-9]+$ ]]; then
+            echo "SKIP: '${link}' (issue link, not required to match PR)"


Allowed to set links to issues. But if this is done, it does not validate anything.
This validation is intended (as it is) for pull request links.

Made with Claude

…_link

mrodm · 2026-05-25T16:18:13Z

/test

elasticmachine · 2026-05-25T16:36:16Z

💔 Build Failed

Buildkite Build
Commit: 9b01ff8

Failed CI Steps

Check integrations netbox

History

💚 Build #43454 succeeded 493516a
💚 Build #43453 succeeded 5dc45eb
💚 Build #43452 succeeded b94cb0c
💔 Build #43450 failed bbc4450
💔 Build #43435 failed 62dd76e

cc @mrodm

mrodm and others added 6 commits May 22, 2026 15:14

Test changelog

c298efa

Test subset of packages

0c20c86

mrodm self-assigned this May 22, 2026

This comment has been minimized.

Sign in to view

mrodm added 2 commits May 22, 2026 16:33

Add github comment

4c44845

Show error

d370bb9

This comment has been minimized.

Sign in to view

Add tools to post comments

261a3f1

This comment has been minimized.

Sign in to view

mrodm and others added 12 commits May 22, 2026 16:54

Add mention

f313165

Refactor notification into a new method

3f820f1

Add another newline

38b4069

Enable soft_fail in the new step

2953319

Test with two packages failing

7102c24

Add buildkite build link in github annotation

6585ae1

[CI] Add collapsible sections in Buildkite output for notifications

93feb1f

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

andrewkroh added Integration:elastic_package_registry Elastic Package Registry Integration:zipkin_input_otel Zipkin OpenTelemetry Input Package labels May 22, 2026

mrodm removed the skip-changelog label May 25, 2026

mrodm added 2 commits May 25, 2026 12:52

Show all changelog files with unexpected links in buildkite output too

1dc0983

Add collapsible section

62dd76e

github-actions Bot mentioned this pull request May 25, 2026

[cloudflare_logpush] fix conflicts in feature branch #19186

Open

mrodm added 3 commits May 25, 2026 15:15

[CI] Split and merge test runners by folder

e7f2775

Made with Claude

[CI] Fix code review findings in test runner refactor

bbc4450

Made with Claude

mrodm commented May 25, 2026

View reviewed changes

Restore packages

b94cb0c

mrodm commented May 25, 2026

View reviewed changes

mrodm added 3 commits May 25, 2026 17:15

[CI] Sort and group non_package_patterns in common.sh

5dc45eb

Made with Claude

Re-order parameters

493516a

[CI] Remove grep filter from list_all_directories

d1893d4

mrodm added ci and removed Integration:elastic_package_registry Elastic Package Registry Integration:zipkin_input_otel Zipkin OpenTelemetry Input Package labels May 25, 2026

Merge remote-tracking branch 'upstream/main' into add_check_changelog…

9b01ff8

…_link

mrodm marked this pull request as ready for review May 25, 2026 16:26

mrodm requested a review from a team as a code owner May 25, 2026 16:26

mrodm mentioned this pull request May 25, 2026

Improve package backport tooling for routine multi-version support #19016

Open

4 tasks

mrodm mentioned this pull request May 26, 2026

backport: add backport_apply.sh to apply a fix to a backport branch as a single command #19212

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Add Buildkite step to validate changelog PR links#19171

[CI] Add Buildkite step to validate changelog PR links#19171
mrodm wants to merge 40 commits into
elastic:mainfrom
mrodm:add_check_changelog_link

mrodm commented May 22, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented May 22, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

This comment has been minimized.

mrodm commented May 22, 2026

Uh oh!

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented May 25, 2026

Uh oh!

mrodm May 25, 2026

Uh oh!

mrodm May 25, 2026

Uh oh!

mrodm May 25, 2026

Uh oh!

mrodm commented May 25, 2026

Uh oh!

elasticmachine commented May 25, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mrodm commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

This comment has been minimized.

This comment has been minimized.

mrodm commented May 22, 2026

Uh oh!

This comment has been minimized.

elastic-vault-github-plugin-prod Bot commented May 25, 2026

Uh oh!

mrodm May 25, 2026

Choose a reason for hiding this comment

Uh oh!

mrodm May 25, 2026

Choose a reason for hiding this comment

Uh oh!

mrodm May 25, 2026

Choose a reason for hiding this comment

Uh oh!

mrodm commented May 25, 2026

Uh oh!

elasticmachine commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💔 Build Failed

Failed CI Steps

History

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrodm commented May 22, 2026 •

edited

Loading

elastic-vault-github-plugin-prod Bot commented May 22, 2026 •

edited

Loading

elasticmachine commented May 25, 2026 •

edited

Loading