Skip to content

[java] add test to prevent updating pinned chrome version before cdp#17425

Open
titusfortner wants to merge 1 commit intotrunkfrom
match_devtools
Open

[java] add test to prevent updating pinned chrome version before cdp#17425
titusfortner wants to merge 1 commit intotrunkfrom
match_devtools

Conversation

@titusfortner
Copy link
Copy Markdown
Member

💥 What does this PR do?

I don't think we should be testing later versions of Chrome than we support with CDP for a few reasons.

🔧 Implementation Notes

Add a python test to scripts so bazel should pick it up when the change is made

🤖 AI assistance

  • No substantial AI assistance used
  • AI assisted (complete below)
    • Tool(s): Codex
    • What was generated: regex/boilerplate
    • I reviewed all AI output and can explain the change

Copilot AI review requested due to automatic review settings May 8, 2026 10:39
@qodo-code-review
Copy link
Copy Markdown
Contributor

Review Summary by Qodo

Add test to validate Chrome versions against CDP support

🧪 Tests ✨ Enhancement

Grey Divider

Walkthroughs

Description
• Add Python test to validate pinned Chrome versions match CDP support
• Prevent accidental updates to Chrome versions unsupported by DevTools Protocol
• Export build files to enable test access to version configurations
• Integrate test into Bazel build system for automated validation

Grey Divider

File Changes

1. scripts/pinned_browsers_test.py 🧪 Tests +23/-0

New test for Chrome version CDP compatibility validation

• New Python test file that validates pinned Chrome versions in repositories.bzl match supported
 versions in versions.bzl
• Uses regex to extract Chrome major versions from repository URLs
• Raises AssertionError if any pinned Chrome version lacks corresponding CDP support
• Prevents regression where Chrome versions could be updated without CDP compatibility verification

scripts/pinned_browsers_test.py


2. common/BUILD.bazel ⚙️ Configuration changes +2/-0

Export repositories.bzl for test access

• Export repositories.bzl file to make it accessible to other build targets
• Enables the new test to read pinned Chrome version configurations

common/BUILD.bazel


3. java/src/org/openqa/selenium/devtools/BUILD.bazel ⚙️ Configuration changes +2/-0

Export versions.bzl for test access

• Export versions.bzl file to make it accessible to other build targets
• Enables the new test to read supported CDP versions

java/src/org/openqa/selenium/devtools/BUILD.bazel


View more (1)
4. scripts/BUILD.bazel ⚙️ Configuration changes +10/-1

Add Bazel test target for Chrome version validation

• Import py_test rule from @rules_python//python:defs.bzl
• Add new pinned_browsers_test target with test source and data dependencies
• Configure test to access exported repositories.bzl and versions.bzl files

scripts/BUILD.bazel


Grey Divider

Qodo Logo

@qodo-code-review
Copy link
Copy Markdown
Contributor

qodo-code-review Bot commented May 8, 2026

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0)

Grey Divider


Action required

1. Pinned Chrome not in CDP 🐞 Bug ≡ Correctness
Description
scripts/pinned_browsers_test.py will raise because common/repositories.bzl pins stable Chrome
milestone 148 while java devtools CDP_VERSIONS only includes 145–147, making
unmatched_chrome_versions non-empty. This makes the PR’s newly-added test fail in the current tree.
Code

scripts/pinned_browsers_test.py[R10-20]

+chrome_versions = set(
+    re.findall(
+        r'name = "(?:linux|mac)_chrome",\s+url = "[^"]+/(\d+)\.[^"]+/(?:linux64|mac-arm64)/chrome-[^"]+\.zip"',
+        repositories,
+    )
+)
+devtools_versions = set(re.findall(r'"v(\d+)"', versions))
+unmatched_chrome_versions = chrome_versions - devtools_versions
+
+if unmatched_chrome_versions:
+    raise AssertionError(
Evidence
The test extracts pinned stable Chrome milestone(s) from repositories.bzl and asserts they are
present in CDP_VERSIONS; currently repositories.bzl pins 148.x while versions.bzl lists only
v145–v147, so the set difference is non-empty and the AssertionError path is taken.

common/repositories.bzl[200-224]
java/src/org/openqa/selenium/devtools/versions.bzl[1-7]
scripts/pinned_browsers_test.py[10-20]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The newly added `//scripts:pinned_browsers_test` fails because the pinned stable Chrome major version extracted from `common/repositories.bzl` is not present in `CDP_VERSIONS`.

### Issue Context
Right now, stable Chrome is pinned to milestone 148 in `common/repositories.bzl`, while `CDP_VERSIONS` only includes v145–v147.

### Fix Focus Areas
- Update pinned stable Chrome milestone OR add corresponding CDP version so the test passes.
- file/path references:
 - common/repositories.bzl[200-276]
 - java/src/org/openqa/selenium/devtools/versions.bzl[1-7]
 - scripts/pinned_browsers_test.py[10-23]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Broken assertion interpolation 🐞 Bug ⚙ Maintainability
Description
The AssertionError message concatenates an f-string with a non-f string, so
"{sorted(devtools_versions)}" is not interpolated and the failure message is misleading. It also
reports sorted(chrome_versions) rather than the actual missing versions (unmatched_chrome_versions).
Code

scripts/pinned_browsers_test.py[R20-23]

+    raise AssertionError(
+        f"Stable pinned Chrome versions {sorted(chrome_versions)} must be present in "
+        "CDP_VERSIONS; found {sorted(devtools_versions)}. "
+    )
Evidence
Only the first literal is prefixed with f, so the second literal’s {sorted(devtools_versions)}
is treated as plain text; additionally, the message prints sorted(chrome_versions) even though the
conditional is based on unmatched_chrome_versions.

scripts/pinned_browsers_test.py[19-23]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The AssertionError message in `pinned_browsers_test.py` is partially not interpolated and reports the wrong set, reducing debuggability.

### Issue Context
The message is built via implicit string literal concatenation, but only the first literal is an f-string.

### Fix Focus Areas
- Make all concatenated parts f-strings (or build a single f-string)
- Report `unmatched_chrome_versions` rather than `chrome_versions`
- file/path references:
 - scripts/pinned_browsers_test.py[19-23]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Regex miss makes test no-op 🐞 Bug ☼ Reliability
Description
If the repositories.bzl format changes such that the regex finds no matches, chrome_versions becomes
empty and the test will pass without validating anything. This can silently disable the protection
the test is meant to provide.
Code

scripts/pinned_browsers_test.py[R10-18]

+chrome_versions = set(
+    re.findall(
+        r'name = "(?:linux|mac)_chrome",\s+url = "[^"]+/(\d+)\.[^"]+/(?:linux64|mac-arm64)/chrome-[^"]+\.zip"',
+        repositories,
+    )
+)
+devtools_versions = set(re.findall(r'"v(\d+)"', versions))
+unmatched_chrome_versions = chrome_versions - devtools_versions
+
Evidence
There is no guard that asserts at least one stable Chrome version was extracted before computing the
set difference; an empty chrome_versions yields an empty unmatched_chrome_versions regardless of
CDP_VERSIONS.

scripts/pinned_browsers_test.py[10-18]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The test can silently pass if the regex fails to match any pinned stable Chrome entries.

### Issue Context
`chrome_versions` is derived from a regex over `repositories.bzl`. If it becomes empty due to formatting changes, the set difference is empty and no assertion is raised.

### Fix Focus Areas
- Add an explicit assertion that `chrome_versions` is non-empty (with a helpful failure message)
- Optionally, tighten extraction by parsing more robustly (or matching both `linux_chrome` and `mac_chrome` blocks separately)
- file/path references:
 - scripts/pinned_browsers_test.py[10-18]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

@selenium-ci selenium-ci added C-java Java Bindings B-build Includes scripting, bazel and CI integrations B-devtools Includes everything BiDi or Chrome DevTools related labels May 8, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Bazel Python test to ensure the repo’s pinned stable Chrome milestone isn’t advanced beyond the Java CDP versions Selenium ships, so updating pinned browsers can’t silently get ahead of CDP support.

Changes:

  • Add //scripts:pinned_browsers_test to compare pinned stable Chrome milestones in common/repositories.bzl against CDP_VERSIONS in java/.../devtools/versions.bzl.
  • Wire the new test into scripts/BUILD.bazel as a py_test.
  • Export repositories.bzl and versions.bzl as files so they can be used as data inputs in Bazel.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
scripts/pinned_browsers_test.py New Bazel test that enforces pinned stable Chrome milestone is present in Java CDP_VERSIONS.
scripts/BUILD.bazel Adds the new py_test target for the pinned browser/CDP consistency check.
java/src/org/openqa/selenium/devtools/BUILD.bazel Exports versions.bzl for Bazel data consumption.
common/BUILD.bazel Exports repositories.bzl for Bazel data consumption.

Comment on lines +21 to +22
f"Stable pinned Chrome versions {sorted(chrome_versions)} must be present in "
"CDP_VERSIONS; found {sorted(devtools_versions)}. "
Comment on lines +10 to +20
chrome_versions = set(
re.findall(
r'name = "(?:linux|mac)_chrome",\s+url = "[^"]+/(\d+)\.[^"]+/(?:linux64|mac-arm64)/chrome-[^"]+\.zip"',
repositories,
)
)
devtools_versions = set(re.findall(r'"v(\d+)"', versions))
unmatched_chrome_versions = chrome_versions - devtools_versions

if unmatched_chrome_versions:
raise AssertionError(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Pinned chrome not in cdp 🐞 Bug ≡ Correctness

scripts/pinned_browsers_test.py will raise because common/repositories.bzl pins stable Chrome
milestone 148 while java devtools CDP_VERSIONS only includes 145–147, making
unmatched_chrome_versions non-empty. This makes the PR’s newly-added test fail in the current tree.
Agent Prompt
### Issue description
The newly added `//scripts:pinned_browsers_test` fails because the pinned stable Chrome major version extracted from `common/repositories.bzl` is not present in `CDP_VERSIONS`.

### Issue Context
Right now, stable Chrome is pinned to milestone 148 in `common/repositories.bzl`, while `CDP_VERSIONS` only includes v145–v147.

### Fix Focus Areas
- Update pinned stable Chrome milestone OR add corresponding CDP version so the test passes.
- file/path references:
  - common/repositories.bzl[200-276]
  - java/src/org/openqa/selenium/devtools/versions.bzl[1-7]
  - scripts/pinned_browsers_test.py[10-23]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

B-build Includes scripting, bazel and CI integrations B-devtools Includes everything BiDi or Chrome DevTools related C-java Java Bindings

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants