[🐛 Bug]: [java] NoSuchElementException no longer thrown when using custom By selectors in 4.39+ (now WebDriverException of "unexpected driver response: null")

[sglajchEG]

Description

We have custom By selectors, for instance By.JQuery. We do this by subclassing the selenium By class. This has worked for a good sized set of tests for a while for us, but 4.39 brought breaking changes. Basically in negative match changes, instead of "NoSuchElementException" being thrown, it throws "WebDriverException" with message "unexpected driver response: null".

This is due to a change in ElementLocation where it used to return the null element (which later gets thrown as NoSuchElementException), but now throws a new message specifically.

The change was in this PR:
9667310#diff-7d37876fdc2e3164ace49703dd9b48334c2ec22ba191b0f522ed2a3aa65cf433R210

Now I realize that subclassing "By" might not be the most common use case, but I feel like it's something others would have done for similar reasons.

For now, as a workaround, we can change our "By" implementation to throw the NoSuchElementException, but I think the change may be worth reverting because in the current Selenium internal code paths, it seems that there's no one place that's responsible for throwing NoSuchElementException. Sometimes it happens in By, and sometimes in other functions.

Below I've given our (minimal) By selector addition for our jquery selector.
You can reproduce this issue by doing driver.findElement(By.JQuery("#idThatDoesntExistHere"))

For now, I've worked around this locally by adding a manual throw of NoSuchElementException within our jquery by findelement calls.

Reproducible Code

package com.apptegic.selenium.utils

import org.openqa.selenium.JavascriptExecutor
import org.openqa.selenium.SearchContext
import org.openqa.selenium.WebElement
import org.openqa.selenium.remote.RemoteWebElement
import org.openqa.selenium.By as SeleniumBy

class By extends SeleniumBy {

    static SeleniumBy JQuery(String selector) {
      return new ByJQuery(selector)
    }

    @Override
    List<WebElement> findElements(SearchContext context) {
        super.findElements(context)
    }

    static class ByJQuery extends SeleniumBy {
        public String selector

        ByJQuery(String selector) {
            this.selector = selector
        }

        @Override
        List<WebElement> findElements(SearchContext context) {
            JavascriptExecutor jsExecutor = (JavascriptExecutor) getWebDriver(context)
            List<WebElement> selectedElements
            if (context instanceof RemoteWebElement) {
                selectedElements = (List<WebElement>) jsExecutor.executeScript("""return jQuery(arguments[0]).find("${selector}").toArray();""", context)
            } else {
                selectedElements = (List<WebElement>) jsExecutor.executeScript("""return jQuery("${selector}").toArray();""")
            }
            return selectedElements
        }

        @Override
        WebElement findElement(SearchContext context) {
            JavascriptExecutor jsExecutor = (JavascriptExecutor) getWebDriver(context)
            WebElement selectedElement
            if (context instanceof RemoteWebElement) {
                selectedElement = (WebElement) jsExecutor.executeScript("""return jQuery(arguments[0]).find("${selector}").get(0);""", context)
            } else {
                selectedElement = (WebElement) jsExecutor.executeScript("""return jQuery("${selector}").get(0);""")
            }
            return selectedElement
        }

        @Override
        String toString() {
            return "By.JQuery: " + selector
        }
    }
}

---

ℹ️ Last known working version: 4.38

[selenium-ci]

@sglajchEG, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide

[github-actions]

⚠️ You reported using Selenium version 4.39, but the latest release is 4.41.

Please verify that this issue still occurs with the latest version. If it no longer applies, you can close this issue or update your comment.

This issue will be marked "awaiting answer" and may be closed automatically if no response is received.

[selenium-ci]

@sglajchEG, thank you for creating this issue. We will troubleshoot it as soon as we can.

Selenium Triage Team: remember to follow the Triage Guide