Wallaby tests started to fail suddenly both on CI and locally

[samuelpordeus]

Elixir and Erlang/OTP versions

Erlang/OTP 27 [erts-15.1] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Elixir 1.17.3 (compiled with Erlang/OTP 25)

Operating system

Mac & Linux box from CI

Browser

Chrome v133

Driver

ChromeDriver v133

Correct Configuration

• I confirm that I have Wallaby configured correctly.

Current behavior

Hey folks!

Last month, on Feb. 21, our wallaby tests suddenly started intermittently failing both locally for all of our team members and on CI with Stale Reference and various race-condition errors. Example:

 1) test *
     test/*
     ** (Wallaby.StaleReferenceError) The element you are trying to reference is stale or no longer attached to the
     DOM. The most likely reason is that it has been removed with JavaScript.

     You can typically solve this problem by using find to block until the DOM is in a
     stable state.
     
     code: |> Browser.text()
     stacktrace:
       (wallaby 0.30.10) lib/wallaby/element.ex:267: Wallaby.Element.raise_error/1

Approximately 60 out of 180 tests were failing each run. We get all sorts of different errors that look like race conditions, like HTML not being updated when it should.

On many occasions, adding something like Process.sleep(1000) fixes the tests. I also noticed that some tests might benefit from waiting until the page is properly rendered. But they were working flawlessly for 4+ years, so it didn't make much sense to update them without a solid reason.

After some digging, we found that locking Chrome and ChromeDriver versions to <=v127 resolves the issues completely. The thing is, nobody on the team upgraded Chrome or ChromeDriver. I was using v133 locally (v134 was already available), and it suddenly started to fail a bunch of tests for everyone both locally and on CI.

So we wondered if it's something like a date-based hard deprecation on the Chrome side or a security patch applied to previous versions, but I've looked through every release note from Chrome for Developers 127 to 134 (here), but couldn't find anything that caught my attention (I'm definitely not a pro when it comes to browser tests 😅)

I only wanted to open this issue after exhausting my options, but here I am 😓 – Let me know if you need more info. Thanks for your time!

Expected behavior

Although wallaby is working fine with Chrome & ChromeDriver on v127, we want to understand what happened and be able to unlock the version in the future.

Test Code & HTML

Some simple tests that were always passing started to randomly break:

test "Required financial info notice" do
  tenant = insert(:tenant, name: "Test Tenant")
  user = insert(:user, tenant_id: tenant.id)

  session =
    tenant.subdomain
    |> build_session()
    |> log_in(email: user.email)

  step "shows a notice" do
    assert has_text?(session, "Financial Information Required")
  end
end

def build_session(host, opts) do
  custom_headers = %{"Host" => host}

  metadata = Phoenix.Ecto.SQL.Sandbox.metadata_for(*.Repo, self())

  {:ok, session} =
    Wallaby.start_session(
      custom_headers: custom_headers,
      metadata: metadata,
      readiness_timeout: 60_000,
      startup_timeout: 60_000,
      window_size:
        if System.get_env("MOBILE") do
          [width: 500, height: 1000]
        else
          [width: 1440, height: 1000]
        end
    )

  session = Browser.visit(session, "http://#{host}")

  if opts[:with_extras?] == true do
    {session, metadata, custom_headers}
  else
    session
  end
end

Error:

  3) test Required financial info notice (*.*Test)
     test/*
     ** (Wallaby.StaleReferenceError) The element you are trying to reference is stale or no longer attached to the
     DOM. The most likely reason is that it has been removed with JavaScript.

     You can typically solve this problem by using `find` to block until the DOM is in a
     stable state.
     
     code: assert has_text?(session, "Financial Information Required")
     stacktrace:
       (wallaby 0.30.10) lib/wallaby/element.ex:267: Wallaby.Element.raise_error/1
       (wallaby 0.30.10) lib/wallaby/browser.ex:1095: anonymous fn/2 in Wallaby.Browser.has_text?/2
       (wallaby 0.30.10) lib/wallaby/browser.ex:148: Wallaby.Browser.retry/2
       (wallaby 0.30.10) lib/wallaby/browser.ex:1094: Wallaby.Browser.has_text?/2
       *:23: (test)

Demonstration Project

No response

[mhanberg]

Could you try and clone the wallaby repo and run the test suite using the suspected Chrome and Chromdriver versions?

I ran them locally with 134 and they all passed.

Should be able to clone, install deps, then run WALLABY_DRIVER=chrome mix test

[samuelpordeus]

........................
Finished in 114.5 seconds (89.1s async, 25.4s sync)
4 features, 255 tests, 0 failures

No failures at all. Same version that fails the tests for our app.

[mhanberg]

Very interesting

This is going to be hard to narrow down without a minimal reproduction case that i can run, if you could somehow create one, that would be awesome.

[samuelpordeus]

yeah, looks like it's going to be tough to understand what's happening without a good example indeed.

I'll see with the team here if we can cook something or other alternatives, since I don't think it's going to be trivial to reproduce it, and I'll update the thread here too.

Thanks, Mitch!

[samuelpordeus]

@mhanberg hey!

we'll have some time to try to replicate it in a demo app in roughly 30 days (2 cycle/sprints), so this will be sitting for a while.

Feel free to close it or leave it open in case anyone else experiences this 🙂

[begedin]

I can confirm I'm seeing the same. My CI uses latest chromedriver and that latest is failing locally for me as well, whereas I was green before the update.

[begedin]

Looks like this is related

teamcapybara/capybara#2800

Which leads me to this

https://issues.chromium.org/issues/402796660

[begedin]

A workaround here could be to set your CI (and local env) to use a custom version of Chrome/Chromedriver, rather than latest. This seems to get the tests into the regular flaky state for me :)

- id: chrome
  uses: browser-actions/setup-chrome@v1
  with:
    chrome-version: 134
    install-chromedriver: true

- name: Override default chrome
  run: sudo ln -sf ${{ steps.chrome.outputs.chrome-path }} /usr/bin/google-chrome