Skip to content

alwaysSaveActualImage: false does not respect comparison threshold larger than 0 #1111

@BosseKarat

Description

@BosseKarat

Environment (please complete the following information):

  • Node.js version: 22.21.1
  • WebdriverIO version: 9.23.2
  • @wdio/visual-service version: 9.1.1

Config of WebdriverIO + @wdio/visual-service

alwaysSaveActualImage: false

Describe the bug
If a comparison result sits somewhere between 0 and the threshold that I have added, an actual image is still generated. Let's say that I have toBeLessThanOrEqual(0.1) and the comparison results in a 0.05 difference, the test will be marked as passed AND an actual image is created

To Reproduce

    it('checkscreen test', async function () {
        await browser.url('https://webdriver.io/docs/api');

        await expect(
            await browser.checkScreen('A TEST CHECKSCREEN', {
            })
        ).toBeLessThanOrEqual(0.9);
    });

Make some adjustments to the baseline image. The adjustments should not be greater than 0.9 so make sure that the test still pass. Once executed with the test passing, you should still have an actual image despite alwaysSaveActualImage: false

Expected behavior
alwaysSaveActualImage: false should respect the threshold set by the user and not create an actual image if the test above does not fail.

Log
n/a

Additional context
n/a

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions