Skip to content

Integration tests should add Consistently checks after Eventually checks to reduce flaky DB behavior #13780

@hanahmily

Description

@hanahmily

Problem

Our integration tests currently rely on Eventually assertions, which verify that a condition becomes true within a timeout window.
This can still allow flaky behavior in database-related tests: a condition may become true briefly and then regress, causing intermittent failures
later in the test flow.

Proposal

After each critical Eventually check, add a follow-up Consistently check to ensure the condition remains stable for a short duration.

Why

  • Eventually confirms convergence.
  • Consistently confirms stability.
  • Using both reduces false positives and flaky failures caused by transient DB states.

Example

Eventually(func() bool {
    return conditionIsMet()
}, timeout, interval).Should(BeTrue())

Consistently(func() bool {
    return conditionIsMet()
}, stableWindow, interval).Should(BeTrue())

Acceptance Criteria

  • Add Consistently checks after key Eventually checks in integration tests that validate DB state.
  • Keep stability windows short but meaningful (enough to catch transient regressions).
  • CI flaky rate for these tests should decrease over time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    databaseBanyanDB - SkyWalking native database

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions