Skip to content

fix: do not exit non-zero on inconclusive bisect#409

Merged
kelhusseiny merged 1 commit intomainfrom
fix-bisect-inconclusive-exit-code
May 5, 2026
Merged

fix: do not exit non-zero on inconclusive bisect#409
kelhusseiny merged 1 commit intomainfrom
fix-bisect-inconclusive-exit-code

Conversation

@kelhusseiny
Copy link
Copy Markdown
Member

Summary

Treat inconclusive bisects as a successful diagnostic outcome instead of a build failure. Reserve non-zero exits for cases where bisect could not actually run.

Motivation

bisect_command currently exit! 1 in two cases that represent successful diagnostic runs:

  1. "The bisection was inconclusive, there might not be any leaky test here." — bisect ran, narrowed candidates, and on final validation the failure did not reproduce on the narrowed-down order. This is the expected outcome for genuinely flaky tests (timing, async, network) rather than order-dependent ones.
  2. "The failing test was the first test in the test order so there is nothing to bisect." — bisect ran and reported that there are no preceding tests to suspect.

Both are valid findings produced by a successful run, not failures.

In practice, callers wire rake test:bisect (or minitest-queue ... bisect directly) into a pipeline that runs leakbot on each detected flaky test. Most flaky tests in a large Rails codebase are flaky for non-order-dependent reasons (timing, shared external state, async). Bisect correctly returns "inconclusive" for those, but the non-zero exit code makes the pipeline report the run as failed even though bisect did exactly what it was supposed to do.

This also makes it impossible to distinguish these legitimate outcomes from genuine harness failures — for example, the lazy_load + non-streaming-queue regression fixed in #400, where bisect bailed immediately with "The failing test does not exist." because Minitest.loaded_tests was empty. Both surfaced as exit 1, so callers had no way to tell "bisect could not run" from "bisect ran and found nothing to pin down."

Changes

After this PR:

Outcome Exit code
Polluter found 0
Failing test fails in isolation 0
Inconclusive (failure did not reproduce) 0 (was 1)
Nothing to bisect (failing test is first) 0 (was 1)
FAILING_TEST missing / not present in the queue 1
Missing arguments 1

In short: exit 0 when bisect ran to completion, regardless of outcome. Exit non-zero only when bisect could not run.

The integration tests in test/integration/minitest_bisect_test.rb are updated to assert the new exit-code contract for each scenario.

Notes

  • User-visible output (the colored step lines, log/test_order.log, log/bisect_test_details.log) is unchanged, so callers can still distinguish outcomes by reading those if they care.
  • This is a behavior change for any caller that today reads exit code 1 as "no polluter found." I'm not aware of any — the typical caller is a CI step wrapper that just propagates the exit code as build status.

cc @andheiberg — follow-up to #400.

Bisect currently exits 1 in two cases that represent successful
diagnostic runs rather than failures:

  1. "The bisection was inconclusive, there might not be any leaky
     test here." - the bisect ran, narrowed candidates, and the
     failure did not reproduce on the final narrowed order. This is
     the expected outcome for genuinely flaky tests (timing, async,
     network) rather than order-dependent ones.

  2. "The failing test was the first test in the test order so there
     is nothing to bisect." - the bisect ran and reported that there
     are no preceding tests to suspect.

Both are valid findings produced by a successful run. Treating them as
build failures forces callers (e.g. CI pipelines that invoke bisect on
flaky tests) to permanently sit at a low pass rate even though bisect
is doing exactly what it is supposed to do, and makes it impossible to
distinguish these outcomes from real harness failures like the one
fixed in #400 (lazy_load leaving Minitest.loaded_tests empty so the
failing test could not be found).

After this change:

  exit 0 - bisect ran to completion (polluter found, inconclusive,
           failing test failed in isolation, or nothing to bisect)
  exit 1 - bisect could not run (FAILING_TEST not present in the
           queue, missing arguments, etc.)

Tests are updated to assert the contract for each scenario.
@kelhusseiny kelhusseiny marked this pull request as ready for review May 5, 2026 16:15
@kelhusseiny kelhusseiny merged commit 5fb5fde into main May 5, 2026
56 of 77 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants