Fix: Prevent flaky tests with random values in parametrize#3
Fix: Prevent flaky tests with random values in parametrize#3MKaczkow merged 2 commits intoMKaczkow:masterfrom
Conversation
📝 WalkthroughWalkthroughA pytest fixture was added to the test file that seeds Python's Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
testing/pytest/test_employee_pytest.py (1)
33-60:⚠️ Potential issue | 🟠 MajorSeed before parametrization, not in a fixture.
pytestfixture bodies run after collection, but therandom.randint()calls on lines 52–60 are evaluated when the decorator arguments are built during module import. This fixture does not make these parameters deterministic.Proposed fix: use a local seeded RNG for parametrized values
-# fixture to ensure reproducible random values -# without seed, random.randint() produces different values each run (flaky tests) -@pytest.fixture(scope="session", autouse=True) -def set_random_seed(): - """make tests reproducible by seeding the random number generator.""" - random.seed(42) - yield +_random_values = random.Random(42) +_random_test_cases = [ + (_random_values.randint(0, 100),) * 2 + for _ in range(9) +] @@ - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), - tuple([random.randint(0, 100)] * 2), + *_random_test_cases,🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@testing/pytest/test_employee_pytest.py` around lines 33 - 60, The parametrized random values are generated at import time so the set_random_seed fixture doesn't make them deterministic; replace those random.randint(...) calls in the pytest.mark.parametrize argument with values produced from a local seeded RNG (e.g. create seeded_rng = random.Random(42) at module scope just above the pytest.mark.parametrize and call seeded_rng.randint(0, 100) for each tuple) and keep the set_random_seed fixture as-is; this ensures deterministic params while keeping set_random_seed for other tests, and refer to the existing set_random_seed fixture and the pytest.mark.parametrize block when making the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@testing/pytest/test_employee_pytest.py`:
- Around line 33-60: The parametrized random values are generated at import time
so the set_random_seed fixture doesn't make them deterministic; replace those
random.randint(...) calls in the pytest.mark.parametrize argument with values
produced from a local seeded RNG (e.g. create seeded_rng = random.Random(42) at
module scope just above the pytest.mark.parametrize and call
seeded_rng.randint(0, 100) for each tuple) and keep the set_random_seed fixture
as-is; this ensures deterministic params while keeping set_random_seed for other
tests, and refer to the existing set_random_seed fixture and the
pytest.mark.parametrize block when making the change.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: fb7b33e4-850c-483a-a1e0-005c4ed7b347
📒 Files selected for processing (1)
testing/pytest/test_employee_pytest.py
|
Yep, thanks for the fix. |
Hey! This PR is part of a research project on Python projects.
I noticed this repository focuses on learning Python concepts, so I wanted to help highlight a common issue in tests.
The test
test_very_simpleuses random values inparametrize. Without a fixed seed, this can lead to flaky tests (different numbers on each run), which makes debugging harder.I understand that in this specific case nothing will break, since the test has only a simple assertion. Still, Im proposing this change as a best practice and as a learning opportunity.
This PR adds a fixture to set a random seed, making tests deterministic while preserving the idea of testing with multiple values.
Summary by CodeRabbit
Release Notes