Split test targets and add nightly runs

[Miyamura80]

Summary

• add explicit fast/slow/nondeterministic test targets and repeat fast tests for flaky detection
• run fast tests in PR CI and add nightly workflow for slow/nondeterministic suites
• document new test targets in CLAUDE.md

Testing

• make test
• make test_fast
• make test_flaky
• make test_slow
• make test_nondeterministic
• make agents_validate
• make ci

[greptile-apps]

Greptile Overview

Greptile Summary

This PR implements a comprehensive test organization strategy by splitting tests into explicit categories (fast, slow, nondeterministic) and creating targeted CI workflows. The changes enable faster PR feedback by running only fast tests in the main CI pipeline, while moving slower and nondeterministic tests to a nightly schedule. The implementation includes:

• New test_fast, test_slow, and test_nondeterministic Make targets with proper pytest marker filtering
• Graceful handling of empty test collections (exit code 5) in slow/nondeterministic targets
• Separation of agents_validate into its own workflow that triggers only on relevant file changes
• Nightly workflow scheduled at 2 AM UTC for comprehensive test coverage
• Updated documentation in CLAUDE.md for all new test targets

The marker-based approach aligns with the existing test infrastructure (pytest.ini has the markers registered, test_template.py provides the decorators). Previous comments about "unreachable code" on lines 151 and 161 are incorrect - the exit $$status statements are properly guarded by the if condition and will execute when status is not 5.

Confidence Score: 5/5

• This PR is safe to merge with no blocking issues
• The implementation is well-designed with proper separation of concerns, correct shell scripting logic, follows existing patterns in the codebase, and includes comprehensive documentation. All workflow files use appropriate action versions and follow GitHub Actions best practices.
• No files require special attention

Important Files Changed

Filename	Overview
.github/workflows/test_target_tests.yaml	Updated to run fast tests only in PR CI, upgraded checkout/setup-python to v4/v5, removed agents_validate (moved to separate workflow)
.github/workflows/nightly_tests.yaml	New nightly workflow for slow/nondeterministic tests scheduled at 2 AM UTC, includes all required API keys
Makefile	Added test_fast, test_slow, test_nondeterministic targets with proper marker filtering and exit code handling for empty test collections

[greptile-apps]

_{2 files reviewed, 2 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

_{4 files reviewed, 4 comments}

_{Edit Code Review Agent Settings | Greptile}

[greptile-apps]

use actions/checkout@v4 for consistency with test_target_tests.yaml

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review.
Path: .github/workflows/agents_validate.yaml
Line: 26:26

Comment:
use `actions/checkout@v4` for consistency with `test_target_tests.yaml`

<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

use actions/setup-python@v5 for consistency with test_target_tests.yaml

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review.
Path: .github/workflows/agents_validate.yaml
Line: 38:38

Comment:
use `actions/setup-python@v5` for consistency with `test_target_tests.yaml`

<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

use actions/checkout@v4 for consistency with test_target_tests.yaml

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review.
Path: .github/workflows/nightly_tests.yaml
Line: 13:13

Comment:
use `actions/checkout@v4` for consistency with `test_target_tests.yaml`

<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>

How can I resolve this? If you propose a fix, please make it concise.

[greptile-apps]

use actions/setup-python@v5 for consistency with test_target_tests.yaml

_{Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!}

Prompt To Fix With AI

This is a comment left during a code review.
Path: .github/workflows/nightly_tests.yaml
Line: 25:25

Comment:
use `actions/setup-python@v5` for consistency with `test_target_tests.yaml`

<sub>Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!</sub>

How can I resolve this? If you propose a fix, please make it concise.