CI: make the nightly sanitizer sweep pass (ASAN gate, TSAN informational)

[ChaoWao]

Summary

The first nightly Sanitizers run (after #915) failed 4/4 cells. This fixes the workflow so ASAN passes reliably and the build/run plumbing actually works.

Root causes & fixes

1. gcc not found (TSAN cells errored at setup). Two layers:
   • sanitizers.yml installed only g++-15, so the unversioned gcc/g++ that _ensure_host_compilers checks were missing on the ubuntu-24 runner → install build-essential + gcc-15 g++-15 + symlink fallbacks.
   • runtime_compiler._find_executable spawned which, which aborts under LD_PRELOAD=libtsan.so (the preloaded runtime kills the uninstrumented child), falsely reporting gcc missing → switched to in-process shutil.which. (This was the real reason TSAN cells died instantly at setup.)
2. ASAN full-suite TIMEOUT @ 1200s. The full examples tests/st livelocks under ASAN — oversubscription-heavy spmd stress cases on a 4-vCPU runner (documented in sim-oversubscription-hang.md). → scope the run to tensormap_and_ringbuffer dynamic_register + prepared_callable, --max-parallel 2, excluding parallel_broadcast and the dlopen_count tests (those assert exact dlopen accounting that the sanitizers perturb).
3. TSAN runs hang even scoped (its ~5-15x slowdown vs ASAN's ~1.7x livelocks the threaded scheduler). → continue-on-error on tsan cells: build validated, run best-effort; ASAN gates the job.

Validated on Linux (docker)

• ASAN scoped run green (prepared_callable + dynamic_register, dlopen_count/parallel_broadcast excluded)
• TSAN builds; the _find_executable fix lets it get past setup (run is informational)
• _find_executable → shutil.which keeps test_sanitizers + unit suite green; pre-commit clean

Testing

• Local docker validation (above)
• Next nightly confirms the ASAN cells go green (TSAN informational)

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d2f7d306-c751-444e-be79-2311309b092a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR hardens nightly sanitizer testing by treating TSAN as best-effort while keeping ASAN as the gating path, refining the test suite to architecture-specific targets with lower parallelism and tighter resource constraints, improving compiler detection to handle gcc-15 symlinks, and switching compiler discovery from subprocess-based which to shutil.which to avoid process aborts under sanitizer runtimes.

Changes

Sanitizer Workflow & Compiler Discovery

Layer / File(s)	Summary
TSAN/ASAN Error Handling & Gating .github/workflows/sanitizers.yml	TSAN matrix job is configured as best-effort with continue-on-error, while ASAN remains failure-gating for the workflow.
Compiler Toolchain Setup with GCC-15 Symlinks .github/workflows/sanitizers.yml	Compiler setup step installs build-essential, ensures gcc-15/g++-15 are available with fallback, and adds symlinks to guarantee unversioned gcc-15/g++-15 commands resolve correctly.
Constrained Sanitized Test Execution .github/workflows/sanitizers.yml	Test execution is refocused to specific tensormap_and_ringbuffer TARGETS with optional dynamic_register, preloads sanitizer runtime, limits devices to 0-7, caps parallelism to 2, excludes problematic tests, and reduces session timeout from 1200 to 600.
Documentation of Sanitizer Sweep docs/ci.md	CI documentation is expanded with specific details of sanitizer-sim runtime paths, LD_PRELOAD gating, test exclusions, parallelism limits, and ASAN/TSAN behavioral differences.
Compiler Discovery using shutil.which simpler_setup/runtime_compiler.py	_find_executable replaces manual file checks and subprocess-spawned which with shutil.which(name) to avoid sanitizer-instrumented process aborts under uninstrumented child processes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

🐰 A workflow refined with grace,
TSAN now best-effort's pace,
Compiler symlinks true and clear,
Test paths narrowed, focused here,
Shutil.which avoids the crash—
Sanitizers tamed in a dash!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: making the nightly sanitizer sweep pass with ASAN as the gate and TSAN as informational, which is the primary objective of this PR.
Description check	✅ Passed	The description provides a detailed explanation of root causes, fixes, and validation steps directly related to the changeset, covering all modified files and their purposes.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request updates the CI documentation to describe the scoped subset of tests run under ASAN/TSAN sanitizers to prevent livelocks, and replaces the external which subprocess call in _find_executable with shutil.which to avoid issues with preloaded sanitizer runtimes. The review feedback suggests simplifying _find_executable by removing redundant file and access checks since shutil.which handles them, and correcting the CMake option name in the documentation from SIMPLER_SANITIZER to SIMPLER_SANITIZERS.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/ci.md`:
- Around line 57-60: Update the docs so the pytest filter and target-selection
text matches the workflow: state that the scoped subset selects tests under
LD_PRELOAD including tensormap_and_ringbuffer and prepared_callable, that
dynamic_register is included only when present (not always), and explicitly
mention that dlopen_count is excluded; also note the use of --max-parallel 2 and
the exclusion of parallel-broadcast cases (a2a3sim/a5sim, ubuntu-only) to match
the actual configuration.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 4839d35f-5359-4ab1-ab75-b65ff126272e

📥 Commits

Reviewing files that changed from the base of the PR and between 39c0445 and 386dd54.

📒 Files selected for processing (3)

• .github/workflows/sanitizers.yml
• docs/ci.md
• simpler_setup/runtime_compiler.py