CI: make the TSAN nightly cell gate (scoped light subset + report races)

[ChaoWao]

Summary

Turns the TSAN nightly cell from continue-on-error (best-effort, was hanging) into a real gating job, after investigating why it hung — rather than filing it as a follow-up.

Finding (docker, Linux): TSAN does not deterministically deadlock. A single light L2 test (prepared_callable::test_run) passes in ~6.5s and surfaces 66 ThreadSanitizer warnings. The nightly hang was oversubscription — TSAN's ~5-15× slowdown throttles the chip-fork L3 cases' internal AICPU/AICore/host threads into a livelock that --max-parallel (cross-case only) can't help — and halt_on_error=1 aborted on the first reported race.

Fix:

• TSAN: scope to the light prepared_callable L2 tests (still surface races), --max-parallel 1, TSAN_OPTIONS=halt_on_error=0 (report races without aborting). Validated green: 2 passed, 66 warnings, no hang.
• ASAN: unchanged scope (prepared_callable + dynamic_register), --max-parallel 2.
• Drop continue-on-error — both sanitizers now gate (on hang/crash).
• Latent bug fixed: a comment block sat between the TSAN_OPTIONS=… \ env prefix and pytest, breaking the \ continuation so pytest would have run without LD_PRELOAD. Restructured to an if/else with exported *_OPTIONS and LD_PRELOAD on the pytest line.

Follow-up (noted, not this PR): triage the 66 reported races into a TSAN suppressions file, then flip halt_on_error=1 so the job gates on new races.

Testing

• docker (Linux): TSAN prepared_callable light subset green (2 passed, 66 warnings); ASAN broad subset green
• YAML parses; bash -n on the run step clean; pre-commit green

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c2555a27-2405-409b-a57c-030bdf8db113

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR refactors the nightly sanitizer workflow to establish both ASAN and TSAN as gating builds, restructure test execution via sanitizer-specific variables (restricting TSAN to serial prepared_callable runs while expanding ASAN to include dynamic_register with controlled parallelism), and updates documentation to match the new scoping and gating policy.

Changes

Sanitizer nightly sweep refactoring

Layer / File(s)	Summary
Gating policy and compiler notes .github/workflows/sanitizers.yml	Comments clarify g++-15 ABI matching, that sanitizer sweeps are not PR gates, and detect_leaks=0 handling. Removed matrix-based continue-on-error behavior; both ASAN and TSAN now gate results.
Sanitizer-specific test execution .github/workflows/sanitizers.yml	Refactored pytest step to branch on ${{ matrix.sanitizer }} via shell variables: TSAN uses prepared_callable targets with serial --max-parallel 1, filters dlopen_count, and sets TSAN_OPTIONS=halt_on_error=0; ASAN optionally adds dynamic_register, uses MAXPAR=2, filters parallel_broadcast and dlopen_count, and sets ASAN_OPTIONS/UBSAN_OPTIONS.
Nightly sanitizer documentation docs/ci.md	Updated to document scoped test subsets per LD_PRELOAD, dlopen_count exclusion, separate ASAN/TSAN test scopes and parallelism, and clarified that both sanitizers gate without continue-on-error.

Possibly related PRs

• hw-native-sys/simpler#931: Both PRs modify .github/workflows/sanitizers.yml to narrow the nightly sanitizer test targets and execution filters per sanitizer (e.g., prepared_callable/dynamic_register, excluding parallel_broadcast/dlopen_count, and using LD_PRELOAD with computed --max-parallel/-k), though the retrieved PR keeps TSAN best-effort while the main PR gates both.

• hw-native-sys/simpler#915: Both PRs adjust the nightly sanitizer "sanitizer-sim" CI behavior/documentation (main PR refactors sanitizers.yml execution/options and updates docs/ci.md, while retrieved PR introduces the sanitizer-sim scheduled job and also updates docs/ci.md).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 Sanitizers aligned, no longer astray,
Both ASAN and TSAN now gate the way,
TSAN's serial dance, ASAN spreads wide,
With docs to match—no more hide,
The nightly sweep runs with purposeful stride! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and specifically summarizes the main change: making TSAN nightly a gating job with a scoped light test subset and race reporting.
Description check	✅ Passed	The description comprehensively explains the problem (TSAN hangs due to oversubscription), the root cause investigation, the implemented fixes, and validation results—all directly related to the changeset.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request updates the documentation in docs/ci.md to clarify the configuration and execution of the nightly ASAN and TSAN sanitizer simulation jobs, detailing their respective parallelism limits, test scopes, and gating behaviors. The review feedback focuses on improving the clarity, readability, and technical writing style of the updated documentation, suggesting minor phrasing adjustments such as clarifying 'Both gate' and correcting colloquial expressions.