[None][fix] Sync disagg KV transfer error handling under ADP across all executor loops

[qiaoxj07]

Summary

• Fixes an MPI collective-mismatch deadlock observed on DSv4 dep16 disagg gen.
• Adds _handle_disagg_cache_errors_synced and wires it into the top of all three executor loops (_executor_loop, _executor_loop_overlap, _executor_loop_pp).

Problem

With ADP enabled in disaggregated serving, a NIXL/UCX KV transfer failure is rank-local: only the failing DP rank sees DISAGG_TRANS_ERROR and enters _check_cache_transfer_errors → _handle_errors → _enqueue_responses → tp_gather. Peer ranks without a local error continue to _can_queue → tp_allgather. The two different TP collectives on the same group hang MPI; the hang detector then dumps stacks after 300s.

Repro log (dep16, batch4, eplb0, mtp3): only one rank in tp_gather from _check_cache_transfer_errors, the other 15 in tp_allgather from _can_queue (_executor_loop_overlap).

Approach

A new _handle_disagg_cache_errors_synced:

1. tp_allgather a single bool (does this rank have local error requests).
2. If no rank reports errors: return.
3. Otherwise all ranks call _handle_errors(...) together, each passing its own (possibly empty) local_error_requests.

Wired into the top of every iteration of _executor_loop, _executor_loop_overlap, and _executor_loop_pp so the downstream tp_gather in _enqueue_responses is reached by all ranks in lockstep.

_check_cache_transfer_errors becomes a no-op under ADP to avoid double-handling; the non-ADP path is unchanged.

Relationship to PR #13900

This is the same root cause and same approach as #13900 by @Shixiaowei02, narrowed to the deadlock fix and extended in two ways:

• [None][fix] Fix dis-agg with ADP and avoid TP collective deadlock #13900 only adds the synced call to _executor_loop. Our hang was observed in _executor_loop_overlap, so this PR also covers the overlap loop and _executor_loop_pp.
• This PR omits [None][fix] Fix dis-agg with ADP and avoid TP collective deadlock #13900's unrelated changes (_pending_transfer_responses for the success path, _adp_dummy_role refactor, _count_schedulable_active_requests GENERATION_TO_COMPLETE exclusion, _pad_attention_dp_dummy_request token_nums, NIXL diagnostic logging) so it can land independently. Happy to drop in favor of [None][fix] Fix dis-agg with ADP and avoid TP collective deadlock #13900 if its scope is rebased to cover the overlap loop.

Cost

Per iteration: one tp_allgather(bool) (~30 bytes pickled). The slow path only runs when at least one rank reports errors.

Test plan

• New unit tests in tests/unittest/_torch/executor/test_py_executor.py:
  • test_synced_error_handler_no_op_when_disagg_disabled
  • test_synced_error_handler_no_op_when_attention_dp_disabled
  • test_synced_error_handler_skips_when_no_rank_has_errors
  • test_synced_error_handler_invokes_handle_errors_when_local_has_errors
  • test_synced_error_handler_runs_on_silent_rank_when_peer_has_errors (deadlock reproducer)
• Stress run dep16 disagg gen (DSv4) to confirm no 300s hang under transfer-error injection