feat(grpo): support async multiple dataloaders

[taivu1998]

Summary

Fixes #2006.

This PR enables standard async GRPO to run with data.use_multiple_dataloader=true while keeping async NeMo-Gym GRPO explicitly unsupported for that path.

Root Cause

The standard async entrypoint blocked multiple dataloaders outright, and the async collector consumed the next prompt batch before reserving the target weight version. That ordering prevented custom multiple-dataloader functions from seeing the async generation target they were sampling for. The wrapper also eagerly held live iterators and Hydra-loaded callables, which made it a poor object to ship into a Ray actor or checkpoint from async collection.

Changes

• Make MultipleDataloaderWrapper lazy, picklable, and checkpointable with per-task dataloader state.
• Reserve async target weight versions before consuming dataloader batches.
• Pass async records into custom multiple-dataloader functions: generation weight version, target weight version, current collector weight version, and expected prompt count.
• Save async multiple-dataloader checkpoint state as train_dataloader_<task>.pt, matching synchronous GRPO resume behavior.
• Remove the standard async GRPO multiple-dataloader guard from examples/run_grpo.py.
• Keep the async NeMo-Gym guard, but make its error message specific to async NeMo-Gym GRPO.
• Add a target-aware custom dataloader example plus unit coverage for lazy/picklable wrapper behavior, state restore, and async record-driven sampling.
• Document async multiple-dataloader configuration, records, checkpointing, and the NeMo-Gym limitation.

Validation

• uvx --python /Users/vuductai/.local/share/uv/python/cpython-3.12-macos-aarch64-none/bin/python3.12 ruff check nemo_rl/data/dataloader.py nemo_rl/algorithms/async_utils.py nemo_rl/algorithms/grpo.py examples/run_grpo.py examples/nemo_gym/run_grpo_nemo_gym.py examples/custom_dataloader/custom_dataloader.py tests/unit/data/test_multiple_dataloader.py
• /Users/vuductai/.local/share/uv/python/cpython-3.12-macos-aarch64-none/bin/python3.12 -m py_compile nemo_rl/data/dataloader.py nemo_rl/algorithms/async_utils.py nemo_rl/algorithms/grpo.py examples/run_grpo.py examples/nemo_gym/run_grpo_nemo_gym.py examples/custom_dataloader/custom_dataloader.py tests/unit/data/test_multiple_dataloader.py tests/unit/algorithms/test_async_utils.py
• git diff --check

uv run pytest tests/unit/data/test_multiple_dataloader.py tests/unit/algorithms/test_async_utils.py is currently blocked on this macOS host before test collection because /usr/local/bin/python3.13 is broken: platform.mac_ver() returns an empty value. The repository requires Python >=3.13.13, and uv python install 3.13.13 reports no matching macOS aarch64 download.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.