[LLM] Add Countdown numbers-game environment

[vmoens]

Stack from ghstack (oldest at bottom):

• [LLM] Wire MATH and Countdown into GRPO and Expert Iteration scripts #3546
• -> [LLM] Add Countdown numbers-game environment #3545
• [LLM] Add MATH (competition mathematics) environment #3544
• [LLM] Simplify IFEval reward aggregator #3543
• [LLM] Rewrite GSM8K reward function to follow standard GRPO conventions #3542

Add CountdownEnv and CountdownRewardParser for the Countdown numbers
game, a popular lightweight problem for GRPO training.

Key features:

• Procedural problem generation (no external dataset required)
• Validates arithmetic expressions: correct operators, each source
  number used at most once, evaluates to target
• Same 0/0.1/1.0 reward convention as GSM8K and MATH parsers
• Configurable problem difficulty (num_count, max_number, max_target)
• Includes unit tests and documentation

Made-with: Cursor

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3545

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 6 Unrelated Failures

As of commit 7b20595 with merge base 491ed0f ():

NEW FAILURES - The following jobs have failed:

• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Generate documentation / build-docs (3.12, 12.8) / linux-job (gh)
  RuntimeError: Command docker exec -t 87646319136b71d8e68673a3abffcb76c8d35460e1b968fd10ab337f1bc3717a /exec failed with exit code 2
• PR Label / add-label (gh)
  Process completed with exit code 1.
• Unit-tests on Linux / tests-cpu (3.14) / linux-job (gh)
  RuntimeError: Command docker exec -t 770b32c715140b2e13ead52fe7376f0ecd4c673ef3041edcabba9c8f151835fb /exec failed with exit code 245

CANCELLED JOB - The following job was cancelled. Please retry:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh) (trunk failure)
  test/test_rb.py::TestStorages::test__rand_given_ndim_recompile
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh) (trunk failure)
  test/test_rb.py::TestStorages::test__rand_given_ndim_recompile
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh) (trunk failure)
  test/test_rb.py::TestStorages::test__rand_given_ndim_recompile
• Unit-tests on Linux / tests-cpu (3.13) / linux-job (gh) (trunk failure)
  test/test_rb.py::TestStorages::test__rand_given_ndim_recompile
• Unit-tests on Linux / tests-olddeps (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_loggers.py::TestCSVLogger::test_log_video[mp4-steps1]
• Unit-tests on Linux / tests-optdeps (3.12, 13.0) / linux-job (gh) (trunk failure)
  test/test_rb.py::TestStorages::test__rand_given_ndim_recompile

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

⚠️ PR Title Label Error

Unknown or invalid prefix [LLM].

Current title: [LLM] Add Countdown numbers-game environment

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
[BugFix]	BugFix	[BugFix] Fix memory leak in collector
[Feature]	Feature	[Feature] Add new optimizer
[Doc] or [Docs]	Documentation	[Doc] Update installation guide
[Refactor]	Refactoring	[Refactor] Clean up module imports
[CI]	CI	[CI] Fix workflow permissions
[Test] or [Tests]	Tests	[Tests] Add unit tests for buffer
[Environment] or [Environments]	Environments	[Environments] Add Gymnasium support
[Data]	Data	[Data] Fix replay buffer sampling
[Performance] or [Perf]	Performance	[Performance] Optimize tensor ops
[BC-Breaking]	bc breaking	[BC-Breaking] Remove deprecated API
[Deprecation]	Deprecation	[Deprecation] Mark old function
[Quality]	Quality	[Quality] Fix typos and add codespell

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).

[github-actions]

⚠️ PR Title Label Error

Unknown or invalid prefix [LLM].

Current title: [LLM] Add Countdown numbers-game environment

Supported Prefixes (case-sensitive)

Your PR title must start with exactly one of these prefixes:

Prefix	Label Applied	Example
[BugFix]	BugFix	[BugFix] Fix memory leak in collector
[Feature]	Feature	[Feature] Add new optimizer
[Doc] or [Docs]	Documentation	[Doc] Update installation guide
[Refactor]	Refactoring	[Refactor] Clean up module imports
[CI]	CI	[CI] Fix workflow permissions
[Test] or [Tests]	Tests	[Tests] Add unit tests for buffer
[Environment] or [Environments]	Environments	[Environments] Add Gymnasium support
[Data]	Data	[Data] Fix replay buffer sampling
[Performance] or [Perf]	Performance	[Performance] Optimize tensor ops
[BC-Breaking]	bc breaking	[BC-Breaking] Remove deprecated API
[Deprecation]	Deprecation	[Deprecation] Mark old function
[Quality]	Quality	[Quality] Fix typos and add codespell

Note: Common variations like singular/plural are supported (e.g., [Doc] or [Docs]).