RLM Tool for SkyRL Environments

[devpatelio]

Working draft and design for RLM (see blog) for SkyRL Gym.

• REPL execution: Executes Python code from blocks in model responses
• Recursive LLM calls: Supports llm_query() for nested LLM calls from code
• Context management: Loads and exposes context data (strings, dicts, lists) to the REPL
• Multi-turn loop: Iterates up to max_turns until a final answer is found
• Final answer extraction: Detects FINAL(...) or FINAL_VAR(name) patterns

Reference Code: rlm/rlm repo

TODO:

• Change the RLM client to be the inference client being used during the generation run
• Test on long-context benchmarks and return results for inference
• Evaluate RLM output "FINAL(answer)" and modify with parser

[devpatelio]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces a Recursive Language Model (RLM) tool and environment for SkyRL Gym. The changes include a new RLMExecutorEnv environment, a powerful RLMExecutorToolGroup with a sandboxed Python REPL for secure code execution, and supporting utilities and configuration. The implementation is comprehensive, with good attention to security aspects like sandboxing code execution. My review focuses on improving code clarity, robustness, and configuration flexibility. I've suggested changes to make parts of the new environment more configurable, improve logging and error handling, and refine some implementation details for efficiency and correctness. I also noted a potentially impactful change in an existing shell script.