Working change for benchmarking new Mujoco Menagerie robot assets on Isaac Lab environments

mtrepte/mujoco-menagerie/README.md

@@ -0,0 +1,59 @@
+# MuJoCo Menagerie Benchmark Branch
+
+This branch ports the MuJoCo Menagerie asset swap onto `mtrepte/mujoco_menagerie_v2`, based on current IsaacLab `develop`.
+
+The goal is to benchmark IsaacLab training environments with robot USD assets from:
+
+```text
+${ISAAC_NUCLEUS_DIR}/Samples/Mujoco_Menagerie
+```
+
+The branch intentionally does not fall back to legacy IsaacLab robot USDs if a Menagerie path or physics variant is missing. Broken paths should fail during launch so they are visible in local smoke runs, Docker runs, and CI-style benchmarks.
+
+## What Changed
+
+- Core robot asset configs now point to MuJoCo Menagerie USD/USDC assets for the covered robots.
+- Training launchers accept `--menagerie-physics-variant` to select Menagerie USD `Physics` variants.
+- The benchmark harness has a Menagerie-specific config at `source/isaaclab_tasks/test/benchmarking/mujoco_configs.yaml`.
+- The benchmark test can emit KPI payloads via `--save_kpi_payload`.
+
+## Primary Smoke Command
+
+Run from the repo root:
+
+```bash
+./isaaclab.sh -p -m pytest -s -x \
+  source/isaaclab_tasks/test/benchmarking/test_environments_training.py \
+  --config_path mujoco_configs.yaml \
+  --mode test \
+  --workflows rsl_rl \
+  --sim-backend physx \
+  --save_kpi_payload
+```
+
+For Newton/MuJoCo physics variant coverage, switch to:
+
+```bash
+--sim-backend newton
+```
+
+For longer KPI collection, switch to:
+
+```bash
+--mode benchmark
+```
+
+## Docker A/B Benchmark Shape
+
+Use the same `mujoco_configs.yaml` in both images:
+
+- Base image: current `develop`, original IsaacLab robot assets.
+- Menagerie image: this branch, Menagerie robot assets and explicit `--sim-backend physx` or `--sim-backend newton`.
+
+Compare generated KPI payloads from `logs/kpi.json`.
+
+## Docs
+
+- `asset_mapping.txt`: IsaacLab robot config to Menagerie asset path mapping.
+- `asset_tasks.txt`: representative tasks that exercise each Menagerie-backed asset.
+- `training_benchmark_catalog.txt`: full benchmark matrix and command notes.

mtrepte/mujoco-menagerie/asset_mapping.txt

@@ -0,0 +1,34 @@
+MuJoCo Menagerie (Nucleus) <-> IsaacLab_1 v2 mapping notes
+Branch intent: mtrepte/mujoco_menagerie_v2
+
+Failure policy
+- No silent fallback. If a resolved path is missing, unreadable, or fails load/sim init, the run must error out immediately so broken assets/paths surface in CI or local runs. Do not substitute legacy USD automatically.
+
+Menagerie root
+  isaaclab.utils.assets.MUJOCO_MENAGERIE_DIR (= ISAAC_NUCLEUS_DIR + "/Samples/Mujoco_Menagerie")
+
+Replaced targets (current code -> Menagerie USD primary)
+  ANYMAL_B_CFG             -> anybotics_anymal_b/anymal_b/anymal_b.usda
+  ANYMAL_C_CFG             -> anybotics_anymal_c/anymal_c/anymal_c.usda
+  SPOT_CFG                 -> boston_dynamics_spot/spot.usda
+  UNITREE_GO2_CFG          -> unitree_go2/go2.usda
+  H1_CFG / H1_MINIMAL_CFG  -> unitree_h1/h1/h1.usda
+  G1_CFG / G1_MINIMAL_CFG  -> unitree_g1/g1/g1.usda
+  G1_29DOF_CFG             -> unitree_g1/usdex/g1_29dof_rev_1_0.usdc
+  ALLEGRO_HAND_CFG         -> wonik_allegro/right_hand/right_hand.usda
+  SHADOW_HAND_CFG          -> shadow_hand/right_hand/right_hand.usda
+
+Physics variants
+- The branch exposes --menagerie-physics-variant with choices auto, physx, mujoco, none.
+- The benchmark pytest --sim-backend option maps physx -> Menagerie Physics=physx and newton -> Menagerie Physics=mujoco.
+- Use explicit --sim-backend for A/B benchmark reproducibility.
+
+Not used in this benchmark matrix
+  apptronik_apollo, booster_t1, universal_robots_ur5e, robotiq_2f85 standalone
+  wonik_allegro left_hand_* unless left-hand tasks are added
+  shadow_hand left_hand_* unless left-hand tasks are added
+
+Ambiguous / split by task
+  G1: velocity envs use G1_MINIMAL_CFG / g1.usda. Locomanipulation uses G1_29DOF_CFG / g1_29dof_rev_1_0.usdc.
+  H1: rough/flat velocity use H1_MINIMAL_CFG today; full H1_CFG also points at the Menagerie H1 USD.
+  Dexsuite: Isaac-Dexsuite-Kuka-Allegro-* uses the KUKA_ALLEGRO_CFG bundle, not the Wonik Menagerie Allegro hand.

mtrepte/mujoco-menagerie/asset_tasks.txt

@@ -0,0 +1,32 @@
+Representative gym tasks that exercise each Menagerie-backed robot asset.
+
+Rule: one primary task id per robot or distinct USD role. Omit -Play and duplicate variants.
+Workflows are the registered training entry points for the task.
+
+task_id | workflows | asset / note
+Isaac-Velocity-Rough-Anymal-B-v0 | rsl_rl, skrl | ANYMAL_B
+Isaac-Velocity-Rough-Anymal-C-v0 | rsl_rl, rl_games, skrl | ANYMAL_C
+Isaac-Velocity-Flat-Spot-v0 | rsl_rl, skrl | SPOT (no rough task in tree)
+Isaac-Velocity-Rough-Unitree-Go2-v0 | rsl_rl, skrl | UNITREE_GO2
+Isaac-Velocity-Rough-H1-v0 | rsl_rl, skrl | H1_MINIMAL velocity
+Isaac-Velocity-Rough-G1-v0 | rsl_rl, skrl | G1_MINIMAL velocity
+Isaac-PickPlace-Locomanipulation-G1-Abs-v0 | robomimic_bc | G1_29DOF distinct Menagerie USD role
+Isaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0 | none registered | G1_29DOF IK-only; add workflows before benchmarking
+Isaac-Repose-Cube-Allegro-v0 | rsl_rl, rl_games, skrl | ALLEGRO_HAND manager-based
+Isaac-Repose-Cube-Allegro-Direct-v0 | rsl_rl, rl_games, skrl | ALLEGRO_HAND direct
+Isaac-Repose-Cube-Shadow-Direct-v0 | rsl_rl, rl_games, skrl | SHADOW_HAND direct
+
+Automated pytest benchmark coverage
+  source/isaaclab_tasks/test/benchmarking/mujoco_configs.yaml currently uses the flat velocity variants plus the direct Allegro and Shadow hand tasks for RSL-RL smoke/benchmark coverage:
+  Isaac-Velocity-Flat-Anymal-B-v0
+  Isaac-Velocity-Flat-Anymal-C-v0
+  Isaac-Velocity-Flat-Spot-v0
+  Isaac-Velocity-Flat-Unitree-Go2-v0
+  Isaac-Velocity-Flat-H1-v0
+  Isaac-Velocity-Flat-G1-v0
+  Isaac-Repose-Cube-Allegro-Direct-v0
+  Isaac-Repose-Cube-Shadow-Direct-v0
+
+Not listed
+- Isaac-Repose-Cube-Shadow-* OpenAI / Vision / Benchmark variants use the same Shadow Menagerie target; add them only if those observation pipelines are in scope.
+- Isaac-Dexsuite-Kuka-Allegro-* uses KUKA_ALLEGRO_CFG / kuka.usd, not the Wonik Menagerie Allegro hand.

mtrepte/mujoco-menagerie/training_benchmark_catalog.txt

@@ -0,0 +1,64 @@
+# Training benchmark catalog - MuJoCo Menagerie asset swap
+
+Purpose
+  Matrix for comparing:
+    A) develop Docker image + original IsaacLab robot USDs
+    B) this branch + Menagerie assets + Physics=mujoco / Newton backend
+    C) this branch + Menagerie assets + Physics=physx / PhysX backend
+
+  Menagerie assets resolve through MUJOCO_MENAGERIE_DIR under Samples/Mujoco_Menagerie.
+  The benchmark should use explicit --sim-backend values so the chosen physics variant is clear.
+
+Workflow column lists gym *_cfg_entry_point keys.
+
+Columns
+  task_id | workflows | Menagerie asset / cfg | notes
+
+--- Locomotion / velocity (Menagerie-backed on this branch) ---
+Isaac-Velocity-Rough-Anymal-B-v0 | rsl_rl, skrl | ANYMAL_B_CFG / anymal_b.usda |
+Isaac-Velocity-Rough-Anymal-C-v0 | rsl_rl, rl_games, skrl | ANYMAL_C_CFG / anymal_c.usda |
+Isaac-Velocity-Flat-Spot-v0 | rsl_rl, skrl | SPOT_CFG / spot.usda | no rough task registered
+Isaac-Velocity-Rough-Unitree-Go2-v0 | rsl_rl, skrl | UNITREE_GO2_CFG / go2.usda |
+Isaac-Velocity-Rough-H1-v0 | rsl_rl, skrl | H1_MINIMAL_CFG -> h1.usda | full H1_CFG also Menagerie
+Isaac-Velocity-Rough-G1-v0 | rsl_rl, skrl | G1_MINIMAL_CFG -> g1.usda |
+
+--- Manipulation / in-hand (Menagerie-backed) ---
+Isaac-Repose-Cube-Allegro-v0 | rsl_rl, rl_games, skrl | ALLEGRO_HAND_CFG / wonik_allegro right hand |
+Isaac-Repose-Cube-Allegro-Direct-v0 | rsl_rl, rl_games, skrl | same as Allegro manager task |
+Isaac-Repose-Cube-Shadow-Direct-v0 | rsl_rl, rl_games, skrl | SHADOW_HAND_CFG / shadow_hand right hand |
+Isaac-PickPlace-Locomanipulation-G1-Abs-v0 | robomimic_bc | G1_29DOF_CFG -> g1_29dof_rev usdc |
+
+--- Not in Menagerie swap matrix ---
+Isaac-Dexsuite-Kuka-Allegro-* | various | KUKA + kuka.usd bundle, not Wonik Menagerie Allegro |
+Isaac-Velocity-*-Unitree-A1-v0 | various | UNITREE_A1_CFG still IsaacLab nucleus a1.usd |
+Isaac-Velocity-*-Anymal-D-v0 | various | ANYMAL_D_CFG still IsaacLab nucleus ANYmal-D USD |
+
+Pytest smoke command
+  ./isaaclab.sh -p -m pytest -s -x \
+    source/isaaclab_tasks/test/benchmarking/test_environments_training.py \
+    --config_path mujoco_configs.yaml \
+    --mode test \
+    --workflows rsl_rl \
+    --sim-backend physx \
+    --save_kpi_payload
+
+Newton / MuJoCo variant
+  Replace --sim-backend physx with --sim-backend newton.
+
+Longer KPI benchmark
+  Replace --mode test with --mode benchmark.
+
+Single-task example
+  ./isaaclab.sh -p -m pytest -s -x \
+    source/isaaclab_tasks/test/benchmarking/test_environments_training.py \
+    --config_path mujoco_configs.yaml \
+    --mode test \
+    --workflows rsl_rl \
+    --sim-backend physx \
+    --save_kpi_payload \
+    -k "Isaac-Velocity-Flat-Anymal-B-v0"
+
+Related
+  asset_mapping.txt - robot config to Menagerie path mapping
+  asset_tasks.txt - representative task per Menagerie-backed robot asset
+  source/isaaclab_tasks/test/benchmarking/mujoco_configs.yaml - automated benchmark config

scripts/demos/markers.py

@@ -38,7 +38,7 @@
 import isaaclab.sim as sim_utils
 from isaaclab.markers import VisualizationMarkers, VisualizationMarkersCfg
 from isaaclab.sim import SimulationContext
-from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, ISAACLAB_NUCLEUS_DIR
+from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, MUJOCO_MENAGERIE_DIR
 from isaaclab.utils.math import quat_from_angle_axis
 
 
@@ -84,7 +84,7 @@ def define_markers() -> VisualizationMarkers:
                 visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(1.0, 0.25, 0.0)),
             ),
             "robot_mesh": sim_utils.UsdFileCfg(
-                usd_path=f"{ISAACLAB_NUCLEUS_DIR}/Robots/ANYbotics/ANYmal-C/anymal_c.usd",
+                usd_path=f"{MUJOCO_MENAGERIE_DIR}/anybotics_anymal_c/anymal_c/anymal_c.usda",
                 scale=(2.0, 2.0, 2.0),
                 visual_material=sim_utils.GlassMdlCfg(glass_color=(0.0, 0.1, 0.0)),
             ),

scripts/demos/multi_asset.py

@@ -55,7 +55,7 @@
 from isaaclab.sim import SimulationContext
 from isaaclab.sim.utils.stage import get_current_stage
 from isaaclab.utils import Timer
-from isaaclab.utils.assets import ISAACLAB_NUCLEUS_DIR
+from isaaclab.utils.assets import ISAACLAB_NUCLEUS_DIR, MUJOCO_MENAGERIE_DIR
 from isaaclab.utils.configclass import configclass
 
 ##
@@ -186,7 +186,7 @@ class MultiObjectSceneCfg(InteractiveSceneCfg):
         prim_path="/World/envs/env_.*/Robot",
         spawn=sim_utils.MultiUsdFileCfg(
             usd_path=[
-                f"{ISAACLAB_NUCLEUS_DIR}/Robots/ANYbotics/ANYmal-C/anymal_c.usd",
+                f"{MUJOCO_MENAGERIE_DIR}/anybotics_anymal_c/anymal_c/anymal_c.usda",
                 f"{ISAACLAB_NUCLEUS_DIR}/Robots/ANYbotics/ANYmal-D/anymal_d.usd",
             ],
             random_choice=True,

scripts/reinforcement_learning/rl_games/train.py

@@ -47,6 +47,7 @@
     resolve_task_config,
     setup_preset_cli,
 )
+from isaaclab_tasks.utils.training_asset_log import log_training_asset_paths
 
 logger = logging.getLogger(__name__)
 
@@ -182,6 +183,8 @@ def main():
         # set the log directory for the environment
         env_cfg.log_dir = os.path.join(log_root_path, log_dir)
 
+        log_training_asset_paths(args_cli.task, env_cfg, "training start (before environment creation)")
+
         # create isaac environment
         env = gym.make(args_cli.task, cfg=env_cfg, render_mode="rgb_array" if args_cli.video else None)
 
@@ -256,10 +259,11 @@ def main():
             else:
                 runner.run({"train": True, "play": False, "sigma": train_sigma})
             print(f"Training time: {round(time.time() - start_time, 2)} seconds")
-            # close the simulator
-            env.close()
         except KeyboardInterrupt:
             pass
+        finally:
+            log_training_asset_paths(args_cli.task, env_cfg, "training end (after training loop)")
+            env.close()
 
 
 if __name__ == "__main__":

scripts/reinforcement_learning/rsl_rl/train.py

@@ -47,6 +47,7 @@
     setup_preset_cli,
 )
 from isaaclab_tasks.utils.hydra import hydra_task_config
+from isaaclab_tasks.utils.training_asset_log import log_training_asset_paths
 
 # local imports
 import cli_args  # isort: skip
@@ -189,6 +190,8 @@ def main(env_cfg: ManagerBasedRLEnvCfg | DirectRLEnvCfg | DirectMARLEnvCfg, agen
         # set the log directory for the environment (works for all environment types)
         env_cfg.log_dir = log_dir
 
+        log_training_asset_paths(args_cli.task, env_cfg, "training start (before environment creation)")
+
         # create isaac environment
         env = gym.make(args_cli.task, cfg=env_cfg, render_mode="rgb_array" if args_cli.video else None)
 
@@ -246,10 +249,11 @@ def main(env_cfg: ManagerBasedRLEnvCfg | DirectRLEnvCfg | DirectMARLEnvCfg, agen
         try:
             runner.learn(num_learning_iterations=agent_cfg.max_iterations, init_at_random_ep_len=True)
             print(f"Training time: {round(time.time() - start_time, 2)} seconds")
-            # close the simulator
-            env.close()
         except KeyboardInterrupt:
             pass
+        finally:
+            log_training_asset_paths(args_cli.task, env_cfg, "training end (after training loop)")
+            env.close()
 
 
 if __name__ == "__main__":

scripts/reinforcement_learning/sb3/train.py

@@ -48,6 +48,7 @@
     resolve_task_config,
     setup_preset_cli,
 )
+from isaaclab_tasks.utils.training_asset_log import log_training_asset_paths
 
 logger = logging.getLogger(__name__)
 
@@ -155,6 +156,8 @@ def main():
         # set the log directory for the environment
         env_cfg.log_dir = log_dir
 
+        log_training_asset_paths(args_cli.task, env_cfg, "training start (before environment creation)")
+
         # create isaac environment
         env = gym.make(args_cli.task, cfg=env_cfg, render_mode="rgb_array" if args_cli.video else None)
 
@@ -213,26 +216,27 @@ def main():
         callbacks = [checkpoint_callback, LogEveryNTimesteps(n_steps=args_cli.log_interval)]
 
         # train the agent
-        with contextlib.suppress(KeyboardInterrupt):
-            agent.learn(
-                total_timesteps=n_timesteps,
-                callback=callbacks,
-                progress_bar=True,
-                log_interval=None,
-            )
-        # save the final model
-        agent.save(os.path.join(log_dir, "model"))
-        print("Saving to:")
-        print(os.path.join(log_dir, "model.zip"))
-
-        if isinstance(env, VecNormalize):
-            print("Saving normalization")
-            env.save(os.path.join(log_dir, "model_vecnormalize.pkl"))
-
-        print(f"Training time: {round(time.time() - start_time, 2)} seconds")
-
-        # close the simulator
-        env.close()
+        try:
+            with contextlib.suppress(KeyboardInterrupt):
+                agent.learn(
+                    total_timesteps=n_timesteps,
+                    callback=callbacks,
+                    progress_bar=True,
+                    log_interval=None,
+                )
+            # save the final model
+            agent.save(os.path.join(log_dir, "model"))
+            print("Saving to:")
+            print(os.path.join(log_dir, "model.zip"))
+
+            if isinstance(env, VecNormalize):
+                print("Saving normalization")
+                env.save(os.path.join(log_dir, "model_vecnormalize.pkl"))
+
+            print(f"Training time: {round(time.time() - start_time, 2)} seconds")
+        finally:
+            log_training_asset_paths(args_cli.task, env_cfg, "training end (after training loop)")
+            env.close()
 
 
 if __name__ == "__main__":

scripts/reinforcement_learning/skrl/train.py

@@ -49,6 +49,7 @@
     resolve_task_config,
     setup_preset_cli,
 )
+from isaaclab_tasks.utils.training_asset_log import log_training_asset_paths
 
 logger = logging.getLogger(__name__)
 
@@ -201,6 +202,8 @@ def main():
         # set the log directory for the environment
         env_cfg.log_dir = log_dir
 
+        log_training_asset_paths(args_cli.task, env_cfg, "training start (before environment creation)")
+
         # create isaac environment
         env = gym.make(args_cli.task, cfg=env_cfg, render_mode="rgb_array" if args_cli.video else None)
 
@@ -250,10 +253,11 @@ def main():
             os.makedirs(os.path.join(log_dir, "checkpoints"), exist_ok=True)
             runner.agent.write_checkpoint(timestep=total_timesteps, timesteps=total_timesteps)
             print(f"[INFO] Saved final agent checkpoint to: {log_dir}/checkpoints")
-            # close the simulator
-            env.close()
         except KeyboardInterrupt:
             pass
+        finally:
+            log_training_asset_paths(args_cli.task, env_cfg, "training end (after training loop)")
+            env.close()
 
 
 if __name__ == "__main__":

source/isaaclab/isaaclab/actuators/actuator_net.py

@@ -81,8 +81,10 @@ def compute(
         self.sea_input[:, 0, 0] = (control_action.joint_positions - joint_pos).flatten()
         self.sea_input[:, 0, 1] = joint_vel.flatten()
 
-        # run network inference
-        with torch.inference_mode():
+        # Run recurrent actuator inference without cuDNN. Some benchmark subprocesses
+        # initialize Newton before the first TorchScript LSTM call, which can leave
+        # cuDNN unavailable even though the non-cuDNN CUDA LSTM path works.
+        with torch.inference_mode(), torch.backends.cudnn.flags(enabled=False):
             torques, (self.sea_hidden_state[:], self.sea_cell_state[:]) = self.network(
                 self.sea_input, (self.sea_hidden_state, self.sea_cell_state)
             )