Add lab desk objects, AprilTag bottle picking, and clean up lab_sim by davetcoleman · Pull Request #512 · PickNikRobotics/moveit_pro_example_ws

davetcoleman · 2026-02-14T08:52:11Z

Summary

Add realistic lab environment objects to MuJoCo scene: desk lamp, laptop with screen/keyboard textures, potted plant, alligator trophy statue on shelf, test tube rack with 6 colored tubes, and 4 manipulable medicine bottles
Add new lab_sim_behaviors package with ComputeTrayPlacePositionsUsingAprilTags behavior plugin for computing tray grid placement positions from AprilTag detections
Add "Pick All Bottles with AprilTags" objective: detects AprilTag-labeled bottles, picks them, and places them in a tray using vision-guided grasping with force-controlled lowering
Rework constrained pick-and-place subtree
Clean up lab_sim objectives: rename files for consistency, remove deprecated objectives (_scan_scene, load_mesh_as_*_pointcloud, pick_and_place_example, scan_multiple_views)
Simplify ur_waypoints.yaml and add new waypoints
Add CLAUDE.md with workspace instructions (MuJoCo keyframes, objective MetadataFields CI requirements)

Dependencies

Requires picknik_accessories#feature/lab-sim-desk-objects for test tube stand repositioning
Related: PR #518 — lab_sim_behaviors package (included here as shared commit)

Test plan

Launch lab_sim in MuJoCo and verify all objects render correctly on the desk
Verify test tubes and medicine bottles are manipulable (have freejoints)
Run "Pick All Bottles with AprilTags" objective end-to-end in simulation
Verify constrained pick-and-place still works after rework
Confirm no objects are floating or falling through the table

🤖 Generated with Claude Code

src/lab_sim/description/scene.xml

davetcoleman · 2026-02-17T19:32:16Z

src/picknik_accessories

@MikeWrock i dont understand why some things live in this picknik_accessories, and others in the example_ws. Can we get rid of picknik_accessories? What's the point? We have GitLFS already.

I'm not sure what things are living in example_ws, but the idea is to have one repo that can hold all our reusable assets, and if you have config specific assets that cannot justifiably be put in the accessories package then it can live in the config package. Due to https://github.com/PickNikRobotics/moveit_pro/issues/13842 we have to do a little CMake hacking so that we can re-use these assets, otherwise they would have to be duplicated within each package that used them and copied out for re-use

But who is reusing these assets? It appears these assets are just used in the example_ws

I propose I have Claude merge these repos together. Maybe @nbbrooks has thoughts.

I've always liked monorepos but we sacrifice all portability for our digital assets. Any mujoco scenes or objects we want to use in more than one package requires duplicating code and files, which then results in bit rot and missed fixes/updates or divergent configurations. Whats the advantage we gain by having the files spread out across multiple packages in the example_ws?

picknik_accessories is where some of our tuned mujoco assets like the robotiq_gripper live

I'd be open to keeping those in accessories for reuse and moving the "example_ws world objects" into example_ws, but we will still have a submodule

i find using picknik_accessories incredibly frustrating, as ive lost some of my work trying to manage PRs between three repos for this tutorial work. i would love to move all of the picknik_accessories content into this repo.

If the contents of picknik_accessories never needs to exist outside of the example_ws then I'm fine with getting rid of it as a submodule but if anything in there gets used in a customer workspace we need to have a clear way of notifying and distributing updates. Maybe we can move everything in to example_ws and then copybara changes out in to picknik_accessories so our user can still get updates?

If we merge accessories into example_ws, people can still grab assets they would like from the example_ws. In most cases I would recommend they just copy paste them into their repo. example_ws and picknik_accessories are only going to get bigger as we go (especially when we get around to adding high quality textures so the simulation looks better). I wouldn't want my custom workspace to pull in the entirety of all or example world textures just to get an april tag or a robotiq mujoco model.

As I mentioned earlier, I could see us eventually making a picknik version of mujoco_menagerie which is just lightweight mujoco xml files for hard-to-tune assets like parallel grippers. It is a little messy using some of their current assets because their actuator names dont match the physical hardware names, so you end up unnecessarily overloading stuff just because of a joint name difference.

Another consideration here given this repo has lots of large assets is if there will be an noticeable difference in retrieval time. @JWhitleyWork do you anticipate any huge retrieval time difference between

Moving picknik_accessories into example_ws and retrieving via git lfs

Distributing picknik_accessories via the apt build farm you set up

I want to note that i don't think there's anything very valuable in picknik_accessories - just some kinda random CAD/MuJoCo files right?

src/lab_sim/description/scene.xml

MikeWrock · 2026-02-18T17:01:28Z

This is failing CI for both integration testing and pre-commit. Please confirm the objectives in lab_sim still run successfully and that the new assets don't introduce any unexpected behaviors (ungraspable objects due to collisions, collisions with new objects due to waypoints, confusing input to ML prompts, poor friction coefficient for new objects). As our flagship config, I wouldn't want to introduce a whole bunch of unpickable objects or add complexity that would make the product look poorly functional.

Please also monitor the change in CPU usage, there are a lot of additional free joints that would significantly increase the CPU load and I want to make sure it's not a huge jump. I noticed some of the free joints (like the alligator) are not even reachable, and would be better as fixed joints.

When this PR is tested and ready for review, please add a story to track your work on the project board

rlpratt12 · 2026-02-27T20:43:19Z

This is for https://github.com/PickNikRobotics/moveit_pro/issues/17169 (priority for sprint 133)

nbbrooks

I still need to test this locally.

nbbrooks · 2026-03-02T20:22:42Z

...lab_sim_behaviors/include/lab_sim_behaviors/compute_tray_place_positions_using_apriltags.hpp

+ * | place_positions         | output    | std::vector<geometry_msgs::msg::PoseStamped>        |
+ */
+class ComputeTrayPlacePositionsUsingAprilTags final
+  : public moveit_pro::behaviors::SharedResourcesNode<BT::SyncActionNode>


I won't block this, but I would have recommended using AddURDF to localize a URDF model of the tray+AprilTag instead of a custom behavior.

This would be a good example of using that for a grasping workflow. Could you make a followup story for Claude to do this?

This isn't just localizing a URDF, it is choosing multiple place positions within the tray.

I don't think trying to make more tiny Behaviors is actually a good idea, I'd like us to move towards thinking about bigger behaviors that are quickly written in code using Claude. @marioprats pointed this out to me this morning - the cost of making a new Behavior plugin is close to zero, and we trying to do every function call as a behavior is incredibly hard and annoying.

I don't think trying to make more tiny Behaviors is actually a good idea

Isn't that what you are doing here? What I suggested adds no new behaviors, and uses a URDF to generate the transforms instead of new code.

What happens if they have a 6 test tube rack and a 12 test tube rack? 2 new behaviors? Or would you parameterize with something describing what the rack geometry looks like (aka a URDF). Again, not going to block this but that is still how I feel about this behavior.

nbbrooks · 2026-03-03T14:45:50Z

I am getting build errors

--- stderr: lab_sim_behaviors                                                                                                        
CMake Deprecation Warning at /opt/overlay_ws/install/moveit_studio_common/share/moveit_studio_common/cmake/moveit_studio_package.cmake:109 (message):
  moveit_studio_package() macro is DEPRECATED.  Please update your package
  to:

  1.  Change <build_depend>moveit_studio_common</build_depend> to
  <build_depend>moveit_pro_package</build_depend>

  2.  Add find_package(moveit_pro_package REQUIRED) to your CMakeLists.txt

  3.  Replace moveit_studio_package() with moveit_pro_package()

  See the moveit_pro_package documentation for migration details.
Call Stack (most recent call first):
  CMakeLists.txt:5 (moveit_studio_package)


In file included from /home/nbb/user_ws/src/lab_sim_behaviors/src/register_behaviors.cpp:11:
/home/nbb/user_ws/src/lab_sim_behaviors/include/lab_sim_behaviors/compute_tray_place_positions_using_apriltags.hpp:10:10: fatal error: moveit_pro_behavior/behaviors/visualization/ros_publisher_handle.hpp: No such file or directory
   10 | #include <moveit_pro_behavior/behaviors/visualization/ros_publisher_handle.hpp>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/lab_sim_behaviors.dir/build.make:90: CMakeFiles/lab_sim_behaviors.dir/src/register_behaviors.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
In file included from /home/nbb/user_ws/src/lab_sim_behaviors/src/compute_tray_place_positions_using_apriltags.cpp:14:
/home/nbb/user_ws/src/lab_sim_behaviors/include/lab_sim_behaviors/compute_tray_place_positions_using_apriltags.hpp:10:10: fatal error: moveit_pro_behavior/behaviors/visualization/ros_publisher_handle.hpp: No such file or directory
   10 | #include <moveit_pro_behavior/behaviors/visualization/ros_publisher_handle.hpp>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/lab_sim_behaviors.dir/build.make:76: CMakeFiles/lab_sim_behaviors.dir/src/compute_tray_place_positions_using_apriltags.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:172: CMakeFiles/lab_sim_behaviors.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
gmake: *** [Makefile:146: all] Error 2
---
Failed   <<< lab_sim_behaviors [28.8s, exited with code 2]

I am on 9.0-rc22 but dont see how that would make a difference

nbbrooks · 2026-03-03T20:55:43Z

@davetcoleman can you clarify the relationship/order of operations between

https://github.com/PickNikRobotics/moveit_pro/pull/16905 (This builds off of all of the below work)
Add lab_sim_behaviors package #518
Add lab desk objects, AprilTag bottle picking, and clean up lab_sim #512 (based on my build error, I think this requries Add lab_sim_behaviors package #518 but that might be unintentional)
Update lab desk objects, textures, and positions picknik_accessories#42 (I believe Add lab desk objects, AprilTag bottle picking, and clean up lab_sim #512 pulls this in)

davetcoleman · 2026-03-04T01:13:04Z

moveit_pro

Tutorial Changes:
https://github.com/PickNikRobotics/moveit_pro/pull/16905

moveit_pro_example_ws

Behavior Plugin. Can be merged in solo, now.
#518

All the other changes to lab_sim. I could combine these two PRs but didn't want too big of a PR, but turns out it still hasn't helped get things merged in after weeks of pinging people. Goes along with the tutorial change PR to moveit_pro
#512

picknik_accessories

Annoying third repo for no benefit that someone else setup for historical reasons. Can be merged in solo, now.
PickNikRobotics/picknik_accessories#42

davetcoleman · 2026-03-04T01:15:54Z

I am getting build errors

From Mr Claude:

OK so the header and library are installed locally. The issue is that your colleague's Docker image doesn't have the latest overlay workspace
installed — ros_publisher_handle was added relatively recently and their pre-built Docker image doesn't include it.

The fix for your colleague:

They need to rebuild the Docker image (or at minimum rebuild moveit_pro_behavior inside the container) so the header gets installed in the overlay
workspace. If lab_sim_behaviors depends on ros_publisher_handle, the overlay must be up to date.
Alternatively, if they're on an older release branch that doesn't have ros_publisher_handle, the lab_sim_behaviors code referencing it needs to be
updated to match that branch.

TL;DR: Their Docker image's pre-built overlay is stale — it doesn't include ros_publisher_handle.hpp which was added in a recent commit. Rebuilding
the Docker image (docker compose build dev) will fix it.

nbbrooks · 2026-03-04T03:06:01Z

Ah, I was thinking I could run this on 9.0, need to run from main.

MikeWrock · 2026-03-04T05:55:38Z

src/lab_sim_behaviors/src/compute_tray_place_positions_using_apriltags.cpp

+  const auto& [detections_array, camera_info] = ports.value();
+
+  const int tray_apriltag_id = getInput<int>(kPortIDTrayApriltagId).value();
+  const int num_rows = getInput<int>(kPortIDNumRows).value();
+  const int columns_per_row = getInput<int>(kPortIDColumnsPerRow).value();
+  const double row_spacing = getInput<double>(kPortIDRowSpacing).value();
+  const double column_spacing = getInput<double>(kPortIDColumnSpacing).value();
+  const double tag_to_tray_center = getInput<double>(kPortIDTagToTrayCenter).value();
+  const double bottle_diameter = getInput<double>(kPortIDBottleDiameter).value();


Suggested change

const auto& [detections_array, camera_info] = ports.value();

const int tray_apriltag_id = getInput<int>(kPortIDTrayApriltagId).value();

const int num_rows = getInput<int>(kPortIDNumRows).value();

const int columns_per_row = getInput<int>(kPortIDColumnsPerRow).value();

const double row_spacing = getInput<double>(kPortIDRowSpacing).value();

const double column_spacing = getInput<double>(kPortIDColumnSpacing).value();

const double tag_to_tray_center = getInput<double>(kPortIDTagToTrayCenter).value();

const double bottle_diameter = getInput<double>(kPortIDBottleDiameter).value();

const auto& [detections_array, camera_info, tray_apriltag_id, num_rows, columns_per_row, row_spacing, column_spacing, tag_to_tray_center, bottle_diameter ] = ports.value();

But it has to match the order you declared the ports in providedPorts

I dont claim to know what our best practices are, but this change seems more error prone to index changes and misalignment. Is the current approach bad?

Also, these plugin changes live in this other PR that needs to be merged in first:
#518

I would love to not have to stack PRs but its been passing for a week and I'm trying to keep PRs from getting too big.

MikeWrock · 2026-03-04T05:58:52Z

CLAUDE.md

+
+After adding or removing bodies with joints, either:
+1. **Remove the keyframe** and let MuJoCo use body `pos=` attributes for initial positions
+2. **Regenerate the keyframe** from the MuJoCo Interactive Viewer (Ctrl+Shift+K) once the scene is stable


Does Claude actually do this somehow?

Yes, Claude built all of this and knows how to handle MuJoCo files completely

How is Claude watching the interactive viewer to see the robot has settled, and how is it executing keystrokes in the GUI? Is it just lying to you and pretending it's doing these actions (while grabbing the information elsewhere)?

Claude's explanation of what it's doing often varies from what it actually does

Oh I see what you are saying. No I'm pretty sure its is not using the Interactive Viewer. As I watch its logic, its doing the math and calculating pos in its mind only.

rlpratt12

This includes custom behaviors, mujoco scene changes, and modified/new objectives. Please break these changes in to multiple PRs per our best practices.

davetcoleman · 2026-03-04T22:12:36Z

@rlpratt12 as stated in the comment above (#512 (comment)) and in the PR desc, I have this effort broken into 4 PRs already, two of which are in this combined PR temporarily. I can break it into more, but I first need the earlier PRs merged in. Managing 4 PRs across 3 repos is a huge pain, and you are unfairly characterizing my efforts here.

davetcoleman · 2026-03-05T00:57:06Z

I've had claude split this out into more PRs, that are all straight forward and ready to merge asap:

This final PR I've marked as draft until those other ones are merged.

… up lab_sim Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JWhitleyWork · 2026-03-05T17:55:30Z

Rebased to pick up CI fix.

davetcoleman mentioned this pull request Feb 14, 2026

Update lab desk objects, textures, and positions PickNikRobotics/picknik_accessories#42

Open

5 tasks

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 3 times, most recently from ff401c8 to 4cee822 Compare February 14, 2026 19:32

davetcoleman requested a review from MikeWrock February 14, 2026 19:32

davetcoleman commented Feb 14, 2026

View reviewed changes

src/lab_sim/description/scene.xml Show resolved Hide resolved

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 2 times, most recently from 17e3dc9 to f06197a Compare February 17, 2026 19:28

davetcoleman added this to the 9.1.0 milestone Feb 17, 2026

davetcoleman commented Feb 17, 2026

View reviewed changes

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 7 times, most recently from b621d9c to 6c53a30 Compare February 18, 2026 05:22

davetcoleman removed the request for review from MikeWrock February 18, 2026 05:22

davetcoleman commented Feb 18, 2026

View reviewed changes

src/lab_sim/description/scene.xml Outdated Show resolved Hide resolved

MikeWrock assigned davetcoleman Feb 18, 2026

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 9 times, most recently from a3d0b64 to 456c578 Compare February 25, 2026 05:27

davetcoleman force-pushed the feature/lab-sim-desk-objects branch from e670555 to c86f75c Compare February 25, 2026 18:51

davetcoleman changed the title ~~Add lab desk objects to MuJoCo scene~~ Add lab desk objects, AprilTag bottle picking, and clean up lab_sim Feb 25, 2026

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 2 times, most recently from 95cabe9 to e35c146 Compare February 27, 2026 05:31

nbbrooks reviewed Mar 2, 2026

View reviewed changes

MikeWrock reviewed Mar 4, 2026

View reviewed changes

rlpratt12 self-requested a review March 4, 2026 21:25

rlpratt12 requested changes Mar 4, 2026

View reviewed changes

davetcoleman force-pushed the feature/lab-sim-desk-objects branch 8 times, most recently from 5b89b7a to 1c644c2 Compare March 5, 2026 00:52

davetcoleman marked this pull request as draft March 5, 2026 00:57

davetcoleman force-pushed the feature/lab-sim-desk-objects branch from 1c644c2 to adf2fbd Compare March 5, 2026 00:57

Add lab desk objects, adjust collision wall/tray positions, and clean…

71b922b

… up lab_sim Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

JWhitleyWork force-pushed the feature/lab-sim-desk-objects branch from adf2fbd to 71b922b Compare March 5, 2026 17:55

Conversation

davetcoleman commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Dependencies

Test plan

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbbrooks Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MikeWrock commented Feb 18, 2026

Uh oh!

rlpratt12 commented Feb 27, 2026

Uh oh!

nbbrooks left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbbrooks Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nbbrooks commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nbbrooks commented Mar 3, 2026

Uh oh!

davetcoleman commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

moveit_pro

moveit_pro_example_ws

picknik_accessories

Uh oh!

davetcoleman commented Mar 4, 2026

Uh oh!

nbbrooks commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rlpratt12 left a comment

Choose a reason for hiding this comment

Uh oh!

davetcoleman commented Mar 4, 2026

Uh oh!

davetcoleman commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

davetcoleman commented Feb 14, 2026 •

edited

Loading

nbbrooks Mar 4, 2026 •

edited

Loading

nbbrooks Mar 4, 2026 •

edited

Loading

nbbrooks commented Mar 3, 2026 •

edited

Loading

davetcoleman commented Mar 4, 2026 •

edited

Loading

nbbrooks commented Mar 4, 2026 •

edited

Loading

davetcoleman commented Mar 5, 2026 •

edited

Loading