feat(lab_sim): add Quest controller teleoperation objective#618
Draft
nbbrooks wants to merge 1 commit into
Draft
feat(lab_sim): add Quest controller teleoperation objective#618nbbrooks wants to merge 1 commit into
nbbrooks wants to merge 1 commit into
Conversation
Adds a single new "Quest Teleop" objective to lab_sim, plus the host-side infrastructure required to pair with the [meta_quest_teleoperation](https://github.com/PickNikRobotics/meta_quest_teleoperation) Unity app. Operators wearing a Quest can teleoperate the simulated UR through clutch-based pose tracking. How the disjoint TF trees are handled The Quest publishes controller pose in a `quest` frame anchored at the headset's spawn location. The robot's `world` frame is anchored at its base. We don't know the geometric relationship between them — it depends on where the operator is physically standing — and we don't try to measure it. The two TF trees are disjoint roots, and the design works by never crossing between them: * On grip press, snapshot both anchors: - controller pose at clutch time, expressed in `quest` - grasp_link pose at clutch time, expressed in `world` * While grip is held, each tick: - read the controller's current pose in `quest` - compute its pose relative to its clutch-time pose — a relative rigid-body transform with no inherent frame attachment - apply that same relative transform to the grasp_link's clutch-time pose; the result is a target pose in `world` - drive VFC at the target A basis change re-expresses the relative transform from Quest FLU (X=forward, Y=left, Z=up) into the IMarker EE convention used by `grasp_link` (X=left, Y=up, Z=forward) so the components line up when applied. The basis change is the conjugation R · X · R⁻¹, expressed with two existing core primitives — `TransformPoseWithPose` (pre-mul by R) followed by `TransformPose` (post-mul by R⁻¹). Specific to this EE — re-derive R for other conventions. Why no v1..v10 history? This is a clean cut. Earlier iterations explored in-app clutch logic, separate-subtree slot architectures, and various basis-change strategies. Only the final design ships: * App publishes raw pose + button state; host BT owns clutch + kinematics. * Disjoint quest/world TF roots are handled by snapshot + relative- transform composition, never by a frame crossing. * Basis-change is expressed via the two-call conjugation using only existing core primitives; no new behavior added. * Per-button slot actions are inline AlwaysSuccess in a single flat tree, so the objective is self-contained — no SubTree-files-to-copy. Files - src/lab_sim/objectives/quest_teleop.xml: the objective. Single flat BehaviorTree, runnable from the UI under the "User Input" subcategory. - src/lab_sim/launch/sim/robot_drivers_to_persist_sim.launch.py: drivers- to-persist launch override. Starts ros_tcp_endpoint (Unity bridge) and a debug-only world->quest static TF so RViz can render the in-quest-frame VisualizePose markers. The static TF is not used by any control path. Persists across agent_bridge restarts so the Quest's TCP socket stays up. - src/lab_sim/config/config.yaml: * simulated_robot_driver_persist_launch_file points at the new launch file (without this, the inherited empty launch wins and the Quest app never sees a peer). * experimental_behaviors loader added to behavior_loader_plugins, which provides the pose / vector / Odometry conversion, snapshot subscriber, and Bool publisher behaviors the objective uses. - .gitmodules: two new submodules under src/external_dependencies/: * moveit_pro_experimental_behaviors, pinned to feat/pose-vector-basis-snapshot-behaviors pending the PR landing on main; switch to `branch = main` once that merges. * ROS-TCP-Endpoint (Unity-Technologies), pinned to main-ros2. For end-to-end setup with the Quest app, see: https://docs.picknik.ai/hardware_guides/input_devices/setting_up_the_meta_quest_for_teleop/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2c0e116 to
082dc8d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a single new "Quest Teleop" objective to lab_sim, plus the host-side infrastructure required to pair with the
meta_quest_teleoperation Unity app.
How the disjoint TF trees are handled
The Quest publishes controller pose in a
questframe anchored at the headset's spawn location. The robot'sworldframe is anchored at its base. We don't know the geometric relationship between them — it depends on where the operator is physically standing — and we don't try to measure it. The two TF trees are disjoint roots, and the design works by never crossing between them:quest- grasp_link pose at clutch time, expressed inworldquestworld- drive VFC at the targetFiles
branch = mainonce that merges.For end-to-end setup with the Quest app, see:
https://docs.picknik.ai/hardware_guides/input_devices/setting_up_the_meta_quest_for_teleop/