Skip to content

feat(lab_sim): add Quest controller teleoperation objective#618

Draft
nbbrooks wants to merge 1 commit into
mainfrom
feat/add-quest-teleop
Draft

feat(lab_sim): add Quest controller teleoperation objective#618
nbbrooks wants to merge 1 commit into
mainfrom
feat/add-quest-teleop

Conversation

@nbbrooks
Copy link
Copy Markdown
Member

Adds a single new "Quest Teleop" objective to lab_sim, plus the host-side infrastructure required to pair with the
meta_quest_teleoperation Unity app.

How the disjoint TF trees are handled

The Quest publishes controller pose in a quest frame anchored at the headset's spawn location. The robot's world frame is anchored at its base. We don't know the geometric relationship between them — it depends on where the operator is physically standing — and we don't try to measure it. The two TF trees are disjoint roots, and the design works by never crossing between them:

  • On grip press, snapshot both anchors: - controller pose at clutch time, expressed in quest - grasp_link pose at clutch time, expressed in world
  • While grip is held, each tick:
    • read the controller's current pose in quest
    • compute its pose relative to its clutch-time pose — a relative rigid-body transform with no inherent frame attachment - apply that same relative transform to the grasp_link's clutch-time pose; the result is a target pose in world - drive VFC at the target

Files

  • src/lab_sim/objectives/quest_teleop.xml: the objective. Single flat BehaviorTree, runnable from the UI under the "User Input" subcategory.
  • src/lab_sim/launch/sim/robot_drivers_to_persist_sim.launch.py: drivers- to-persist launch override. Starts ros_tcp_endpoint (Unity bridge) and a debug-only world->quest static TF so RViz can render the in-quest-frame VisualizePose markers. The static TF is not used by any control path. Persists across agent_bridge restarts so the Quest's TCP socket stays up.
  • src/lab_sim/config/config.yaml:
    • simulated_robot_driver_persist_launch_file points at the new launch file (without this, the inherited empty launch wins and the Quest app never sees a peer).
    • experimental_behaviors loader added to behavior_loader_plugins, which provides the pose conversion, basis change, snapshot subscriber, and Bool/Empty publisher behaviors the objective uses.
  • .gitmodules: two new submodules under src/external_dependencies/:
    • moveit_pro_experimental_behaviors, pinned to feat/pose-vector-basis-snapshot-behaviors pending the PR landing on main; switch to branch = main once that merges.
    • ROS-TCP-Endpoint (Unity-Technologies), pinned to main-ros2.

For end-to-end setup with the Quest app, see:
https://docs.picknik.ai/hardware_guides/input_devices/setting_up_the_meta_quest_for_teleop/

Adds a single new "Quest Teleop" objective to lab_sim, plus the host-side
infrastructure required to pair with the
[meta_quest_teleoperation](https://github.com/PickNikRobotics/meta_quest_teleoperation)
Unity app. Operators wearing a Quest can teleoperate the simulated UR
through clutch-based pose tracking.

How the disjoint TF trees are handled

The Quest publishes controller pose in a `quest` frame anchored at the
headset's spawn location. The robot's `world` frame is anchored at its
base. We don't know the geometric relationship between them — it depends
on where the operator is physically standing — and we don't try to
measure it. The two TF trees are disjoint roots, and the design works by
never crossing between them:

  * On grip press, snapshot both anchors:
      - controller pose at clutch time, expressed in `quest`
      - grasp_link pose at clutch time, expressed in `world`
  * While grip is held, each tick:
      - read the controller's current pose in `quest`
      - compute its pose relative to its clutch-time pose — a relative
        rigid-body transform with no inherent frame attachment
      - apply that same relative transform to the grasp_link's
        clutch-time pose; the result is a target pose in `world`
      - drive VFC at the target

A basis change re-expresses the relative transform from Quest FLU
(X=forward, Y=left, Z=up) into the IMarker EE convention used by
`grasp_link` (X=left, Y=up, Z=forward) so the components line up when
applied. The basis change is the conjugation R · X · R⁻¹, expressed with
two existing core primitives — `TransformPoseWithPose` (pre-mul by R)
followed by `TransformPose` (post-mul by R⁻¹). Specific to this EE —
re-derive R for other conventions.

Why no v1..v10 history? This is a clean cut. Earlier iterations explored
in-app clutch logic, separate-subtree slot architectures, and various
basis-change strategies. Only the final design ships:
  * App publishes raw pose + button state; host BT owns clutch + kinematics.
  * Disjoint quest/world TF roots are handled by snapshot + relative-
    transform composition, never by a frame crossing.
  * Basis-change is expressed via the two-call conjugation using only
    existing core primitives; no new behavior added.
  * Per-button slot actions are inline AlwaysSuccess in a single flat tree,
    so the objective is self-contained — no SubTree-files-to-copy.

Files
- src/lab_sim/objectives/quest_teleop.xml: the objective. Single flat
  BehaviorTree, runnable from the UI under the "User Input" subcategory.
- src/lab_sim/launch/sim/robot_drivers_to_persist_sim.launch.py: drivers-
  to-persist launch override. Starts ros_tcp_endpoint (Unity bridge) and a
  debug-only world->quest static TF so RViz can render the in-quest-frame
  VisualizePose markers. The static TF is not used by any control path.
  Persists across agent_bridge restarts so the Quest's TCP socket stays
  up.
- src/lab_sim/config/config.yaml:
    * simulated_robot_driver_persist_launch_file points at the new launch
      file (without this, the inherited empty launch wins and the Quest
      app never sees a peer).
    * experimental_behaviors loader added to behavior_loader_plugins,
      which provides the pose / vector / Odometry conversion, snapshot
      subscriber, and Bool publisher behaviors the objective uses.
- .gitmodules: two new submodules under src/external_dependencies/:
    * moveit_pro_experimental_behaviors, pinned to
      feat/pose-vector-basis-snapshot-behaviors pending the PR landing on
      main; switch to `branch = main` once that merges.
    * ROS-TCP-Endpoint (Unity-Technologies), pinned to main-ros2.

For end-to-end setup with the Quest app, see:
https://docs.picknik.ai/hardware_guides/input_devices/setting_up_the_meta_quest_for_teleop/

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@nbbrooks nbbrooks force-pushed the feat/add-quest-teleop branch from 2c0e116 to 082dc8d Compare May 11, 2026 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant