docs: Hand-Eye calibration guide #479

docs/core/examples/guides/fiducial-markers.md

@@ -0,0 +1,109 @@
+---
+sidebar_position: 14
+title: Fiducial Markers
+---
+
+import stagDetectorExample from './assets/stag-detector-example.png'
+import stagMarkerDetection from './assets/stag-marker-detection.webm'
+import stagMarkerNumOne from './assets/stagDetector-predicates_1.png'
+import stagMarkerNumZero from './assets/stagDetector-predicates_0.png'
+
+# Fiducial Markers
+
+Different types of fiducial markers are used in robotics to provide precise 3D pose estimation and identification for cameras, enabling or improving robotic calibration and object manipulation.
+
+AICA's `core-vision` package gives you the choice between using two commonly used markers, the STag and ArUco.
+
+:::tip
+Performing the [intrinsic calibration](./camera-calibration.md) of the camera improves the precision for fiducial marker detection and tracking.
+:::
+
+This guide provides an example of STag marker detection. Using the ArUco marker follows a very similar process.
+
+## Preparing fiducial markers
+
+A fiducial marker is an object placed in the field of view of an image for use as a point of reference or a measure. STag and ArUco markers are two of the common types of fiducial marker systems used for real-time 6D pose estimation. This section explains how to obtain, download and print these markers.
+
+### Obtaining markers
+
+- **ArUco marker**: ArUco markers can be generate online (e.g., from [here](https://chev.me/arucogen/)), which permits choosing the dictionary, marker ID, and marker size. It can be then exported as PDF or SVG for printing.
+
+- **STag marker**: STag marker set can be either downloaded from [public Google Drive](https://drive.google.com/drive/folders/0ByNTNYCAhWbIV1RqdU9vRnd2Vnc?resourcekey=0-9ipvecbezW8EWUva5GBQTQ) or obtained from the [ROS2 STag project repository](https://github.com/usrl-uofsc/stag_ros/tree/ros2-devel) or the generator/reference files linked by the project. In practice, you’ll want to obtain the marker PDF/SVG or generate the markers from the project’s reference generator, then print them at true size.
+
+### Printing markers
+
+After choosing the marker family, selecting the library/dictionary, and the marker ID, download it as PDF or SVG. Use the actual size of the marker (100% scale) for printing, so the black border and marker geometry are not resized. Also it is recommended to print with high contrast and avoid compression artifacts.
+
+If possible, print on a rigid and flat sheet of paper to reduce warping, since fiducial detection is sensitive to distortion. As another solution, you can fix the printed marker on a rigid surface, such as a piece of wood or cardboard.
+
+After printing, measure the marker’s outer dimensions and compare them with the intended size from the generator. This matters because calibration will be wrong if the marker size in the software does not match the physical print.
+
+## Using the STag detector
+
+Launch AICA Studio with a configuration that contains the `core-vision` package and create a new application.
+
+1. Remove the hardware interface that is included in new applications by default.
+2. From the `Scene` menu, use the `Add Component` tab and look for the **Camera Streamer** and **STag Detector** components, either by searching
+   or by manually going under the `Core Vision Components` menu. Add both of them to the graph.
+3. Next, connect both components to the start block. Moreover, connect the outputs of the Camera Streamer to the relevant inputs of the STag Detector.
+4. Enable **auto-configure** and **auto-activate** on both components.
+5. By selecting any of the components, you can find all the available component parameters in the right panel under Settings.
+6. If an intrinsic camera calibration is performed prior to this, add the file path of the camera configuration file as a parameter to the **Camera Streamer** component.
+
+By this point, you should have something like the following:
+
+<div class="text--center">
+  <img src={stagDetectorExample} alt="CameraStreamer configuration alongside STagDetector component" style={{ borderRadius: "8px" }}/>
+</div>
+
+:::info
+The Camera Streamer parameters are explained in the [CameraStreamer component guide](./camera-streamer.md).
+:::
+
+## STag Detector parameters
+
+- **Rate**: The rate parameter doesn't affect the behavior of the component as the detection process
+  occurs on reception of a new image.
+- **Bundle file**: The filepath to a predefined marker bundle configuration. This additional feature is described in a separate guide (coming soon).
+- **Marker selection**: The name(s) of the marker(s) that we want to recognize. If any of these markers enters the camera frame, the `is_any_selected_marker_detected` predicate is set to **True**. Also if a decision needs to be made based on the existence of a specific STag marker in the camera frame, its name should be indicated in this parameter. The markers name should always be prepended with the value of `Prefix`.
+- **Marker size**: Determines the side length of a square that specifies the marker in meters.
+- **Library**: This is the ID number of the HD library utilized by STag markers. The allowed numbers are `[11, 13, 15, 17, 19, 21, 23]`.
+- **Error correction**:
+- **Prefix**: This prefix is used for marker names.
+
+## STag Detector predicates
+
+- **Is any marker detected**: This predicate will be set to **True** if any marker is detected in the camera frame, even though its name is not indicated in the 'Marker Selection' parameter.
+  As you can see in the screenshot below, the marker name specified in the `Marker selection` parameter is stag_1, but the marker recognized in the camera frame is stag_0, yet `Is any marker detected` predicate is set to **True**.
+
+<div class="text--center">
+  <img src={stagMarkerNumOne} alt="Is any marker detected at all" style={{ borderRadius: "8px" }}/>
+</div>
+
+- **Is any selected marker detected**: If one or more marker names are indicated in the `Marker selection` parameter, and if any of them appears in the camera frame, this predicate with be set to **True**. If the names of none of the markers present in the camera frame is indicated as a parameter, this predicate will remain **False**.
+  In the screenshot below the name of the marker appearing in the camera frame matches the name indicated in the `Marker selection` parameter.
+
+<div class="text--center">
+  <img src={stagMarkerNumZero} alt="Is any of the selected markers detected" style={{ borderRadius: "8px" }}/>
+</div>
+
+- **Is a marker bundle detected**: If a registered group of markers is detected by the camera, this parameter will be set to **True**. Otherwise it will remain **False**.
+
+After setting up the proper parameters for Camera Streamer and STag Detector:
+
+1. Press **Start** to start the application.
+2. To see the live camera feed, select **Launch RViz** from the Launcher settings
+3. In RViz, select _Add > By topic > /stag_detector/annotated_image > Image_. This adds a panel that shows the live image. The marker should be detected in the camera.
+
+<div style={{ display: "flex", justifyContent: "center" }}>
+  <video autoPlay loop muted playsInline style={{ maxWidth: "100%", borderRadius: "8px" }}>
+    <source src={stagMarkerDetection} type="video/webm" />
+    STag marker detection video.
+  </video>
+</div>
+
+:::info
+
+The process for using ArUco markers follows a similar process.
+
+:::

docs/core/examples/guides/hand-eye-calibration.md

@@ -0,0 +1,250 @@
+---
+sidebar_position: 15
+title: Hand-Eye calibration
+---
+
+import RobotCalibrationConfiguration from './assets/robot-calibration-configuration.png'
+import RobotCameraCalibration from './assets/robot-camera-calibration-component.png'
+import URCalibrationFile from './assets/UR-calibration-file.png'
+
+# Hand-Eye calibration
+
+Robot calibration is a fundamental prerequisite for any robotic system that relies on precise coordination between a vision sensor and a manipulator. It establishes the spatial transformation between the robot’s end-effector and the camera frame, enabling accurate mapping between observed features and actionable robot coordinates. Without proper calibration, even high-quality perception or motion planning algorithms can yield significant positioning errors.
+
+The AICA's `core-vision` package provides a structured workflow for performing hand–eye calibration efficiently and reproducibly.
+
+Accurate hand–eye calibration is critical in tasks such as visual servoing, object manipulation, inspection, and assembly. This tool is designed to minimize discrepancies and provide reliable calibration outputs suitable for industrial environments.
+
+## Robot Camera Calibration component
+
+The Robot Camera Calibration component is the component that does the main job of calculating the transformation between the `camera` and the `robot end-effector`.
+
+Click on the Robot Camera Calibration component block to view and edit the available parameters.
+
+<div class="text--center">
+  <img src={RobotCameraCalibration} alt="Robot Camera Calibration parameters" style={{ borderRadius: "8px" }}/>
+</div>
+
+The parameters of the Robot Camera Calibration component are defined as follows:
+
+- **Rate**: Determines the frequency at which transformations are acquired. This parameter does not affect the component’s behavior.
+- **Bundle file**: The filepath to a predefined marker bundle configuration. This additional feature is described in a separate guide (coming soon).
+- **Camera frame**: The name of the camera frame used in the application. This can be retrieved from the list of frames in RViz.
+- **Marker frame**: The name of the marker detected by the camera. It is indicated in the STag or ArUco marker detector component used in the application.
+- **Robot base frame**: The name of the robot's base frame, which can be found in RViz.
+- **Robot end-effector frame**: The name of the robot’s end-effector frame. This is also available in the RViz frame list.
+- **Number of recorded points**: The number of transformations from `camera` to `robot end-effector` that are recorded. These are computed from multiple pairs of `robot end-effector -> robot base` and `camera -> marker` transformations.
+- **Distance between points**: The minimum spatial difference required between two consecutive transformations for them to be recorded.
+- **Epsilon time**:
+- **Calibration folder path**: The directory where the final calibration file will be stored.
+- **Calibration file**: The name of the output file generated after calibration, containing the computed calibration data.
+- **Points dataset file**: A file containing previously recorded transformations of the robot end-effector or marker, which can be reused if available.
+- **Reset calibration**: Setting this component to `True` resets the calibration. Meaning it starts the calibration process again, even if the calibration matrices are already set.
+- **Reset dataset**: If set to `True`, the recording process will re-start, even if the dataset of points already exists.
+- **Is camera attached**: This parameter should be set to `True` if a physical camera is attached to the to the robot end-effector.
+
+## Robot Calibration using AICA Studio and a marker
+
+After completing the camera calibration as described in the [Camera Calibration example](./camera-calibration.md), and verifying marker detection as outlined in the [Fiducial Markers](./fiducial-markers.md) section, you can proceed with the hand–eye calibration process.
+
+This example demonstrates the eye-in-hand configuration (camera mounted on the robot arm). The procedure for the eye-to-hand configuration (static camera) follows a similar workflow.
+
+- Ensure that the camera is properly configured and operational.
+- Connect all required components and controllers in the AICA application. Refer to the system setup illustration below for guidance.
+- Place the marker within the robot workspace, ensuring it is fully visible to the camera. To monitor the live camera feed, enable `Launch RViz` from the Launcher settings, as described in the [marker detection](./marker-detection.md) guide.
+- Run the program and move the robot TCP (Tool Center Point) to capture images of the marker from multiple perspectives. The application starts capturing images automatically.
+- The robot TCP can be moved by jogging or Freedrive mode using the robot's teach pendant. In the case of using a Universal Robot, AICA Studio offers the option of `Hand Guiding Controller` which facilitates and accelerates the process. This controller is described in the [Hand Guiding Controller](./ur-harware-interface.md) page.
+- Ensure sufficient variation in position and orientation to improve calibration accuracy.
+- Once the number of captured images reaches the `Number of recorded points` indicated in the Robot Camera Calibration component parameters, the system automatically generates a calibration file in YAML format in the following directory:
+
+```bash
+/tmp/calibration/camera_calibration.yaml
+```
+
+An example of the calibration file:
+
+<div class="text--center">
+  <img src={URCalibrationFile} alt="An example showing a final calibration file" style={{ borderRadius: "8px" }}/>
+</div>
+
+In the screenshot below you can see an example of the components configuration for the Hand-Eye calibration, that resulted to the output above. Notice that the Camera and the marker detector components might differ based on the type of hardware being used.
+
+<div class="text--center">
+  <img src={RobotCalibrationConfiguration} alt="The configuration required for hand-eye calibration" style={{ borderRadius: "8px" }}/>
+</div>
+
+The following YAML snippet contains the full application of the image above:
+
+<details>
+  <summary>Example application, Hand-Eye calibration</summary>
+
+```yaml
+schema: 2-0-6
+dependencies:
+  core: v5.1.0
+on_start:
+  load:
+    - component: orbbec_camera
+    - component: robot_camera_calibration
+    - component: stag_detector
+    - hardware: hardware
+components:
+  orbbec_camera:
+    component: orbbec_camera::OBCameraNodeDriver
+    display_name: Orbbec Camera
+    outputs:
+      color_image: /orbbec_camera/color_image
+      color_camera_info: /orbbec_camera/color_camera_info
+  robot_camera_calibration:
+    component: core_vision_components::calibration::RobotCameraCalibration
+    display_name: Robot Camera Calibration
+    events:
+      transitions:
+        on_load:
+          lifecycle:
+            component: robot_camera_calibration
+            transition: configure
+        on_configure:
+          lifecycle:
+            component: robot_camera_calibration
+            transition: activate
+    parameters:
+      camera_frame:
+        value: orbbec_camera_link
+        type: string
+      marker_frame:
+        value: stag_0
+        type: string
+      robot_base_frame:
+        value: world
+        type: string
+      robot_ee_frame:
+        value: ur_tool0
+        type: string
+      is_camera_attached:
+        value: true
+        type: bool
+  stag_detector:
+    component: core_vision_components::pose_detection::STagDetector
+    display_name: STag Detector
+    events:
+      transitions:
+        on_load:
+          lifecycle:
+            component: stag_detector
+            transition: configure
+        on_configure:
+          lifecycle:
+            component: stag_detector
+            transition: activate
+    parameters:
+      marker_selection:
+        value:
+          - stag_0
+        type: string_array
+    inputs:
+      image: /orbbec_camera/color_image
+      camera_info: /orbbec_camera/color_camera_info
+hardware:
+  hardware:
+    display_name: Hardware Interface
+    urdf: Universal Robots 5e
+    rate: 500
+    events:
+      transitions:
+        on_load:
+          load:
+            - controller: robot_state_broadcaster
+              hardware: hardware
+            - controller: ur_hand_guiding_controller
+              hardware: hardware
+    parameters:
+      robot_ip: 192.168.42.20
+    controllers:
+      robot_state_broadcaster:
+        plugin: aica_core_controllers/RobotStateBroadcaster
+        events:
+          transitions:
+            on_load:
+              switch_controllers:
+                hardware: hardware
+                activate: robot_state_broadcaster
+      ur_hand_guiding_controller:
+        plugin: aica_ur_controllers/URHandGuidingController
+        parameters:
+          ft_sensor_name:
+            value: ur_tcp_fts_sensor
+            type: string
+          ft_sensor_reference_frame:
+            value: ur_tool0
+            type: string
+          force_limit:
+            value:
+              - 20
+              - 20
+              - 20
+              - 2
+              - 2
+              - 2
+            type: vector
+        events:
+          transitions:
+            on_load:
+              switch_controllers:
+                hardware: hardware
+                activate: ur_hand_guiding_controller
+graph:
+  positions:
+    on_start:
+      x: 120
+      y: 0
+    stop:
+      x: 120
+      y: 100
+    components:
+      orbbec_camera:
+        x: 340
+        y: 0
+      robot_camera_calibration:
+        x: 340
+        y: 320
+      stag_detector:
+        x: 800
+        y: -80
+    hardware:
+      hardware:
+        x: 1260
+        y: -120
+  edges:
+    on_start_on_start_robot_camera_calibration_robot_camera_calibration:
+      path:
+        - x: 260
+          y: 60
+        - x: 260
+          y: 380
+    on_start_on_start_stag_detector_stag_detector:
+      path:
+        - x: 260
+          y: 60
+        - x: 260
+          y: -20
+    on_start_on_start_hardware_hardware:
+      path:
+        - x: 260
+          y: 60
+        - x: 260
+          y: -60
+```
+
+</details>
+
+:::tip
+Don't forget to modify the `robot_ip` according to the ip of the robot that you are using.
+
+:::
+
+There are several ways to use the transformation information obtained in the calibration file:
+
+1. Adding the camera link to the URDF.
+2. Publish the transformation with a `FrameBroadcaster` manually.
+3. Using the calibration file path directly as a Frame Broadcaster component parameter to publish the transformations.