Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
4dd39c8
hand-eye calibration branch created on relevant order between other d…
MithraGhlm Apr 9, 2026
c23f24a
Added the STag detector doc and rearranged the numbers
MithraGhlm Apr 20, 2026
8790c37
some text written for the STag marker detector.Related image added. T…
MithraGhlm Apr 20, 2026
a459c04
The STag detection video added.
MithraGhlm Apr 21, 2026
bd5c193
WIP: initial documentation draft
MithraGhlm Apr 21, 2026
0613b51
info for Aruco marker added.
MithraGhlm Apr 21, 2026
81dbf13
Decoupled the explanation of Marker detection from Hand-Eye calibration
MithraGhlm Apr 27, 2026
b5952af
Applied some grammatical, structural, and definitional improvements.
MithraGhlm Apr 27, 2026
413e081
canceled the small change made to the text.
MithraGhlm Apr 27, 2026
743eede
Renamed the file to show the marker detection process is generic
MithraGhlm Apr 27, 2026
a9a0028
The first version of introduction for the robot calibration guide.
MithraGhlm Apr 28, 2026
c0b4a83
added the screenshot of component configuration for the Hand-Eye cali…
MithraGhlm Apr 28, 2026
d904aeb
added calibration workflow.
MithraGhlm Apr 28, 2026
426fef6
one word replaced by a better one.
MithraGhlm Apr 28, 2026
c4da9af
added things that can be done with the acquired calibration info
MithraGhlm Apr 28, 2026
9ce7fec
corrected capital letter which was in the middle of the sentence.
MithraGhlm Apr 29, 2026
f2e20b7
adding a small frame to the image
MithraGhlm Apr 29, 2026
b12fd16
Added the missing word, Launcher.
MithraGhlm Apr 29, 2026
4f58281
sidebar position modified
MithraGhlm Apr 30, 2026
b290070
sidebar position modified
MithraGhlm Apr 30, 2026
8ec6d43
sidebar position modified
MithraGhlm Apr 30, 2026
94c8d66
The structure of the sentence improved.
MithraGhlm Apr 30, 2026
f256e5a
added fiducial to marker
MithraGhlm Apr 30, 2026
ce36b54
changed the name of the file to Fiducial Markers
MithraGhlm Apr 30, 2026
93f6f3f
structure of the sentence improved
MithraGhlm Apr 30, 2026
004055e
The structure of the sentence improved.
MithraGhlm Apr 30, 2026
b1d7970
Misspelling corrected
MithraGhlm Apr 30, 2026
07e4020
reminder to add the camera configuration file to the Camera Streamer …
MithraGhlm Apr 30, 2026
a4fa172
corrected the definition of Rate parameter of the STag Detector compo…
MithraGhlm Apr 30, 2026
d529009
the word stag_ replaced by the word Prefix
MithraGhlm Apr 30, 2026
43c9874
small typo corrected
MithraGhlm May 1, 2026
c59aa4a
adding the explanation of the Marker Size parameter
MithraGhlm May 1, 2026
e22827b
moved the robot camera calibration component explanation to the prope…
MithraGhlm May 1, 2026
a3bd7fc
screenshot of Robot Camera Calibration parameters added
MithraGhlm May 1, 2026
9d55015
app added for the HE calibration
MithraGhlm May 1, 2026
addc15b
the structure of some sentences enhanced
MithraGhlm May 1, 2026
6fea105
added the reminder to turn on the camera attached check
MithraGhlm May 1, 2026
c07b8c2
added the definition of Bundle file
MithraGhlm May 4, 2026
a747428
added the explanation for obtaining, downloading, and printing markers
MithraGhlm May 4, 2026
941b84b
rectified the file that Bundle File was explained in
MithraGhlm May 4, 2026
5417117
The wrong image removed
MithraGhlm May 4, 2026
b7b0bfc
typo correction
MithraGhlm May 8, 2026
346be2b
Some typo and font inconsistency corrected. Some definitions and pict…
MithraGhlm May 8, 2026
9c2edee
text added to explain the screenshot, HE caliberation configuration e…
MithraGhlm May 8, 2026
6ca8d8e
Bolding parameters
MithraGhlm May 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 13 additions & 13 deletions docs/core/examples/guides/abb-hardware-interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ ABB permits remote control of its manipulator range using the Externally Guided
provides external devices with the ability to send commands and control ABB robotic arms, using Google's Protocol
Buffers (Protobuf) serialization library to transport information through UDP sockets. For more information, check out
the
[official product documentation](https://library.e.abb.com/public/344f15f0f43341eb944fe35279d9fa2e/3HAC073319+AM+Externally+Guided+Motion+RW6-en.pdf?x-sign=WlxgV7Vao27KV3d3hlsfaoykgctYqoA0F98ch89S%2FPEaGwQg47ou%2FioylQtzvLaV).
[official product documentation](https://library.e.abb.com/public/344f15f0f43341eb944fe35279d9fa2e/3HAC073319+AM+Externally+Guided+Motion+RW6-en.pdf?x-sign=WlxgV7Vao27KV3d3hlsfaoykgctYqoA0F98ch89S%2FPEaGwQg47ou%2FioylQtzvLaV).

:::warning

EGM is an optional add-in and has to be purchased separately.
EGM is an optional add-in and has to be purchased separately.

:::

Expand All @@ -68,8 +68,8 @@ steps, which will be explained in the following sections. More information can b
### RAPID

RAPID is the programming language of ABB robots. Users can utilize RAPID to set up and execute their workflows and
processes. This is enabled by user-defined libraries called *Modules*, that contain variables and functions or
processes (*PROCs*). Modules can then be loaded in controller *Tasks*, and called as required.
processes. This is enabled by user-defined libraries called _Modules_, that contain variables and functions or
processes (_PROCs_). Modules can then be loaded in controller _Tasks_, and called as required.

## Connecting to a robot

Expand All @@ -93,7 +93,7 @@ virtual machine.
Setting up a virtual workstation and controller can be achieved by following the next steps:

1. In RobotStudio, navigate to the **Add-Ins** tab and go to **Gallery**. There is a list of all available robot models
and RobotWare versions, the internal controller software. To ensure consistency between simulation and reality, make
and RobotWare versions, the internal controller software. To ensure consistency between simulation and reality, make
sure to install the versions matching the real robot controller, if one is available.
<div class="text--center">
<img src={abbInstallAddins} alt="Install necessary addins in RobotStudio." />
Expand All @@ -109,14 +109,14 @@ Setting up a virtual workstation and controller can be achieved by following the
controller. Then select **Apply and Reset** to finalize.
:::note
The **3119-1 RobotStudio Connect** add-in is required to connect a controller to RobotStudio over a public network.
For more information, see the RobotStudio instruction manual.
For more information, see the RobotStudio instruction manual.
:::
<div class="text--center">
<img src={abbAdditionalOptions} alt="Additional options in the RobotStudio project." />
</div>
7. Disable the Windows firewall on the network where the PC running AICA Core is connected to.
8. Finally, the PC running AICA Core has to be whitelisted to communicate with RobotStudio. As explained
[here](https://forums.robotstudio.com/discussion/12082/using-robotwebservices-to-access-a-remote-virtual-controller),
[here](https://forums.robotstudio.com/discussion/12082/using-robotwebservices-to-access-a-remote-virtual-controller),
create a file called `vcconf.xml` under `C:/Users/<user>/AppData/Roaming/ABB Industrial IT/Robotics IT/RobVC` with
the content below. Replace `<user>` in the path above with your Windows user and the IP address in the snippet below with
the IP of the PC running AICA Core (in this example 192.168.137.100).
Expand Down Expand Up @@ -387,24 +387,24 @@ robot. The majority of the hardware interface parameters enable connection to EG
<img src={abbHIParameters} alt="ABB Hardware interface parameters" />
</div>

- RWS IP & port: the address and port of the RWS server, e.g. the address of the real robot or the RobotStudio device
- **RWS IP & port**: the address and port of the RWS server, e.g. the address of the real robot or the RobotStudio device
:::note
OmniCore controllers and RobotWare 7.x versions by default listen on HTTPS and port 80 for RobotStudio and 443 for the
real robot. If necessary, the port numbers can be modified by following the instructions in this
[forum post](https://forums.robotstudio.com/discussion/12177/how-to-change-the-listening-port-of-the-virtual-controller-robotware-6-x-and-7-x).
:::
- EGM port: the port of the EGM server, e.g. the *Remote Port Number* of the UDPUC device defined in RobotStudio above
- Connection timeout: the amount of time the hardware interface tries to connect to the RWS and EGM servers before
- **EGM port**: the port of the EGM server, e.g. the _Remote Port Number_ of the UDPUC device defined in RobotStudio above
- **Connection timeout**: the amount of time the hardware interface tries to connect to the RWS and EGM servers before
reporting an error
- Controller Task and Main Module name: Depends on the controller configuration and usually defaults to `T_ROB1` and
- **Controller Task and Main Module name**: Depends on the controller configuration and usually defaults to `T_ROB1` and
`AICA_EGM`, respectively.
- Uc Device: The name of the UDPUC device configured above.
- **Uc Device**: The name of the UDPUC device configured above.

Before starting an application with an ABB hardware interface in AICA Studio, the motors and RAPID program on the robot
must be started manually through the teach pendant or RobotStudio. After that, running the application will connect to
the robot and get information about the mechanical setup of the robot being used.

<!-- TODO: link example -->
<!-- TODO: link example -->

<!-- <div class="text--center">
<img src={abbRSSuccessfulConnection} alt="Connected to RobotStudio successfully." />
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
36 changes: 19 additions & 17 deletions docs/core/examples/guides/fanuc-hardware-interface.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ FANUC is one of the largest and most established industrial robotic manipulator
technology and robust performance for a wide range of automation tasks.

FANUC is one of the first major robot brands that enables accelerating Physical AI implementation through an official
first-party ROS driver in the `ros2_control` framework
first-party ROS driver in the `ros2_control` framework
([article from Dec 2025](https://www.fanuc.co.jp/en/product/new_product/2025/202512_robot_physicalai.html)). This allows
advanced programming platforms like the AICA System to control the robots in real-time and deploy complex behaviors
without custom implementations.
Expand All @@ -27,6 +27,7 @@ hardware compatibility. The minimum robot controller software versions are:
- R-50iA series: V10.10P/26

Additionally, one of the following software options are required:

- J519 Stream Motion and R912 Remote Motion
- S636 External Control Package, which includes the previous

Expand Down Expand Up @@ -67,16 +68,16 @@ robot:
<img src={fanucHI} alt="FANUC hardware interface." style={{ borderRadius: "8px" }} />
</div>

- Robot IP: the IP address of the robot
- RMI port: the port of the Remote Motion Interface (keep default unless otherwise configured on the robot controller)
- Stream Motion Port: the port of Stream Motion (keep default unless otherwise configured on the robot controller)
- GPIO configuration (v1.1.0 and above): an absolute path to a YAML file containing the GPIOs that should be configured
- **Robot IP**: the IP address of the robot
- **RMI port**: the port of the Remote Motion Interface (keep default unless otherwise configured on the robot controller)
- **Stream Motion Port**: the port of Stream Motion (keep default unless otherwise configured on the robot controller)
- **GPIO configuration (v1.1.0 and above)**: an absolute path to a YAML file containing the GPIOs that should be configured
(see more information below)
- Payload Schedule: the number of the payload to be set on the robot
- Out Cmd Interp Buff Target: Output command interpolation buffer target size for stream motion control
- Force Sensor Type (v1.1.0 and above): The type of force torque sensor to configure (see more information below)
- **Payload Schedule**: the number of the payload to be set on the robot
- **Out Cmd Interp Buff Target**: Output command interpolation buffer target size for stream motion control
- **Force Sensor Type (v1.1.0 and above)**: The type of force torque sensor to configure (see more information below)

Click **Start** to start the application and connect to the robot.
Click **Start** to start the application and connect to the robot.

### GPIO configuration

Expand All @@ -90,14 +91,14 @@ the driver starts. For detailed description, refer to the relevant sections in t
The official documentation states the behavior as follows:

> When outputs or numeric registers are added to the command section of the GPIO configuration YAML file, once the ROS 2
driver is launched those output and numeric register values in the controller will be set to false or zero.
> driver is launched those output and numeric register values in the controller will be set to false or zero.

However, the version of the driver provided by AICA takes additional care **not** to overwrite the value of the GPIOs on startup,
and instead persists the initial value set on the robot controller.

:::

A example configuration of a GPIO configuration file could look as follows:
An example configuration of a GPIO configuration file could look as follows:

```yaml
gpio_topic_config:
Expand All @@ -115,10 +116,12 @@ gpio_topic_config:
```

This configuration claims reads interfaces

- DI[101], DI[102], and DI[103] (`start` is 101 and `length` is 3)
- DO[101] (`start` is 101 and `length` is 1)

and writes to interfaces

- DO[101] and DO[102] (`start` is 101 and `length` is 2)

In order for this configuration to be accepted by the hardware interface, it is necessary to add matching state and
Expand All @@ -143,21 +146,20 @@ command interfaces to the `ros2_control` section of the URDF as follows:

### Force torque sensor configuration

Starting with driver version `collections/fanuc:v1.1.0` and robot controller software V9.40/P85, it is possible to read
force torque sensor values directly from the robot. The type of force sensor can be configured through the `force_sensor_type`
Starting with driver version `collections/fanuc:v1.1.0` and robot controller software V9.40/P85, it is possible to read force torque sensor values directly from the robot. The type of force sensor can be configured through the `force_sensor_type`
hardware parameter. Available types are:

- 0: Unselected
- 1: Embedded: for all CRX models with inbuilt sensor
- 2: External: for robots with an external FANUC force torque sensor mounted at the flange
- **0**: Unselected
- **1**: Embedded: for all CRX models with inbuilt sensor
- **2**: External: for robots with an external FANUC force torque sensor mounted at the flange

:::note

Users without software V9.40/P85 should keep the type at 0 to be able to start the hardware interface.

:::

When the `force_sensor_type` is set to 1 or 2, the URDF must include the following `sensor` interfaces within the `ros2_control`
When the `force_sensor_type` is set to 1 or 2, the URDF must include the following `sensor` interfaces within the `ros2_control`
element.

```xml
Expand Down
107 changes: 107 additions & 0 deletions docs/core/examples/guides/fiducial-markers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
---
sidebar_position: 14
title: Fiducial Markers
---

import stagDetectorExample from './assets/stag-detector-example.png'
import stagMarkerDetection from './assets/stag-marker-detection.webm'

# Fiducial Markers

Different types of fiducial markers are used in robotics to provide precise 3D pose estimation and identification for cameras, enabling or improving robotic calibration and object manipulation.

AICA's `core-vision` package gives you the choice between using two commonly used markers, the STag and ArUco.

:::tip
Performing the [intrinsic calibration](./camera-calibration.md) of the camera improves the precision for fiducial marker detection and tracking.
:::

This guide provides an example of STag marker detection. Using the ArUco marker follows a very similar process.

## Preparing fiducial markers

A fiducial marker is an object placed in the field of view of an image for use as a point of reference or a measure. STag and ArUco markers are two of the common types of fiducial marker systems used for real-time 6D pose estimation. This section explains how to obtain, download and print these markers.

### Obtaining markers

- **ArUco marker**: ArUco markers can be generate online (e.g., from [here](https://chev.me/arucogen/)), which permits choosing the dictionary, marker ID, and marker size. It can be then exported as PDF or SVG for printing.

- **STag marker**: STag marker set can be either downloaded from [google public drive](https://drive.google.com/drive/folders/0ByNTNYCAhWbIV1RqdU9vRnd2Vnc?resourcekey=0-9ipvecbezW8EWUva5GBQTQ) or obtained from the [STag project repository](https://github.com/usrl-uofsc/stag_ros) or the generator/reference files linked by the project. The [STag project repository for ROS](https://github.com/usrl-uofsc/stag_ros) is the other place to look for marker-generation assets. In practice, you’ll want to obtain the marker PDF/SVG or generate the markers from the project’s reference generator, then print them at true size.

### Printing markers

After choosing the marker family, selecting the library/dictionary, and the marker ID, download it as PDF or SVG. Use the Actual size of the marker (100% scale) for printing, so the black border and marker geometry are not resized. Also it is recommended to print with high contrast and avoid compression artifacts.

If possible, print on a rigid and flat sheet of paper to reduce warping, since fiducial detection is sensitive to distortion. As another solution, you can fix the printed marker on a rigid surface, such as a piece of wood or cardboard.

After printing, measure the marker’s outer dimensions and compare them with the intended size from the generator. This matters because calibration will be wrong if the marker size in the software does not match the physical print.

## Using the STag detector

Launch AICA Studio with a configuration that contains the `core-vision` package and create a new application.

1. Remove the hardware interface that is included in new applications by default.
2. From the `Scene` menu, use the `Add Component` tab and look for the **Camera Streamer** and **STag Detector** components, either by searching
or by manually going under the `Core Vision Components` menu. Add both of them to the graph.
3. Next, connect both components to the start block. Moreover, connect the outputs of the Camera Streamer to the relevant inputs of the STag Detector.
4. Enable **auto-configure** and **auto-activate** on both components.
5. By selecting any of the components, you can find all the available component parameters in the right panel under Settings.
6. If an intrinsic camera calibration is performed prior to this, add the file path of the camera configuration file as a parameter to the **Camera Streamer** component.

By this point, you should have something like the following:

<div class="text--center">
<img src={stagDetectorExample} alt="CameraStreamer configuration alongside STagDetector component" />
</div>

:::info
The Camera Streamer parameters are explained in the [CameraStreamer component guide](./camera-streamer.md).
:::

STag Detector component has three predicates that are as follow:

- **Is any marker detected**: This predicate will be set to `True` if any marker is detected in the camera frame.
- **Is any selected marker detected**: If one or more marker names are indicated in the `Marker selection` parameter, this predicate with be set to `True`. If the names of none of the markers present in the camera frame is indicated as a parameter, this predicate will remain `False`.
- **Is a marker bundle detected**: If a registered group of markers is detected by the camera, this parameter will be set to `True`.

Now let us go through the parameters of the STag Detector component:

- **Rate**: The rate parameter doesn't affect the behavior of the component as the detection process
occurs on reception of a new image.
- **Bundle file**: The filepath to a predefined marker bundle configuration. This additional feature is described in a separate guide (coming soon).
- **Marker selection**: The name(s) of the marker(s) that we intend to be recognized. If any of these markers enters the camera frame, the `is_any_selected_marker_detected` predicate is set to **True**. This name should always be prepended with the value of `Prefix`.
- **Marker size**: Determines the side length of a square that specifies the marker in meters.
- **Library**: This is the ID number of the HD library utilized by STag markers. The allowed numbers are `[11, 13, 15, 17, 19, 21, 23]`.
- **Error correction**:
- **Prefix**: This prefix is used for marker names.

:::tip

If a decision needs to be made based on the existence of a specific STag marker in the camera frame, its name should be indicated in the `Marker Selection` parameter.

:::

:::info

If any marker is detected in the camera frame at all, the `is_any_marker_detected` predicate is set to **True**.

:::

After setting up the proper parameters for Camera Streamer and STag Detector:

1. Press **Start** to start the application.
2. To see the live camera feed, select **Launch RViz** from the Launcher settings
3. In RViz, select _Add > By topic > /stag_detector/annotated_image > Image_. This adds a panel that shows the live image. The marker should be detected in the camera.

<div style={{ display: "flex", justifyContent: "center" }}>
<video autoPlay loop muted playsInline style={{ maxWidth: "100%", borderRadius: "8px" }}>
<source src={stagMarkerDetection} type="video/webm" />
STag marker detection video.
</video>
</div>

:::info

The process for using ArUco markers follows a similar process.

:::
Loading