Alignment mode broken for Orbbec Femto Bolt recording

[NIkolaStaykov]

Hi all,
I use your Orbbec wrapper to record data with a Femto Bolt camera. Guided by your recording example here: recording script, I am trying to set config.alignedDepth = true; . When I do that, however, obPipeline->getCameraParam(); return only zeroes after the spectacularAI::orbbecPlugin::Pipeline vioPipeline(*obPipeline, config, mappingCallback); initialization. After that, a crash follows. The error can be reproduced directly with the script mentioned above.
I checked that the depth stream resolution (512x512) supports depth alignment with the OrbbecSdk. Moreover, the ROS1 Wrapper for OrbbecSDK produces correctly aligned images with the same settings. This is why I consider this a SAI issue.

The SDK version is 1.34

PS. Can I set my custom camera intrinsics calibration with your SDK? I find the factory one to be non-satisfactory.

[kaatrasa]

Hey!

It's a known issue to us that the alignedDepth mode is broken with Orbbec Femto Bolt. However, I think the issue is on Orbbec side since the alignedDepth mode does work with Femto Mega & Astra 2.

Internally, all the flag does is set:

if (configuration.alignedDepth) {
    obConfig->setAlignMode(ALIGN_D2C_HW_MODE);
} else {
    obConfig->setAlignMode(ALIGN_DISABLE);
}

We currently don't have a Femto Bolt device, so I cannot check if the issue has been fixed if we updated our OrbbecSDK version (currently v1.10.5). Previously, enabling ALIGN_D2C_HW_MODE mode caused camera intrinsics to be 0s with Femto Bolt like you mentioned.

In any case, I would recommend not using the alignedDepth mode with Femto Mega/Bolt since the depth camera FOV is much greater than color camera, and in alignedDepth mode much of the depth information is lost that is very useful for our SDK.

PS. Can I set my custom camera intrinsics calibration with your SDK? I find the factory one to be non-satisfactory.

Currently, this isn't possible through our Orbbec wrapper, but for recordings, you can modify the calibration.json file with your custom calibration. See: https://github.com/SpectacularAI/docs/blob/main/pdf/calibration_manual.pdf

[NIkolaStaykov]

Thanks for the reply!
I'll keep in mind that alignment eliminates lots of depth information. As for the correct implementation of depth alignment with the Orbbec SDK, I figured the ORBBEC ROS1 repo is a good reference point, as they already correctly implemented it to pass through the ros launch parameters. They use an alignment filter object, as is visible here; maybe you guys will find that useful in the future as well.

[kaatrasa]

Thanks for pointing that out! Looks like that may be the correct way to enable depth->color alignment in OrbbecSDK.
When I get a chance, I will try updating our Orbbec wrapper's alignedDepth mode to use that approach instead. We are currently quite busy with other projects, but I will let you know here when the fix is available.

[123orrin]

Hello,

Just wondering if there are any updates on this issue? I am running into a similar issue where setting alignedDepth = true for the Orbbec Femto Bolt results in the following error: terminate called after throwing an instance of 'std::runtime_error' what(): SpectacularAI error: Camera 0 has invalid focalLengthX (0.000000).

Additional Info
Ubuntu 22.04
SpectacularAI SDK v1.36.0
OrbbecSDK v1.10.12