Unable to Retrieve annotation.segmentation from RoboInterData

[RZkiller]

I followed the tutorial provided in the README of RoboInterData to extract annotation.segmentation, but when I try to print it, the result is None. Could you please clarify why this happens? Additionally, could you guide me on where I can access the mask annotations like those provided in the Video Annotation Visualizer demo?

codes:
for batch in dataloader:
images = batch["observation.images.primary"] # (B, H, W, 3)
actions = batch["action"] # (B, 16, 7)
trace = batch["annotation.trace"] # JSON strings
skill = batch["annotation.primitive_skill"] # List[str]
mask = batch["annotation.segmentation"]
print("mask:", mask)
break

Thank you for your time and assistance. I look forward to your guidance.

[LiHaoHN]

Hello! I guess you’re reading the LeRobot dataset directly. In LeRobot we mainly store per-frame annotations aligned with per-step actions, but we do not store segmentation masks in the LeRobot format because the files are very large and would significantly impact the usability of other annotation types.

We have provided a mapping dict in both our Annotation_raw file and a JSON file here:
https://huggingface.co/datasets/InternRobotics/RoboInter-Data/blob/main/VideoID_2_SegmentationNPZ.json. The raw segmentation files are packaged in:
https://huggingface.co/datasets/InternRobotics/RoboInter-Data/tree/main/Annotation_raw
(specifically segmentation_npz.zip.00 ~ segmentation_npz.zip.02)

[LiHaoHN]

You can load the mask for the episodes you need from these raw files. Please note: this segmentation data is raw and not aligned per-frame with the manipulated objects (especially for multiple objects). You can use the data (segmentation_npz.zip) and the contact frame annotation (https://huggingface.co/datasets/JeasLee/RoboInter-Data_sup/blob/main/start_frame_2_obj_id.json) to align. Here, we provide an example code:

import numpy as np
import pickle

def get_mask(path):
mask = np.load(path)
return mask['masks']

def get_anno_info(path):
anno_info = np.load(path)['arr_0']
return pickle.loads(anno_info)

def get_obj_masks(mask_path, anno_path):
mask = get_mask(mask_path) #(1, 287, 1, 360, 640)
video_length = mask.shape[1]
anno_info = get_anno_info(anno_path)
start_frame_2_obj_id = anno_info['start_frame_2_obj_id']
start_end_range_2_obj_id = {}
for idx, start_frame in enumerate(start_frame_2_obj_id.keys()):
obj_id = start_frame_2_obj_id[start_frame]
if idx == len(start_frame_2_obj_id) - 1:
end_frame = video_length
else:
end_frame = list(start_frame_2_obj_id.keys())[idx + 1]
start_end_range_2_obj_id[(start_frame-1, end_frame-1)] = obj_id
video_masks = []
for se_pair, obj_id in start_end_range_2_obj_id.items():
start = se_pair[0]
end = se_pair[1]
cur_obj_mask = mask[obj_id-1, start:end, ...]
video_masks.append(cur_obj_mask)
video_masks = np.concatenate(video_masks, axis=0)
return video_masks

[LiHaoHN]

Additionally, you can also try add the segmentation mask to Lerobot Data yourself by modifying the conversion script (

RoboInter/RoboInterData/convert_to_lerobot/convert_droid_to_lerobot_anno_fast.py

Line 410 in 481acfb

frame_data = {

) in our repository.

If you have any further questions, feel free to ask!

[RZkiller]

@LiHaoHN Thank you for your prompt reply! I have one more question: which annotations are only provided for the primary view, and which are available for both the primary and wrist views?

From my print outputs, it seems that affordance_box is only available for the primary view, while annotations like contact points seem to exist for both views. I would like to confirm whether my understanding is correct. Thank you!

[LiHaoHN]

@RZkiller. In fact, all of our annotations are based on the primary (main) camera view. The wrist-view images change very drastically over time, which makes it difficult to maintain stable bounding boxes or consistent affordance box tracking. We include wrist-view images in the LeRobot data mainly to match the existing application pipeline for this robot data, and we did not create any dedicated annotations specifically for the wrist view.

However, some annotations that are independent of RGB images, such as subtasks, primitive skills, contact frames, etc. can be directly aligned to the wrist view via the corresponding timestamps.