spin_some not thread-safe, triggers TSan

[muji-4ok]

Generated by Generative AI

No response

Operating System:

Linux dd3a63f3db39 6.14.0-28-generic #28~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 25 10:47:01 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

ROS version or commit hash:

kilted

RMW implementation (if applicable):

rmw_fastrtps_cpp

RMW Configuration (if applicable):

No response

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

ros2 doctor --report

   NETWORK CONFIGURATION
inet         : 127.0.0.1
inet4        : ['127.0.0.1']
inet6        : ['::1']
netmask      : 255.0.0.0
device       : lo
flags        : UP,LOOPBACK,RUNNING
mtu          : 65536
inet         : 172.17.0.2
inet4        : ['172.17.0.2']
ether        : 02:42:ac:11:00:02
netmask      : 255.255.0.0
device       : eth0
flags        : UP,BROADCAST,RUNNING,MULTICAST
mtu          : 1500
broadcast    : 172.17.255.255

   PACKAGE VERSIONS
ament_cmake_ros                           : latest=0.14.3, local=0.14.3
geometry_msgs                             : latest=5.5.0, local=5.5.0
rosidl_dynamic_typesupport_fastrtps       : latest=0.4.1, local=0.4.1
rosidl_pycommon                           : latest=4.9.5, local=4.9.4
ros_environment                           : latest=4.3.1, local=4.3.1
rclcpp_components                         : latest=29.5.2, local=29.5.0
tinyxml2_vendor                           : latest=0.10.0, local=0.10.0
composition_interfaces                    : latest=2.3.0, local=2.3.0
ros2run                                   : latest=0.38.0, local=0.38.0
rosidl_adapter                            : latest=4.9.5, local=4.9.4
ament_lint_auto                           : latest=0.19.2, local=0.19.2
sensor_msgs                               : latest=5.5.0, local=5.5.0
rpyutils                                  : latest=0.6.2, local=0.6.2
rmw_implementation                        : latest=3.0.4, local=3.0.4
rosidl_runtime_c                          : latest=4.9.5, local=4.9.4
ament_cmake_test                          : latest=2.7.3, local=2.7.3
rmw_implementation_cmake                  : latest=7.8.2, local=7.8.2
std_msgs                                  : latest=5.5.0, local=5.5.0
ament_cmake_gen_version_h                 : latest=2.7.3, local=2.7.3
type_description_interfaces               : latest=2.3.0, local=2.3.0
rosidl_parser                             : latest=4.9.5, local=4.9.4
launch_ros                                : latest=0.28.2, local=0.28.1
ament_cmake_gmock                         : latest=2.7.3, local=2.7.3
ament_cmake_export_include_directories    : latest=2.7.3, local=2.7.3
rosidl_runtime_cpp                        : latest=4.9.5, local=4.9.4
rclcpp_lifecycle                          : latest=29.5.2, local=29.5.0
ros2node                                  : latest=0.38.0, local=0.38.0
rosidl_typesupport_cpp                    : latest=3.3.3, local=3.3.3
ros2cli_common_extensions                 : latest=0.4.0, local=0.4.0
rcutils                                   : latest=6.9.8, local=6.9.5
ament_cmake_include_directories           : latest=2.7.3, local=2.7.3
uncrustify_vendor                         : latest=3.1.0, local=3.1.0
ament_lint                                : latest=0.19.2, local=0.19.2
rosidl_typesupport_fastrtps_c             : latest=3.8.0, local=3.8.0
rosidl_cmake                              : latest=4.9.5, local=4.9.4
ament_cmake_export_definitions            : latest=2.7.3, local=2.7.3
ament_flake8                              : latest=0.19.2, local=0.19.2
tracetools                                : latest=8.6.0, local=8.6.0
rosidl_core_runtime                       : latest=0.3.1, local=0.3.1
ros2multicast                             : latest=0.38.0, local=0.38.0
ament_pep257                              : latest=0.19.2, local=0.19.2
ros2cli                                   : latest=0.38.0, local=0.38.0
ros2launch                                : latest=0.28.2, local=0.28.1
sros2                                     : latest=0.15.2, local=0.15.2
rclcpp                                    : latest=29.5.2, local=29.5.0
launch_xml                                : latest=3.8.2, local=3.8.1
ament_cmake_xmllint                       : latest=0.19.2, local=0.19.2
ros2doctor                                : latest=0.38.0, local=0.38.0
nav_msgs                                  : latest=5.5.0, local=5.5.0
rcl                                       : latest=10.1.1, local=10.1.0
service_msgs                              : latest=2.3.0, local=2.3.0
ros2service                               : latest=0.38.0, local=0.38.0
ros_core                                  : latest=0.12.0, local=0.12.0
rclcpp_action                             : latest=29.5.2, local=29.5.0
ament_copyright                           : latest=0.19.2, local=0.19.2
rmw_dds_common                            : latest=3.2.1, local=3.2.1
rosgraph_msgs                             : latest=2.3.0, local=2.3.0
ament_cmake_ros_core                      : latest=0.14.3, local=0.14.3
ament_cmake_export_link_flags             : latest=2.7.3, local=2.7.3
rcl_action                                : latest=10.1.1, local=10.1.0
rosidl_dynamic_typesupport                : latest=0.3.1, local=0.3.1
rosidl_generator_py                       : latest=0.24.1, local=0.24.1
ament_cmake_export_dependencies           : latest=2.7.3, local=2.7.3
rosidl_typesupport_introspection_cpp      : latest=4.9.5, local=4.9.4
rosidl_generator_type_description         : latest=4.9.5, local=4.9.4
ament_lint_common                         : latest=0.19.2, local=0.19.2
stereo_msgs                               : latest=5.5.0, local=5.5.0
std_srvs                                  : latest=5.5.0, local=5.5.0
ament_cmake                               : latest=2.7.3, local=2.7.3
class_loader                              : latest=2.8.0, local=2.8.0
zenoh_cpp_vendor                          : latest=0.6.3, local=0.6.1
sros2_cmake                               : latest=0.15.2, local=0.15.2
rosidl_typesupport_interface              : latest=4.9.5, local=4.9.4
ament_lint_cmake                          : latest=0.19.2, local=0.19.2
ament_cmake_pytest                        : latest=2.7.3, local=2.7.3
rosidl_generator_cpp                      : latest=4.9.5, local=4.9.4
ros2topic                                 : latest=0.38.0, local=0.38.0
ament_cmake_core                          : latest=2.7.3, local=2.7.3
rcl_interfaces                            : latest=2.3.0, local=2.3.0
rmw_fastrtps_cpp                          : latest=9.3.2, local=9.3.2
rmw_zenoh_cpp                             : latest=0.6.3, local=0.6.1
ament_cmake_libraries                     : latest=2.7.3, local=2.7.3
rosidl_typesupport_fastrtps_cpp           : latest=3.8.0, local=3.8.0
diagnostic_msgs                           : latest=5.5.0, local=5.5.0
ament_cmake_copyright                     : latest=0.19.2, local=0.19.2
ros2action                                : latest=0.38.0, local=0.38.0
rosidl_default_runtime                    : latest=1.7.1, local=1.7.1
ament_cmake_cppcheck                      : latest=0.19.2, local=0.19.2
rclpy                                     : latest=9.1.1, local=9.1.0
ros2param                                 : latest=0.38.0, local=0.38.0
rmw_test_fixture_implementation           : latest=0.14.3, local=0.14.3
ros2component                             : latest=0.38.0, local=0.38.0
lifecycle_msgs                            : latest=2.3.0, local=2.3.0
ament_cmake_gtest                         : latest=2.7.3, local=2.7.3
ament_cpplint                             : latest=0.19.2, local=0.19.2
ament_cmake_lint_cmake                    : latest=0.19.2, local=0.19.2
ros2lifecycle                             : latest=0.38.0, local=0.38.0
rmw                                       : latest=7.8.2, local=7.8.2
visualization_msgs                        : latest=5.5.0, local=5.5.0
ament_index_python                        : latest=1.11.0, local=1.11.0
ament_cmake_auto                          : latest=2.7.3, local=2.7.3
rmw_test_fixture                          : latest=0.14.3, local=0.14.3
rosidl_generator_c                        : latest=4.9.5, local=4.9.4
unique_identifier_msgs                    : latest=2.7.0, local=2.7.0
ament_cmake_version                       : latest=2.7.3, local=2.7.3
launch_testing                            : latest=3.8.2, local=3.8.1
ament_cmake_pep257                        : latest=0.19.2, local=0.19.2
rosidl_runtime_py                         : latest=0.14.1, local=0.14.1
libstatistics_collector                   : latest=2.0.1, local=2.0.1
rmw_fastrtps_shared_cpp                   : latest=9.3.2, local=9.3.2
ros2pkg                                   : latest=0.38.0, local=0.38.0
rosidl_core_generators                    : latest=0.3.1, local=0.3.1
ament_cppcheck                            : latest=0.19.2, local=0.19.2
ament_xmllint                             : latest=0.19.2, local=0.19.2
ament_cmake_export_interfaces             : latest=2.7.3, local=2.7.3
rcl_lifecycle                             : latest=10.1.1, local=10.1.0
launch_testing_ros                        : latest=0.28.2, local=0.28.1
rosidl_typesupport_introspection_c        : latest=4.9.5, local=4.9.4
rosidl_default_generators                 : latest=1.7.1, local=1.7.1
launch_yaml                               : latest=3.8.2, local=3.8.1
ament_cmake_export_libraries              : latest=2.7.3, local=2.7.3
shape_msgs                                : latest=5.5.0, local=5.5.0
rosidl_cli                                : latest=4.9.5, local=4.9.4
statistics_msgs                           : latest=2.3.0, local=2.3.0
common_interfaces                         : latest=5.5.0, local=5.5.0
rmw_security_common                       : latest=7.8.2, local=7.8.2
launch                                    : latest=3.8.2, local=3.8.1
osrf_pycommon                             : latest=2.1.6, local=2.1.6
ament_index_cpp                           : latest=1.11.0, local=1.11.0
ament_cmake_cpplint                       : latest=0.19.2, local=0.19.2
ament_cmake_export_targets                : latest=2.7.3, local=2.7.3
ament_cmake_flake8                        : latest=0.19.2, local=0.19.2
console_bridge_vendor                     : latest=1.8.0, local=1.8.0
trajectory_msgs                           : latest=5.5.0, local=5.5.0
rcl_logging_spdlog                        : latest=3.2.2, local=3.2.2
pluginlib                                 : latest=5.6.0, local=5.6.0
rcl_logging_interface                     : latest=3.2.2, local=3.2.2
ros2interface                             : latest=0.38.0, local=0.38.0
ament_package                             : latest=0.17.2, local=0.17.2
actionlib_msgs                            : latest=5.5.0, local=5.5.0
rcpputils                                 : latest=2.13.4, local=2.13.4
launch_testing_ament_cmake                : latest=3.8.2, local=3.8.1
ament_cmake_uncrustify                    : latest=0.19.2, local=0.19.2
libyaml_vendor                            : latest=1.7.1, local=1.7.1
ament_cmake_target_dependencies           : latest=2.7.3, local=2.7.3
rosidl_typesupport_c                      : latest=3.3.3, local=3.3.3
ament_uncrustify                          : latest=0.19.2, local=0.19.2
ros_workspace                             : latest=1.0.3, local=1.0.3
rcl_yaml_param_parser                     : latest=10.1.1, local=10.1.0
spdlog_vendor                             : latest=1.7.0, local=1.7.0
builtin_interfaces                        : latest=2.3.0, local=2.3.0
ament_cmake_python                        : latest=2.7.3, local=2.7.3
action_msgs                               : latest=2.3.0, local=2.3.0
tlsf_cpp                                  : latest=0.18.3, local=0.18.3
depthimage_to_laserscan                   : latest=2.5.1, local=2.5.1
rqt_image_view                            : latest=1.3.3, local=1.3.0
rviz2                                     : latest=15.0.5, local=15.0.2
tango_icons_vendor                        : latest=0.4.0, local=0.4.0
rviz_ogre_vendor                          : latest=15.0.5, local=15.0.2
rviz_assimp_vendor                        : latest=15.0.5, local=15.0.2
intra_process_demo                        : latest=0.36.1, local=0.36.0
demo_nodes_cpp_native                     : latest=0.36.1, local=0.36.0
rviz_default_plugins                      : latest=15.0.5, local=15.0.2
rqt_bag_plugins                           : latest=2.0.2, local=2.0.2
gz_cmake_vendor                           : latest=0.2.2, local=0.2.2
turtlesim_msgs                            : latest=1.9.3, local=1.9.3
python_qt_binding                         : latest=2.3.1, local=2.3.1
dummy_sensors                             : latest=0.36.1, local=0.36.0
examples_rclpy_minimal_action_server      : latest=0.20.5, local=0.20.5
map_msgs                                  : latest=2.5.0, local=2.5.0
demo_nodes_py                             : latest=0.36.1, local=0.36.0
point_cloud_transport                     : latest=5.1.3, local=5.1.3
rviz_common                               : latest=15.0.5, local=15.0.2
rqt_bag                                   : latest=2.0.2, local=2.0.2
teleop_twist_keyboard                     : latest=2.4.0, local=2.4.0
examples_rclpy_minimal_client             : latest=0.20.5, local=0.20.5
dummy_map_server                          : latest=0.36.1, local=0.36.0
sdl2_vendor                               : latest=3.3.0, local=3.3.0
rqt_reconfigure                           : latest=1.7.0, local=1.7.0
rqt_py_console                            : latest=1.4.0, local=1.4.0
quality_of_service_demo_cpp               : latest=0.36.1, local=0.36.0
examples_rclcpp_minimal_composition       : latest=0.20.5, local=0.20.5
examples_rclcpp_minimal_service           : latest=0.20.5, local=0.20.5
examples_rclcpp_minimal_action_client     : latest=0.20.5, local=0.20.5
tlsf                                      : latest=0.10.1, local=0.10.1
rqt_topic                                 : latest=1.8.1, local=1.8.1
image_tools                               : latest=0.36.1, local=0.36.0
examples_rclcpp_minimal_publisher         : latest=0.20.5, local=0.20.5
libcurl_vendor                            : latest=3.7.0, local=3.7.0
rqt_plot                                  : latest=1.6.3, local=1.6.3
laser_geometry                            : latest=2.10.1, local=2.10.0
dummy_robot_bringup                       : latest=0.36.1, local=0.36.0
pcl_msgs                                  : latest=1.0.0, local=1.0.0
lifecycle                                 : latest=0.36.1, local=0.36.0
image_transport                           : latest=6.1.2, local=6.1.1
rqt_msg                                   : latest=1.6.0, local=1.6.0
action_tutorials_cpp                      : latest=0.36.1, local=0.36.0
rqt_graph                                 : latest=1.7.1, local=1.7.0
examples_rclcpp_minimal_client            : latest=0.20.5, local=0.20.5
examples_rclcpp_minimal_timer             : latest=0.20.5, local=0.20.5
rttest                                    : latest=0.18.3, local=0.18.3
rqt_py_common                             : latest=1.9.0, local=1.9.0
examples_rclcpp_minimal_action_server     : latest=0.20.5, local=0.20.5
qt_gui                                    : latest=2.9.1, local=2.9.1
examples_rclcpp_minimal_subscriber        : latest=0.20.5, local=0.20.5
rqt_gui_py                                : latest=1.9.0, local=1.9.0
pendulum_control                          : latest=0.36.1, local=0.36.0
logging_demo                              : latest=0.36.1, local=0.36.0
rviz_resource_interfaces                  : latest=15.0.5, local=15.0.2
pendulum_msgs                             : latest=0.36.1, local=0.36.0
example_interfaces                        : latest=0.13.0, local=0.13.0
rqt_common_plugins                        : latest=1.2.0, local=1.2.0
examples_rclpy_minimal_subscriber         : latest=0.20.5, local=0.20.5
rqt_action                                : latest=2.3.0, local=2.3.0
resource_retriever                        : latest=3.7.0, local=3.7.0
rqt_console                               : latest=2.3.1, local=2.3.1
rqt_gui_cpp                               : latest=1.9.0, local=1.9.0
examples_rclpy_minimal_action_client      : latest=0.20.5, local=0.20.5
joy                                       : latest=3.3.0, local=3.3.0
topic_monitor                             : latest=0.36.1, local=0.36.0
examples_rclpy_minimal_service            : latest=0.20.5, local=0.20.5
action_tutorials_py                       : latest=0.36.1, local=0.36.0
qt_gui_cpp                                : latest=2.9.1, local=2.9.1
rqt_publisher                             : latest=1.9.0, local=1.9.0
composition                               : latest=0.36.1, local=0.36.0
pcl_conversions                           : latest=2.7.3, local=2.7.3
image_geometry                            : latest=4.1.0, local=4.1.0
angles                                    : latest=1.16.0, local=1.16.0
quality_of_service_demo_py                : latest=0.36.1, local=0.36.0
demo_nodes_cpp                            : latest=0.36.1, local=0.36.0
rviz_rendering                            : latest=15.0.5, local=15.0.2
turtlesim                                 : latest=1.9.3, local=1.9.3
examples_rclcpp_multithreaded_executor    : latest=0.20.5, local=0.20.5
gz_utils_vendor                           : latest=0.2.2, local=0.2.2
qt_gui_py_common                          : latest=2.9.1, local=2.9.1
teleop_twist_joy                          : latest=2.6.5, local=2.6.3
rqt_gui                                   : latest=1.9.0, local=1.9.0
gz_math_vendor                            : latest=0.2.3, local=0.2.3
rqt_srv                                   : latest=1.3.0, local=1.3.0
qt_dotgraph                               : latest=2.9.1, local=2.9.1
desktop                                   : latest=0.12.0, local=0.12.0
examples_rclpy_executors                  : latest=0.20.5, local=0.20.5
rqt_service_caller                        : latest=1.4.0, local=1.4.0
interactive_markers                       : latest=2.7.0, local=2.7.0
examples_rclpy_minimal_publisher          : latest=0.20.5, local=0.20.5
rqt_shell                                 : latest=1.3.1, local=1.3.1
cv_bridge                                 : latest=4.1.0, local=4.1.0
urdf                                      : latest=2.12.2, local=2.12.2
rosbag2_py                                : latest=0.32.0, local=0.32.0
python_orocos_kdl_vendor                  : latest=0.7.1, local=0.7.1
tf2_tools                                 : latest=0.41.2, local=0.41.0
geometry2                                 : latest=0.41.2, local=0.41.0
sensor_msgs_py                            : latest=5.5.0, local=5.5.0
urdf_parser_plugin                        : latest=2.12.2, local=2.12.2
rosbag2_transport                         : latest=0.32.0, local=0.32.0
ros_base                                  : latest=0.12.0, local=0.12.0
orocos_kdl_vendor                         : latest=0.7.1, local=0.7.1
tf2_msgs                                  : latest=0.41.2, local=0.41.0
keyboard_handler                          : latest=0.4.0, local=0.4.0
tf2_eigen                                 : latest=0.41.2, local=0.41.0
eigen3_cmake_module                       : latest=0.4.0, local=0.4.0
zstd_vendor                               : latest=0.32.0, local=0.32.0
tf2_ros                                   : latest=0.41.2, local=0.41.0
yaml_cpp_vendor                           : latest=9.1.0, local=9.1.0
tf2_ros_py                                : latest=0.41.2, local=0.41.0
tf2_py                                    : latest=0.41.2, local=0.41.0
pybind11_vendor                           : latest=3.2.0, local=3.2.0
rosbag2_storage                           : latest=0.32.0, local=0.32.0
rosbag2_storage_mcap                      : latest=0.32.0, local=0.32.0
mcap_vendor                               : latest=0.32.0, local=0.32.0
tf2_kdl                                   : latest=0.41.2, local=0.41.0
kdl_parser                                : latest=2.12.1, local=2.12.1
rosbag2_storage_default_plugins           : latest=0.32.0, local=0.32.0
liblz4_vendor                             : latest=0.32.0, local=0.32.0
rosbag2_compression_zstd                  : latest=0.32.0, local=0.32.0
rosbag2_interfaces                        : latest=0.32.0, local=0.32.0
message_filters                           : latest=7.1.1, local=7.1.1
rosbag2_compression                       : latest=0.32.0, local=0.32.0
rosbag2_cpp                               : latest=0.32.0, local=0.32.0
rosbag2                                   : latest=0.32.0, local=0.32.0
sqlite3_vendor                            : latest=0.32.0, local=0.32.0
rosbag2_storage_sqlite3                   : latest=0.32.0, local=0.32.0
tf2_sensor_msgs                           : latest=0.41.2, local=0.41.0
tf2_geometry_msgs                         : latest=0.41.2, local=0.41.0
tf2                                       : latest=0.41.2, local=0.41.0
ros2bag                                   : latest=0.32.0, local=0.32.0
tf2_bullet                                : latest=0.41.2, local=0.41.0
tf2_eigen_kdl                             : latest=0.41.2, local=0.41.0
robot_state_publisher                     : latest=3.4.2, local=3.4.2

   PLATFORM INFORMATION
system           : Linux
platform info    : Linux-6.14.0-28-generic-x86_64-with-glibc2.39
release          : 6.14.0-28-generic
processor        : x86_64

   QOS COMPATIBILITY LIST
compatibility status    : No publisher/subscriber pairs found

   RMW MIDDLEWARE
middleware name    : rmw_fastrtps_cpp

   ROS 2 INFORMATION
distribution name      : kilted
distribution type      : ros2
distribution status    : active
release platforms      : {'debian': ['bookworm'], 'rhel': ['9'], 'ubuntu': ['noble']}

   TOPIC LIST
topic               : none
publisher count     : 0
subscriber count    : 0

Steps to reproduce issue

1. Create a project with ros2 pkg create --build-type ament_cmake --license Apache-2.0 model_samples --dependencies rclcpp std_msgs
2. Create files sample_2.cpp and CMakeLists.txt from the gist I linked below
3. Build the application with colcon build --mixin tsan
4. Run it

Expected behavior

Using spin_some manually in a loop on multiple threads without any inherent shared variables should not trigger TSan

Actual behavior

TSan reports some errors

Additional information

Here's code for reproduction:

https://gist.github.com/muji-4ok/6b02f0de0a23a6a5cbf7ef07f4ead4fc

There's a commented out lock_guard, which stops the reproduction

And here's logs I get:

https://gist.github.com/muji-4ok/d646f95dfead075b40432351e1f03ede

Some context:

I was experimenting with ros2 and TSan for my project. For that I needed a simple way to test out running multiple nodes in one process with CPU affinity

I've read https://docs.ros.org/en/kilted/Concepts/Intermediate/About-Executors.html and https://docs.ros.org/en/kilted/How-To-Guides/Using-callback-groups.html

I'm assuming from the docs that callback groups should be used if your business logic itself requires some mutual exclusion. But from the example I provided I see that the middleware also requires some mutual exclusion to work correctly. This seems like buggy behaviour IMHO

[jmachowinski]

This looks like a bug in fastDDS.
Do your get the same errors if you switch to cyclone ?

export RMW_IMPLEMENTATION="rmw_cyclonedds_cpp"
./your_test

In theory your code should work as expected. Note, that rclcpp::spin_come is deprecated in rolling.
The replacement code is :

SingleThreadedExecutor executor;
executor.add_node(node);
while (should_spin)
{
    executor.spin_some();
}

[muji-4ok]

Hmm, I also get an error with it, a different one however: https://gist.github.com/muji-4ok/d6b0914ce8f38e3ff86d3c7990f7c839

[muji-4ok]

Rewriting the code to use the SingleThreadedExecutor does not change anything:

https://gist.github.com/muji-4ok/3467e4d31279e57485b8f6d13b20df5e

[skyegalaxy]

We took a look at some of the tsan logs today. Once the call stack gets past the rclcpp layer, it's hard to tell what's going on. Is there any way you could build rmw_fastrtps / fast-dds / fast-cdr with debug flags and TSan enabled and post the output logs from that?