onNewCacheChangeAdded can be called after node shuts down

[dhood]

Summary: we have "guard condition handle not from this implementation" errors in tests sometimes. it seems to me that the rmw_fastrtps_cpp code is doing the right thing, but it's a race condition in fastrtps. I opened eProsima/Fast-DDS#134 with a fix that works for me locally, this ticket is for local tracking.

A crash that can occur when the onNewCacheChageAdded of the custom ReaderListener in ROS 2 (used for listening to info about the graph e.g. topic names and types) is being called after we have removed the listener.

An example of when this is likely to happen is an executable that creates a publisher but is in the process of shutting down before it finally gets advertised. I can reproduce it pretty often with ~/ros2_ws/build_isolated/test_rclcpp/gtest_local_parameters__rmw_fastrtps_cpp --gtest_filter=test_local_parameters__rmw_fastrtps_cpp.set_parameter_if_not_set, and
here's example output from test_publisher__rmw_fastrtps_cpp on the buildfarm:

[ RUN      ] TestPublisherFixture__rmw_fastrtps_cpp.test_publisher_nominal_string
[ERROR] [rmw_fastrtps_cpp]: failed to trigger graph guard condition: guard condition handle not from this implementation, at /home/rosbuild/ci_scripts/ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/rmw_trigger_guard_condition.cpp:31 (onNewCacheChangeAdded() at /home/rosbuild/ci_scripts/ws/src/ros2/rmw_fastrtps/rmw_fastrtps_cpp/src/types/writer_info.hpp:105)
-- run_test.py: return code -11

That error output is from when the guard condition is triggered from within onNewCacheChageAdded, or if you're unlucky you can get:

*** Error in `/home/dhood/ros2_ws/build_isolated/test_rclcpp/gtest_local_parameters__rmw_fastrtps_cpp': double free or corruption (fasttop): 0x00007febcc007dc0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7febe483a7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8037a)[0x7febe484337a]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7febe484753c]
/home/dhood/ros2_ws/build_isolated/test_rclcpp/gtest_local_parameters__rmw_fastrtps_cpp(_ZN9__gnu_cxx13new_allocatorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE10deallocateEPS6_m+0x20)[0x425e64]
/home/dhood/ros2_ws/build_isolated/test_rclcpp/gtest_local_parameters__rmw_fastrtps_cpp(_ZNSt16allocator_traitsISaINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEE10deallocateERS6_PS5_m+0x2b)[0x424b6e]
/home/dhood/ros2_ws/build_isolated/test_rclcpp/gtest_local_parameters__rmw_fastrtps_cpp(_ZNSt12_Vector_baseINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE13_M_deallocateEPS5_m+0x32)[0x4238aa]
/home/dhood/ros2_ws/install_isolated/rclcpp/lib/librclcpp.so(_ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE19_M_emplace_back_auxIIRKS5_EEEvDpOT_+0x121)[0x7febe5447071]
/home/dhood/ros2_ws/install_isolated/rclcpp/lib/librclcpp.so(_ZNSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS5_EE9push_backERKS5_+0x69)[0x7febe54441af]
/home/dhood/ros2_ws/install_isolated/rmw_fastrtps_cpp/lib/librmw_fastrtps_cpp.so(+0x3f9c7)[0x7febe2e1a9c7]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZN8eprosima8fastrtps4rtps20EDPSimpleSUBListener21onNewCacheChangeAddedEPNS1_10RTPSReaderEPKNS1_13CacheChange_tE+0x56f)[0x7febe28eaccb]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZN8eprosima8fastrtps4rtps14StatefulReader15change_receivedEPNS1_13CacheChange_tEPNS1_11WriterProxyERSt11unique_lockISt15recursive_mutexE+0x28c)[0x7febe28597d2]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZN8eprosima8fastrtps4rtps14StatefulReader14processDataMsgEPNS1_13CacheChange_tE+0x3ee)[0x7febe2858944]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZN8eprosima8fastrtps4rtps15MessageReceiver16proc_Submsg_DataEPNS1_12CDRMessage_tEPNS1_18SubmessageHeader_tEPb+0xee7)[0x7febe2872b01]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZN8eprosima8fastrtps4rtps15MessageReceiver13processCDRMsgERKNS1_12GuidPrefix_tEPNS1_9Locator_tEPNS1_12CDRMessage_tE+0x6e5)[0x7febe287104d]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZN8eprosima8fastrtps4rtps19RTPSParticipantImpl22performListenOperationEPNS1_20ReceiverControlBlockENS1_9Locator_tE+0xab)[0x7febe28865cb]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZNKSt12_Mem_fn_baseIMN8eprosima8fastrtps4rtps19RTPSParticipantImplEFvPNS2_20ReceiverControlBlockENS2_9Locator_tEELb1EEclIJS5_S6_EvEEvPS3_DpOT_+0xb1)[0x7febe288be13]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN8eprosima8fastrtps4rtps19RTPSParticipantImplEFvPNS3_20ReceiverControlBlockENS3_9Locator_tEEEPS4_S6_S7_EE9_M_invokeIJLm0ELm1ELm2EEEEvSt12_Index_tupleIJXspT_EEE+0x7b)[0x7febe288bd57]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZNSt12_Bind_simpleIFSt7_Mem_fnIMN8eprosima8fastrtps4rtps19RTPSParticipantImplEFvPNS3_20ReceiverControlBlockENS3_9Locator_tEEEPS4_S6_S7_EEclEv+0x2c)[0x7febe288bbbe]
/home/dhood/ros2_ws/install_isolated/fastrtps/lib/libfastrtps.so.1(_ZNSt6thread5_ImplISt12_Bind_simpleIFSt7_Mem_fnIMN8eprosima8fastrtps4rtps19RTPSParticipantImplEFvPNS5_20ReceiverControlBlockENS5_9Locator_tEEEPS6_S8_S9_EEE6_M_runEv+0x1c)[0x7febe288bb4e]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7febe4e5bc80]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7febe57466ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7febe48ca3dd]

[mikaelarguedas]

fixed in eProsima/Fast-DDS#134