Test for multithreaded execution #72

jacquelinekay · 2015-11-12T02:47:22Z

This PR tests MultiThreadedExecutor spin and spin_some in 4 test cases:

Create 1 publisher and 2*N subscribers where N is the hardware concurrency of the machine, in an attempt to saturate the execution with work that must be done.
Same as above, but with intraprocess turned on.
Create 1 service and 2*N clients.
Create 1 publisher, 1 subscriber, and N timers that all attempt to publish a message on the publisher at the same time.

dirk-thomas · 2015-11-12T16:13:10Z

test_rclcpp/test/test_multithreaded.cpp

+#endif
+
+static inline void multi_consumer_pub_sub_test(bool intra_process) {
+  rclcpp::init(0, nullptr);


This calls init twice which currently is not a problem but might be as soon as the function does some real initialization work.

But I don't have a custom main which calls init before starting the tests. Where does init get called twice?

Also, I call shutdown in all of the test cases, generally to stop the while loop in spin. Do I have to call init at the beginning of each test case since it is likely that the previous test case called shutdown?

Both tests multi_consumer_single_producer and multi_consumer_intraprocess are calling this function and therefore init.

Currently we don't have a function symmetric to init. So I can't recommend anything but calling it only once in the main.

Why can't shutdown be made symmetric to init?

It could be made symmetric. I was just referring to that it currently is not.

This (calling init, then shutdown, then init) actually is giving me a problem when I run all the test cases at once. I believe ros2/rclcpp#152 fixes the problem.

dirk-thomas · 2015-11-13T23:23:13Z

test_rclcpp/CMakeLists.txt

+    ament_add_gtest(
+      gtest_multithreaded__${middleware_impl}
+      "test/test_multithreaded.cpp"
+      TIMEOUT 30


How long is the expected runtime? Could the timeout be shorter?

jacquelinekay · 2015-11-16T17:13:54Z

I'm moving this back to In Progress to work on a segfault I've discovered in this test.

jacquelinekay · 2015-12-02T23:25:06Z

With the fix @gerkey pushed yesterday, I ran this test on repeat and got a segfault in the multi_access_publisher_intra_process case after 245 iterations. Previously it would take 1-10 iterations to get a segfault on my machine. I also no longer see deadlock.

I will investigate the segfault and try to reproduce it. The core dump appears to have a corrupt stack and the code wasn't built in Debug mode.

dirk-thomas · 2015-12-02T23:30:22Z

For better debugging I would also recommend to consider using a debug build of the rmw impl. you are using.

gerkey · 2015-12-03T00:06:43Z

@jacquelinekay, are you testing with the changes in ros2/rclcpp#165?

jacquelinekay · 2015-12-03T00:07:12Z

With the changes made in ros2/rclcpp#165, I have run the test for over 400 iterations without seeing a segfault or deadlock.

jacquelinekay · 2015-12-03T00:59:18Z

I've rebased this branch intra_process_lock since that set of branches is required to make this test pass reliably.

gerkey · 2015-12-03T02:22:20Z

+1 (with ros2/rclcpp#165 required for reliable execution)

This test helped us find some nasty races. Good stuff.

dirk-thomas · 2015-12-03T05:35:39Z

test_rclcpp/CMakeLists.txt

@@ -12,6 +12,9 @@ endif()

 find_package(ament_cmake REQUIRED)

+#set (CMAKE_CXX_COMPILER "clang++")
+#set (CMAKE_LINKER "llvm-ld")
+


This should be removed before merging.

Test for multithreaded execution

dirk-thomas · 2015-12-03T19:55:32Z

This PR seem to be the reason that the master build on Windows is failing now: http://ci.ros2.org/job/ci_windows_opensplice/733/

Why did the previous CI jobs pass? Did you change further stuff after that?

dirk-thomas · 2015-12-03T20:24:03Z

With the PR ros2/rclcpp#167 the code builds at least. But some tests fail: http://ci.ros2.org/job/ci_windows_opensplice/734/testReport/

jacquelinekay · 2015-12-03T21:20:18Z

Those test cases pass reliably on my Linux machine. I suspect the failures could be due to quirks in the Windows thread scheduler.

dirk-thomas · 2015-12-03T22:01:44Z

While running these tests I also saw the following output. The test passed but the values for the first iterations look pretty wrong. Shouldn't this fail the test somehow?

[ RUN      ] test_multithreaded__rmw_connext_dynamic_cpp.multi_consumer_intra_process
RTI Data Distribution Service Evaluation License issued to OSRF (OSRF01) dthomas@osrfoundation.org For non-production use only.
Expires on 08-dec-2015 See www.rti.com for more information.
callback()    1 with message data 3223089
callback()    2 with message data 3223089
callback()    3 with message data 3223089
callback()    4 with message data 3223089
callback()    1 with message data 1
callback()    2 with message data 1
callback()    3 with message data 1
callback()    4 with message data 1
callback()    5 with message data 2
callback()    6 with message data 2
callback()    7 with message data 2
callback()    8 with message data 2
callback()    9 with message data 3
callback()   10 with message data 3
callback()   11 with message data 3
callback()   12 with message data 3
callback()   13 with message data 4
callback()   14 with message data 4
callback()   15 with message data 4
callback()   16 with message data 4
callback()   17 with message data 5
callback()   18 with message data 5
callback()   19 with message data 5
callback()   20 with message data 5
[       OK ] test_multithreaded__rmw_connext_dynamic_cpp.multi_consumer_intra_process (3875 ms)

jacquelinekay · 2015-12-03T22:08:57Z

The tests actually never check the data, they only check the number of times the callback was called (store in atomic uint called counter).

I also never see that error on my machine. Which platform are you running on? And is it every time you run the test case or does it happen randomly?

dirk-thomas · 2015-12-03T22:19:36Z

The snippet is from http://ci.ros2.org/job/ci_osx/541/consoleFull at time 33:57.272.

jacquelinekay added the in progress Actively being worked on (Kanban column) label Nov 12, 2015

jacquelinekay self-assigned this Nov 12, 2015

dirk-thomas reviewed Nov 12, 2015
View reviewed changes

jacquelinekay mentioned this pull request Nov 12, 2015

Add new case to multiple_service_calls for "n" clients (currently 5) #71

Merged

jacquelinekay added in review Waiting for review (Kanban column) and removed in progress Actively being worked on (Kanban column) labels Nov 13, 2015

dirk-thomas reviewed Nov 13, 2015
View reviewed changes

jacquelinekay added in progress Actively being worked on (Kanban column) and removed in review Waiting for review (Kanban column) labels Nov 16, 2015

jacquelinekay force-pushed the multithreaded branch 2 times, most recently from 580c94e to aebfb2d Compare November 19, 2015 16:45

jacquelinekay mentioned this pull request Dec 1, 2015

Replace shared pointers in AnyExecutable with raw pointers ros2/rclcpp#164

Closed

jacquelinekay force-pushed the multithreaded branch from ee18287 to 154e6c3 Compare December 1, 2015 23:10

gerkey mentioned this pull request Dec 2, 2015

Make WaitSet static in rmw_wait ros2/rmw_opensplice#97

Merged

gerkey mentioned this pull request Dec 3, 2015

Make intra-process manager thread safe, rename IPMState to IPMImpl ros2/rclcpp#165

Merged

jacquelinekay force-pushed the multithreaded branch from 154e6c3 to 6aacc17 Compare December 3, 2015 00:58

gerkey mentioned this pull request Dec 3, 2015

trivial style fixes #78

Closed

dirk-thomas reviewed Dec 3, 2015
View reviewed changes

jacquelinekay force-pushed the multithreaded branch from 80b57dc to 31e9014 Compare December 3, 2015 17:49

Multithreaded pub/sub, client/service, and intra-process tests

7feaba6

jacquelinekay added in review Waiting for review (Kanban column) and removed in progress Actively being worked on (Kanban column) labels Dec 3, 2015

jacquelinekay force-pushed the multithreaded branch from dbe538c to 7feaba6 Compare December 3, 2015 18:06

jacquelinekay added a commit that referenced this pull request Dec 3, 2015

Merge pull request #72 from ros2/multithreaded

d585bff

Test for multithreaded execution

jacquelinekay merged commit d585bff into master Dec 3, 2015

jacquelinekay deleted the multithreaded branch December 3, 2015 18:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test for multithreaded execution #72

Test for multithreaded execution #72

jacquelinekay commented Nov 12, 2015

dirk-thomas Nov 12, 2015

jacquelinekay Nov 12, 2015

dirk-thomas Nov 12, 2015

jacquelinekay Nov 12, 2015

dirk-thomas Nov 12, 2015

jacquelinekay Nov 12, 2015

dirk-thomas Nov 13, 2015

jacquelinekay commented Nov 16, 2015

jacquelinekay commented Dec 2, 2015

dirk-thomas commented Dec 2, 2015

gerkey commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

gerkey commented Dec 3, 2015

dirk-thomas Dec 3, 2015

dirk-thomas commented Dec 3, 2015

dirk-thomas commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

dirk-thomas commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

dirk-thomas commented Dec 3, 2015

Test for multithreaded execution #72

Test for multithreaded execution #72

Conversation

jacquelinekay commented Nov 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacquelinekay commented Nov 16, 2015

jacquelinekay commented Dec 2, 2015

dirk-thomas commented Dec 2, 2015

gerkey commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

gerkey commented Dec 3, 2015

Choose a reason for hiding this comment

dirk-thomas commented Dec 3, 2015

dirk-thomas commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

dirk-thomas commented Dec 3, 2015

jacquelinekay commented Dec 3, 2015

dirk-thomas commented Dec 3, 2015