Print console output for pytest to give context for timeouts

[dhood]

Otherwise tests that timeout (as opposed to failing) are killed when they time out and their output is not shown in reports

This is the pytest equivalent of #98

Timeout without this PR: http://ci.ros2.org/view/nightly/job/nightly_linux_repeated/841/testReport/junit/(root)/projectroot/test_composition__rmw_fastrtps_cpp/

Timeout with this PR:
http://ci.ros2.org/job/ci_windows/3678/testReport/junit/(root)/projectroot/test_composition__rmw_fastrtps_cpp/ (you can see the output)

[mikaelarguedas]

I think the reason for removing this is that the junit files end up empty if we don't capture:
ament/ament_tools@a3a9584.

Before this PR all the non-timed out jobs had content in the result file: http://ci.ros2.org/view/nightly/job/nightly_linux_debug/655/testReport/composition.build.composition/test_composition__rmw_fastrtps_cpp_Debug/test_api_pubsub_composition/

With this PR they end up empty: http://ci.ros2.org/job/ci_windows/3678/testReport/composition.build.composition/test_composition__rmw_connext_cpp_Release/test_api_pubsub_composition/

So I think we need to find another way (dont know which one) to get content in the result files and console output if timeout

[dhood]

Hmm ok, you're right.
I'll have to see if there's a way to get the best of both worlds (probably by not having cmake kill them when they timeout).

If we desperately want this, then the passing tests do have their output captured: http://ci.ros2.org/job/ci_windows/3678/testReport/(root)/projectroot/test_composition__rmw_connext_cpp/
but it's the output of how the test was run as opposed to the generated file for each test, so not ideal.

[mikaelarguedas]

If we desperately want this, then the passing tests do have their output captured: http://ci.ros2.org/job/ci_windows/3678/testReport/(root)/projectroot/test_composition__rmw_connext_cpp/
but it's the output of how the test was run as opposed to the generated file for each test, so not ideal.

I was more concerned about failing tests (before timeout) rather than passing tests (even if it's always good to have result files for all of them). I couldnt find a test failing without timeout in the job you posted that's why I picked a passing one. Do you know if the result files are empty for failing tests as well?

I think it could be fine for jenkins display as each test output can be accessed fairly easily. For the case of running local tests It'll be more of a pain as the result files will be empty and people will need to browse their console output to figure out why a specific test fail without having to rerun only that test.

We could conclude (for now) that having console output for all tests in Jenkins outweigh the lack of content in the result files.

I'll place this in progress for now

[dhood]

FYI the --show-capture option added to pytest that @dirk-thomas mentioned applies to failed tests. Unfortunately it doesn't help with this issue because we don't have pytest failures but timeouts. Think we'd need something more like this pytest feature to get the best of both worlds.

What if we add the pytest-timeout pip package so pytest doesn't get SIGTERMed by ctest? https://pypi.python.org/pypi/pytest-timeout

[dirk-thomas]

Closing due to no activity. Also just skipping capturing isn't the right solution anyway.