Agent crash #157

ted-miller · 2022-06-30T17:57:09Z

Describe the bug
The agent crashes.

On Windows: [ros2run]: Process exited with failure 3221225477
On Ubuntu (using docker image): [ros2run]: Segmentation fault

To Reproduce
Not sure.

For the first crash (using Windows), my robot connected to the agent just fine. Then I opened another command prompt and tried ros2 topic echo /joint_states. The agent immediately failed with [ros2run]: Process exited with failure 3221225477.

Then I switched to my Ubuntu machine and started up the docker image.

docker run \
  -it \
  --rm \
  --net=host \
  microros/micro-ros-agent:humble \
    udp4 \
    --port 8888

It seemed to be working, so I assumed it was something wrong with the Windows version.

I left the agent connected with the robot running. (Robot might have been rebooted at some point; I don't really know.) Some time later, I went to shut down the Ubuntu machine and saw the agent had failed. [ros2run] Segmentation fault

So, I figured I would try it on Windows again to get a procedure to reproduce the error. But, I can't make it happen again.

System information (please complete the following information):

OS: Windows 10 and Ubuntu Jammy
ros2-humble-20220523-windows-release-amd64
Agent v3.0.3

Additional context
Up until now, I had been using a galactic version of the Agent. I had not had any problems with it. This is my first time using Humble.

The text was updated successfully, but these errors were encountered:

pablogs9 · 2022-07-01T05:50:07Z

Please provide instructions for replicating

ted-miller · 2022-07-01T12:56:44Z

Well that's part of the problem. I cannot reliably reproduce the issue on-demand. I don't know what is triggering the failure.

But here's what I did, verbatim.

Windows:

Open command prompt.

call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
call "C:\Users\millete\Projects\ROS-2\ros2-humble\local_setup.bat"

call "C:\path\to\ros2-humble\local_setup.bat"
mkdir "%USERPROFILE%\micro_ros_agent_ws\src"
git clone ^
  -b humble ^
  https://github.com/micro-ROS/micro_ros_msgs.git ^
  "%USERPROFILE%\micro_ros_agent_ws\src\micro_ros_msgs"
git clone ^
  -b humble ^
  https://github.com/micro-ROS/micro-ROS-Agent.git ^
  "%USERPROFILE%\micro_ros_agent_ws\src\micro-ROS-Agent"
cd "%USERPROFILE%\micro_ros_agent_ws"
colcon build ^
  --merge-install ^
  --packages-up-to micro_ros_agent ^
  --cmake-args ^
  "-DUAGENT_BUILD_EXECUTABLE=OFF" ^
  "-DUAGENT_P2P_PROFILE=OFF" ^
  "--no-warn-unused-cli"

cd "%USERPROFILE%\micro_ros_agent_ws
call install\setup.bat
ros2 run micro_ros_agent micro_ros_agent udp4 --port 8888

Robot connects to agent.

Open another command prompt

call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
call "C:\Users\millete\Projects\ROS-2\ros2-humble\local_setup.bat"

ros2 topic echo /joint_states

Agent crashes.

I'll continue testing and see if I can come up with something more definitive.

ted-miller · 2022-07-01T13:41:01Z

I have found a reliable way to make it crash.

It seems to be related to the hard-liveliness-check. If I turn off my robot and wait ten seconds, the agent stops with [ros2run]: Process exited with failure 3221225477.

I don't know that this is the same issue as the first crash I experienced. But, it is a reproducible issue.

EDIT: (This is on Windows 10)

pablogs9 · 2022-07-04T05:27:53Z

I'll try to reproduce it.

pablogs9 · 2022-07-04T06:32:36Z

Could you confirm if this issue is replicable under Linux?

gavanderhoorn · 2022-07-04T07:09:16Z

@pablogs9: apologies for the 'vague' description here. It's just that we don't have any additional information ourselves, so we can not share anything more than @ted-miller already did.

Could you confirm if this issue is replicable under Linux?

In the meantime I've been able to reproduce this on Linux as well, specifically using the humble Docker image.

A sequence of starting the humble image, having the Client connect and then disappear (ie: not disconnect properly) while having subscribers active (one more multiple) seems to trigger it often. But not always.

Sometimes the Agent just hangs (no symptoms other than a non-responding Agent), other times it hangs while consuming 100% of a CPU core (and being non-responsive).

Sometimes it crashes with a SEGFAULT.

It never really seems to print anything related to this, but I haven't ran it with increased debug level (so there might be something, but I haven't seen it).

I cannot reproduce it using the galactic image.

Both Docker images were pulled last Friday (the 1st).

One complicating factor: we are not completely up-to-date on the Client side, and given the fact there seems to be some implication it has something to do with the liveness check (which appears to have had some PRs merged), it could be our Client is misbehaving (at least from the perspective of the Agent).

Again, our apologies for the vague report. I'll update when we have more information.

Edit: oh and when the Agent becomes unresponsive, the Client application receives non-zero return values from rclc_node_init_default(..) and rclc_support_init_with_options(..).

And to clarify: while the Agent is unresponsive on the PC side, apparently it is responsive enough to still let the Client progress through at least a few phases in the connection handshake. It doesn't complete it though, as evidenced by the failures in rclc_node_init_default(..) and rclc_support_init_with_options(..).

pablogs9 · 2022-07-04T07:31:33Z

I'm going to try with some subscribers on the client-side.

Update: Working as expected with subscribers.

In which version of the Micro XRCE-DDS Client are you?

gavanderhoorn · 2022-07-04T07:46:30Z

eProsima/Micro-XRCE-DDS-Client@e3f6439

we had to track develop for a while, but I believe this is in ros2 as well now.

I'm going to try with some subscribers on the client-side.

The subscribers I mentioned are on the Agent side (ie: ROS PC).

Update: Working as expected with subscribers.

just to make sure: it takes a few disappearances of the Client to trigger this. It's not 100% reproducible all the time.

pablogs9 · 2022-07-04T07:51:59Z

just to make sure: it takes a few disappearances of the Client to trigger this. It's not 100% reproducible all the time.

How many aprox? 10? 100?

gavanderhoorn · 2022-07-04T07:52:15Z

2 to 5 when I tried to reproduce it on Friday.

And we're using UDPv4.

Command used to start the Agent (in my case, @ted-miller's is different, as he's on Windows, but he's shown the command he's using in #157 (comment)):

docker run \
  -it \
  --rm \
  --net=host \
  microros/micro-ros-agent:humble \
    udp4 \
    --port 8888

gavanderhoorn · 2022-07-05T14:09:46Z

I've just updated our XRCE-DDS Client to v2.2.0, and can still reproduce this with the humble image.

Sequence of steps/commands:

start the micro-ros-agent:humble image using the command mentioned earlier
start the Client
start a nr of subscribers on the ROS side (I used two JointState subscribers, both on the same topic), keep them running
reboot the robot (session is not shutdown nicely, so timeout teardown is needed)
Agent notices Client disappeared, tears down session (prints "destroyed due to liveliness timeout")
robot finished reboot, Client attempts to reconnect
reconnection succeeds, but ROS subscribers don't get new data
at this point Agent crashes (with a SEGFAULT in this case)
Client notices Agent has disappeared (as expected)

I'll see if I can run the Agent in gdb, but as I'm using the Docker image, that may not reveal too much.

pablogs9 · 2022-07-05T14:11:48Z

Ok, we are going to replicate this scenario.

pablogs9 · 2022-07-06T08:28:28Z

Replicated, working on it.

pablogs9 · 2022-07-06T08:29:08Z

Could you check if you can replicate this using a bare Micro XRCE-DDS Agent instead of the micro-ROS Agent?

gavanderhoorn · 2022-07-06T08:43:18Z

I'm participating in World ROS-I day today, but I believe @ted-miller will be online in about 6 hours from now.

Would you have an idea / hunch already?

ted-miller · 2022-07-06T14:41:30Z

I can try this today

ted-miller · 2022-07-06T14:52:23Z

BTW: Last week, I opened the uros Agent Visual Studio solution in the hopes that it would show me where the crash occurs. The call stack seems to indicate that the exception is coming from the fastrtps library. But, I didn't have the debug symbols, so I'm not getting any real useful info.

I tried building a version with debug symbols. But, I couldn't get it to link everything properly.

EDIT: The call stack also included the micro xrce-dds agent library too. So, it could be an issue in the xrce agent that passed invalid data to fastrtps.

ted-miller · 2022-07-06T15:28:58Z

I was not able to reproduce the issue using the micro-xrce-dds agent.

cd xrce_ws/src
git clone -b ros2 https://github.com/eProsima/Micro-XRCE-DDS-Agent.git
cd ../
colcon build
source install/local_setup.bash
cd install/microxrcedds_agent/bin
./MicroXRCEAgent udp4 --port 8888

I have connected (using the same client application on the robot as previous tests) 10 times without issue.

gavanderhoorn · 2022-07-06T16:56:16Z

@ted-miller: but there were no subscribers that time, were there?

ted-miller · 2022-07-06T16:58:56Z

Correct, there were no subscribers.

But, I didn't have any subscribers in previous reproductions either.

gavanderhoorn · 2022-07-06T17:03:10Z

Ah, ok.

I've always had subscribers, and that seems to 'reliably' trigger it.

gavanderhoorn · 2022-07-07T13:41:39Z

Another data point: just had the Agent SEGFAULT on me as I was creating a new subscriber.

So same steps as in #157 (comment), but I didn't make it past step 3.

Same Client version, same version of the Docker image (humble).

pablogs9 · 2022-07-11T05:40:29Z

We replicated this using your instructions, and the issue seems to be identified by the Fast DDS team.

CC: @EduPonz @MiguelCompany @jsantiago-eProsima

gavanderhoorn · 2022-07-11T05:53:09Z

@pablogs9: are eProsima/Fast-DDS#2794, eProsima/Fast-DDS#2801 and eProsima/Fast-DDS#2828 related?

gavanderhoorn · 2022-07-17T15:14:26Z

friendly ping @pablogs9.

pablogs9 · 2022-07-18T05:42:05Z

Not sure @gavanderhoorn, maybe @EduPonz @MiguelCompany @jsantiago-eProsima can tell you

EduPonz · 2022-07-18T06:00:36Z

are eProsima/Fast-DDS#2794, eProsima/Fast-DDS#2801 and eProsima/Fast-DDS#2828 related?

It sure looks like it! We are bundling a Fast DDS v2.6.2 by the end of this week so it'll be included in the next Humble sync.

gavanderhoorn · 2022-07-18T08:59:53Z

@EduPonz thanks.

Was this a problem specific to Humble (ie: Fast-DDS on Humble)? I've not been able to reproduce the problem(s) with a Galactic image.

gavanderhoorn · 2022-07-25T08:42:58Z

The Agent Docker images also get FastDDS from the OR repositories, correct?

pablogs9 · 2022-07-26T05:55:05Z

@gavanderhoorn micro-ROS Agent uses the installed Fast DDS version if it is available. So in the docker, it uses the OSRF distributed binary, yes.

gavanderhoorn · 2022-07-26T07:49:52Z

Have the PRs related to the problem discussed in this issue been merged upstream already? I'd like to test whether the crash has been solved.

From your description it sounds like I could force using a from-source build of FastRTPS by avoiding installing the binary packages. Correct?

pablogs9 · 2022-07-26T07:51:47Z

Check this and this

pablogs9 · 2022-08-23T10:43:55Z

Could you check if #169 solves this issue?

ted-miller · 2022-08-23T13:54:28Z

Yes, #169 appears to have fixed the issue. Thank you.

gavanderhoorn · 2022-08-23T14:07:00Z

Thanks for the fix.

Have/will the docker image(s) be(en) updated?

pablogs9 · 2022-08-23T14:12:35Z

@gavanderhoorn ongoing generation:https://github.com/micro-ROS/docker/actions/runs/2912224500

Sorry for the delay in the fix!

pablogs9 mentioned this issue Aug 8, 2022

micro-ROS agent fails sometimes if ROS2 application is opened or closed micro-ROS/micro_ros_setup#558

Closed

ted-miller closed this as completed Aug 23, 2022

Agent crash #157

Agent crash #157

Comments

ted-miller commented Jun 30, 2022

pablogs9 commented Jul 1, 2022

ted-miller commented Jul 1, 2022

ted-miller commented Jul 1, 2022 • edited Loading

pablogs9 commented Jul 4, 2022

pablogs9 commented Jul 4, 2022

gavanderhoorn commented Jul 4, 2022 • edited Loading

pablogs9 commented Jul 4, 2022 • edited Loading

gavanderhoorn commented Jul 4, 2022 • edited Loading

pablogs9 commented Jul 4, 2022

gavanderhoorn commented Jul 4, 2022 • edited Loading

gavanderhoorn commented Jul 5, 2022 • edited Loading

pablogs9 commented Jul 5, 2022

pablogs9 commented Jul 6, 2022

pablogs9 commented Jul 6, 2022

gavanderhoorn commented Jul 6, 2022 • edited Loading

ted-miller commented Jul 6, 2022

ted-miller commented Jul 6, 2022 • edited Loading

ted-miller commented Jul 6, 2022

gavanderhoorn commented Jul 6, 2022

ted-miller commented Jul 6, 2022

gavanderhoorn commented Jul 6, 2022

gavanderhoorn commented Jul 7, 2022

pablogs9 commented Jul 11, 2022

gavanderhoorn commented Jul 11, 2022 • edited Loading

gavanderhoorn commented Jul 17, 2022

pablogs9 commented Jul 18, 2022

EduPonz commented Jul 18, 2022

gavanderhoorn commented Jul 18, 2022

gavanderhoorn commented Jul 25, 2022

pablogs9 commented Jul 26, 2022

gavanderhoorn commented Jul 26, 2022

pablogs9 commented Jul 26, 2022

pablogs9 commented Aug 23, 2022

ted-miller commented Aug 23, 2022

gavanderhoorn commented Aug 23, 2022

pablogs9 commented Aug 23, 2022

ted-miller commented Jul 1, 2022 •

edited

Loading

gavanderhoorn commented Jul 4, 2022 •

edited

Loading

pablogs9 commented Jul 4, 2022 •

edited

Loading

gavanderhoorn commented Jul 4, 2022 •

edited

Loading

gavanderhoorn commented Jul 4, 2022 •

edited

Loading

gavanderhoorn commented Jul 5, 2022 •

edited

Loading

gavanderhoorn commented Jul 6, 2022 •

edited

Loading

ted-miller commented Jul 6, 2022 •

edited

Loading

gavanderhoorn commented Jul 11, 2022 •

edited

Loading