Sending large ros2 messages over ethernet slows down simulation

[MateuszWasilewski]

Describe the bug
ROS2SystemComponent::OnTick takes a long time when ros2 component send big messages Ex. ros2 image > 5Mb.
This slows simulation

Steps to reproduce
Steps to reproduce the behavior:

1. Create empty Level
2. Add ROS2CameraSensorComponent
   Image should be large Ex. 3440x1440
3. Launch simulation
4. Check current FPS in the Editor (ros2 images are generated in the background)
5. Listen to camera topic on separate PC
6. Check current FPS (they are substantially lower than before listening on topic

Expected behavior
Simulation performance shouldn't be affected by ros2 config

Actual behavior
Simulation slows down substantially, slowdown is correlated with message size

Additional context

• Issue was found while using cyclone_dds rmw implementation (which does not support async publishing)
• I suggest creating a separate thread to perform rclpp::Node::spin_once.

[adamdbrw]

Are you using synchronous publisher in the main thread?

[MateuszWasilewski]

I am using standard publisher created using rclcpp::Node::create_publisher(...) is not synchronous (method publish does not block). But according to issue CycloneDDS does not support async publishing.

Furthermore in ROS2SystemComponent
Blocking action (executor::spin_some) is performed - ros2 docs spin_some

void ROS2SystemComponent::OnTick([[maybe_unused]] float deltaTime, [[maybe_unused]] AZ::ScriptTimePoint time)
   {
       if (rclcpp::ok())
       {
           m_dynamicTFBroadcaster->sendTransform(m_frameTransforms);
           m_frameTransforms.clear();

           m_simulationClock->Tick();
           m_executor->spin_some();
       }
   }

[adamdbrw]

There is already an issue for spin thread, and publishing can be done in a thread or through DDS vendor features. It is not a bug

[albertdai-rf]

@MateuszWasilewski How did you resolve the issue finally? Switching to rmw fastrtps (with async pub) or just publishing the message in a separated thread?

[MateuszWasilewski]

Hi, I solved the issue by creating a separate thread which handles node spinning and publishing. This reduced waiting in the main thread to just pushing a future (task to send msg) to a threadpool.
Spawned thread has negligible CPU usage (is mostly waiting for new task / publishing)
This approach isn't the best but requires the least amount of work while still supporting CycloneDDS