High latency on multi-process simple pub/sub with DYNAMIC_REUSABLE

[mauropasse]

On rolling, I'm seeing high latency when using DYNAMIC_REUSABLE as history memory policy & zero copy.
In the other hand, latency seems OK when using PREALLOCATED_WITH_REALLOC.
Also latency seems OK when the system is single-process, regardless of the policy (still using loaned msgs).

The system tested uses shared memory to pass a 4MB loaned message between processes:

Multi-process system:
- Process 1: Publisher of 4Mb Loaned messages (10Hz)
- Process 2: Subscription of 4Mb messages
- Platforms: x86 & RaspberryPi4

As reference I'm also showing latency of IPC OFF, i.e. not using Loaned messages with shared memory (normal serialized msg).

Below I provide a CPU flamegraph (from RPi) for both policies:

• Left: DYNAMIC_REUSABLE

  • There seem to be a copy of the 4Mb message on eprosima::fastdds::rtps::UDPChannelResource::perform_listen_operation.

• Right: PREALLOCATED_WITH_REALLOC

The XML config file looks like this for both reader & writer:

  <qos><data_sharing><kind>AUTOMATIC</kind></data_sharing></qos>
  <historyMemoryPolicy>DYNAMIC_REUSABLE</historyMemoryPolicy>
  <!-- other: PREALLOCATED_WITH_REALLOC -->

Since in this case we're aiming for zero-copy using shared memory with loaned messages, I'm guessing this copy is undesired, i.e. a bug? What do you think?

[MiguelCompany]

One of the data-sharing constraints is that the memory policy of the DataWriter should be either PREALLOCATED_MEMORY_MODE or PREALLOCATED_WITH_REALLOC_MEMORY_MODE

This means data-sharing delivery is not used with DYNAMIC_REUSABLE

[fujitatomoya]

@mauropasse thanks for sharing detailed performance analysis always!

Since in this case we're aiming for zero-copy using shared memory with loaned messages, I'm guessing this copy is undesired, i.e. a bug? What do you think?

i think this is designed behavior, but constraints.
With data-sharing mode, DataWriter and DataReader must share the DataWriter's HistoryCache. So DataWriter needs to preallocate the shared memory segment before data communication takes place.

https://fast-dds.docs.eprosima.com/en/latest/fastdds/transport/datasharing.html#constraints

btw, did you enable loaned message on subscription during this analysis? (https://docs.ros.org/en/rolling/How-To-Guides/Configure-ZeroCopy-loaned-messages.html#subscriptions) if not, probably that is not true zero copy, there will be a copy from DataReader to ROS 2 subscription.

[fujitatomoya]

Ah, @MiguelCompany sorry, i did not refresh the browser... one step behind, thanks!

[mauropasse]

Thanks @MiguelCompany and @fujitatomoya for your answers!

@fujitatomoya: btw, did you enable loaned message on subscription during this analysis?

Yes, I run export ROS_DISABLE_LOANED_MESSAGES=0 on terminals for pub & sub processes.

So, since data-sharing delivery is not used with DYNAMIC_REUSABLE. Would it make sense to add some warning or log on the publisher API borrow_loaned_message(), if conditions are not met?

For example, enabling rclcpp intra-process and trying to loan a message results in:

[rclcpp]: Currently used middleware can't loan messages. Local allocator will be used.

In the other hand, I see the default is PREALLOCATED_WITH_REALLOC, so one could think that if the user sets DYNAMIC_REUSABLE, is for some specific reason and it should know or expect that shared memory won't be used in that case. In this case, I'll be closing this issue.

[MiguelCompany]

So, since data-sharing delivery is not used with DYNAMIC_REUSABLE. Would it make sense to add some warning or log on the publisher API borrow_loaned_message(), if conditions are not met?

The only condition to allow borrowing a loaned message is that the type is plain. Data-sharing is not a requirement for borrowing a loaned message. This is done this way since it allows removing the serialization of data, thus reducing one copy for all communication scenarios (network, interprocess, and intra-process).

if the user sets DYNAMIC_REUSABLE, is for some specific reason and it should know or expect that shared memory won't be used in that case.

This is exactly what I think. Setting this requires writing an XML, and the user should know the caveats before changing it from the default.