Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove topic partitions #170

Merged
merged 5 commits into from
Jun 7, 2018
Merged

Remove topic partitions #170

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f283d03
removal of topic partitions, inital draft
rohitsalem Apr 21, 2018
c1a9e1e
Updated alternative strategy of using partitions
rohitsalem Apr 25, 2018
6c41d4b
fixed suggestions, added limits for service_names
rohitsalem Apr 26, 2018
3ff8d1c
Mikael/remove topic partitions (#171)
mikaelarguedas Apr 30, 2018
09dfbd2
highlight rti connext limitations
Karsten1987 May 11, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 58 additions & 78 deletions articles/140_topic_and_service_name_mapping.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,50 +202,16 @@ The ROS topic and service name constraints allow more types of characters than t
These must be substituted or otherwise removed during the process of converting the topic or service name to DDS concepts.
Since ROS 2 topic and service names are expanded to fully qualified names, any balanced bracket (`{}`) substitutions and tildes (`~`) will have been expanded.
Additionally any URL related syntax, e.g. the `rostopic://` prefix, will be removed once parsed.
Therefore only forward slashes (`/`) need to be addressed when converting to DDS concepts.

### ROS Namespaces with DDS Partitions

The proposed strategy to address the forward slashes (`/`) which are present in ROS names, is to separate the ROS name into the "namespace" and the "base name", and then place the namespace, stripped of leading and trailing forward slashes (`/`), into a single DDS partition entry and the remaining base name into the DDS topic name.
This addresses the issue because the ROS name's base name will not contain any forward slashes (`/`) by definition and so there are no longer any disallowed characters in the DDS topic name.
The DDS partition will contain the ROS name's namespace, including any forward slashes (`/`) that make up the namespace and were not at the beginning or the end of the namespace.
That is acceptable because DDS partitions are allowed to contain forward slashes (`/`) unlike the DDS topics.

This strategy also avoids DDS partitions from being exposed to the ROS API's.
This is because the splitting of the ROS name into namespace and base name can occur inside the implementation of the ROS middleware interface, and so the ROS topic or service name will remain intact throughout the middleware agnostic code.
This means that if someone chooses to implement ROS 2 on top of something other than DDS, then this strategy will not impact that implementation, as it applies only to DDS implementations.

#### DDS Partitions Characteristics

DDS partitions are implemented as an array of strings within the `DDS::Publisher` and `DDS::Subscriber` QoS settings.
The array allows up to 64 items, and each string in the array is an individual partition.
This means that every string of the array is an independently matchable entity when paired with the topic name and the other QoS settings.
Put another way, a DDS DataWriter and DataReader will not connect to one another unless at least one of the partition strings in the array match, even if the topic names match.
Sometimes special characters are used in the partition strings to express regular expressions matching (such as `+`, `*`, `^`), but they are not officially supported by the standard.
The maximum size of each string in the array may depend on the DDS or RTPS implementation.
You can read more about partitions in RTI's documentation:

- [PARTITION_QosPolicy](https://community.rti.com/static/documentation/connext-dds/5.2.3/doc/manuals/connext_dds/html_files/RTI_ConnextDDS_CoreLibraries_UsersManual/Content/UsersManual/PARTITION_QosPolicy.htm)

The array of strings have no hierarchy and the order does not matter.
Each entry in the partition array is directly combined with the DDS topic and they are not sequentially combined.
If a publisher has two partition entries, e.g. `foo` and `bar` with a base name of `baz`, this would be equivalent to having two different publishers on these topics: `/foo/baz` and `/bar/baz`.
Therefore we cannot use the array of strings to represent our implied namespace hierarchy and we have to continue to use the forward slash to accomplish this.

#### Hierarchy with DDS Partitions

Since DDS partitions have no hierarchy or order to the array of strings, this proposal would use only one of the strings in the array to hold the entire ROS name's namespace.
Therefore we preserve the forward slashes (`/`) that exist in the ROS name's namespace already.
Previously forward slashes (`/`) were disallowed in DDS topic names, now the restriction has been lifted (see [issue](https://issues.omg.org/issues/lists/dds-rtf5#issue-42236) on omg.org) and therefore the ROS topic names are first prefixed with ROS Specific Namespace prefix (described below) are then mapped directly into DDS topic names.

### ROS Specific Namespace Prefix

In order to differentiate ROS topics easily, all DDS topics created by ROS shall be automatically prefixed with a namespace like `/rX`, where `X` is a single character that indicates to which subsystem of ROS the topic belongs.
At this point it's important to remember that when the namespace part of the ROS name is put into the DDS partition it is stripped of leading and trailing forward slashes (`/`).
For example, a plain topic called `/foo` would translate to a DDS partition `rt` and a DDS topic `foo`, which is the result of implicitly adding `/rt` to the namespace of a ROS topic which is in the root namespace `/` and has a base name `foo`, then the namespace getting the leading and trailing forward slashes (`/`) stripped.
As another example, a topic called `/left/image_raw` would translate to a DDS partition `rt/left` and a DDS topic `image_raw`, which is the result of implicitly adding `/rt` to the namespace of a ROS topic which is in the namespace `/left` and has a base name `image_raw`, again stripping the leading and trailing forward slashes (`/`) from the namespace.
For example, a plain topic called `/foo` would translate to a DDS topic `rt/foo`, which is the result of implicitly adding `rt` to the namespace of a ROS topic which is in the root namespace `/` and has a base name `foo`.
As another example, a topic called `/left/image_raw` would translate to a DDS topic `rt/left/image_raw`, which is the result of implicitly adding `rt` to the namespace of a ROS topic which is in the namespace `/left` and has a base name `image_raw`.

For systems where Services are implemented with topics (like with OpenSplice), a different subsystem character can be used: `rq` for the request topic and `rr` for the response topic.
On systems where the implementation is handled for us by DDS (like with Connext), we use `rs` as the common prefix.
On systems where Services are handled explicitly implemented, we consider a separate prefix, e.g. `rs`.

Here is a non-exhaustive list of prefixes:

Expand All @@ -265,68 +231,56 @@ The standard reserves the right to use up to 8 characters for the prefix in case

Here are some examples of how a fully qualified ROS name would be broken down into DDS concepts:

| ROS Name | DDS Topic | DDS Partition |
|---------------------------------|-------------|-------------------------------|
| `/foo` | `foo` | \[`rt`\] |
| `rostopic:///foo/bar` | `bar` | \[`rt/foo`\] |
| `/robot1/camera_left/image_raw` | `image_raw` | \[`rt/robot1/camera_left`\] |
| ROS Name | DDS Topic |
|---------------------------------|-----------------------------------|
| `/foo` | `rt/foo` |
| `rostopic:///foo/bar` | `rt/foo/bar` |
| `/robot1/camera_left/image_raw` | `rt/robot1/camera_left/image_raw` |

### Changing ROS Names During Runtime

If the ROS name needs to be changed during runtime (after the publisher or subscriber has already been created and is in use), then some details about DDS partitions might be useful to consider.
### ROS Topic and Service Name Length Limit

Since this proposal splits the ROS name into namespace and base name, we'll consider how that might affect DDS partitions and DDS topic names respectively.
When changing the namespace of a ROS name, i.e. everything but the base name, then this could be done by changing just the existing partition string.
As partitions are part of QoS settings, it technically doesn't require a full restart of the publisher and subscription when changing the partitions field, however we cannot change the DDS topic name on the fly.
With this in mind, we may want to always require a full restart of the publisher or subscription instance to avoid inconsistency in how these changes behave from the ROS user's perspective.
The length of the DDS topic must not exceed 256 characters. Therefore the length of a ROS Topic, including the namespace hierarchy, the base name of the topic and any ros specific prefixes must not exceed 256 characters since this is mapped directly as DDS topic.

### ROS Topic and Service Name Length Limit
#### Considerations for RTI Connext

The length of the DDS topic must not exceed 256 characters.
The actual length of a partition field may be limited to 256 characters, however this varies drastically depending on the vendor.
RTI Connext does not allow a creation of a publisher/subscription with a partition length of 248 characters, whereas FastRTPS does not have any limitation in length.
Please bare in mind, that the length of the partition gets further diminished due to the introduction of a ROS specific prefix.
The actual length of a ROS Topic, including the namespace hierarchy and the base name of the topic, may thus be varying in length as well.
Yet, the base name token must not exceed the length of 256 characters as this is getting mapped directly as the DDS topic.
While testing our implementation with Connext, we encountered some additional limitations when it comes to the topic length.
That is, for example the length of a service name has tighter limits than the length of the ROS Topics.
The RTI Connext implementation for service names are suffixed with the GUID value of the DDS participant for the service response topic.
Additionally, a content filtered topic (max length 256 characters) is created which is mapped from the suffixed service name.
Therefore when linking against `rmw_connext_c` or `rmw_connext_cpp`, service names cannot be longer than 185 characters including the namespace hierarchy and any ros specific prefixes.

### Communicating with Non-ROS Topics

Since all ROS topics are prefixed when being converted to DDS topic names, it makes it impossible to subscribe to existing DDS topics which do not follow the same naming pattern.
For example, if an existing DDS program is publishing on the `image` topic (and is using the DDS equivalent to the ROS message type) then a ROS program could not subscribe to it because of the name mangling produced by the implicit ROS specific namespace.
Therefore to allow ROS programs to interoperate with "native" DDS topic names the API should provide a way to skip the ROS specific prefixing.

#### Examples and Ideas for Communicating with Non-ROS Topics

Note that these are not part of the proposal, but only possible solutions to the issue of communicating with "native" DDS topics.
This section should be update when/if a complete solution is found.
There is an option in the API, a boolean `avoid_ros_namespace_convention` in the qos_profile which can be set to `false` to use ROS prefix and `true` to not using ROS namespace prefixing.

It may be possible, through an option in the API or through some name syntax, to get a similar break down but without the ROS specific prefix.
For example:

For example with an option in the API:
| ROS Name | avoid_ros_namespace_conventions | DDS Topic |
|-----------------------|------------------------------------|-------------|
| `rostopic://image` | `false` | `rt/image` |
| `rostopic://image` | `true` | `image` |
Copy link
Member

@mikaelarguedas mikaelarguedas May 15, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my understanding is that this will result in /image (based on discussion from ros2/rmw_fastrtps#192 (comment)).
We can leave this as is in the design doc if we ticket the missing bit that is to pass the user-specified topic name to rmw instead of the expanded one.
Otherwise we should correct it here

(same below)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes that's correct I think it should be /image instead of image, as of now by default the topic name being passed here will already have a / appended to it (if / is not already present) be a fully qualified as ros topic isn't it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ros2/ros2#496 has been created to track what is described here but not yet implemented


| ROS Name | ROS Prefix | DDS Topic | DDS Partition |
|-----------------------|------------|-------------|-------------------------------|
| `rostopic://image` | with | `image` | \[`rt`\] |
| `rostopic://image` | without | `image` | \[\] |
#### Alternative(Idea)

Note that the alternative below is not part of the proposal, but only possible solutions to the issue of communicating with "native" DDS topics.
Another option would be to have some markup in the scheme name, for example:

| ROS Name | DDS Topic | DDS Partition |
|---------------------------------------|---------------------|-------------------------------|
| `rostopic://image` | `image` | \[`rt`\] |
| `rostopic+exact://image` | `image` | \[\] |
| `rostopic+exact://camera_left/image` | `camera_left/image` | \[\] |

Considering the third case, it's not clear if the `/` should be respected or if additional syntax is required to support partitions in the case of "exact" topic names.
You could also imagine:

| ROS Name | DDS Topic | DDS Partition |
|-----------------------------------------|---------------------|-------------------------------|
| `rostopic+exact+ns://camera_left/image` | `image` | \[`camera_left`\] |
| ROS Name | DDS Topic |
|---------------------------------------|---------------------|
| `rostopic://image` | `rt/image` |
| `rostopic+exact://image` | `image` |
| `rostopic+exact://camera_left/image` | `camera_left/image` |
| `rostopic+exact:///camera_left/image` | `camera_left/image` |

## Compare and Contrast with ROS 1

In order to support a mapping to the more restrictive DDS topic name rules, these rules are in some ways more constraining than the rules for ROS 1.
In order to support a mapping to the - slightly more - restrictive DDS topic name rules, these rules are in some ways more constraining than the rules for ROS 1.
Other changes have been proposed for convenience or to remove a point of confusion that existed in ROS 1.
In ROS 2, topic and service names differ from ROS 1 in that they:

Expand Down Expand Up @@ -391,6 +345,32 @@ This compromise was made so that when more work is done on substitutions, it wou

There was some discussion of alternatives and concerns with respect to the ROS -> DDS translation.

#### Alternative using DDS Partitions

Previously the usage of forward slashes (`/`) was disallowed in DDS topic name and hence a strategy was proposed which used DDS partitions to address the forward slashes (`/`) which are present in ROS names. The main idea was to separate the ROS name into the "namespace" and the "base name", and then place the namespace, stripped of leading and trailing forward slashes (`/`), into a single DDS partition entry and the remaining base name into the DDS topic name.
This addressed the issue because the ROS name's base name will not contain any forward slashes (`/`) by definition and so there are no longer any disallowed characters in the DDS topic name.
The DDS partition would contain the ROS name's namespace, including any forward slashes (`/`) that made up the namespace and were not at the beginning or the end of the namespace.
That is acceptable because DDS partitions are allowed to contain forward slashes (`/`) unlike the DDS topics previously but now DDS topic names allow forward slashes (`/`).

DDS partitions are implemented as an array of strings within the `DDS::Publisher` and `DDS::Subscriber` QoS settings and have no hierarchy or order, Each entry in the partition array is directly combined with the DDS topic and they are not sequentially combined.
If a publisher has two partition entries, e.g. `foo` and `bar` with a base name of `baz`, this would be equivalent to having two different publishers on these topics: `/foo/baz` and `/bar/baz`.
Therefore this proposal used only one of the strings in the partitions array to hold the entire ROS name's namespace.

You can read more about partitions in RTI's documentation:

- [PARTITION_QosPolicy](https://community.rti.com/static/documentation/connext-dds/5.2.3/doc/manuals/connext_dds/html_files/RTI_ConnextDDS_CoreLibraries_UsersManual/Content/UsersManual/PARTITION_QosPolicy.htm)

Trade-offs (in comparison to using the whole ROS name along with the namespaces):

- Splitting the ROS name into "namespace" and "base name", and placing the complete namespace into a field designed for another purpose seemed incorrect.
- In general partitions are recommended to be used as a spare, but using partitions for all ROS names suggested otherwise.
- Major concern was reported in this [issue](https://github.com/ros2/rmw_connext/issues/234), where having two topics with same base name, although different namespace and different types caused problem. For example: topicA is `/camera/data` of type `Image` and topicB is `/imu/data` of type `Imu`. The base names for both topicA and topicB is `data`, generated errors as described in the [issue](https://github.com/ros2/rmw_connext/issues/234).
- Newer standards such as [DDS-XRCE](https://www.omg.org/spec/DDS-XRCE) might not have partitions at all.
- Using the complete ROS name in the later strategy will cause a tighter length limit on base name because the DDS topic name would contain ROS prefix, namespace along with the base name which should not exceed DDS topic name limitation which is 256 characters.

Rationale:
- With the decision from the DDS vendors to allow forward slashes (`/`) in DDS topic names, using the complete ROS name seemed simple and more intuitive than using partitions.

#### Alternative Substitute the Namespace Delimiter

A previous proposal was to substitute the namespace delimiter, i.e. forward slash (`/`), with something that is allowed in DDS topic names, and then only use the DDS topic name to represent the full ROS name.
Expand Down