[Discussion] LiDAR Centerpoint Inference Process

[twbabyduck]

The idea is improving the documentation in the lidar_centerpoint package and if possible improve the model accuracy.
https://github.com/autowarefoundation/autoware.universe/tree/main/perception/lidar_centerpoint

• Instructions how to use https://github.com/open-mmlab/mmdetection3d to train on customized dataset.
• A simple chart to explain the relations between ONNX, TensorRT (.engine) files.
• Reproduce inference results with lidar_centerpoint
• Improve lidar_centerpoint trained model accuracy

@yukke42

[twbabyduck]

A simple way to test the inference using lidar_centerpoint package is shown as follows: (will edit later in this week)

Command
ros2 launch lidar_centerpoint lidar_centerpoint.launch.xml input/pointcloud:=/INPUT_TOPIC model_name:=centerpoint

Terminal

Loading from /home/$USER/autoware/install/lidar_centerpoint/share/lidar_centerpoint/data/pts_voxel_encoder_centerpoint.engine
Loading from /home/$USER/autoware/install/lidar_centerpoint/share/lidar_centerpoint/data/backbone_neck_head_centerpoint.engine

INPUT
sensor_msgs::msg::PointCloud2

OUTPUT
autoware_auto_perception_msgs::msg::DetectedObjects

Note (Expected Result):
If the lidar point cloud is given then the inference code should be able to publish the DetectedObjects through above commands.

@yukke42

[twbabyduck]

@yukkysaito
There is no issue to follow the tutorial and it can definitely detected the objects. The question is the lidar_centerpoint should be able to execute separately. Please confirm that if you can execute and get the detected objects from lidar_centerpoint node.

ros2 launch lidar_centerpoint lidar_centerpoint.launch.xml input/pointcloud:=/INPUT_TOPIC model_name:=centerpoint

[yukkysaito]

It need tf (world to base link, base link to sensor frame id) to concatenate multi frame.
So, you need to try with sensing and localization module.

if multi frame is enable, it need tf.
But now, in single frame mode, it need tf. I think it should be changed not to need tf.

[yukkysaito]

@yukke42 How about?

[twbabyduck]

Thanks for clarification! As we discussed in Discord, do we want to make a plan to provide single frame mode.

[twbabyduck]

As discussed in the discord channel. If developer choose to user their own rosbag to test autoware perception package instead of using the data from this tutorial

They have to check the following steps:
(1) lanelet2_map.osm and pointcloud_map.pcd for the map
(2) a rosbag recorded at the same place of (1) so that autoware's localization module can publish tf information

However, in the step (1) some developers might not have corresponding (.osm) and considering they only want to test perception, it appears that the (.osm) should not be the necessary information. So, in this case should we provide instruction about this?

As for the step (2), what are the related packages have to modify in order to test the perception code so that the tf can properly generated from localization and sensing module with their own rosbag? (e.g. Modifications of concatenating point clouds, ...)

[twbabyduck]

A simple way to test the inference using lidar_centerpoint package is shown as follows:

Command
ros2 launch lidar_centerpoint lidar_centerpoint.launch.xml input/pointcloud:=/INPUT_TOPIC model_name:=centerpoint

Terminal

Loading from /home/$USER/autoware/install/lidar_centerpoint/share/lidar_centerpoint/data/pts_voxel_encoder_centerpoint.engine
Loading from /home/$USER/autoware/install/lidar_centerpoint/share/lidar_centerpoint/data/backbone_neck_head_centerpoint.engine

INPUT
sensor_msgs::msg::PointCloud2

OUTPUT
autoware_auto_perception_msgs::msg::DetectedObjects

Note (Expected Result):
If the lidar point cloud is given then the inference code should be able to publish the DetectedObjects through above commands.

Update
The original setup in lidar_centerpoint by default its densification process is enabled, so it means that the input of lidar_centerpoint requires the localization module to publish tf info

For those who want to test the lidar_centerpoint performance, you can use the above commands with the following steps:
(1) Check your input point cloud message's frame_id if you want to use localization module for the densification process make sure frame_id is base_link

(2) If your point cloud simply wants to check lidar_centerpoint visualization and ignore thr other setups, just to make sure go into the autoware workspace to enable rviz plugin visualization, then you can check the output topic named /objects with the corresponding frame_id in your point cloud's topic.

Hope the above description can assist those who want to test the lidar_centerpoint separately.
The details of the steps will update soon 😃

[YueChuCheng]

And I'm curious. Can I test lidar_centerpoint with local data? For example, input .pcd or .bin into centerpoint model and output detection result?

[yukke42]

@YueChuCheng
Sorry for the late reply. We now trying to support to inference with rosbag w/o tf or pcd file.

[lhakim85]

@YueChuCheng,
I try this run with my rosbag, I can get the ouput, but the result not so ok, sometime there is bounding box, sometime, no, maybe because input feed to the lidar centerpoint is raw pointcloud, not the rectified, need to figure it out.
My step

1. ros2 bag play coplace_0.db3 -l --remap /velodyne_points:=/sensing/lidar/top/rectified/pointcloud_ex
2. edit /config/obstacle_segmentation/ground_segmentation/ground_segmentation.param.yaml to include base_frame: velodyne
3. run ros2 launch perception.launch.xml
4. open rviz2, change the fixed frame to velodyne frame
5. add object detection by topic

[YueChuCheng]

@lhakim85 Thanks for your reply, but how can you visualize the "autoware_auto_perception_msgs/msg/DetectedObjects" topic with rviz2? I mean, I can't find any topic for bbox results.

[lhakim85]

@YueChuCheng, for LiDAR Centerpoint, the topic is /perception/object_recognition/detection/centerpoint/objects.