Code repository for the paper CORSAIR: Convolutional Object Retrieval and Symmetry-AIded Registration.
@INPROCEEDINGS{CORSAIR,
author = {Zhao, Tianyu and Feng, Qiaojun and Jadhav, Sai and Atanasov, Nikolay},
booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
title = {{CORSAIR: Convolutional Object Retrieval and Symmetry-AIded Registration}},
year = {2021},
pages = {47-54},
doi = {10.1109/IROS51168.2021.9636347}
}
git clone --recursive --branch main https://github.com/ExistentialRobotics/CORSAIR.git
cd CORSAIR/docker
# install docker and nvidia-container-toolkit if not installed
./install_docker_ubuntu.bash
reboot # reboot the system to apply the changes
./build.bash # builds the docker image
Install Docker, and build docker images following instructions from the previous section. For pose estimation, two categories are available, namely chair and table. Execute one of the commands below to run the demos.
cd CORSAIR/docker
# Pose estimation
./run.bash #runs the docker image for category chair by default
ENTRYPOINT="/home/user/CORSAIR/entrypoint_table.bash" ./run.bash #runs the docker image for category table
Install Docker, and build docker images following instructions from the previous section. We demonstrate shape estimation on the chair category. Execute the command below to run the demo.
cd CORSAIR/docker
# Shape estimation on chair
./run_shape.bash # Runs the docker image for shape estimation with Chamfer distance metric (KETI Metric 11)
On the first run, the script will download the test data as Docker images, and mount those as volumes. After running the demo, it may be necessary to clean up the data volumes. To do so:
docker volume prune
git clone --recursive --branch main https://github.com/ExistentialRobotics/CORSAIR.git
pipenv lock --verbose
pipenv sync --verbose
pipenv shell
cd deps/MinkowskiEngine # 0.5.5
pip install . --verbose
The PyTorch
and CUDA
versions should match otherwise the installation of MinkowskiEngine
will fail. To install
older versions of CUDA
compatible with PyTorch
, here is an example showing how to install CUDA 12.1
for PyTorch 2.3
on Arch Linux
:
wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda_12.1.0_530.30.02_linux.run
chmod +x cuda_12.1.0_530.30.02_linux.run
sudo ./cuda_12.1.0_530.30.02_linux.run --toolkitpath=/opt/cuda-12.1 --toolkit --override
paru -S gcc12 gcc12-libs # Max GCC version supported by CUDA 12.1
Then, install MinokowskiEngine
with the following command:
CC=gcc-12 CXX=g++-12 CUDA_HOME=/opt/cuda-12.1 pip install . --verbose
- Download ShapeNetCore.v2.PC15k
from here and extract it
to
data/ShapeNetCore.v2.PC15k
. - Download pre-processed Scan2CAD data
from here and extract it
to
data/Scan2CAD_pc
. - Download Scan2CAD annotation data
from here and extract it
to
data/Scan2CAD_annotations
.
Evaluate CORSAIR
options:
-h, --help show this help message and exit
--shapenet-pc15k-root SHAPENET_PC15K_ROOT
Path to ShapeNetCore.v2.PC15k
--scan2cad-pc-root SCAN2CAD_PC_ROOT
Path to Scan2CAD
--scan2cad-annotation-root SCAN2CAD_ANNOTATION_ROOT
Path to Scan2CAD annotations
--category {table,chair}
Category to evaluate
--checkpoint CHECKPOINT
Path to the checkpoint
--cache-dir CACHE_DIR
Path to load / save the result of registration.
--register-gt Registering gt CAD model
--device {cuda,cpu} Device to use for evaluation
--ignore-cache Ignore cached results
For example,
python evaluation.py --shapenet-pc15k-root data/ShapeNetCore.v2.PC15k --scan2cad-pc-root data/Scan2CAD_pc --scan2cad-annotation-root data/Scan2CAD_annotations --category chair --checkpoint ckpts/scannet_pose_chair_best --cache-dir data/cache_pose_best --register-gt --device cuda
will evaluate the model on the chair category and register the ground-truth CAD model using the GPU.
- Press Left/Right to navigate through the results.
- Drag the mouse to rotate the object.
Note: the Predicted Closest CAD PC
may not align with the query point cloud well when the matching point cloud is
not the ground truth.
Note: the numbers of Top1-Prediction
tend to be lower because the predicted CAD model may not be the ground-truth
match, making the relative pose estimate less accurate.
Checkpoint | Precision@10% | Top1-CD |
---|---|---|
scannet_pose_chair_best | 22.55% | 0.17 |
scannet_pose_chair | 22.71% | 0.17 |
scannet_ret_chair_best | 31.27% | 0.15 |
scannet_ret_chair | 31.30% | 0.15 |
Checkpoint | Precision@10% | Top1-CD |
---|---|---|
scannet_pose_table_best | 31.89% | 0.22 |
scannet_pose_table | 31.70% | 0.22 |
scannet_ret_table_bset | 41.13% | 0.19 |
scannet_ret_table | 41.05% | 0.19 |
- RRE: relative rotation error
Checkpoint | Registration Target | Sym. | Mean RRE | RRE |
RRE |
RRE |
---|---|---|---|---|---|---|
scannet_pose_chair_best | Top1-Prediction | N | 6.64% | 54.78% | 80.36% | |
Y | 9.87% | 59.82% | 81.17% | |||
G.T. Matching | N | 12.69% | 75.33% | 92.15% | ||
Y | 20.24% | 80.87% | 91.54% | |||
scannet_pose_chair | Top1-Prediction | N | 8.06% | 54.98% | 79.46% | |
Y | 9.77% | 59.82% | 79.56% | |||
G.T. Matching | N | 13.39% | 73.11% | 91.64% | ||
Y | 19.54% | 79.15% | 91.84% | |||
scannet_ret_chair_best | Top1-Prediction | N | 7.96% | 60.12% | 83.79% | |
Y | 10.98% | 66.26% | 83.38% | |||
G.T. Matching | N | 13.19% | 74.62% | 91.64% | ||
Y | 19.54% | 80.26% | 92.04% | |||
scannet_ret_chair | Top1-Prediction | N | 8.26% | 64.65% | 84.49% | |
Y | 10.78% | 68.38% | 84.19% | |||
G.T. Matching | N | 13.39% | 74.92% | 91.14% | ||
Y | 19.33% | 81.77% | 91.54% |
Checkpoint | Registration Target | Sym. | Mean RRE | RRE |
RRE |
RRE |
---|---|---|---|---|---|---|
scannet_pose_table_best | Top1-Prediction | N | 11.68% | 42.61% | 59.79% | |
Y | 22.33% | 50.52% | 61.51% | |||
G.T. Matching | N | 18.21% | 59.11% | 74.57% | ||
Y | 33.33% | 69.41% | 76.98% | |||
scannet_pose_table | Top1-Prediction | N | 13.40% | 42.27% | 56.01% | |
Y | 22.68% | 47.42% | 58.08% | |||
G.T. Matching | N | 21.31% | 62.89% | 76.29% | ||
Y | 35.05% | 70.45% | 75.60% | |||
scannet_ret_table_best | Top1-Prediction | N | 13.40% | 44.33% | 58.42% | |
Y | 21.65% | 51.55% | 61.86% | |||
G.T. Matching | N | 20.27% | 62.89% | 73.88% | ||
Y | 39.52% | 73.88% | 78.01% | |||
scannet_ret_table | Top1-Prediction | N | 11.34% | 36.77% | 54.30% | |
Y | 20.27% | 45.02% | 57.04% | |||
G.T. Matching | N | 20.27% | 62.89% | 73.54% | ||
Y | 35.05% | 70.10% | 74.57% |
- RTE: relative translation error
Checkpoint | Registration Target | Sym. | Mean RTE | RTE |
RTE |
RTE |
RTE |
---|---|---|---|---|---|---|---|
scannet_pose_chair_best | Top1-Prediction | N | 0.28 | 0.20% | 3.93% | 20.95% | 43.61% |
Y | 0.27 | 0.30% | 4.53% | 23.36% | 47.33% | ||
G.T. Matching | N | 0.15 | 1.31% | 13.70% | 49.35% | 73.21% | |
Y | 0.15 | 1.61% | 17.52% | 58.01% | 78.15% | ||
scannet_pose_chair | Top1-Prediction | N | 0.28 | 0.30% | 4.73% | 22.86% | 43.00% |
Y | 0.27 | 0.30% | 4.83% | 24.37% | 45.02% | ||
G.T. Matching | N | 0.16 | 1.10% | 13.19% | 46.83% | 73.41% | |
Y | 0.15 | 1.61% | 18.12% | 56.60% | 78.05% | ||
scannet_ret_chair_best | Top1-Prediction | N | 0.25 | 0.30% | 4.33% | 26.38% | 47.23% |
Y | 0.24 | 0.50% | 6.24% | 29.90% | 51.56% | ||
G.T. Matching | N | 0.16 | 1.21% | 11.78% | 47.63% | 72.21% | |
Y | 0.15 | 2.01% | 18.03% | 55.39% | 78.75% | ||
scannet_ret_chair | Top1-Prediction | N | 0.23 | 0.40% | 6.64% | 29.00% | 53.47% |
Y | 0.23 | 0.30% | 7.96% | 33.94% | 56.90% | ||
G.T. Matching | N | 0.16 | 1.11% | 13.60% | 49.85% | 74.22% | |
Y | 0.15 | 1.41% | 16.92% | 57.00% | 79.25% |
Checkpoint | Registration Target | Sym. | Mean RTE | RTE |
RTE |
RTE |
RTE |
---|---|---|---|---|---|---|---|
scannet_pose_table_best | Top1-Prediction | N | 0.35 | 0.34% | 5.15% | 21.31% | 35.40% |
Y | 0.34 | 1.37% | 5.84% | 23.37% | 37.46% | ||
G.T. Matching | N | 0.28 | 2.41% | 12.71% | 39.86% | 56.01% | |
Y | 0.24 | 3.78% | 25.77% | 56.01% | 67.70% | ||
scannet_pose_table | Top1-Prediction | N | 0.39 | 0.69% | 5.84% | 18.90% | 30.24% |
Y | 0.38 | 0.69% | 6.19% | 21.31% | 38.14% | ||
G.T. Matching | N | 0.25 | 1.72% | 13.06% | 40.21% | 59.79% | |
Y | 0.23 | 2.06% | 25.77% | 53.61% | 67.70% | ||
scannet_ret_table_best | Top1-Prediction | N | 0.37 | 0.34% | 4.81% | 20.27% | 33.33% |
Y | 0.36 | 0.34% | 8.25% | 27.15% | 37.46% | ||
G.T. Matching | N | 0.27 | 1.37% | 14.09% | 41.24% | 58.42% | |
Y | 0.22 | 3.09% | 28.18% | 56.70% | 73.20% | ||
scannet_ret_table | Top1-Prediction | N | 0.40 | 0.34% | 4.81% | 16.49% | 28.18% |
Y | 0.39 | 0.00% | 6.19% | 20.27% | 34.36% | ||
G.T. Matching | N | 0.27 | 0.34% | 11.68% | 41.24% | 61.17% | |
Y | 0.25 | 4.12% | 23.02% | 54.30% | 68.38% |
The above registration results are affected by the quality of the predicted CAD model and the accuracy of the ground
truth poses. To evaluate the registration more fairly with accurate ground truth and without the affect of bad object
retrieval, please run the evalution-shapenet.py
script. e.g.
python evaluation-shapenet.py --shapenet-root ./Data/ShapeNetCore.v2.PC15k --category chair --model-ckpt ckpts/scannet_ret_chair --n-models 100
python evaluation-shapenet.py --shapenet-root ./Data/ShapeNetCore.v2.PC15k --category table --model-ckpt ckpts/scannet_pose_table_best --n-models 100
The numbers vary slightly due to the unpredictable parallelism behavior of the Open3D
RANSAC
algorithm.
Checkpoint | Sym. | RRE |
RRE |
RRE |
RTE |
RTE |
RTE |
RTE |
---|---|---|---|---|---|---|---|---|
scannet_ret_chair | N | 72% | 98% | 99% | 30% | 93% | 98% | 100% |
scannet_ret_chair | Y | 93% | 100% | 100% | 64% | 97% | 100% | 100% |
scannet_pose_table_best | N | 84% | 99% | 99% | 32% | 88% | 100% | 100% |
scannet_pose_table_best | Y | 92% | 98% | 98% | 69% | 95% | 99% | 100% |
We gratefully acknowledge support from ARL DCIST CRA W911NF17-2-0181 and ONR SAI N00014-18-1-2828 and the Technology Innovation Program (20018112, Development of autonomous manipulation and gripping technology using imitation learning based on visual and tactile sensing) funded by the Ministry of Trade, Industry & Energy (MOTIE), Korea.