Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Which one is AC metrics? #6

Closed
KJ-Falloutlast opened this issue Feb 13, 2025 · 2 comments
Closed

Which one is AC metrics? #6

KJ-Falloutlast opened this issue Feb 13, 2025 · 2 comments
Labels

Comments

@KJ-Falloutlast
Copy link

Hello Dr. Hu! I have long admired your work, and I would like to extend my congratulations on your recent acceptance! There are a few questions I would like to clarify:

  1. In the eval_result.txt file, is F1 intended to represent what you referred to as the AC metric? I couldn't locate a metric specifically named AC.
  2. Regarding the relationship between map accuracy and the AC or CD metrics: From my understanding, when our algorithm produces a more accurate map, both the AC value should decrease and so should the CD value. Is this interpretation correct?
  3. However, over time, multiple AC and CD values can be obtained. For instance, here is an array of CD values: CD: 0.15422, 0.07356, 0.03098, 0.00980, 0.00357. When comparing different algorithms, which value should we choose to best represent our final result?

Image

@JokerJohn
Copy link
Owner

JokerJohn commented Feb 14, 2025

Hi, @KJ-Falloutlast ,
Thank you for your interest in our work. I have updated the code to remove the confusing parts.

  1. F1 and AC are unrelated metrics. While F1 was initially considered, we found it unsuitable for evaluating SLAM point cloud maps, unlike object-level 3D reconstruction. This is because SLAM maps and their corresponding ground truth are often captured using different devices, resulting in varying scanning patterns and viewpoints. Overlap-based metrics become less meaningful in this context. Therefore, we have removed this metric.

  2. AC is the RMSE between inliers within a specified accuracy threshold (accuracy_level), while CD directly measures the bidirectional minimum average error between all points in two point clouds. AC < CD is expected. However, CD is sensitive to:

  • Point cloud noise (e.g., glass reflections)
  • Point density variations
  • Outliers (even a small cluster of distant outliers can significantly increase CD).
    These points often don't exist in ground truth and shouldn't be included in evaluation (refer to Section VI-A). You're correct that generally, higher map accuracy results in lower AC and CD values.
  1. Multiple AC values are calculated for different accuracy thresholds. We typically use only the first value, corresponding to the first element in accuracy_level. CD is calculated as a single value.

@KJ-Falloutlast
Copy link
Author

Dear Dr. Hu, thank you very much for your prompt reply! My questions have been fully addressed through both the updated code and your detailed explanation. I appreciate your help immensely.

Thank you once again for your time and assistance!

@JokerJohn JokerJohn pinned this issue Feb 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants