Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(perception_online_evaluator): add metric_value not only stat #7100

Merged
merged 1 commit into from
May 24, 2024

Conversation

kosuke55
Copy link
Contributor

@kosuke55 kosuke55 commented May 23, 2024

Description

Previously, only stat with max, mean, and min as keys could be handled in diagnostics, but this PR allows metric_value with a single value.
This is to allow only one value to be included in metrics such as object counts, since all stat values are the same, which is redundant.

  • stat type
- level: "\0"
  name: predicted_path_deviation_variance_CAR_5.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.039821'
  - key: max
    value: '0.039821'
  - key: mean
    value: '0.039821'
  • metric_value type
- level: "\0"
  name: interval_average_objects_count_UNKNOWN_r100.00_h10.00
  message: ''
  hardware_id: ''
  values:
  - key: metric_value
    value: '4.000000'

image

Tests performed

psim

Effects on system behavior

Not applicable.

Interface changes

no interface changes for autoware.

but the diagnostic key used for tools (driving_log_replayer etc) is chaged

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

  • There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

@kosuke55 kosuke55 marked this pull request as ready for review May 23, 2024 03:01
@github-actions github-actions bot added the component:evaluator Evaluation tools for planning, localization etc. (auto-assigned) label May 23, 2024
@kosuke55 kosuke55 force-pushed the feat/metric_value branch from b300c36 to d14f690 Compare May 23, 2024 03:17
@kosuke55 kosuke55 added the run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci) label May 23, 2024
Signed-off-by: kosuke55 <kosuke.tnp@gmail.com>
@kosuke55 kosuke55 force-pushed the feat/metric_value branch from d14f690 to 3ecee88 Compare May 23, 2024 03:29
@kosuke55 kosuke55 requested a review from MasatoSaeki May 23, 2024 05:47
Copy link
Contributor

@hayato-m126 hayato-m126 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked the source code.
I also confirmed that the following two types of DianosticsStatus are produced by actually running the code.

  • min, max, mean
  • metric_value
❯ ros2 topic echo /diagnostic/perception_online_evaluator/metrics
1716517666.266979 [77]       ros2: determined eno1 (udp/10.0.55.137) as highest quality interface, selected for automatic interface.
header:
  stamp:
    sec: 1649138862
    nanosec: 528072108
  frame_id: ''
status:
- level: "\0"
  name: lateral_deviation_CAR
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.002937'
  - key: max
    value: '0.005215'
  - key: mean
    value: '0.004076'
- level: "\0"
  name: yaw_deviation_CAR
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.014042'
  - key: max
    value: '0.038703'
  - key: mean
    value: '0.026373'
- level: "\0"
  name: predicted_path_deviation_variance_CAR_5.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '3.730826'
  - key: max
    value: '6.671615'
  - key: mean
    value: '5.201220'
- level: "\0"
  name: predicted_path_deviation_variance_CAR_3.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.903402'
  - key: max
    value: '1.781432'
  - key: mean
    value: '1.342417'
- level: "\0"
  name: predicted_path_deviation_CAR_3.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '1.064637'
  - key: max
    value: '1.540737'
  - key: mean
    value: '1.302687'
- level: "\0"
  name: predicted_path_deviation_CAR_5.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '2.362636'
  - key: max
    value: '3.284624'
  - key: mean
    value: '2.823630'
- level: "\0"
  name: predicted_path_deviation_CAR_2.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.526803'
  - key: max
    value: '0.812374'
  - key: mean
    value: '0.669589'
- level: "\0"
  name: predicted_path_deviation_variance_CAR_2.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.172313'
  - key: max
    value: '0.507762'
  - key: mean
    value: '0.340038'
- level: "\0"
  name: predicted_path_deviation_variance_CAR_1.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.033420'
  - key: max
    value: '0.054184'
  - key: mean
    value: '0.043802'
- level: "\0"
  name: predicted_path_deviation_CAR_1.00
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.252737'
  - key: max
    value: '0.285710'
  - key: mean
    value: '0.269224'
- level: "\0"
  name: yaw_rate_UNKNOWN
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.000008'
  - key: max
    value: '0.000008'
  - key: mean
    value: '0.000008'
- level: "\0"
  name: yaw_rate_CAR
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.000485'
  - key: max
    value: '0.001173'
  - key: mean
    value: '0.000829'
- level: "\0"
  name: yaw_rate_BUS
  message: ''
  hardware_id: ''
  values:
  - key: min
    value: '0.000009'
  - key: max
    value: '0.000009'
  - key: mean
    value: '0.000009'
- level: "\0"
  name: total_objects_count_PEDESTRIAN_r150.00_h10.00
  message: ''
  hardware_id: ''
  values:
  - key: metric_value
    value: '0.000000'

@kosuke55 kosuke55 merged commit fe30833 into autowarefoundation:main May 24, 2024
26 of 28 checks passed
@kosuke55 kosuke55 deleted the feat/metric_value branch May 24, 2024 07:47
karishma1911 pushed a commit to Interplai/autoware.universe that referenced this pull request Jun 3, 2024
a-maumau pushed a commit to a-maumau/autoware.universe that referenced this pull request Jun 7, 2024
a-maumau pushed a commit to a-maumau/autoware.universe that referenced this pull request Jun 7, 2024
hayato-m126 pushed a commit that referenced this pull request Jun 7, 2024
…)(#7118) (revert of revert) (#7167)

* Revert "fix(perception_online_evaluator): revert "add metric_value not only s…"

This reverts commit d827b1b.

* Update evaluator/perception_online_evaluator/include/perception_online_evaluator/perception_online_evaluator_node.hpp

* Update evaluator/perception_online_evaluator/src/perception_online_evaluator_node.cpp

* use emplace back

Signed-off-by: kosuke55 <kosuke.tnp@gmail.com>

---------

Signed-off-by: kosuke55 <kosuke.tnp@gmail.com>
Co-authored-by: Kotaro Uetake <60615504+ktro2828@users.noreply.github.com>
KhalilSelyan pushed a commit that referenced this pull request Jul 22, 2024
)

Signed-off-by: kosuke55 <kosuke.tnp@gmail.com>
KhalilSelyan pushed a commit that referenced this pull request Jul 22, 2024
…)(#7118) (revert of revert) (#7167)

* Revert "fix(perception_online_evaluator): revert "add metric_value not only s…"

This reverts commit d827b1b.

* Update evaluator/perception_online_evaluator/include/perception_online_evaluator/perception_online_evaluator_node.hpp

* Update evaluator/perception_online_evaluator/src/perception_online_evaluator_node.cpp

* use emplace back

Signed-off-by: kosuke55 <kosuke.tnp@gmail.com>

---------

Signed-off-by: kosuke55 <kosuke.tnp@gmail.com>
Co-authored-by: Kotaro Uetake <60615504+ktro2828@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component:evaluator Evaluation tools for planning, localization etc. (auto-assigned) run:build-and-test-differential Mark to enable build-and-test-differential workflow. (used-by-ci)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants