From b2973fb4e670f65cc368edff1043d2356669ecdb Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 20:51:47 +0900 Subject: [PATCH 01/51] feat(how-to-guides): add new pages for important temporal performance metrics. Signed-off-by: Takayuki AKAMINE --- .../tuning-parameters-and-performance/.pages | 1 + ...odel-for-obstacle-segmentation-metrics.svg | 4 + .../model-for-perception-metrics.svg | 4 + .../model-for-top-level-metrics.svg | 4 + .../important-temporal-performance-metrics.md | 116 ++++++++++++++++++ 5 files changed, 129 insertions(+) create mode 100644 docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg create mode 100644 docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg create mode 100644 docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg create mode 100644 docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages index 9cb01955bdf..d42af4c125d 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages @@ -1,3 +1,4 @@ nav: - evaluating-controller-performance.md - evaluating-real-time-performance.md + - important-temporal-performance-metrics.md diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg new file mode 100644 index 00000000000..9d6913ccd06 --- /dev/null +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg @@ -0,0 +1,4 @@ + + + +
LiDARs
LiDARs
CPS #0
CPS #0
Sensing (preprocessing and concatenate)
Sensing (preproces...
CPS #1
CPS #1
CPS #2
CPS #2
Occupancy grid map
Occupancy grid map
Obstacle segmentation
Obstacle segmentation
Planning/behavior path planner
Planning/behavior pat...
subscription
callback
subscription...
timer callback
timer callback
Planning/behavior velocity planner
Planning/behavior vel...
subscription
callback
subscription...
subscription
callback
subscription...
CPS #3
CPS #3
CPS #4
CPS #4
CPS #5
CPS #5
CPS #6
CPS #6
CPS #7
CPS #7
CPS #8
CPS #8
CPS #9
CPS #9
CPS #10
CPS #10
CPS #11
CPS #11
CPS #12
CPS #12
CPS #13
CPS #13
CPS #14
CPS #14
CPS #: Checkpoint for Segmentation and planning components
CPS #: Checkpoint for Segmentation and planni...
Legend
Legend
Key Component
Key Component
Callback
Callback
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg new file mode 100644 index 00000000000..feb695c7c48 --- /dev/null +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg @@ -0,0 +1,4 @@ + + + +
LiDARs
LiDARs
CPP #6
CPP #6
Sensing (preprocessing and concatenate)
Sensing (preprocessing...
Clustering
(including
- occupancy grid map
- obstacle segmentation)
Clustering...
DNN-based
 Object Recognition
(Centerpoint)
DNN-based...
Camera-based
Object Recognition
Camera-based...
OR Cameras
OR Cameras
Sensor Fusion
Sensor Fusion
Detection by Tracker
Detection by Tracker
Association
Merger
Association...
Tracking
Tracking
Prediction
Prediction
Planning
(path_planner)
Planning...
TLR Cameras
TLR Cameras
Camera-based
Traffic Light Recognition
Camera-based...
CPP #P #0
CPP #P #0
CPP #1
CPP #1
CPP #2
CPP #2
CPP #3
CPP #3
CPP #4
CPP #4
CPP #5
CPP #5
CPP #7
CPP #7
CPP #8
CPP #8
CPP #9
CPP #9
CPP #10
CPP #10
CPP #11
CPP #11
CPP #12
CPP #12
CPP #13
CPP #13
CPP #14
CPP #14
CPP #15
CPP #15
CPP #16
CPP #16
CPP #19
CPP #19
CPP #20
CPP #20
CPP #24
CPP #24
CPP #25
CPP #25
CPP #26
CPP #26
timer callback
timer callback
CPP #22
CPP #22
subscription
callback
subscription...
CPP #21
CPP #21
CPP #23
CPP #23
subscription
callback
subscription...
subscription
callback
subscription...
timer callback
(10 Hz)
timer callback...
CPP #28
CPP #28
CPP #30
CPP #30
CPP #27
CPP #27
CPP #29
CPP #29
CPP #31
CPP #31
Validation
Validation
CPP #17
CPP #17
CPP #18
CPP #18
CPP #: Checkpoint for Perception components
CPP #: Checkpoint for Perception components
Legend
Legend
Key Component
Key Component
Callback
Callback
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg new file mode 100644 index 00000000000..b6ec6119716 --- /dev/null +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg @@ -0,0 +1,4 @@ + + + +
Sensing
Sensing
Sensor
(LiDARs)
Sensor...
Localization
/NDT Scan Matcher
Localization/NDT Scan...
Planning
Planning
Control
Control
Vehicle Interface
Vehicle Interface
CPA #0
CPA #0
CPA #: Checkpoint for Autoware overview
CPA #: Checkpoint for Autoware overview
Vehicle
ECU
Vehicle...
Localization
/Stop Filter
Localization...
CPA #1
CPA #1
CPA #2
CPA #2
CPA #10
CPA #10
CPA #3
CPA #3
CPA #9
CPA #9
CPA #11
CPA #11
CPA #16
CPA #16
CPA #17
CPA #17
subscription
callback
subscription...
subscription
callback
subscription...
CPA #12
CPA #12
Perception/
Detection
Perception/...
Perception/
Tracking
Perception/...
Perception/
Prediction
Perception/...
subscription
callback
subscription...
timer
callback
timer...
CPA #4
CPA #4
CPA #5
CPA #5
CPA #7
CPA #7
CPA #6
CPA #6
CPA #8
CPA #8
Legend
Legend
Key Component
Key Component
Callback
Callback
CPA #13
CPA #13
subscription
callback
subscription...
timer
callback
timer...
CPA #14
CPA #14
Localization
/EKF Localizer
Localization/EKF...
CPA #18
CPA #18
subscription
callback
subscription...
CPA #19
CPA #19
subscription
callback
subscription...
CPA #20
CPA #20
CPA #21
CPA #21
CPA #15
CPA #15
CPA #22
CPA #22
subscription
callback
subscription...
CPA #23
CPA #23
CPA #24
CPA #24
CPA #25
CPA #25
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...
CPA #26
CPA #26
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...
a chain of callback driven by a timer
a chain of callback dr...
a chain of callback driven by a timer
a chain of callback dr...Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md new file mode 100644 index 00000000000..96e857b4967 --- /dev/null +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -0,0 +1,116 @@ +# Important temporal performance metrics on components + +## Motivation to list temporal performance metrics + +### Objective of the page + +This page introduces important metrics to evaluate temporal performance on components of Autoware. + +It is expected that most algorithms employed for Autoware are executed with high frequency and short response time as possible. In order to achieve safe autonomous driving, one of the desired outcomes is no time gap between perceived and actual situation. The time gap is commonly referred to as delay. If the delay is significant, the system may determine trajectory and maneuver based on outdated situation. Consequently, if the actual situation differs from the perceived one due to the delay, the system may make unexpected decisions. + +As mentioned earlier, this page presents lists of temporal performance metrics that are crucial for the main functionalities of Autoware: Localization, Perception, Planning, and Control. + +!!! Note + + Other functionalities, such as system components for diagnosis, are excluded currently. However they will be taken into account in the near future. + +### Contribution of the temporal performance metrics + +Temporal performance metrics are important for evaluating Autoware. These metrics are particularly useful for assessing delays caused by new algorithms and logic. They can be employed when comparing the temporal performance of software on a desktop computer with that on a vehicle during the vehicle integration phase. + +In addition, these metrics are useful for designers and evaluators of middleware, operating systems, and computers. They are selected based on user and product requirements. One of these requirements is to provide sufficient temporal performance for executing Autoware. "Sufficient temporal performance" is defined as a temporal performance requirement, but it can be challenging to define the requirement because it varies depending on the product type, operational design domain (ODD), and other factors. Then, this page specifically focuses on temporal performance metrics rather than requirements. + +Temporal performance metrics are important for evaluating the reliability of Autoware. However, ensuring the reliability of Autoware requires consideration of not only temporal performance metrics but also other metrics. + +## Temporal performance metrics list + +This section is divided into multiple subsections, each containing a model diagram and an accompanying list that explains the important temporal performance metrics. Each model is equipped with checkpoints that serve as indicators for these metrics. + +The first subsection presents the top-level temporal performance metrics, which are depicted in the abstract structure of Autoware as a whole. The detailed metrics are not included in the model as they would add complexity to it. Instead, the subsequent section introduces the detailed metrics. The detailed metrics are subject to more frequent updates compared to the top-level ones, which is another reason for categorizing them separately. + +Each list includes a column for the reference value. The reference value represents the observed value of each metric when Autoware is running according to the tutorial. It is important to note that the reference value is not a required value, meaning that Autoware does not necessarily fail in the tutorial execution if certain metrics do not fulfill the reference value. + +### Top-level temporal performance metrics for Autoware + +The diagram below introduces the model for top-level temporal performance metrics. + +![Model for top-level temporal performance metrics](./images/important-temporal-performance-metrics/model-for-top-level-metrics.svg) + +The following three policies assist in selecting the top-level performance metrics: + +- Splitting Autoware based on components that consume observed values, such as sensor data, and considering the processing frequency and response time around these components +- Dividing Autoware based on the entry point of Planning and Control and considering the processing frequency and response time around these components +- Showing the minimum metrics for the Vehicle Interface, as they may vary depending on the target vehicle + +Additionally, it is assumed that algorithms are implemented as multiple nodes and function as a pipeline processing system. + +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | +| ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object Recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | +| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Performance requirement varies depending on vehicle type. | + +### Detailed temporal performance metrics for Perception + +The diagram below introduces the model for temporal performance metrics for Perception. + +![Model for Perception temporal performance metrics](./images/important-temporal-performance-metrics/model-for-perception-metrics.svg) + +The following two policies assist in selecting the performance metrics: + +- Regarding the frequency and response time at which Recognition results from Object Recognition and Traffic Light Recognition are consumed in Planning +- Splitting the merging point of data from multiple processing paths and considering the frequency and response time around that point + +The following list shows the temporal performance metrics for Perception. + +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | +| ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | +| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | +| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | +| APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | +| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | +| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | +| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | +| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | +| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | +| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | + +### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning + +Obstacle segmentation, which is a crucial part of Perception, transmits data to Planning. The figure below illustrates the model that takes into account performance metrics related to Obstacle segmentation and Planning. + +![Model for Obstacle segmentation temporal performance metrics](./images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg) + +!!! Note + Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. + +The following list shows the temporal performance metrics around Obstacle segmentation and Planning. + +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | +| ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | +| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-004 | Response time from CPS #0 to CPS #13
or
CPS #1 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From d50d7e3ecf45477e623c824f7a8ad630e1e3a1ce Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 11:54:58 +0000 Subject: [PATCH 02/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 92 +++++++++---------- 1 file changed, 46 insertions(+), 46 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 96e857b4967..ea3ba70674c 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -44,25 +44,25 @@ The following three policies assist in selecting the top-level performance metri Additionally, it is assumed that algorithms are implemented as multiple nodes and function as a pipeline processing system. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | -| ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object Recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | -| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | -| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | -| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | -| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | -| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | -| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | -| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | -| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Performance requirement varies depending on vehicle type. | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | ----------------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object Recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | +| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Performance requirement varies depending on vehicle type. | ### Detailed temporal performance metrics for Perception -The diagram below introduces the model for temporal performance metrics for Perception. +The diagram below introduces the model for temporal performance metrics for Perception. ![Model for Perception temporal performance metrics](./images/important-temporal-performance-metrics/model-for-perception-metrics.svg) @@ -73,29 +73,29 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | -| ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | -| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | -| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | -| APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | -| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | -| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | -| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | -| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | -| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | -| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------- | --- | ---------------------------------------------------------------- | --- | +| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | +| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | +| APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | +| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | +| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | +| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | +| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | +| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | +| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning @@ -104,13 +104,13 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to ![Model for Obstacle segmentation temporal performance metrics](./images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg) !!! Note - Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. +Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. The following list shows the temporal performance metrics around Obstacle segmentation and Planning. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | -| ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | -| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | -| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-004 | Response time from CPS #0 to CPS #13
or
CPS #1 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ---- | +| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-004 | Response time from CPS #0 to CPS #13
or
CPS #1 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From 33758cb89b6ecdf5b7194ddb07e3bd7b67c6c76c Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:16:39 +0900 Subject: [PATCH 03/51] fix: removed unnecessary sub component names. Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 96e857b4967..19a8bbffaca 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -46,11 +46,11 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | | ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object Recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | -| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | +| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | | AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | | AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | | AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | @@ -60,6 +60,9 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | | AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Performance requirement varies depending on vehicle type. | +!!! Note + There is an assumption that LiDAR outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. This is represented by CPA #1 in the model. + ### Detailed temporal performance metrics for Perception The diagram below introduces the model for temporal performance metrics for Perception. From 85fe3d457792e0df24b2149c9870b6be0098dbac Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 12:20:17 +0000 Subject: [PATCH 04/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index fcdc026cbbe..d4093ea8c5c 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -44,24 +44,24 @@ The following three policies assist in selecting the top-level performance metri Additionally, it is assumed that algorithms are implemented as multiple nodes and function as a pipeline processing system. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric |Note | -| ----------- | ------------------ | ------------ | --------------- | -------- |------ | ------------- | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | -| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | -| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | -| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | -| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | -| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | -| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | -| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | -| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Performance requirement varies depending on vehicle type. | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | +| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Performance requirement varies depending on vehicle type. | !!! Note - There is an assumption that LiDAR outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. This is represented by CPA #1 in the model. +There is an assumption that LiDAR outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. This is represented by CPA #1 in the model. ### Detailed temporal performance metrics for Perception From d048e5442ba7ea4d11bdcb652f401354f832fffc Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:34:15 +0900 Subject: [PATCH 05/51] fixed section for top-level temporal performance metrics Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index d4093ea8c5c..1f073413b83 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -10,7 +10,7 @@ It is expected that most algorithms employed for Autoware are executed with high As mentioned earlier, this page presents lists of temporal performance metrics that are crucial for the main functionalities of Autoware: Localization, Perception, Planning, and Control. -!!! Note +!!! note Other functionalities, such as system components for diagnosis, are excluded currently. However they will be taken into account in the near future. @@ -47,7 +47,7 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20
or
from CPA #1 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | | AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | | AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | | AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | @@ -58,10 +58,11 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | | AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | | AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Performance requirement varies depending on vehicle type. | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Temporal performance requirement varies depending on vehicle type. | -!!! Note -There is an assumption that LiDAR outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. This is represented by CPA #1 in the model. +!!! note + + There is an assumption that LiDAR outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. This is represented by CPA #1 in the model. ### Detailed temporal performance metrics for Perception @@ -106,8 +107,8 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to ![Model for Obstacle segmentation temporal performance metrics](./images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg) -!!! Note -Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. +!!! ote + Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. The following list shows the temporal performance metrics around Obstacle segmentation and Planning. From 497f2455f68746a820674d6d8796e4ec89cd639e Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 12:34:52 +0000 Subject: [PATCH 06/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 32 +++++++++---------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 1f073413b83..a7a3bc64e65 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -44,21 +44,21 @@ The following three policies assist in selecting the top-level performance metri Additionally, it is assumed that algorithms are implemented as multiple nodes and function as a pipeline processing system. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | -------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | -| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | -| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | -| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | -| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | -| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | -| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | -| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | -| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Temporal performance requirement varies depending on vehicle type. | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | +| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Temporal performance requirement varies depending on vehicle type. | !!! note @@ -108,7 +108,7 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to ![Model for Obstacle segmentation temporal performance metrics](./images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg) !!! ote - Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. +Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. The following list shows the temporal performance metrics around Obstacle segmentation and Planning. From f0e40065997cfc9c993a9f01a789dc18671f2eab Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:36:01 +0900 Subject: [PATCH 07/51] fixed note Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index a7a3bc64e65..ca966aaa910 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -107,8 +107,9 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to ![Model for Obstacle segmentation temporal performance metrics](./images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg) -!!! ote -Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. +!!! note + + Both the Obstacle grid map and Obstacle segmentation transmit data to multiple sub-components of Planning. However, not all of these sub-components are described in the model. This is because our primary focus is on the paths from LiDAR to Planning via Obstacle segmentation. The following list shows the temporal performance metrics around Obstacle segmentation and Planning. From 78a88cb51ab34649d243fcc3a891818b578566c0 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:39:51 +0900 Subject: [PATCH 08/51] fixed table for Perception Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index ca966aaa910..183da405534 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -78,7 +78,7 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------- | --- | ---------------------------------------------------------------- | --- | +| -------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | From 335d3ab5e583c2bb518a73166905afe57c108dcf Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 12:40:23 +0000 Subject: [PATCH 09/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 183da405534..ca966aaa910 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -78,7 +78,7 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------- | --- | ---------------------------------------------------------------- | --- | | APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | From ebe16b76c2c3495d5fc76757702d6fc0018ba2b1 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:45:55 +0900 Subject: [PATCH 10/51] fix table of Perception Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index ca966aaa910..8d262e68850 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -78,7 +78,7 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------- | --- | ---------------------------------------------------------------- | --- | +| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | @@ -92,14 +92,14 @@ The following list shows the temporal performance metrics for Perception. | APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | | APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | | APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other.| | | APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | | APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | | APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | -| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning From b8e20626633055160282215ac25fc0d5af86d644 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 12:46:31 +0000 Subject: [PATCH 11/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 8d262e68850..b53dfc0eaca 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -92,14 +92,14 @@ The following list shows the temporal performance metrics for Perception. | APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | | APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | | APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other.| | +| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | | APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | | APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | -| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning From 62d286795eefb3eb1e569e215b9b81c0361075be Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:48:18 +0900 Subject: [PATCH 12/51] fixed table of Perception Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index b53dfc0eaca..d7dd4f2fdd0 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -78,7 +78,7 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| ---| --- | ---- | ---- | ------- | ------- | ------ | | APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | From 8df42707c7ed8485ba34ce613b0805fdbf7e285d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 12:50:43 +0000 Subject: [PATCH 13/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index d7dd4f2fdd0..b53dfc0eaca 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -78,7 +78,7 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| ---| --- | ---- | ---- | ------- | ------- | ------ | +| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | From fa9fd406911817b5fbab118f985c1a35a4c4242c Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:53:24 +0900 Subject: [PATCH 14/51] fixed table of Perception Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index d7dd4f2fdd0..ae294588d76 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -79,12 +79,12 @@ The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | ---| --- | ---- | ---- | ------- | ------- | ------ | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | N/A It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | | APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | -| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | +| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | | APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | | APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | | APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | @@ -94,7 +94,7 @@ The following list shows the temporal performance metrics for Perception. | APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | | APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | | APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | From 82c3c9cb0cec3d5e6d6c30a58312819802357d3e Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 12:55:21 +0000 Subject: [PATCH 15/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 50 ++++++++++--------- 1 file changed, 26 insertions(+), 24 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index a7793075492..dd840d0ef44 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -77,34 +77,36 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | <<<<<<< HEAD | ---| --- | ---- | ---- | ------- | ------- | ------ | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | N/A It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | N/A It is essential to update traffic light Recognition frequently and seamlessly. | | ======= + | -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | ->>>>>>> takam5f2/add-performance-metrics-page -| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | -| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | -| APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | -| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | -| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | -| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | -| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | -| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | -| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | + +> > > > > > > takam5f2/add-performance-metrics-page +> > > > > > > | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | +> > > > > > > | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | +> > > > > > > | APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | +> > > > > > > | APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | +> > > > > > > | APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +> > > > > > > | APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | +> > > > > > > | APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +> > > > > > > | APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +> > > > > > > | APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | +> > > > > > > | APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +> > > > > > > | APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +> > > > > > > | APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +> > > > > > > | APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +> > > > > > > | APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +> > > > > > > | APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +> > > > > > > | APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +> > > > > > > | APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | +> > > > > > > | APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | +> > > > > > > | APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | +> > > > > > > | APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning From 3b389dfd34e9a316bb6d68262d81908f5fcc40fd Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 21:55:57 +0900 Subject: [PATCH 16/51] fixed not saved Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index a7793075492..ae294588d76 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -78,13 +78,8 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -<<<<<<< HEAD | ---| --- | ---- | ---- | ------- | ------- | ------ | | APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | N/A It is essential to update traffic light Recognition frequently and seamlessly. | | -======= -| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | It is essential to update traffic light Recognition frequently and seamlessly. | | ->>>>>>> takam5f2/add-performance-metrics-page | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | | APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | From ddd7223816d6bed3d39810e76f43ceb9ea710cfd Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 13:02:14 +0000 Subject: [PATCH 17/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 46 +++++++++---------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 4d78974841d..07b54fab037 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -77,29 +77,29 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| ---| --- | ---- | ---- | ------- | ------- | ------ | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | -| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | -| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | -| APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | -| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | -| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | -| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | -| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | -| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | -| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | +| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | +| APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | +| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | +| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | +| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | +| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | +| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | +| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning From 94080708b1b61524b5a2dcd899cbb8b8cdb5d8ec Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 22:04:49 +0900 Subject: [PATCH 18/51] fixed table of obstacle segmentation Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 07b54fab037..6eb032cbfc2 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -116,6 +116,6 @@ The following list shows the temporal performance metrics around Obstacle segmen | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ---- | | OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | | OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | | OSEG-004 | Response time from CPS #0 to CPS #13
or
CPS #1 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From 084c992eb7c51e94250588f11aac7e94388ab43d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 13:05:45 +0000 Subject: [PATCH 19/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 6eb032cbfc2..4b1fc036b8d 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -113,9 +113,9 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to The following list shows the temporal performance metrics around Obstacle segmentation and Planning. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ---- | -| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | -| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-004 | Response time from CPS #0 to CPS #13
or
CPS #1 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | +| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-004 | Response time from CPS #0 to CPS #13
or
CPS #1 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From 64bcee055e43a2b98bc5e3cc8aedac18c63472d1 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 22:11:13 +0900 Subject: [PATCH 20/51] removed or Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 4b1fc036b8d..b302e0adaef 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -62,7 +62,7 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an !!! note - There is an assumption that LiDAR outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. This is represented by CPA #1 in the model. + There is an assumption that each of LiDARs outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. That is represented by CPA #1 in the model. The detailed metrics employs the idea as well. ### Detailed temporal performance metrics for Perception @@ -82,9 +82,9 @@ The following list shows the temporal performance metrics for Perception. | APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | -| APER-004 | Response time from CPP #0 to CPP #30
or
from CPP #1 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | +| APER-004 | Response time from CPP #0 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | | APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | Response time from CPP #0 to CPP #21
or
CPP #1 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +| APER-006 | Response time from CPP #0 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | | APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | | APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | | APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | @@ -116,6 +116,6 @@ The following list shows the temporal performance metrics around Obstacle segmen | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | | OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-002 | Response time from CPS #0 to CPS #9
or
CPS #1 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-002 | Response time from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | | OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-004 | Response time from CPS #0 to CPS #13
or
CPS #1 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-004 | Response time from CPS #0 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From 08b2b5dabba9a8f0ba4fd1bfc68e30d08f4d985e Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 13:11:53 +0000 Subject: [PATCH 21/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 54 +++++++++---------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index b302e0adaef..9805e1551f9 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -77,29 +77,29 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | -| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | -| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | -| APER-004 | Response time from CPP #0 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | -| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | Response time from CPP #0 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | -| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | -| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | -| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | -| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | -| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | +| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | +| APER-004 | Response time from CPP #0 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | +| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | +| APER-006 | Response time from CPP #0 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | +| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | +| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | +| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | +| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | +| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning @@ -113,9 +113,9 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to The following list shows the temporal performance metrics around Obstacle segmentation and Planning. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | -| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ------------------------------------ | ---------------------------------------------------------------------------------------- | -------------------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | +| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | | OSEG-002 | Response time from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | -| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | | OSEG-004 | Response time from CPS #0 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From e4d3892ab39ce6a7ae183eb01ea4b1563ecf07a7 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 22:18:18 +0900 Subject: [PATCH 22/51] replaced LiDAR by sensor Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 9805e1551f9..10949b1a029 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -62,7 +62,7 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an !!! note - There is an assumption that each of LiDARs outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the LiDARs are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. That is represented by CPA #1 in the model. The detailed metrics employs the idea as well. + There is an assumption that each of sensors, such as LiDARs and cameras, outputs a set of pointcloud with a timestamp. CPA #0 is observed with the timestamp. If the sensors are not configured to output the timestamp, the time when Autoware receives the pointcloud is used instead. That is represented by CPA #1 in the model. The detailed metrics employs the idea as well. ### Detailed temporal performance metrics for Perception From 12a366191ffb1e848b8ba0c36bd040411c61d430 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 5 Jun 2023 23:50:12 +0900 Subject: [PATCH 23/51] add some minor fix Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 52 +++++++++---------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 10949b1a029..6f63ab6d91d 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -73,33 +73,33 @@ The diagram below introduces the model for temporal performance metrics for Perc The following two policies assist in selecting the performance metrics: - Regarding the frequency and response time at which Recognition results from Object Recognition and Traffic Light Recognition are consumed in Planning -- Splitting the merging point of data from multiple processing paths and considering the frequency and response time around that point +- Splitting Perception component on merging points of data from multiple processing paths and considering the frequency and response time around that point The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| APER-001 | Message rate from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Perception/Traffic light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | -| APER-002 | Response time from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Perception/Traffic light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | -| APER-003 | Message rate from CPP #26 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Perception/Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | -| APER-004 | Response time from CPP #0 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Perception/Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | -| APER-005 | Response time from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Perception/Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | Response time from CPP #0 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception/Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | -| APER-007 | Message rate from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Perception/Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | -| APER-008 | Message rate from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Perception/Object Recognition | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-009 | Message rate from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | | -| APER-010 | Message rate from CPP #18 to CPP #19 | Update rate of result from Validation | Perception/Object Recognition. | 10 Hz | Association Merger expects three inputs to synchronize with each others. | -| APER-011 | Response time from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-012 | Response time from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-013 | Response time from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | Message rate from CPP #10 to CPP #13 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-015 | Response time from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-016 | Message rate from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Perception/Object Recognition | 10 Hz | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-017 | Response time from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Perception/Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-018 | Message rate from CPP #10 to CPP #17 | Update rate of result from Clustering. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | Response time from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | -| APER-020 | Message rate from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Perception/Object Recognition | 10 Hz | Validator expects two inputs to synchronize with each other. | -| APER-021 | Response time from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Perception/Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | +| APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | +| APER-004 | **Response time** from CPP #6 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | +| APER-005 | **Response time** from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | +| APER-006 | **Response time** from CPP #6 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | +| APER-007 | **Message rate** from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | +| APER-008 | **Message rate** from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | +| APER-009 | **Message rate** from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Object Recognition. | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | +| APER-010 | **Message rate** from CPP #18 to CPP #19 | Update rate of result from Validation | Object Recognition. | 10 Hz | Association Merger wants each input to be updated at expected frequency. | +| APER-011 | **Response time** from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-012 | **Response time** from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-013 | **Response time** from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | +| APER-014 | **Message rate** from CPP #10 to CPP #13 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | +| APER-015 | **Message rate** from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | +| APER-016 | **Response time** from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-017 | **Response time** from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | +| APER-018 | **Message rate** from CPP #10 to CPP #17 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | **Message rate** from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | +| APER-020 | **Response time** from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | +| APER-021 | **Response time** from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning @@ -115,7 +115,7 @@ The following list shows the temporal performance metrics around Obstacle segmen | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ------------------------------------ | ---------------------------------------------------------------------------------------- | -------------------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | -| OSEG-001 | Message rate from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-002 | Response time from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | -| OSEG-003 | Message rate from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Perception/Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-004 | Response time from CPS #0 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Perception/Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Occupancy rid map must be updated frequently and smoothly. | | +| OSEG-002 | **Response time** from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map. | | +| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | +| OSEG-004 | **Response time** from CPS #0 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From 783c78c8923451dd329a790634ff666b51cdbeb4 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 5 Jun 2023 14:52:06 +0000 Subject: [PATCH 24/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 6f63ab6d91d..7ee27778cda 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -77,8 +77,8 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | ------------------------------------ | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------- | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | | APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | @@ -86,18 +86,18 @@ The following list shows the temporal performance metrics for Perception. | APER-005 | **Response time** from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | | APER-006 | **Response time** from CPP #6 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | | APER-007 | **Message rate** from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | -| APER-008 | **Message rate** from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | -| APER-009 | **Message rate** from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Object Recognition. | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | +| APER-008 | **Message rate** from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | +| APER-009 | **Message rate** from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Object Recognition. | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | | APER-010 | **Message rate** from CPP #18 to CPP #19 | Update rate of result from Validation | Object Recognition. | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | APER-011 | **Response time** from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | | APER-012 | **Response time** from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | | APER-013 | **Response time** from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | **Message rate** from CPP #10 to CPP #13 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | -| APER-015 | **Message rate** from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | +| APER-014 | **Message rate** from CPP #10 to CPP #13 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | +| APER-015 | **Message rate** from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | | APER-016 | **Response time** from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | | APER-017 | **Response time** from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-018 | **Message rate** from CPP #10 to CPP #17 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | **Message rate** from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | +| APER-018 | **Message rate** from CPP #10 to CPP #17 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | **Message rate** from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | | APER-020 | **Response time** from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | | APER-021 | **Response time** from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | @@ -113,9 +113,9 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to The following list shows the temporal performance metrics around Obstacle segmentation and Planning. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ------------------------------------ | ---------------------------------------------------------------------------------------- | -------------------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | -| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Occupancy rid map must be updated frequently and smoothly. | | -| OSEG-002 | **Response time** from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | +| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Occupancy rid map must be updated frequently and smoothly. | | +| OSEG-002 | **Response time** from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map. | | | OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | | OSEG-004 | **Response time** from CPS #0 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From 89433a6891f638355d26e8ef576f0c8013ac059f Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Tue, 6 Jun 2023 15:11:24 +0900 Subject: [PATCH 25/51] add high-level component name Signed-off-by: Takayuki AKAMINE --- .../model-for-obstacle-segmentation-metrics.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg index 9d6913ccd06..4b294e262c6 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg @@ -1,4 +1,4 @@ -
LiDARs
LiDARs
CPS #0
CPS #0
Sensing (preprocessing and concatenate)
Sensing (preproces...
CPS #1
CPS #1
CPS #2
CPS #2
Occupancy grid map
Occupancy grid map
Obstacle segmentation
Obstacle segmentation
Planning/behavior path planner
Planning/behavior pat...
subscription
callback
subscription...
timer callback
timer callback
Planning/behavior velocity planner
Planning/behavior vel...
subscription
callback
subscription...
subscription
callback
subscription...
CPS #3
CPS #3
CPS #4
CPS #4
CPS #5
CPS #5
CPS #6
CPS #6
CPS #7
CPS #7
CPS #8
CPS #8
CPS #9
CPS #9
CPS #10
CPS #10
CPS #11
CPS #11
CPS #12
CPS #12
CPS #13
CPS #13
CPS #14
CPS #14
CPS #: Checkpoint for Segmentation and planning components
CPS #: Checkpoint for Segmentation and planni...
Legend
Legend
Key Component
Key Component
Callback
Callback
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...Text is not SVG - cannot display \ No newline at end of file +
LiDARs
LiDARs
CPS #0
CPS #0
Sensing (preprocessing and concatenate)
Sensing (preproces...
CPS #1
CPS #1
CPS #2
CPS #2
Perception/Occupancy grid map
Perception/Occupancy...
Perception/Obstacle segmentation
Perception/Obstacle seg...
Planning/Behavior path planner
Planning/Behavior pat...
subscription
callback
subscription...
timer callback
timer callback
Planning/Behavior velocity planner
Planning/Behavior vel...
subscription
callback
subscription...
subscription
callback
subscription...
CPS #3
CPS #3
CPS #4
CPS #4
CPS #5
CPS #5
CPS #6
CPS #6
CPS #7
CPS #7
CPS #8
CPS #8
CPS #9
CPS #9
CPS #10
CPS #10
CPS #11
CPS #11
CPS #12
CPS #12
CPS #13
CPS #13
CPS #14
CPS #14
CPS #: Checkpoint for Segmentation and planning components
CPS #: Checkpoint for Segmentation and planni...
Legend
Legend
Key Component
Key Component
Callback
Callback
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...Text is not SVG - cannot display \ No newline at end of file From 4bfcfa8dba99bcb3846d985c121998f121b227af Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:36:33 +0900 Subject: [PATCH 26/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 7ee27778cda..3d7259f89b2 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -48,7 +48,7 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | -------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | | AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | | AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | | AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | | AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | From d86811f93d2c742621199fb540ede0be20d048c2 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:36:56 +0900 Subject: [PATCH 27/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 3d7259f89b2..aad619ff3c0 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -47,7 +47,7 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception if delay compensation is disabled in Tracking. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If delay compensation is disabled in Tracking. | | AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | | AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | | AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | From 3fbb402fc26706d9c2ed869cd7b1c35f34e65ea6 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:37:10 +0900 Subject: [PATCH 28/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index aad619ff3c0..6579f8d534e 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -49,7 +49,7 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | | AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If delay compensation is disabled in Tracking. | | AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | -| AWOV-004 | **Response time** from CPA #0 to CPA #6
or
CPA #1 to CPA #6 | Duration to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If sensor provides timestamp, CPA #0 is starting point. Otherwise, #1 is used instead. | +| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | | AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | | AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | | AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | From d2077b664a3fa379a38bdc024da1cf07c0f8821b Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:37:19 +0900 Subject: [PATCH 29/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 6579f8d534e..1d53b818864 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -58,7 +58,7 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | | AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | | AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency | Temporal performance requirement varies depending on vehicle type. | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | !!! note From e64eb1c8918cd618ec08523785ea756a85aa14a0 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:37:27 +0900 Subject: [PATCH 30/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 1d53b818864..8839bffabfc 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -79,7 +79,7 @@ The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------- | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | It is essential to update traffic light Recognition frequently and seamlessly. | | +| APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | It is essential to update Traffic Light Recognition frequently and seamlessly. | | | APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | | APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | | APER-004 | **Response time** from CPP #6 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | From 684c6e134d4b3f91c37f6a6a1ead751bce3b0ddc Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 6 Jun 2023 09:37:48 +0000 Subject: [PATCH 31/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 8839bffabfc..a6ba064dcad 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -44,21 +44,21 @@ The following three policies assist in selecting the top-level performance metri Additionally, it is assumed that algorithms are implemented as multiple nodes and function as a pipeline processing system. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If delay compensation is disabled in Tracking. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | -| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | -| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | -| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | -| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | -| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | -| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | -| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | -| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If delay compensation is disabled in Tracking. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | +| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | !!! note From e561d8ce2cb02ccdfe209525e1ac2857bafc01b7 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:48:19 +0900 Subject: [PATCH 32/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index a6ba064dcad..9fefd7a61d6 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -81,7 +81,7 @@ The following list shows the temporal performance metrics for Perception. | -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------- | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | | APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | It is essential to update Traffic Light Recognition frequently and seamlessly. | | | APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | -| APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | +| APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object Recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | | APER-004 | **Response time** from CPP #6 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | | APER-005 | **Response time** from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | | APER-006 | **Response time** from CPP #6 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | From 5183baf2d86abbe07fd8eac59c9b1f4e86bdc766 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:48:28 +0900 Subject: [PATCH 33/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 9fefd7a61d6..a762bbdc01b 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -115,7 +115,7 @@ The following list shows the temporal performance metrics around Obstacle segmen | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | -| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Occupancy rid map must be updated frequently and smoothly. | | +| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Occupancy grid map must be updated frequently and smoothly. | | | OSEG-002 | **Response time** from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map. | | | OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | | OSEG-004 | **Response time** from CPS #0 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From fdacf4e2d3ab13975b6b1bff492c1f3c19be9d33 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE <38586589+takam5f2@users.noreply.github.com> Date: Tue, 6 Jun 2023 18:49:16 +0900 Subject: [PATCH 34/51] Update docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md Co-authored-by: takeshi-iwanari --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index a762bbdc01b..7e9ee28ea8d 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -118,4 +118,4 @@ The following list shows the temporal performance metrics around Obstacle segmen | OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Occupancy grid map must be updated frequently and smoothly. | | | OSEG-002 | **Response time** from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map. | | | OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-004 | **Response time** from CPS #0 to CPS #13 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-004 | **Response time** from CPS #0 to CPS #11 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | From 1aff59d02547ca26db0103e1edc42efd5842fbcb Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Wed, 7 Jun 2023 10:27:37 +0900 Subject: [PATCH 35/51] fixed reason expression. Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 76 +++++++++---------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 7e9ee28ea8d..182bfc7da48 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -46,19 +46,19 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Perceived data from Prediction must be updated frequently and smoothly. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Processing time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception. | If delay compensation is disabled in Tracking. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Processing time in main body of Perception. | Perception | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | -| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | If delay compensation is enabled in Tracking. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly. | | -| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer requires fresh and up-to-date observed data from sensors. | | -| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer expects observed data to be updated frequently. | | -| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from the EKF Localizer smoothly for interpolation purposes. | | -| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning requires Localization to update the estimated pose frequently. | | -| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | Trajectory message should be updated within a short time frame to achieve appropriate driving behavior. | | -| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | Trajectory message should be updated frequently to achieve optimal driving behavior. | | -| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Sampling frequency influences on control stability so that Control should run at expected frequency. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle expects Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Response time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is disabled in Tracking. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Response time from Tracking output of Tracking to its data consumption in Planning. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is enabled in Tracking. | +| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is used if delay compensation is enabled in Tracking. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from EKF Localizer smoothly for linear interpolation. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning relies on Localization to update the estimated pose frequently. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | The vehicle relies on Planning to update trajectory within a short time frame to achieve safe driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | The vehicle relies on Planning to update trajectory frequently to achieve safe driving behavior. | | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Control stability and comfort relies on sampling frequency of Control. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | !!! note @@ -79,27 +79,27 @@ The following list shows the temporal performance metrics for Perception. | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------- | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | It is essential to update Traffic Light Recognition frequently and seamlessly. | | -| APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition. | | -| APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object Recognition | 10 Hz | Object Recognition must be updated frequently and smoothly. | Same as AWOV-001 | -| APER-004 | **Response time** from CPP #6 to CPP #30 | Processing time in main body of Object Recognition if delay compensation is disabled in Tracking. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle Recognition. | Same as AWOV-002 | -| APER-005 | **Response time** from CPP #23 to CPP #30 | Processing time Object Recognition if delay compensation is enabled in Tracking | Object Recognition | N/A | Planning relies on Obstacle Recognition to provide real-time and up-to-date perceived data. | Same as AWOV-003 | -| APER-006 | **Response time** from CPP #6 to CPP #21 | Time to process pointcloud data in Sensing and Detection if delay compensation is enabled in Tracking. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | Same as AWOV-004 | -| APER-007 | **Message rate** from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Object Recognition | 10 Hz | Tracking expects Detection to update perceived data frequently and smoothly | Same as AWOV-005 | -| APER-008 | **Message rate** from CPP #14 to CPP #19 | Update rate of result from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | -| APER-009 | **Message rate** from CPP #16 to CPP #19 | Update rate of result from Detection by Tracker. | Object Recognition. | 10 Hz | Association Merger wants each input to be updated at expected frequency. | | -| APER-010 | **Message rate** from CPP #18 to CPP #19 | Update rate of result from Validation | Object Recognition. | 10 Hz | Association Merger wants each input to be updated at expected frequency. | -| APER-011 | **Response time** from CPP #6 to CPP #19 via CPP #14 | Response time for result from Sensor Fusion after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-012 | **Response time** from CPP #6 to CPP #19 via CPP #16 | Response time for result from Detection by Tracker after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-013 | **Response time** from CPP #6 to CPP #19 via CPP #18 | Response time for result from Validator after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger expects three inputs to synchronize with each others. | | -| APER-014 | **Message rate** from CPP #10 to CPP #13 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | -| APER-015 | **Message rate** from CPP #5 to CPP #13 | Update rate of result from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion wants two inputs to be updated at expected frequency. | | -| APER-016 | **Response time** from CPP #6 to CPP #13 | Response time for result from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-017 | **Response time** from CPP #3 to CPP #13 | Response time for result from Camera-based Object detection after Cameras output images. | Object Recognition | N/A | Sensor Fusion expects two inputs to synchronize with each other. | | -| APER-018 | **Message rate** from CPP #10 to CPP #17 | Update rate of result from Clustering. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | **Message rate** from CPP #12 to CPP #17 | Update rate of result from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator wants two inputs to be updated at expected frequency. | -| APER-020 | **Response time** from CPP #6 to CPP #17 via CPP #10 | Response time for result from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | It seems similar to APER-015, but the topic message is different. | -| APER-021 | **Response time** from CPP #6 to CPP #17 via CPP #12 | Response time for result from DNN-based Object Recognition after LiDARs output pointcloud. | Object Recognition | N/A | Validator expects two inputs to synchronize with each other. | | +| APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | | +| APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | | +| APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-001. | +| APER-004 | **Response time** from CPP #6 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-002 and used if delay compensation is disabled in Tracking. | +| APER-005 | **Response time** from CPP #23 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-003 and used if delay compensation is enabled in Tracking. | +| APER-006 | **Response time** from CPP #6 to CPP #21 | Duration to process pointcloud data in Sensing and Detection. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | The metrics is same as AWOV-004 and used if delay compensation is enabled in Tracking. | +| APER-007 | **Message rate** from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Object Recognition | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is same as AWOV-005 | +| APER-008 | **Message rate** from CPP #14 to CPP #19 | Update rate of data sent from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | | +| APER-009 | **Message rate** from CPP #16 to CPP #19 | Update rate of data sent from Detection by Tracker. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | | +| APER-010 | **Message rate** from CPP #18 to CPP #19 | Update rate of data sent from Validation | Object Recognition. | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | +| APER-011 | **Response time** from CPP #6 to CPP #19 via CPP #14 | Response time to consume data sent from Sensor Fusion after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | +| APER-012 | **Response time** from CPP #6 to CPP #19 via CPP #16 | Response time to consume data sent from Detection by Tracker after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | +| APER-013 | **Response time** from CPP #6 to CPP #19 via CPP #18 | Response time to consume data sent from Validator after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | +| APER-014 | **Message rate** from CPP #10 to CPP #13 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | | +| APER-015 | **Message rate** from CPP #5 to CPP #13 | Update rate of data sent from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | | +| APER-016 | **Response time** from CPP #6 to CPP #13 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | | +| APER-017 | **Response time** from CPP #3 to CPP #13 | Response time to consume data sent from Camera-based Object detection after Cameras output images. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | | +| APER-018 | **Message rate** from CPP #10 to CPP #17 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | **Message rate** from CPP #12 to CPP #17 | Update rate of data sent from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | +| APER-020 | **Response time** from CPP #6 to CPP #17 via CPP #10 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. | It seems similar to APER-015, but the topic message is different. | +| APER-021 | **Response time** from CPP #6 to CPP #17 via CPP #12 | Response time to consume data sent from DNN-based Object Recognition after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning @@ -115,7 +115,7 @@ The following list shows the temporal performance metrics around Obstacle segmen | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | -| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Occupancy grid map must be updated frequently and smoothly. | | -| OSEG-002 | **Response time** from CPS #0 to CPS #9 | Processing time to output occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map. | | -| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly. | | -| OSEG-004 | **Response time** from CPS #0 to CPS #11 | Processing time to output obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation. | | +| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of Occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Planning relies on occupancy grid map to be updated frequently and smoothly for creating accurate trajectory. | | +| OSEG-002 | **Response time** from CPS #0 to CPS #9 via CPS #7 | Response time to consume Occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map for creating accurate trajectory.. | | +| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly for creating accurate trajectory.. | | +| OSEG-004 | **Response time** from CPS #0 to CPS #13 via CPS #11 | Response time to consume Obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation for creating accurate trajectory.. | | From 2815580fbbdc01806f3e22bab367d6f0d8b2f525 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 7 Jun 2023 01:28:09 +0000 Subject: [PATCH 36/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 84 +++++++++---------- 1 file changed, 42 insertions(+), 42 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 182bfc7da48..5b97fd8941a 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -46,19 +46,19 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Response time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is disabled in Tracking. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Response time from Tracking output of Tracking to its data consumption in Planning. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is enabled in Tracking. | -| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is used if delay compensation is enabled in Tracking. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | | -| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | -| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | -| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from EKF Localizer smoothly for linear interpolation. | | -| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning relies on Localization to update the estimated pose frequently. | | -| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | The vehicle relies on Planning to update trajectory within a short time frame to achieve safe driving behavior. | | -| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | The vehicle relies on Planning to update trajectory frequently to achieve safe driving behavior. | | -| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Control stability and comfort relies on sampling frequency of Control. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Response time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is disabled in Tracking. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Response time from Tracking output of Tracking to its data consumption in Planning. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is enabled in Tracking. | +| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is used if delay compensation is enabled in Tracking. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from EKF Localizer smoothly for linear interpolation. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning relies on Localization to update the estimated pose frequently. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | The vehicle relies on Planning to update trajectory within a short time frame to achieve safe driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | The vehicle relies on Planning to update trajectory frequently to achieve safe driving behavior. | | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Control stability and comfort relies on sampling frequency of Control. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | !!! note @@ -77,29 +77,29 @@ The following two policies assist in selecting the performance metrics: The following list shows the temporal performance metrics for Perception. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------- | --------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | -| APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | | -| APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | | -| APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-001. | -| APER-004 | **Response time** from CPP #6 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-002 and used if delay compensation is disabled in Tracking. | -| APER-005 | **Response time** from CPP #23 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-003 and used if delay compensation is enabled in Tracking. | -| APER-006 | **Response time** from CPP #6 to CPP #21 | Duration to process pointcloud data in Sensing and Detection. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | The metrics is same as AWOV-004 and used if delay compensation is enabled in Tracking. | -| APER-007 | **Message rate** from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Object Recognition | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is same as AWOV-005 | -| APER-008 | **Message rate** from CPP #14 to CPP #19 | Update rate of data sent from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | | -| APER-009 | **Message rate** from CPP #16 to CPP #19 | Update rate of data sent from Detection by Tracker. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | | -| APER-010 | **Message rate** from CPP #18 to CPP #19 | Update rate of data sent from Validation | Object Recognition. | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | -| APER-011 | **Response time** from CPP #6 to CPP #19 via CPP #14 | Response time to consume data sent from Sensor Fusion after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | -| APER-012 | **Response time** from CPP #6 to CPP #19 via CPP #16 | Response time to consume data sent from Detection by Tracker after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | -| APER-013 | **Response time** from CPP #6 to CPP #19 via CPP #18 | Response time to consume data sent from Validator after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | -| APER-014 | **Message rate** from CPP #10 to CPP #13 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | | -| APER-015 | **Message rate** from CPP #5 to CPP #13 | Update rate of data sent from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | | -| APER-016 | **Response time** from CPP #6 to CPP #13 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | | -| APER-017 | **Response time** from CPP #3 to CPP #13 | Response time to consume data sent from Camera-based Object detection after Cameras output images. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | | -| APER-018 | **Message rate** from CPP #10 to CPP #17 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | It seems similar to APER-014, but the topic message is different. | -| APER-019 | **Message rate** from CPP #12 to CPP #17 | Update rate of data sent from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | -| APER-020 | **Response time** from CPP #6 to CPP #17 via CPP #10 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. | It seems similar to APER-015, but the topic message is different. | -| APER-021 | **Response time** from CPP #6 to CPP #17 via CPP #12 | Response time to consume data sent from DNN-based Object Recognition after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------- | --------------- | ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| APER-001 | **Message rate** from CPP #2 to CPP #26 | Update rate of Traffic Light Recognition. | Traffic Light Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | | +| APER-002 | **Response time** from CPP #0 to CPP #30 | Response time from camera input to consumption of the result in Planning. | Traffic Light Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Traffic Light Recognition for making precise decisions. | | +| APER-003 | **Message rate** from CPP #25 to CPP #28 | Update rate of result from Prediction (Object Recognition) to Planning. | Object Recognition | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-001. | +| APER-004 | **Response time** from CPP #6 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-002 and used if delay compensation is disabled in Tracking. | +| APER-005 | **Response time** from CPP #23 to CPP #30 | Response time from Tracking output of Tracking to its data consumption in Planning. | Object Recognition | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is same as AWOV-003 and used if delay compensation is enabled in Tracking. | +| APER-006 | **Response time** from CPP #6 to CPP #21 | Duration to process pointcloud data in Sensing and Detection. | Object Recognition | N/A | Tracking relies on Detection to provide real-time and up-to-date perceived data. | The metrics is same as AWOV-004 and used if delay compensation is enabled in Tracking. | +| APER-007 | **Message rate** from CPP #20 to CPP #21 | Update rate of Detection result received by Tracking. | Object Recognition | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is same as AWOV-005 | +| APER-008 | **Message rate** from CPP #14 to CPP #19 | Update rate of data sent from Sensor Fusion. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | | +| APER-009 | **Message rate** from CPP #16 to CPP #19 | Update rate of data sent from Detection by Tracker. | Object Recognition | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | | +| APER-010 | **Message rate** from CPP #18 to CPP #19 | Update rate of data sent from Validation | Object Recognition. | 10 Hz | Association Merger relies on the data to be updated at expected frequency for data synchronization. | +| APER-011 | **Response time** from CPP #6 to CPP #19 via CPP #14 | Response time to consume data sent from Sensor Fusion after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | +| APER-012 | **Response time** from CPP #6 to CPP #19 via CPP #16 | Response time to consume data sent from Detection by Tracker after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | +| APER-013 | **Response time** from CPP #6 to CPP #19 via CPP #18 | Response time to consume data sent from Validator after LiDARs output pointcloud. | Object Recognition | N/A | Association Merger relies on fresh and up-to-date data for data synchronization. | | +| APER-014 | **Message rate** from CPP #10 to CPP #13 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | | +| APER-015 | **Message rate** from CPP #5 to CPP #13 | Update rate of data sent from Camera-based Object detection. | Object Recognition | 10 Hz | Sensor Fusion relies on the data to be updated at expected frequency for data synchronization. | | +| APER-016 | **Response time** from CPP #6 to CPP #13 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | | +| APER-017 | **Response time** from CPP #3 to CPP #13 | Response time to consume data sent from Camera-based Object detection after Cameras output images. | Object Recognition | N/A | Sensor Fusion relies on fresh and up-to-date data for data synchronization. | | +| APER-018 | **Message rate** from CPP #10 to CPP #17 | Update rate of data sent from Clustering. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | It seems similar to APER-014, but the topic message is different. | +| APER-019 | **Message rate** from CPP #12 to CPP #17 | Update rate of data sent from DNN-based Object Recognition. | Object Recognition | 10 Hz | Validator relies on the data to be updated at expected frequency for data synchronization. | +| APER-020 | **Response time** from CPP #6 to CPP #17 via CPP #10 | Response time to consume data sent from Clustering after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. | It seems similar to APER-015, but the topic message is different. | +| APER-021 | **Response time** from CPP #6 to CPP #17 via CPP #12 | Response time to consume data sent from DNN-based Object Recognition after LiDARs output pointcloud. | Object Recognition | N/A | Validator relies on fresh and update-date data for data synchronization. | | ### Detailed temporal performance metrics for Paths between Obstacle segmentation and Planning @@ -113,9 +113,9 @@ Obstacle segmentation, which is a crucial part of Perception, transmits data to The following list shows the temporal performance metrics around Obstacle segmentation and Planning. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ---------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | --------------- | ---------------------------------------------------------------------------------- | ---- | -| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of Occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Planning relies on occupancy grid map to be updated frequently and smoothly for creating accurate trajectory. | | -| OSEG-002 | **Response time** from CPS #0 to CPS #9 via CPS #7 | Response time to consume Occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map for creating accurate trajectory.. | | -| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly for creating accurate trajectory.. | | -| OSEG-004 | **Response time** from CPS #0 to CPS #13 via CPS #11 | Response time to consume Obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation for creating accurate trajectory.. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------- | ---- | +| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of Occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Planning relies on occupancy grid map to be updated frequently and smoothly for creating accurate trajectory. | | +| OSEG-002 | **Response time** from CPS #0 to CPS #9 via CPS #7 | Response time to consume Occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map for creating accurate trajectory.. | | +| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly for creating accurate trajectory.. | | +| OSEG-004 | **Response time** from CPS #0 to CPS #13 via CPS #11 | Response time to consume Obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation for creating accurate trajectory.. | | From 5a235b7be3588f769a08b3bb92ec6b5216932cd8 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Wed, 7 Jun 2023 11:57:07 +0900 Subject: [PATCH 37/51] fixed model for perception Signed-off-by: Takayuki AKAMINE --- .../model-for-perception-metrics.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg index feb695c7c48..9c4f2ba732f 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg @@ -1,4 +1,4 @@ -
LiDARs
LiDARs
CPP #6
CPP #6
Sensing (preprocessing and concatenate)
Sensing (preprocessing...
Clustering
(including
- occupancy grid map
- obstacle segmentation)
Clustering...
DNN-based
 Object Recognition
(Centerpoint)
DNN-based...
Camera-based
Object Recognition
Camera-based...
OR Cameras
OR Cameras
Sensor Fusion
Sensor Fusion
Detection by Tracker
Detection by Tracker
Association
Merger
Association...
Tracking
Tracking
Prediction
Prediction
Planning
(path_planner)
Planning...
TLR Cameras
TLR Cameras
Camera-based
Traffic Light Recognition
Camera-based...
CPP #P #0
CPP #P #0
CPP #1
CPP #1
CPP #2
CPP #2
CPP #3
CPP #3
CPP #4
CPP #4
CPP #5
CPP #5
CPP #7
CPP #7
CPP #8
CPP #8
CPP #9
CPP #9
CPP #10
CPP #10
CPP #11
CPP #11
CPP #12
CPP #12
CPP #13
CPP #13
CPP #14
CPP #14
CPP #15
CPP #15
CPP #16
CPP #16
CPP #19
CPP #19
CPP #20
CPP #20
CPP #24
CPP #24
CPP #25
CPP #25
CPP #26
CPP #26
timer callback
timer callback
CPP #22
CPP #22
subscription
callback
subscription...
CPP #21
CPP #21
CPP #23
CPP #23
subscription
callback
subscription...
subscription
callback
subscription...
timer callback
(10 Hz)
timer callback...
CPP #28
CPP #28
CPP #30
CPP #30
CPP #27
CPP #27
CPP #29
CPP #29
CPP #31
CPP #31
Validation
Validation
CPP #17
CPP #17
CPP #18
CPP #18
CPP #: Checkpoint for Perception components
CPP #: Checkpoint for Perception components
Legend
Legend
Key Component
Key Component
Callback
Callback
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...Text is not SVG - cannot display \ No newline at end of file +
LiDARs
LiDARs
CPP #6
CPP #6
Sensing (preprocessing and concatenate)
Sensing (preprocessing...
Clustering
(including
- occupancy grid map
- obstacle segmentation)
Clustering...
DNN-based
 Object Recognition
(Centerpoint)
DNN-based...
Camera-based
Object Recognition
Camera-based...
OR Cameras
OR Cameras
Sensor Fusion
Sensor Fusion
Detection by Tracker
Detection by Tracker
Association
Merger
Association...
Tracking
Tracking
Prediction
Prediction
Planning
(path_planner)
Planning...
TLR Cameras
TLR Cameras
Camera-based
Traffic Light Recognition
Camera-based...
CPP #P #0
CPP #P #0
CPP #1
CPP #1
CPP #2
CPP #2
CPP #3
CPP #3
CPP #4
CPP #4
CPP #5
CPP #5
CPP #7
CPP #7
CPP #8
CPP #8
CPP #9
CPP #9
CPP #10
CPP #10
CPP #11
CPP #11
CPP #12
CPP #12
CPP #13
CPP #13
CPP #14
CPP #14
CPP #15
CPP #15
CPP #16
CPP #16
CPP #19
CPP #19
CPP #20
CPP #20
CPP #24
CPP #24
CPP #25
CPP #25
CPP #26
CPP #26
timer callback
timer callback
CPP #22
CPP #22
subscription
callback
subscription...
CPP #21
CPP #21
CPP #23
CPP #23
subscription
callback
subscription...
subscription
callback
subscription...
timer callback
(10 Hz)
timer callback...
CPP #28
CPP #28
CPP #30
CPP #30
CPP #27
CPP #27
CPP #29
CPP #29
CPP #31
CPP #31
Validator
Validator
CPP #17
CPP #17
CPP #18
CPP #18
CPP #: Checkpoint for Perception components
CPP #: Checkpoint for Perception components
Legend
Legend
Key Component
Key Component
Callback
Callback
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...Text is not SVG - cannot display \ No newline at end of file From ddb6754c8031389b287e6dce896f0df7f2bc1fd3 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Wed, 7 Jun 2023 11:58:28 +0900 Subject: [PATCH 38/51] replace the by a Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 5b97fd8941a..90b0f0a5867 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -55,10 +55,10 @@ Additionally, it is assumed that algorithms are implemented as multiple nodes an | AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | | AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from EKF Localizer smoothly for linear interpolation. | | | AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning relies on Localization to update the estimated pose frequently. | | -| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | The vehicle relies on Planning to update trajectory within a short time frame to achieve safe driving behavior. | | -| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | The vehicle relies on Planning to update trajectory frequently to achieve safe driving behavior. | | +| AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | A vehicle relies on Planning to update trajectory within a short time frame to achieve safe driving behavior. | | +| AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | A vehicle relies on Planning to update trajectory frequently to achieve safe driving behavior. | | | AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Control stability and comfort relies on sampling frequency of Control. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | Vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | A vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | !!! note From c016d85290ce82e6f5785d0fedf60aca5d3dc87b Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 7 Jun 2023 02:59:22 +0000 Subject: [PATCH 39/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 90b0f0a5867..f790b956c4c 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -44,21 +44,21 @@ The following three policies assist in selecting the top-level performance metri Additionally, it is assumed that algorithms are implemented as multiple nodes and function as a pipeline processing system. -| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | -| -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | -| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | | -| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Response time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is disabled in Tracking. | -| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Response time from Tracking output of Tracking to its data consumption in Planning. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is enabled in Tracking. | -| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is used if delay compensation is enabled in Tracking. | -| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | | -| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | -| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | -| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from EKF Localizer smoothly for linear interpolation. | | -| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning relies on Localization to update the estimated pose frequently. | | +| ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | +| -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | --------------------- | --------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ | +| AWOV-001 | **Message rate** from CPA #9 to CPA #18 | Update rate of result from Prediction to Planning. | Perception | 10 Hz | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | | +| AWOV-002 | **Response time** from CPA #0 to CPA #20 via CPA #18 | Response time in main body of Perception. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is disabled in Tracking. | +| AWOV-003 | **Response time** from CPA #7 to CPA #20 | Response time from Tracking output of Tracking to its data consumption in Planning. | Perception | N/A | Planning relies on fresh and up-to-date perceived data from Perception for creating accurate trajectory. | The metric is used if delay compensation is enabled in Tracking. | +| AWOV-004 | **Response time** from CPA #0 to CPA #6 | Duration to process pointcloud data in Sensing and Detection. | Perception | N/A | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | The metric is used if delay compensation is enabled in Tracking. | +| AWOV-005 | **Message rate** from CPA #4 to CPA #5 | Update rate of Detection result received by Tracking. | Perception | 10 Hz | Tracking relies on detection to provide real-time and up-to-date sensed data for accurate tracking. | | +| AWOV-006 | **Response time** from CPA #0 to CPA #14 | Response time from output of observed data from LiDARs to its consumption in EKF Localizer via NDT Scan Matcher. | Localization | N/A | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | +| AWOV-007 | **Message rate** from CPA #11 to CPA #13 | Update rate of pose estimated by NDT Scan Matcher. | Localization | 10 Hz | EKF Localizer relies on fresh and up-to-date observed data from sensors for accurate estimation of self pose. | | +| AWOV-008 | **Message rate** from CPA #15 to CPA #12 | Update rate of feed backed pose estimated by EKF Localizer. | Localization | 50 Hz | NDT Scan Matcher relies on receiving estimated pose from EKF Localizer smoothly for linear interpolation. | | +| AWOV-009 | **Message rate** from CPA #17 to CPA #19 | Update rate of Localization result received by Planning. | Localization | 50 Hz | Planning relies on Localization to update the estimated pose frequently. | | | AWOV-010 | **Response time** from CPA #20 to CPA #23 | Processing time from beginning of Planning to consumption of Trajectory message in Control. | Planning | N/A | A vehicle relies on Planning to update trajectory within a short time frame to achieve safe driving behavior. | | | AWOV-011 | **Message rate** from CPA #21 to CPA #22 | Update rate of Trajectory message from Planning. | Planning | 10 Hz | A vehicle relies on Planning to update trajectory frequently to achieve safe driving behavior. | | -| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Control stability and comfort relies on sampling frequency of Control. | | -| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | A vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | +| AWOV-012 | **Message rate** from CPA #24 to CPA #25 | Update rate of Control command. | Control | 33 Hz | Control stability and comfort relies on sampling frequency of Control. | | +| AWOV-013 | **Message rate** between CPA #26 and Vehicle | Communication rate between Autoware and Vehicle. | Vehicle Interface | N/A | A vehicle requires Autoware to communicate with each other at predetermined frequency. | Temporal performance requirement varies depending on vehicle type. | !!! note From bb8d212db7d7362c81862330f7b1d5d3514ef81a Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Wed, 7 Jun 2023 12:01:27 +0900 Subject: [PATCH 40/51] fixed the sentence Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index 90b0f0a5867..20c858fb066 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -115,7 +115,7 @@ The following list shows the temporal performance metrics around Obstacle segmen | ID | Representation in the model | Metric meaning | Related functionality | Reference value | Reason to choose it as a metric | Note | | -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------- | ---- | -| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of Occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Planning relies on occupancy grid map to be updated frequently and smoothly for creating accurate trajectory. | | -| OSEG-002 | **Response time** from CPS #0 to CPS #9 via CPS #7 | Response time to consume Occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from occupancy grid map for creating accurate trajectory.. | | -| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Obstacle segmentation must be updated frequently and smoothly for creating accurate trajectory.. | | +| OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of Occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Planning relies on Occupancy grid map to be updated frequently and smoothly for creating accurate trajectory. | | +| OSEG-002 | **Response time** from CPS #0 to CPS #9 via CPS #7 | Response time to consume Occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Occupancy grid map for creating accurate trajectory.. | | +| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Planning relies on Obstacle segmentation to be updated frequently and smoothly for creating accurate trajectory. | | | OSEG-004 | **Response time** from CPS #0 to CPS #13 via CPS #11 | Response time to consume Obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation for creating accurate trajectory.. | | From 57ed04da69a735a689eb55ee2fd89baf09b9f372 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Wed, 7 Jun 2023 03:02:16 +0000 Subject: [PATCH 41/51] style(pre-commit): autofix --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index db39ff914c4..e182348e3bd 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -117,5 +117,5 @@ The following list shows the temporal performance metrics around Obstacle segmen | -------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------------- | --------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------- | ---- | | OSEG-001 | **Message rate** from CPS #4 to CPS #7 | Update rate of Occupancy grid map received by Planning (`behavior_path_planner`) | Obstacle segmentation | 10 Hz | Planning relies on Occupancy grid map to be updated frequently and smoothly for creating accurate trajectory. | | | OSEG-002 | **Response time** from CPS #0 to CPS #9 via CPS #7 | Response time to consume Occupancy grid map after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Occupancy grid map for creating accurate trajectory.. | | -| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Planning relies on Obstacle segmentation to be updated frequently and smoothly for creating accurate trajectory. | | +| OSEG-003 | **Message rate** from CPS #6 to CPS #11 | Update rate of obstacle segmentation received by Planning (`behavior_velocity_planner`). | Obstacle segmentation | 10 Hz | Planning relies on Obstacle segmentation to be updated frequently and smoothly for creating accurate trajectory. | | | OSEG-004 | **Response time** from CPS #0 to CPS #13 via CPS #11 | Response time to consume Obstacle segmentation after LiDARs output sensing data. | Obstacle segmentation | N/A | Planning relies on fresh and up-to-date perceived data from Obstacle segmentation for creating accurate trajectory.. | | From 35bc0dec7c4b2191aff44a40f943339eaa4cf1a8 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Wed, 7 Jun 2023 18:03:00 +0900 Subject: [PATCH 42/51] replace image by correct one Signed-off-by: Takayuki AKAMINE --- .../model-for-perception-metrics.svg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg index 9c4f2ba732f..68e52c545db 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg @@ -1,4 +1,4 @@ -
LiDARs
LiDARs
CPP #6
CPP #6
Sensing (preprocessing and concatenate)
Sensing (preprocessing...
Clustering
(including
- occupancy grid map
- obstacle segmentation)
Clustering...
DNN-based
 Object Recognition
(Centerpoint)
DNN-based...
Camera-based
Object Recognition
Camera-based...
OR Cameras
OR Cameras
Sensor Fusion
Sensor Fusion
Detection by Tracker
Detection by Tracker
Association
Merger
Association...
Tracking
Tracking
Prediction
Prediction
Planning
(path_planner)
Planning...
TLR Cameras
TLR Cameras
Camera-based
Traffic Light Recognition
Camera-based...
CPP #P #0
CPP #P #0
CPP #1
CPP #1
CPP #2
CPP #2
CPP #3
CPP #3
CPP #4
CPP #4
CPP #5
CPP #5
CPP #7
CPP #7
CPP #8
CPP #8
CPP #9
CPP #9
CPP #10
CPP #10
CPP #11
CPP #11
CPP #12
CPP #12
CPP #13
CPP #13
CPP #14
CPP #14
CPP #15
CPP #15
CPP #16
CPP #16
CPP #19
CPP #19
CPP #20
CPP #20
CPP #24
CPP #24
CPP #25
CPP #25
CPP #26
CPP #26
timer callback
timer callback
CPP #22
CPP #22
subscription
callback
subscription...
CPP #21
CPP #21
CPP #23
CPP #23
subscription
callback
subscription...
subscription
callback
subscription...
timer callback
(10 Hz)
timer callback...
CPP #28
CPP #28
CPP #30
CPP #30
CPP #27
CPP #27
CPP #29
CPP #29
CPP #31
CPP #31
Validator
Validator
CPP #17
CPP #17
CPP #18
CPP #18
CPP #: Checkpoint for Perception components
CPP #: Checkpoint for Perception components
Legend
Legend
Key Component
Key Component
Callback
Callback
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...Text is not SVG - cannot display \ No newline at end of file +
LiDARs
LiDARs
CPP #6
CPP #6
Sensing (preprocessing and concatenate)
Sensing (preprocessing...
Clustering
(including
- occupancy grid map
- obstacle segmentation)
Clustering...
DNN-based
 Object Recognition
(Centerpoint)
DNN-based...
Camera-based
Object Recognition
Camera-based...
OR Cameras
OR Cameras
Sensor Fusion
Sensor Fusion
Detection by Tracker
Detection by Tracker
Association
Merger
Association...
Tracking
Tracking
Prediction
Prediction
Planning/Behavior Path Planner
Planning/Behavior Pa...
TLR Cameras
TLR Cameras
Camera-based
Traffic Light Recognition
Camera-based...
CPP #P #0
CPP #P #0
CPP #1
CPP #1
CPP #2
CPP #2
CPP #3
CPP #3
CPP #4
CPP #4
CPP #5
CPP #5
CPP #7
CPP #7
CPP #8
CPP #8
CPP #9
CPP #9
CPP #10
CPP #10
CPP #11
CPP #11
CPP #12
CPP #12
CPP #13
CPP #13
CPP #14
CPP #14
CPP #15
CPP #15
CPP #16
CPP #16
CPP #19
CPP #19
CPP #20
CPP #20
CPP #24
CPP #24
CPP #25
CPP #25
CPP #26
CPP #26
timer callback
timer callback
CPP #22
CPP #22
subscription
callback
subscription...
CPP #21
CPP #21
CPP #23
CPP #23
subscription
callback
subscription...
subscription
callback
subscription...
timer callback
(10 Hz)
timer callback...
CPP #28
CPP #28
CPP #30
CPP #30
CPP #27
CPP #27
CPP #29
CPP #29
CPP #31
CPP #31
Validator
Validator
CPP #17
CPP #17
CPP #18
CPP #18
CPP #: Checkpoint for Perception components
CPP #: Checkpoint for Perception components
Legend
Legend
Key Component
Key Component
Callback
Callback
inter-component
communication
inter-componentcom...
intra-component
communication
intra-componentcom...
callbacks denote when input messages are consumed rather than received.
callbacks denote when inpu...Text is not SVG - cannot display \ No newline at end of file From 3a56e0195e4df1d231e3c70602528c8bb9578a77 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Wed, 7 Jun 2023 19:12:32 +0900 Subject: [PATCH 43/51] fixed missing link Signed-off-by: Takayuki AKAMINE --- .../important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md index e182348e3bd..56652a9bea3 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md @@ -28,7 +28,7 @@ This section is divided into multiple subsections, each containing a model diagr The first subsection presents the top-level temporal performance metrics, which are depicted in the abstract structure of Autoware as a whole. The detailed metrics are not included in the model as they would add complexity to it. Instead, the subsequent section introduces the detailed metrics. The detailed metrics are subject to more frequent updates compared to the top-level ones, which is another reason for categorizing them separately. -Each list includes a column for the reference value. The reference value represents the observed value of each metric when Autoware is running according to the tutorial. It is important to note that the reference value is not a required value, meaning that Autoware does not necessarily fail in the tutorial execution if certain metrics do not fulfill the reference value. +Each list includes a column for the reference value. The reference value represents the observed value of each metric when Autoware is running according to [the tutorial](../../../tutorials/). It is important to note that the reference value is not a required value, meaning that Autoware does not necessarily fail in [the tutorial](../../../tutorials/) execution if certain metrics do not fulfill the reference value. ### Top-level temporal performance metrics for Autoware From d5c7936981444dc06bff684f945767fe6fab88f3 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 12 Jun 2023 13:29:30 +0900 Subject: [PATCH 44/51] chore: mv the pages to how-to-guides/others Signed-off-by: Takayuki AKAMINE --- .../tuning-parameters-and-performance/.pages | 1 - .../model-for-obstacle-segmentation-metrics.svg | 0 .../model-for-perception-metrics.svg | 0 .../model-for-top-level-metrics.svg | 0 .../important-temporal-performance-metrics.md | 2 +- 5 files changed, 1 insertion(+), 2 deletions(-) rename docs/how-to-guides/{integrating-autoware/tuning-parameters-and-performance => others}/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg (100%) rename docs/how-to-guides/{integrating-autoware/tuning-parameters-and-performance => others}/images/important-temporal-performance-metrics/model-for-perception-metrics.svg (100%) rename docs/how-to-guides/{integrating-autoware/tuning-parameters-and-performance => others}/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg (100%) rename docs/how-to-guides/{integrating-autoware/tuning-parameters-and-performance => others}/important-temporal-performance-metrics.md (98%) diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages index d42af4c125d..9cb01955bdf 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages +++ b/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/.pages @@ -1,4 +1,3 @@ nav: - evaluating-controller-performance.md - evaluating-real-time-performance.md - - important-temporal-performance-metrics.md diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg b/docs/how-to-guides/others/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg similarity index 100% rename from docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg rename to docs/how-to-guides/others/images/important-temporal-performance-metrics/model-for-obstacle-segmentation-metrics.svg diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg b/docs/how-to-guides/others/images/important-temporal-performance-metrics/model-for-perception-metrics.svg similarity index 100% rename from docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-perception-metrics.svg rename to docs/how-to-guides/others/images/important-temporal-performance-metrics/model-for-perception-metrics.svg diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg b/docs/how-to-guides/others/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg similarity index 100% rename from docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg rename to docs/how-to-guides/others/images/important-temporal-performance-metrics/model-for-top-level-metrics.svg diff --git a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md b/docs/how-to-guides/others/important-temporal-performance-metrics.md similarity index 98% rename from docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md rename to docs/how-to-guides/others/important-temporal-performance-metrics.md index 56652a9bea3..5b1d1a5bbc9 100644 --- a/docs/how-to-guides/integrating-autoware/tuning-parameters-and-performance/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/others/important-temporal-performance-metrics.md @@ -28,7 +28,7 @@ This section is divided into multiple subsections, each containing a model diagr The first subsection presents the top-level temporal performance metrics, which are depicted in the abstract structure of Autoware as a whole. The detailed metrics are not included in the model as they would add complexity to it. Instead, the subsequent section introduces the detailed metrics. The detailed metrics are subject to more frequent updates compared to the top-level ones, which is another reason for categorizing them separately. -Each list includes a column for the reference value. The reference value represents the observed value of each metric when Autoware is running according to [the tutorial](../../../tutorials/). It is important to note that the reference value is not a required value, meaning that Autoware does not necessarily fail in [the tutorial](../../../tutorials/) execution if certain metrics do not fulfill the reference value. +Each list includes a column for the reference value. The reference value represents the observed value of each metric when Autoware is running according to [the tutorial](../../tutorials/). It is important to note that the reference value is not a required value, meaning that Autoware does not necessarily fail in [the tutorial](../../tutorials/) execution if certain metrics do not fulfill the reference value. ### Top-level temporal performance metrics for Autoware From 4b8e58d0d521d45349bfe131c1878d1a93773914 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Mon, 12 Jun 2023 13:59:33 +0900 Subject: [PATCH 45/51] chore: add page item to the list Signed-off-by: Takayuki AKAMINE --- docs/how-to-guides/index.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/how-to-guides/index.md b/docs/how-to-guides/index.md index 8d5aabb0274..e94ba6b67ef 100644 --- a/docs/how-to-guides/index.md +++ b/docs/how-to-guides/index.md @@ -14,6 +14,7 @@ - [Advanced usage of colcon](others/advanced-usage-of-colcon.md) - [Applying Clang-Tidy to ROS packages](others/applying-clang-tidy-to-ros-packages.md) - [Using Eagleye in Autoware](others/eagleye-integration-guide.md) +- [Important temporal performance metrics on components](others/important-temporal-performance-metrics.md) TODO: Write the following contents. From 30bb7b5ddaaffa5b67fb5f416a2597036a7c5008 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Tue, 20 Jun 2023 13:38:44 +0900 Subject: [PATCH 46/51] chore: add the reason why temporal performance is used. Signed-off-by: Takayuki AKAMINE --- .../others/important-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/others/important-temporal-performance-metrics.md b/docs/how-to-guides/others/important-temporal-performance-metrics.md index 5b1d1a5bbc9..d9b7fdbc17f 100644 --- a/docs/how-to-guides/others/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/others/important-temporal-performance-metrics.md @@ -4,7 +4,7 @@ ### Objective of the page -This page introduces important metrics to evaluate temporal performance on components of Autoware. +This page introduces important metrics to evaluate temporal performance on components of Autoware. The term "temporal performance" is often used in the page in order to distinguish between functional performance, which referred to as accuracy as well, and time-related performance. It is expected that most algorithms employed for Autoware are executed with high frequency and short response time as possible. In order to achieve safe autonomous driving, one of the desired outcomes is no time gap between perceived and actual situation. The time gap is commonly referred to as delay. If the delay is significant, the system may determine trajectory and maneuver based on outdated situation. Consequently, if the actual situation differs from the perceived one due to the delay, the system may make unexpected decisions. From eeac4f06c44d8fdef670f95b9bd9a2532b559f98 Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Tue, 20 Jun 2023 14:35:37 +0900 Subject: [PATCH 47/51] chore: add guide to performance evaluation tools Signed-off-by: Takayuki AKAMINE --- .../others/important-temporal-performance-metrics.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/how-to-guides/others/important-temporal-performance-metrics.md b/docs/how-to-guides/others/important-temporal-performance-metrics.md index d9b7fdbc17f..6b56ebb17dc 100644 --- a/docs/how-to-guides/others/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/others/important-temporal-performance-metrics.md @@ -4,7 +4,7 @@ ### Objective of the page -This page introduces important metrics to evaluate temporal performance on components of Autoware. The term "temporal performance" is often used in the page in order to distinguish between functional performance, which referred to as accuracy as well, and time-related performance. +This page introduces important metrics to evaluate temporal performance on components of Autoware. The term "temporal performance" is often used throughout the page in order to distinguish between functional performance, which referred to as accuracy as well, and time-related performance. It is expected that most algorithms employed for Autoware are executed with high frequency and short response time as possible. In order to achieve safe autonomous driving, one of the desired outcomes is no time gap between perceived and actual situation. The time gap is commonly referred to as delay. If the delay is significant, the system may determine trajectory and maneuver based on outdated situation. Consequently, if the actual situation differs from the perceived one due to the delay, the system may make unexpected decisions. @@ -22,6 +22,16 @@ In addition, these metrics are useful for designers and evaluators of middleware Temporal performance metrics are important for evaluating the reliability of Autoware. However, ensuring the reliability of Autoware requires consideration of not only temporal performance metrics but also other metrics. +### Tools for evaluating the metrics + +There are several tools available for evaluating Autoware according to the metrics listed in the page. For example, both [CARET](https://github.com/tier4/caret) and [ros2_tracing](https://github.com/ros2/ros2_tracing) are recommended options when evaluating Autoware on Linux and ROS 2. If you want to measure the metrics with either of these tools, refer to the corresponding user guide for instructions. It's important to note that if you import Autoware to a platform other than Linux and ROS 2, you will need to choose a supported tool for evaluation. + +!!! note + + TIER IV plans to measure Autoware and provide a performance evaluation report periodically. An example of such a report can be found [here](https://tier4.github.io/CARET_report/), although it may not include all of the metrics listed. + +The page does not aim to provide instructions on how to use these tools or measure the metrics. its primary focus is on the metrics themselves, as they are more important than the specific tools used. These metrics retain their relevance regardless of the employed platform. + ## Temporal performance metrics list This section is divided into multiple subsections, each containing a model diagram and an accompanying list that explains the important temporal performance metrics. Each model is equipped with checkpoints that serve as indicators for these metrics. From abd79111e75b199c99d8a1e2646af1f4b00bb0fc Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Thu, 22 Jun 2023 18:58:01 +0900 Subject: [PATCH 48/51] add explanation of definition policy Signed-off-by: Takayuki AKAMINE --- ... defining-temporal-performance-metrics.md} | 34 +++++++++++++------ 1 file changed, 24 insertions(+), 10 deletions(-) rename docs/how-to-guides/others/{important-temporal-performance-metrics.md => defining-temporal-performance-metrics.md} (87%) diff --git a/docs/how-to-guides/others/important-temporal-performance-metrics.md b/docs/how-to-guides/others/defining-temporal-performance-metrics.md similarity index 87% rename from docs/how-to-guides/others/important-temporal-performance-metrics.md rename to docs/how-to-guides/others/defining-temporal-performance-metrics.md index 6b56ebb17dc..f35eb8c6981 100644 --- a/docs/how-to-guides/others/important-temporal-performance-metrics.md +++ b/docs/how-to-guides/others/defining-temporal-performance-metrics.md @@ -1,14 +1,14 @@ -# Important temporal performance metrics on components +# Defining temporal performance metrics on components -## Motivation to list temporal performance metrics +## Motivation to defining temporal performance metrics ### Objective of the page -This page introduces important metrics to evaluate temporal performance on components of Autoware. The term "temporal performance" is often used throughout the page in order to distinguish between functional performance, which referred to as accuracy as well, and time-related performance. +This page introduces policies to define metrics to evaluate temporal performance on components of Autoware. The term "temporal performance" is often used throughout the page in order to distinguish between functional performance, which referred to as accuracy as well, and time-related performance. -It is expected that most algorithms employed for Autoware are executed with high frequency and short response time as possible. In order to achieve safe autonomous driving, one of the desired outcomes is no time gap between perceived and actual situation. The time gap is commonly referred to as delay. If the delay is significant, the system may determine trajectory and maneuver based on outdated situation. Consequently, if the actual situation differs from the perceived one due to the delay, the system may make unexpected decisions. +It is expected that most algorithms employed for Autoware are executed with as high frequency and short response time as possible. In order to achieve safe autonomous driving, one of the desired outcomes is no time gap between perceived and actual situation. The time gap is commonly referred to as delay. If the delay is significant, the system may determine trajectory and maneuver based on outdated situation. Consequently, if the actual situation differs from the perceived one due to the delay, the system may make unexpected decisions. -As mentioned earlier, this page presents lists of temporal performance metrics that are crucial for the main functionalities of Autoware: Localization, Perception, Planning, and Control. +As mentioned above, this page presents the policies to define metrics. Besides, the page contains lists of sample metrics that are crucial for the main functionalities of Autoware: Localization, Perception, Planning, and Control. !!! note @@ -18,7 +18,7 @@ As mentioned earlier, this page presents lists of temporal performance metrics t Temporal performance metrics are important for evaluating Autoware. These metrics are particularly useful for assessing delays caused by new algorithms and logic. They can be employed when comparing the temporal performance of software on a desktop computer with that on a vehicle during the vehicle integration phase. -In addition, these metrics are useful for designers and evaluators of middleware, operating systems, and computers. They are selected based on user and product requirements. One of these requirements is to provide sufficient temporal performance for executing Autoware. "Sufficient temporal performance" is defined as a temporal performance requirement, but it can be challenging to define the requirement because it varies depending on the product type, operational design domain (ODD), and other factors. Then, this page specifically focuses on temporal performance metrics rather than requirements. +In addition, these metrics are useful for designers and evaluators of middleware, operating systems, and computers. They are selected based on user and product requirements. One of these requirements is to provide sufficient temporal performance for executing Autoware. "Sufficient temporal performance" is defined as a temporal performance requirement, but it can be challenging to define the requirement because it varies depending on the product type, Operational Design Domain (ODD), and other factors. Then, this page specifically focuses on temporal performance metrics rather than requirements. Temporal performance metrics are important for evaluating the reliability of Autoware. However, ensuring the reliability of Autoware requires consideration of not only temporal performance metrics but also other metrics. @@ -28,13 +28,27 @@ There are several tools available for evaluating Autoware according to the metri !!! note - TIER IV plans to measure Autoware and provide a performance evaluation report periodically. An example of such a report can be found [here](https://tier4.github.io/CARET_report/), although it may not include all of the metrics listed. + TIER IV plans to measure Autoware, which is running according to [the tutorial](../../tutorials/), and provide a performance evaluation report periodically. An example of such a report can be found [here](https://tier4.github.io/CARET_report/), although it may not include all of the metrics listed. -The page does not aim to provide instructions on how to use these tools or measure the metrics. its primary focus is on the metrics themselves, as they are more important than the specific tools used. These metrics retain their relevance regardless of the employed platform. +The page does not aim to provide instructions on how to use these tools or measure the metrics. Its primary focus is on the metrics themselves, as they are more important than the specific tools used. These metrics retain their relevance regardless of the employed platform. -## Temporal performance metrics list +## Policies to define temporal performance metrics -This section is divided into multiple subsections, each containing a model diagram and an accompanying list that explains the important temporal performance metrics. Each model is equipped with checkpoints that serve as indicators for these metrics. +As mentioned above, the configuration of Autoware varies according to the product type, ODD, and other factors. The variety of configurations makes it difficult to define the uniform metrics for evaluating Autoware. +However, the policies to define them are basically reused even when the configuration changed. Each of temporal performance metrics is categorized into two types: execution frequency and response time. Although there are many types of metrics, such as communication latency, the only two types are taken into account for ease. +Execution frequency is observed using rate of message for Inter-Process Communication (IPC). You will find an enormous number of messages in Autoware, but you don't have to take care of all. Some messages might be crucial to functionality and they should be picked up for evaluation. +Response time is duration elapsed through a series of processing. A series of processing is mentioned as a path. Response time is calculated from timestamps of starting and ending of a path. Thought a lot of paths can be defined in Autoware, you have to choose significant paths. + +As a hint, here are several characteristics of message and path in order to pick up metrics. + +1. Messages and paths on boundaries where observed values from sensors are consumed +1. Messages and paths on boundaries of functions, e.g. a boundary of perception and planning +1. Messages and paths on boundaries where two different messages are synchronized and merged +1. Messages who must be transmitted at expected frequency for example vehicle command messages + +## List of sample metrics + +This section demonstrates how to metrics according to the policies explained. section is divided into multiple subsections, each containing a model diagram and an accompanying list that explains the important temporal performance metrics. Each model is equipped with checkpoints that serve as indicators for these metrics. The first subsection presents the top-level temporal performance metrics, which are depicted in the abstract structure of Autoware as a whole. The detailed metrics are not included in the model as they would add complexity to it. Instead, the subsequent section introduces the detailed metrics. The detailed metrics are subject to more frequent updates compared to the top-level ones, which is another reason for categorizing them separately. From b04f4b639f180c1978c1c084756d181b21c2770e Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Fri, 23 Jun 2023 11:25:57 +0900 Subject: [PATCH 49/51] fix: completed writing policies Signed-off-by: Takayuki AKAMINE --- .../defining-temporal-performance-metrics.md | 23 +++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/how-to-guides/others/defining-temporal-performance-metrics.md b/docs/how-to-guides/others/defining-temporal-performance-metrics.md index f35eb8c6981..c0def9b7f2b 100644 --- a/docs/how-to-guides/others/defining-temporal-performance-metrics.md +++ b/docs/how-to-guides/others/defining-temporal-performance-metrics.md @@ -34,21 +34,26 @@ The page does not aim to provide instructions on how to use these tools or measu ## Policies to define temporal performance metrics -As mentioned above, the configuration of Autoware varies according to the product type, ODD, and other factors. The variety of configurations makes it difficult to define the uniform metrics for evaluating Autoware. -However, the policies to define them are basically reused even when the configuration changed. Each of temporal performance metrics is categorized into two types: execution frequency and response time. Although there are many types of metrics, such as communication latency, the only two types are taken into account for ease. -Execution frequency is observed using rate of message for Inter-Process Communication (IPC). You will find an enormous number of messages in Autoware, but you don't have to take care of all. Some messages might be crucial to functionality and they should be picked up for evaluation. -Response time is duration elapsed through a series of processing. A series of processing is mentioned as a path. Response time is calculated from timestamps of starting and ending of a path. Thought a lot of paths can be defined in Autoware, you have to choose significant paths. +As mentioned above, the configuration of Autoware varies by the product type, ODD, and other factors. The variety of configurations makes it difficult to define the uniform metrics for evaluating Autoware. +However, the policies used to define them are basically reused even when the configuration changes. Each of temporal performance metrics is categorized into two types: **execution frequency** and **response time**. Although there are many types of metrics, such as communication latency, the only two types are considered for simplicity. +Execution frequency is observed using rate of Inter-Process Communication (IPC) messages. You will find an enormous number of messages in Autoware, but you don't have to take care of all. Some messages might be critical to functionality and they should be chosen for evaluation. +Response time is duration elapsed through a series of processing. A series of processing is referred to as a path. Response time is calculated from timestamps of start and end of a path. Althought many paths can be defined in Autoware, you have to choose significant paths. -As a hint, here are several characteristics of message and path in order to pick up metrics. +As a hint, here are some characteristics of message and path in order to choose metrics. 1. Messages and paths on boundaries where observed values from sensors are consumed -1. Messages and paths on boundaries of functions, e.g. a boundary of perception and planning -1. Messages and paths on boundaries where two different messages are synchronized and merged -1. Messages who must be transmitted at expected frequency for example vehicle command messages +2. Messages and paths on boundaries of functions, e.g., a boundary of perception and planning +3. Messages and paths on boundaries where timer-based frequency is switched +4. Messages and paths on boundaries where two different messages are synchronized and merged +5. Messages that must be transmitted at expected frequency, e.g., vehicle command messages + +Those hints would be helpful for most configurations but there may be exclusions. Defining metrics precisely requires an understanding of configuration. + +In addition, it is recommended that metrics be determined incrementally from the architectural level to the detailed design and implementation level. Mixing metrics at different levels of granularity can be confusing. ## List of sample metrics -This section demonstrates how to metrics according to the policies explained. section is divided into multiple subsections, each containing a model diagram and an accompanying list that explains the important temporal performance metrics. Each model is equipped with checkpoints that serve as indicators for these metrics. +This section demonstrates how to define metrics according to the policies explained and has lists of the metrics for Autoware launched according to [the tutorial](../../tutorials/). The section is divided into multiple subsections, each containing a model diagram and an accompanying list that explains the important temporal performance metrics. Each model is equipped with checkpoints that serve as indicators for these metrics. The first subsection presents the top-level temporal performance metrics, which are depicted in the abstract structure of Autoware as a whole. The detailed metrics are not included in the model as they would add complexity to it. Instead, the subsequent section introduces the detailed metrics. The detailed metrics are subject to more frequent updates compared to the top-level ones, which is another reason for categorizing them separately. From 01acf607bbc364f9f8fa7de8bb6da313854f13b2 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Fri, 23 Jun 2023 02:26:21 +0000 Subject: [PATCH 50/51] style(pre-commit): autofix --- .../others/defining-temporal-performance-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/others/defining-temporal-performance-metrics.md b/docs/how-to-guides/others/defining-temporal-performance-metrics.md index c0def9b7f2b..2a17029fbe6 100644 --- a/docs/how-to-guides/others/defining-temporal-performance-metrics.md +++ b/docs/how-to-guides/others/defining-temporal-performance-metrics.md @@ -47,7 +47,7 @@ As a hint, here are some characteristics of message and path in order to choose 4. Messages and paths on boundaries where two different messages are synchronized and merged 5. Messages that must be transmitted at expected frequency, e.g., vehicle command messages -Those hints would be helpful for most configurations but there may be exclusions. Defining metrics precisely requires an understanding of configuration. +Those hints would be helpful for most configurations but there may be exclusions. Defining metrics precisely requires an understanding of configuration. In addition, it is recommended that metrics be determined incrementally from the architectural level to the detailed design and implementation level. Mixing metrics at different levels of granularity can be confusing. From 3d4c03cda9f833ca27ddf842013ae82b1055d75e Mon Sep 17 00:00:00 2001 From: Takayuki AKAMINE Date: Fri, 23 Jun 2023 13:15:43 +0900 Subject: [PATCH 51/51] chore: fixed missing link. Signed-off-by: Takayuki AKAMINE --- docs/how-to-guides/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/how-to-guides/index.md b/docs/how-to-guides/index.md index e94ba6b67ef..1efda48eeb7 100644 --- a/docs/how-to-guides/index.md +++ b/docs/how-to-guides/index.md @@ -14,7 +14,7 @@ - [Advanced usage of colcon](others/advanced-usage-of-colcon.md) - [Applying Clang-Tidy to ROS packages](others/applying-clang-tidy-to-ros-packages.md) - [Using Eagleye in Autoware](others/eagleye-integration-guide.md) -- [Important temporal performance metrics on components](others/important-temporal-performance-metrics.md) +- [Defining temporal performance metrics on components](others/defining-temporal-performance-metrics.md) TODO: Write the following contents.