Migration of autoware_auto_msgs to autoware_msgs in scenario_simulator_v2

[kielczykowski-rai]

Introduction

Since the migration of autoware_auto_msgs to autoware_msgs in Autoware will happen I am currently working on converting the autoware_auto_msgs to autoware_msgs in scenario_simulator_v2. I have discovered some message types missing or their definitions changes so I wanted to ask for more context.

Questions

1. I see that the following message definitions are missing in autoware_msgs and I wanted to ask if they are going to be supported or there was a design change which justified their absence in Autoware?

• VehicleStateCommand
• EmergencyState
• DetectedObjects, DetectedObject and DetectedObjectKinematics
  • I saw that there is PredictedObject message however it looks like it better maps to TrackedObject than the DetectedObject. So my question is, is the DetectedObject message type still supported?

2. Some messages attributes were changed, in comparison to previous versions so I wanted to check if my understanding is correct:

• TrackedObject conversion to PredictedObject
  • I am not sure if the conversion from TrackedObjectKinematics to PredictedObjectKinematics covers the same information
    • PredictedObjectKinematics misses is_stationary boolean parameter but it contains predicted_paths. I was thinking if the is_stationary attribute set to true can be equivalent to predicted_paths empty array?
    • PredictedObjectKinematics misses the orientation_availability boolean parameter, but I am not sure if it is required anywhere in the Autoware system?

[yukkysaito]

I see that the following message definitions are missing in autoware_msgs and I wanted to ask if they are going to be supported or there was a design change which justified their absence in Autoware?

Regarding VehicleStateCommand and EmergencyState, they have been present since the era of autoware.auto and it's debatable whether they should remain the same type in autoware core/universe. VehicleStateCommand appears to be fine as it is, but as for EmergencyState, it's uncertain if it will be directly carried over, as the MRM interface is still being organized.

As for DetectedObject and TrackedObject, since they are not the final output of the Perception component, the current definition is that of a PredictedObject, which is a certain final output. In the future, it's expected that Detection, Tracking, and Prediction will become a single neural network, so it's worth discussing whether to explicitly prepare them as autoware_msgs interfaces. For example, if Perception itself is carried out by an End to End neural network from Detection to Prediction, and Tracked Object is used in something like scenario_simulator, it could cease to function.
cc @mitsudome-r @xmfcx

2. Some messages attributes were changed, in comparison to previous versions so I wanted to check if my understanding is correct:

• TrackedObject conversion to PredictedObject
• I am not sure if the conversion from TrackedObjectKinematics to PredictedObjectKinematics covers the same information
  * PredictedObjectKinematics misses is_stationary boolean parameter but it contains predicted_paths. I was thinking if the is_stationary attribute set to true can be equivalent to predicted_paths empty array?
  * PredictedObjectKinematics misses the orientation_availability boolean parameter, but I am not sure if it is required anywhere in the Autoware system?

Regarding is_stationary, it's a remnant from autoware.auto and isn't used in the current implementation. Also, it seems that whether an object is stationary can be determined by PredictedObject, so it's unlikely to be added.

There was a previous discussion about orientation_availability (I am currently searching for the thread). The existence of orientation_availability is solely due to the current algorithms being poor, and it has been suggested that "for the component interface between Perception and Planning, the orientation should be determined." Therefore, for now, it's decided not to include orientation_availability in PredictedObject.

[xmfcx]

EmergencyState

As for EmergencyState, it's uncertain if it will be directly carried over, as the MRM interface is still being organized.

@yukkysaito where is this being discussed?

To not have delays, for ongoing topics, let's port the old messages as they are.

And when the discussions are settled, we can update the messages.

Inter-component vs Intra-component messages for perception

Inter-component: eg. between perception and planning
Intra-component: eg. between perception nodes

As for DetectedObject and TrackedObject, since they are not the final output of the Perception component, the current definition is that of a PredictedObject, which is a certain final output.

I see that that some messages are being removed because they are only used in the perception:

• feat(autoware_perception_msgs): remove traffic light messages autoware_msgs#70

And TrackedObject and DetectedObject messages are not ported:

• https://github.com/autowarefoundation/autoware_msgs/tree/224bd3a87fb539630f5a09409c809a81f8bb9118/autoware_perception_msgs/msg

Won't we put any Intra-component messages in the autoware_msgs?

cc. @mitsudome-r @isamu-takagi

[xmfcx]

We talked with @mitsudome-r and we decided to make this separation as follows:

• If a message is needed as a part of interface between Autoware and a simulator, it should be in the autoware_msgs.
  • DetectedObject
  • TrackedObject
• If the message is just used within a component, and a simulator is not expected to interface with it, then it should be put to something like autoware_internal_msgs (the name is not decided).

@yukkysaito @isamu-takagi what do you think about this proposal?

[xmfcx]

We can remove the VehicleStateCommand altogether:

• https://github.com/tier4/autoware_auto_msgs/blob/tier4/main/autoware_auto_vehicle_msgs/msg/VehicleStateCommand.idl
• This message should be replaced by the smaller messages, this was an old decision but I don't know why this message came back.
• Very old MR that removed this message: https://gitlab.com/autowarefoundation/autoware.auto/autoware_auto_msgs/-/merge_requests/20/diffs#0c674dabdaca7dff8a97521076dbadcc7a15890a
• Related Autoware.Auto issue: https://gitlab.com/autowarefoundation/autoware.auto/AutowareAuto/-/issues/890

Related PR:

• feat(autoware_vehicle_msgs): add autoware_vehicle_msgs autoware_msgs#78

cc. @yukkysaito

[kielczykowski-rai]

@yukkysaito @xmfcx
Thank you for the informations.

• If a message is needed as a part of interface between Autoware and a simulator, it should be in the autoware_msgs.
  • DetectedObject
  • TrackedObject
• If the message is just used within a component, and a simulator is not expected to interface with it, then it should be put to something like autoware_internal_msgs (the name is not decided).

As for this matter, I can see that the scenario_simulator_v2 publishes the DetectionObject and TrackedObjects to feed the perception pipeline by publishing similar information as the original sensors (link). It seems that this message types are needed in scenario_simulator_v2 so the first mentioned approach would be easier to implement. As soon as the perception pipeline becomes unified in a single neural network, the scenario_simulator_v2 will follow the change. However, for now, it feels like a subject for the future.

Please let me know if any decision/work has been made regarding the perception pipeline messages port into autoware_msgs package.

We can remove the VehicleStateCommand altogether:

As I checked the VehicleStateCommand is not used anywhere in scenario_simulator_v2 project. Sorry for bringing up this matter, but it was worth double-checking.

[kielczykowski-rai]

I wanted to double check on the matters which still seem to be unresolved.

@xmfcx cc @mitsudome-r
Is there any work regarding the matter of porting the DetectedObject and TrackedObject msgs from autoware_auto_msgs to autoware_msgs repository? Context

@yukkysaito @xmfcx

Regarding VehicleStateCommand and EmergencyState, they have been present since the era of autoware.auto and it's debatable whether they should remain the same type in autoware core/universe. VehicleStateCommand appears to be fine as it is, but as for EmergencyState, it's uncertain if it will be directly carried over, as the MRM interface is still being organized.

and

To not have delays, for ongoing topics, let's port the old messages as they are.

Is there any progress on the decision regarding the future of EmergencyState?

[mitsudome-r]

We have decided to port DetectedObject and TrackedObject messages as well by creating the same message under autoware_perception_msgs